--- name: benchmarks-0.17.0 description: "The Andrey 0.17.0 release card - SCORE and NoGAM, NOTEARS against its authors' code, EM in more searches, and path claims in more methods." meta: type: guide tags: [benchmarks, functional causal models, SCORE, NoGAM, NOTEARS, AGES, missing values, background knowledge] --- # 0.17.0 release card :::::{div} release-opening :::{card} :class-card: release-cover release-default :shadow: none Release card

Andrey

0.17.0 ::: ::::{div} release-opening-text 0.17.0 adds SCORE and NoGAM, two methods for nonlinear data with additive noise. Each reads a causal order from the score of the data, the gradient of its log density, and then prunes the order to a DAG. SCORE assumes Gaussian noise, and NoGAM works under additive noise of any distribution. Both are supported. NOTEARS, previously experimental, is supported, as is AGES for linear Gaussian data where some effects are weak. DAGMA stays experimental. Its fit now leaves the diagonal of the weight matrix free, as the paper's algorithm does. It is for data whose units are meaningful. The release also continues the work from 0.16.0. `missing="em"` now runs the alternating structural EM of Friedman (1997) and is supported in GES, BOSS, and GRaSP, as it already was in NOTEARS, hill climbing, and Tabu search. Path and confounder claims are supported in more methods, and a path claim now directs an undirected edge of a graph that has no consistent extension. `andrey.data` draws more noise families and mechanisms. :::: ::::: ## The benchmarks The earlier release cards keep the numbers of the builds they measured. This release's summary adds the comparison of SCORE with its authors' code and of NoGAM with dodiscover ([SCORE and NoGAM](#score-and-nogam)), which the homepage shows beside them. It also adds the comparison of NOTEARS with its authors' code ([NOTEARS](#notears)) and the comparison of AGES with pcalg ([AGES](#ages)). DAGMA was measured beside NOTEARS. With the diagonal of its weight matrix left free, it ties its authors' package. It stays experimental with a stated scope ([DAGMA](#dagma)). EM and the path and confounder claims were measured for this release against reference packages ([Missing data](#missing-data), [Path and confounder claims](#path-and-confounder-claims)). ## SCORE and NoGAM `andrey.score_matching` is SCORE (Rolland et al., 2022), and `andrey.nogam` is NoGAM (Montagna et al., 2023). Both find a causal order from the leaf up with a kernel estimate of the score and prune the complete DAG of that order with the tests of `cam`. They return a DAG and a causal order, take `knowledge`, and give the same result in any units and column order. Both are supported. The [functional causal models post](../../blog/20261103-andrey-0-13-0.md#score-and-nogam) describes them beside CAM and RESIT, and the [API reference](../api.md#supported-methods) describes each. We compared SCORE with its authors' code and NoGAM with dodiscover on the simulation of their papers, on 20 datasets per setting, with every column z-scored and with each variable rescaled: 12 settings for SCORE and 24 for NoGAM. Neither method made more errors (SHD) than its reference in any setting, and neither ordered fewer true edges in a setting that repeated on fresh datasets. NoGAM made fewer errors than dodiscover in 16 of its 24 settings, on the first datasets and again on 20 fresh ones, and ordered more true edges in 18. SCORE's median fit was 4 to 9 times faster than its authors' code, and NoGAM's was 1.6 to 3 times faster than dodiscover. The [comparison page][sn] has every setting and number. The table shows z-scored data with 1,000 rows. SHD is the mean number of edge errors, and the empty graph's SHD is the number of edges. Order share is the share of true edges that the causal order puts in the true direction. | Method, noise | Variables | Edges | SHD: Andrey | Reference | Order share: Andrey | Reference | Fit, s: Andrey | Reference | |---|---:|---:|---:|---:|---:|---:|---:|---:| | SCORE, Gaussian | 10 | 10 | 2.8 | 3.1 | 0.81 | 0.79 | 1.0 | 4.0 | | SCORE, Gaussian | 10 | 40 | 22.1 | 22.4 | 0.97 | 0.96 | 0.9 | 4.0 | | SCORE, Gaussian | 20 | 20 | 7.8 | 8.2 | 0.73 | 0.71 | 3.0 | 14.3 | | SCORE, Gaussian | 20 | 80 | 51.9 | 51.5 | 0.90 | 0.91 | 2.7 | 14.1 | | SCORE, Gaussian | 50 | 50 | 25.0 | 24.4 | 0.65 | 0.66 | 28.2 | 262.3 | | SCORE, Gaussian | 50 | 200 | 142.7 | 141.2 | 0.85 | 0.87 | 32.4 | 251.8 | | NoGAM, Laplace | 10 | 10 | 1.2 | 2.9 | 0.90 | 0.77 | 9.4 | 20.4 | | NoGAM, Laplace | 10 | 40 | 19.9 | 20.4 | 0.98 | 0.98 | 9.7 | 23.7 | | NoGAM, Laplace | 20 | 20 | 4.2 | 5.8 | 0.86 | 0.81 | 37.9 | 79.2 | | NoGAM, Laplace | 20 | 80 | 48.8 | 51.1 | 0.94 | 0.93 | 37.8 | 108.3 | The measurements also show the limits of the two methods: - On sparse graphs, CAM is more accurate. With as many edges as variables, SCORE's order put 65% to 81% of the true edges in the true direction and NoGAM's 81% to 92%, compared with CAM's 90% to 98%. With four times as many edges, both ordered as many true edges as CAM or more. - Neither method works well on every additive-noise model. On Andrey's own additive-noise simulation, with links `z + sin z`, SCORE ordered 41% to 53% of the true edges correctly and NoGAM 47% to 62%, close to a random order, as the reference implementations do. CAM ordered 76% to 94% there. Use CAM on such data. - SCORE's rule is exact under Gaussian noise only. NoGAM was measured at 10 and 20 variables, and a single fit at 50 variables took about three minutes. ## NOTEARS `andrey.notears` is NOTEARS (Zheng et al., 2018). It fits a weighted DAG to linear data by minimizing a least-squares loss with an `l1` penalty under a smooth acyclicity constraint. It returns a DAG with the coefficient of each edge. It is supported. The [API reference](../api.md#supported-methods) describes it. We compared it with its authors' code on the simulation of the paper: linear models on Erdos-Renyi graphs with twice as many edges as variables and on scale-free graphs with four times as many. The data had 10 to 100 variables, Gaussian, exponential, or Gumbel noise, and 1,000 rows, with every column z-scored. We used 20 datasets per setting. In those 24 settings, NOTEARS made no more errors (SHD) than the authors' code: the two tied in 23, and NOTEARS made fewer in 1, which we did not rerun. It made fewer errors than the empty graph in 15 of the 24, and the authors' code did so in 16. The [comparison page][nd] has every setting and number, including those of DAGMA, which stays experimental. The table shows z-scored data with Gaussian noise. SHD is the mean number of edge errors, and the empty graph's SHD is the number of edges. Same graph counts the datasets on which both return the same DAG, and the fit time is the median. | Graphs | Variables | Edges | SHD: Andrey | Reference | Same graph | Fit, s: Andrey | Reference | |---|---:|---:|---:|---:|---:|---:|---:| | Erdos-Renyi | 10 | 20 | 18.4 | 18.3 | 18 of 20 | 2.4 | 1.5 | | Erdos-Renyi | 20 | 40 | 37.4 | 37.5 | 15 of 20 | 7.6 | 8.7 | | Erdos-Renyi | 50 | 100 | 88.0 | 87.9 | 10 of 20 | 40.3 | 63.4 | | Erdos-Renyi | 100 | 200 | 157.2 | 156.7 | 7 of 20 | 239.9 | 392.4 | | Scale-free | 10 | 24 | 23.1 | 23.3 | 15 of 20 | 2.0 | 1.5 | | Scale-free | 20 | 64 | 64.2 | 64.0 | 10 of 20 | 10.8 | 13.7 | | Scale-free | 50 | 184 | 180.8 | 181.0 | 8 of 20 | 38.3 | 63.3 | | Scale-free | 100 | 384 | 379.4 | 379.9 | 2 of 20 | 327.8 | 467.9 | The measurements also show where NOTEARS is weak: - On the scale-free graphs, NOTEARS was within 5% of the empty graph in all 12 settings and made fewer errors than it in 3. On the Erdos-Renyi graphs, it made 6% to 22% fewer errors in all 12. - On data in the simulation's own units, where a variable's variance grows along the causal order, NOTEARS made far fewer errors: 2.5 to 83.5 against the empty graph's 20 to 184, at 10 to 50 variables. Sorting the columns by variance and regressing each on those before it made 4.3 to 150 errors on the same data, so most of that accuracy comes from the data's units. - On sparser graphs, with as many edges as variables, `ges` and `calm` are more accurate. On z-scored columns at 10 to 100 variables, the median SHD against the true CPDAG was 7 to 92 for NOTEARS, 2 to 79 for `ges`, and 3.5 to 36 for `calm`. ### DAGMA `andrey.dagma` (Bello et al., 2022) stays experimental. At its default settings, it does no better than the empty graph on z-scored columns or any data whose units are arbitrary. It made fewer errors than the empty graph in none of the 24 settings above and more in 16, exactly as its authors' package did. Its accuracy on the simulation's own units comes from the columns' variances following the causal order (varsortability; Reisach et al., 2021), which z-scoring removes. In this release, DAGMA's fit leaves the diagonal of the weight matrix free, as the paper's algorithm does. It now ties its authors' package in all 24 settings; before, it made more errors in 3. At 0.05, the penalty used in the DAGMA paper's experiments, DAGMA made fewer errors than the empty graph in 6 of the 24 settings and more in 6. The [comparison page][nd] has every setting, the trace of why the least-squares score prefers another graph on z-scored columns, and the literature. ## AGES `andrey.ages` is AGES (Eigenmann, Nandy, and Maathuis, 2017), for linear Gaussian data where some effects are weak. It overlays the directions of the CPDAGs that GES finds under higher penalties on the GES CPDAG. This keeps the GES skeleton and can direct more of its edges. It is supported. The [API reference](../api.md#supported-methods) describes it. We compared it with pcalg's `ages`, the authors' implementation, on the paper's simulation with 10 variables, 23 settings of strong and weak edges, and 10,000, 1,000, or 100 rows. We used 20 datasets per setting. Andrey returned pcalg's graph on 1,378 of the 1,380 datasets, was no less accurate in any of the 69 settings, and fit 7 to 12 times faster. The [comparison page][ag] has every setting and number. The table shows 10,000 rows. Strong and weak are the shares of variable pairs with a strong and a weak edge. SHD is the mean number of edge errors; a correct direction beyond the CPDAG does not count as an error. Recall is the share of true edges directed correctly, and precision is the share of directed edges that are correct. GES runs at the same penalty. | Strong | Weak | Edges | Same graph | SHD: Andrey | pcalg | Precision: AGES | GES | Recall: AGES | GES | Fit, s: Andrey | pcalg | |---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:| | 0.1 | 0.7 | 37.1 | 19 of 20 | 21.3 | 21.3 | 0.51 | 0.51 | 0.49 | 0.45 | 0.17 | 1.28 | | 0.3 | 0.3 | 27.9 | 20 of 20 | 18.4 | 18.4 | 0.52 | 0.52 | 0.59 | 0.57 | 0.15 | 1.27 | | 0.3 | 0.7 | 45.0 | 19 of 20 | 26.1 | 26.1 | 0.46 | 0.46 | 0.38 | 0.36 | 0.22 | 1.86 | | 0.5 | 0.5 | 45.0 | 20 of 20 | 25.6 | 25.6 | 0.47 | 0.47 | 0.41 | 0.36 | 0.21 | 1.64 | | 0.5 | 0.0 | 21.7 | 20 of 20 | 12.2 | 12.2 | 0.58 | 0.59 | 0.66 | 0.63 | 0.14 | 1.09 | | 1.0 | 0.0 | 45.0 | 20 of 20 | 24.6 | 24.6 | 0.50 | 0.51 | 0.42 | 0.39 | 0.21 | 1.48 | - With 10,000 rows, AGES directed more edges correctly than GES in 20 of the 23 settings, and more edges wrongly in the same 20. It directed 49% of the true edges correctly against 46%, at the same precision, 49%. With 1,000 rows, the shares were 44% against 43%; with 100 rows, there was no gain. - In 7 of the 69 settings, all dense graphs with few or no weak edges, neither package's errors were clearly below the empty graph's. - The evidence is at 10 variables. The paper's 100-variable setting uses a variant of the search that Andrey does not implement. ## Missing data `missing="em"` now runs the alternating structural EM of Friedman (1997) in every method that takes it. Before each search after the first, 5 parameter-only EM steps refit the current graph. It is supported in GES, BOSS, and GRaSP, as it already was in NOTEARS, hill climbing, and Tabu search. A fit with `missing="em"` can return a different graph than in 0.16.0. Results on complete data do not change. The [Ready for real data post](../../blog/20261029-andrey-0-11-0.md#why-values-go-missing) describes the scheme, and [Missing values](../guides/getting-started.md#missing-values) in the getting-started guide explains what it assumes. We compared each search with a reference implementation on simulated data with values missing completely at random or at random, on 20 datasets per cell. GES, BOSS, and GRaSP ran at the standard BIC, with 10 to 50 variables, beside Tetrad 8.0.0's FGES, BOSS, and GRaSP under its EM covariance policy, with its sample size at the mean pairwise count. A cell that was worse ran again on 20 fresh datasets. The [comparison page][md] has every lane and number. | Method | Reference | Cells better | Tied | Worse | Worse again | Below the empty graph | Status | |---|---|---|---|---|---|---|---| | `ges(missing="em")` | Tetrad's FGES, EM covariance | 90 of 132 | 42 | 0 | 0 | 73 of 132 | Supported | | `boss(missing="em")` | Tetrad's BOSS, EM covariance | 72 of 132 | 59 | 1 | 0 | 122 of 132 | Supported | | `grasp(missing="em")` | Tetrad's GRaSP, EM covariance | 68 of 132 | 61 | 3 | 0 | 124 of 132 | Supported | | `notears(missing="em")` | MissDAG | 23 of 24 | 1 | 0 | 0 | 24 of 24 | Supported | | `hc(missing="em")` | bnlearn's structural EM | 28 of 36 | 8 | 0 | 0 | 22 of 36 | Supported | | `tabu(missing="em")` | bnlearn's structural EM | 31 of 36 | 5 | 0 | 0 | 29 of 36 | Supported | The measurements also show what to expect from a fit: - A fit runs 56 E-steps instead of 11 and takes about 1.5 to 5 times as long as in 0.16.0. - EM does not always settle on one graph within its 10 rounds. The last round still changed the graph in 2 to 6% of the benchmark's fits and in 20% of GRaSP's. Such a fit logs a warning, and `metadata["em"]["graph_changed"]` records it. - Tetrad's policy estimates one covariance by EM before the search and is not structural EM, so part of the margin over it is due to the reference's weakness. The default `ebic_gamma=1.0` and the 100-variable settings were not compared. - XGES, exact search, DAGMA, and CALM with EM stay experimental because no reference implementation exists to compare them with. ## Path and confounder claims 0.16.0 left the path and confounder claims experimental in 14 methods that its benchmark did not run. 0.17.0 measures them, and each of the 14 now supports at least one part. A path claim also acts where a fit used to drop it. When the tests of PC, CD-NOD, IAMB, Inter-IAMB, or MMPC contradict each other, the graph they return has no consistent extension: it stands for no DAG. On such a graph, a claim on a pair joined by an undirected edge now directs that edge, and `metadata["edge_claims"]` lists the claims applied this way. [Background knowledge](../guides/getting-started.md#background-knowledge) in the getting-started guide shows each part, and the [Ready for real data post](../../blog/20261029-andrey-0-11-0.md#what-you-already-know) uses them on a survey. We gave each method true claims on simulated data, one part at a time, and counted its errors against the truth with and without the claims, on 20 datasets per cell. A cell counts as worse only when it is worse again on 20 fresh datasets. We also ran causal-learn, bnlearn, Tetrad, and lingam with the same knowledge. No supported part was less accurate than its reference in a cell that repeated. The [comparison page][kp] has every lane and number. | Method | Supported parts | Cells worse with true claims | Cells better | Experimental parts | |---|---|---|---|---| | CD-NOD | two path parts | 0 of 72 | 53 | | | GRaSP | two path parts | 0 of 72 | 71 | | | XGES, hill climbing | two path parts | 0 of 36 each | 36 each | | | Exact search | two path parts | 0 of 36 | 34 | | | Multi-group DirectLiNGAM | two path parts | 0 of 24 | 0 | | | FCI-JCI | all three | 0 of 24 | 12 | | | GRaSP-FCI | all three | 0 of 24 | 8 | | | Tabu search | `forbidden_paths` | 0 of 18 | 18 | `required_paths` | | MMHC, RSMAX2 | `forbidden_paths` | 0 of 36 each | 31 each | `required_paths` | | IAMB, Inter-IAMB, MMPC | `required_paths` | 0 of 36 each | 35 each | `forbidden_paths` | The table counts the cells of the supported parts. Cells better counts the cells with fewer errors against the truth with the claims. The measurements also show what a path claim cannot do: - `required_paths` stays experimental in Tabu search, MMHC, and RSMAX2. bnlearn, given the required edges those paths imply, made fewer errors there in cells that repeated. - `forbidden_paths` stays experimental in IAMB, Inter-IAMB, and MMPC. With forbidden paths alone at 50 variables and mean degree 2, their CPDAG was 0.65 to 1.95 edges farther from the class of graphs the claims define in 3 or 4 of 6 settings over 40 datasets. - On a graph with no consistent extension, a claim acts only on an edge, and most claims name a pair that is not adjacent. The five methods drop 76 to 81% of the required and 96 to 97% of the forbidden paths there. The result is not guaranteed to stand for a DAG, and applying a claim to such a graph is Andrey's own choice. ## Other changes in 0.17.0 `andrey.data` draws Student-t and Gaussian-mixture noise, functions from random Fourier features, three more post-nonlinear transforms, and measurement columns. Each is off by default, so every seed drawn without them gives the same data as before. MMHC, and RSMAX2 with `restrict="mmpc"` or `"pc"`, now pass background knowledge to their restrict phase, which removed 0.9 to 1.9 errors on average on graphs of mean degree 4 with required edges. The Limitations of {py:func}`~andrey.grasp` say when one start is not enough, and those of {py:func}`~andrey.pc` say why `missing="mvpc"` loses to test-wise deletion on larger graphs. On the data the GIES benchmark currently draws, `gies` and pcalg's GIES return the same graph on 200 of 200 datasets, so the rows of the [0.7.0 release card](0.7.0.md) remain valid ([#868](https://github.com/Abel-ai-lab/andrey/issues/868)). The [changelog](../changelog.md) lists every change. [sn]: https://github.com/Abel-ai-lab/andrey/blob/main/benchmarks/comparisons/score-nogam.md [nd]: https://github.com/Abel-ai-lab/andrey/blob/main/benchmarks/comparisons/notears-dagma.md [ag]: https://github.com/Abel-ai-lab/andrey/blob/main/benchmarks/comparisons/ages.md [md]: https://github.com/Abel-ai-lab/andrey/blob/main/benchmarks/comparisons/missing-data.md [kp]: https://github.com/Abel-ai-lab/andrey/blob/main/benchmarks/comparisons/knowledge-paths.md