--- name: benchmarks-0.16.0 description: "The Andrey 0.16.0 release card - MVPC and EM for missing data, path and confounder claims in background knowledge, LOCI, intervention marks, and HSIC p-values." meta: type: guide tags: [benchmarks, missing values, background knowledge, pairwise, drawing, HSIC, FCI] --- # 0.16.0 release card :::::{div} release-opening :::{card} :class-card: release-cover release-default :shadow: none Release card

Andrey

0.16.0 ::: ::::{div} release-opening-text 0.16.0 extends three earlier themes. For missing data, you can simulate values that go missing for a known reason, correct PC when missingness depends on other variables with MVPC, and fit the score searches, NOTEARS, DAGMA, and CALM on incomplete data with EM. Background knowledge can now say that one variable causes another by some path, that it does not, or that two share a hidden common cause. LOCI reads the direction of a pair whose noise grows or shrinks with the cause. Drawings also mark intervened variables as causal-discovery figures do. Every method that tests with HSIC keeps its p-values precise far below 1e-16, and FCI's documentation says when a fit is not a PAG. LOCI and MVPC are supported, and so is EM in NOTEARS, hill climbing, and Tabu search; EM in the other searches is experimental. The new knowledge parts are supported in PC, GES, BOSS, FCI, GFCI, BOSS-FCI, LPCMCI, and DirectLiNGAM, and experimental in the other methods. :::: ::::: ## The benchmarks The earlier release cards keep the numbers of the builds they measured, and those numbers hold for 0.16.0. This release's summary adds LOCI's comparison with its authors' code ([LOCI](#loci)), which the homepage shows beside them. Of the changes in 0.16.0, only the HSIC p-values can affect a fit those cards measured. On 970 pairs drawn as the ANM and PNL benchmarks draw theirs, no direction changed ([HSIC p-values](#hsic-p-values)). The comparison of J-PCMCI+ with tigramite now reads each conflict as tigramite marks it, `x-x`: 7 of the 21 same-graph counts in the 0.15.0 claim go up, and no claim changes. MVPC and EM were measured for this release against reference implementations ([What we measured on missing data](#what-we-measured-on-missing-data)). ## Missing data 0.16.0 lets you simulate the reasons values go missing with `andrey.data.missingness` and gives you two methods that correct for them: `pc(data, missing="mvpc")`, Missing-Value PC (Tu et al., 2019), and `missing="em"`, structural EM (Friedman, 1997; Gao et al., 2022) for the score searches, NOTEARS, DAGMA, and CALM. Every method that accepts missing values now names its handling with one argument, `missing`. Its default, `None`, keeps each method's own handling, so no result changes. The [Ready for real data post](../../blog/20261029-andrey-0-11-0.md#why-values-go-missing) now describes them beside test-wise deletion on a real survey, and [Missing values](../guides/getting-started.md#missing-values) in the getting-started guide explains each one and what it assumes. ### What we measured on missing data We measured both corrections on simulated data, each against a reference implementation and against four baselines fit on the same files: test-wise PC, the method after dropping every incomplete row, the method after filling in column means, and the empty graph. A method is better or worse than another in a cell when the 95% bootstrap interval of their paired difference in errors (SHD) lies wholly on one side of zero. A worse cell counts only when it is worse again on 20 fresh datasets. The [comparison page][md] has every lane and number, from 192,700 fits. | Method | Reference | Cells better | Tied | Worse | Below the empty graph | Status | |---|---|---|---|---|---|---| | `pc(missing="mvpc")` | The MVPC authors' R code | 28 of 32 | 4 | 0 | 32 of 32 | Supported | | `notears(missing="em")` | MissDAG | 19 of 24 | 5 | 0 | 22 of 24 | Supported | | `hc(missing="em")` | bnlearn's structural EM | 29 of 36 | 7 | 0 | 23 of 36 | Supported | | `tabu(missing="em")` | bnlearn's structural EM | 32 of 36 | 4 | 0 | 27 of 36 | Supported | In its paper's setting, at 50 variables and 10,000 rows with values missing at random, MVPC makes 5.1 errors against test-wise PC's 14.3, in 2.5 s against the R code's 85.5 s. The median fit of NOTEARS with EM took 8 to 42 s against MissDAG's 49 to 114 s. The measurements also show where each method stops helping: - MVPC does not help when each missingness indicator depends weakly on many variables. On a wider grid of our own, it ties test-wise PC in 125 of 141 such cells. With 100 variables and 10,000 rows, 77 fits exceeded the cap of 30 minutes while searching for the indicators' parents. `max_depth` bounds this search. - With 50 or 100 variables and 20 or 30% of the values missing, MVPC is worse than test-wise PC in 12 cells of that grid, on the first datasets and again on 20 fresh ones, by 0.7 to 3.6 errors. In the three of those cells where we also ran the authors' code, its MVPC ties its own test-wise PC, and Andrey's MVPC is no less accurate than it. In these settings, use test-wise deletion. - With 15 variables and 100 rows, hill climbing and Tabu search with EM are often no better than the empty graph, as is the plain search on complete data. Hill climbing is below it in 11 of 24 cells and Tabu search in 15. With 20 variables and 500 or 2,000 rows, both are below it in 12 of 12 cells. - With half of the values missing completely at random on the two denser graphs, neither NOTEARS with EM nor MissDAG is below the empty graph. - GES, XGES, BOSS, GRaSP, exact search, CALM, and DAGMA with EM stay experimental because no reference implementation exists to compare them with. The comparison page has their results against the baselines. With 100 rows and 40 to 50% of the values missing, GES, BOSS, and GRaSP return almost the empty graph at the default penalty. ## Path and confounder claims `andrey.Knowledge` takes three more parts: `required_paths`, the pairs in which one variable causes the other, directly or through other variables; `forbidden_paths`, the pairs in which it does not; and `confounded`, the pairs that share a hidden common cause. PC, GES, BOSS, and the other methods that return a CPDAG honor the two path parts, as does DirectLiNGAM. FCI and the other PAG methods honor all three. [Background knowledge](../guides/getting-started.md#background-knowledge) in the getting-started guide shows each part and how each method honors it, and the [Ready for real data post](../../blog/20261029-andrey-0-11-0.md#what-you-already-know) uses them on a survey. ### What we measured We checked them three ways: against exact answers, against the same fit without the claims, and against reference packages given the same knowledge. The [comparison page][kp] has every lane and number. With d-separation in place of the tests, the CPDAG step gave the exact answer on all 131,296 claim sets over 32,824 graphs. Every mark that FCI, GFCI, and BOSS-FCI set from true claims matched the true graph's mark, on 14,946 fits each. On simulated data, we gave each method true claims on 5 to 30% of the variable pairs, one part at a time and all together. We counted its errors against the truth with and without the claims, on 20 datasets per cell. A cell counts as worse only when it is worse again on 20 fresh datasets. | Method | Parts | Cells worse with true claims | Cells better | Claims dropped | Status | |---|---|---|---|---|---| | FCI | all three | 0 of 24 | 6 | 0 to 9.7% | Supported | | GFCI | all three | 0 of 24 | 6 | 0 to 7.9% | Supported | | BOSS-FCI | all three | 0 of 24 | 6 | 0 to 7.0% | Supported | | LPCMCI | all three | 0 of 24 | 10 | 0 to 2.9% | Supported | | DirectLiNGAM | two path parts | 0 of 24 | 1 | 0 to 0.6% | Supported | | PC | two path parts | 0 of 72 | 65 | 23.8 to 59.9% | Supported | | GES | two path parts | 0 of 36 | 36 | 1.3 to 7.0% | Supported | | BOSS | two path parts | 0 of 72 | 72 | 0.8 to 2.9% | Supported | A cell here is a graph size, a sample size, and a share of pairs claimed, with one part. Claims dropped is the share of true claims a method dropped because the graph it found does not meet them, reported for the parts with the lowest and highest shares. For PC, GES, and BOSS, Cells better counts the cells closer to the true DAG with the claims. No fit dropped a `confounded` pair. The Status column also reflects the reference packages. We ran Tetrad, pcalg, tigramite, and lingam on the same datasets with the same knowledge. Tetrad and pcalg take edges, not paths, so they and Andrey's methods received the edges the claims imply. With the same knowledge, no method was less accurate than its reference in a cell that repeated. The same held without knowledge. DirectLiNGAM also fit the 140 of 240 datasets with forbidden paths on which lingam raised an error. The measurements also show what a path claim cannot do: - The CPDAG methods restrict the graph the search found. On finite data, that graph can be wrong, and a true claim then directs some edges the wrong way: 5.8 to 10.9% of the edges the claims directed with PC, 1.7 to 4.1% with BOSS, and 1.1 to 1.7% with GES. Every one was in a fit whose graph had a missing or extra edge or an incorrect collider. - A path claim says less than an edge and cannot restore an edge the search left out. Given required paths, PC, GES with the plain BIC, FCI, and GFCI made more errors than their references given the edges those paths imply, in cells that repeated. BOSS did too in one cell with all parts together; with forbidden paths none did. When you know the edge itself, state it as `required`. - GES is judged on the sparser graphs, at mean degree 2; mean degree 4 is outside the setting it suits. - We have not measured the parts in the methods this benchmark does not run, where they are experimental: CD-NOD, XGES, hill climbing, Tabu search, MMHC, RSMAX2, GRaSP, exact search, IAMB, Inter-IAMB, MMPC, FCI-JCI, GRaSP-FCI, and multi-group DirectLiNGAM. [kp]: https://github.com/Abel-ai-lab/andrey/blob/main/benchmarks/comparisons/knowledge-paths.md [md]: https://github.com/Abel-ai-lab/andrey/blob/main/benchmarks/comparisons/missing-data.md ## LOCI `andrey.loci` decides the direction of a variable pair under the location-scale noise model of Immer et al. (2023), `y = f(x) + g(x) n`, in which the spread of the effect changes with the cause. It decides by an HSIC test of the scaled residual against the cause, `criterion="independence"`, the default, or by the larger likelihood, `criterion="likelihood"`. Like ANM and PNL, it reads the rows where both columns are observed and takes `missing`. The [pairwise direction post](../../blog/20261015-andrey-0-4-0.md) describes LOCI beside ANM and PNL. LOCI is supported. We compared each criterion with the same criterion of the LOCI authors' code, LOCI-H for independence and LOCI-M for likelihood, on 100 pairs in each of 26 settings. Neither criterion made significantly more errors than the authors' code in any setting we judged, 39 comparisons in all. On 8 of the 10 location-scale comparisons we ran again on 100 fresh pairs, it made fewer errors both times (5 with the default criterion, 3 with the likelihood). Andrey's median fit was 40 to 912 times faster with the default criterion and 376 to 1,948 times faster with the likelihood. The [comparison page][loci] lists the settings and every row. The table shows pairs with the true direction, of 100, at 1,000 rows. | Pairs, 1,000 rows | Independence: Andrey | LOCI-H | Likelihood: Andrey | LOCI-M | |---|---:|---:|---:|---:| | Location-scale, Gaussian noise | 94 | 87 | 100 | 99 | | Location-scale, Laplace noise | 91 | 83 | 44\* | 35\* | | Location-scale, Gumbel noise | 95 | 79 | 37\* | 35\* | | Additive, Gaussian noise | 90 | 89 | 93 | 87 | | Tagasovska et al. AN | 100 | 97 | 100 | 96 | | Tagasovska et al. AN-s | 100 | 99 | 100 | 99 | | Tagasovska et al. LS | 94 | 86 | 100 | 92 | | Tagasovska et al. LS-s | 100 | 50 | 100 | 96 | | Tagasovska et al. MN-U | 99 | 96 | 100\* | 96\* | \* These results are outside the likelihood criterion's Gaussian model, so they are not judged. Under Gumbel noise, it correctly identified the direction of 35 to 38 pairs of 100 at every size, fewer than expected from random guessing, and the authors' LOCI-M did so for 31 to 37. Keep the default criterion unless you know the noise is Gaussian: the independence criterion correctly identified the direction of most pairs in all 26 settings, 79 to 100 of 100. LOCI decides one pair; for more variables, use CAM. The cause must range without bounds, so a uniform cause is outside its model. Under `criterion="independence"`, the HSIC test's time and memory grow with the square of the number of rows. On 20 pairs, the median fit took 60 s and 10 GB at 16,000 rows. At 32,000 rows, every fit exceeded a 28 GB cap, while the likelihood criterion took 1.4 s. [loci]: https://github.com/Abel-ai-lab/andrey/blob/main/benchmarks/comparisons/loci.md ## Other changes in 0.16.0 ### Intervention marks `andrey.viz.draw(structure, targets=[...])` now marks each edge that has an arrowhead into an intervention target with a cross in its middle. A target that no arrowhead points into is marked by its square alone. `andrey.viz.to_graphviz(structure, targets=[...])` draws the targets too, as `square` nodes. The graph types guide has a new section, [How to read a causal graph](../guides/graph-types.md#how-to-read-a-causal-graph), with a legend of every mark `draw` uses. ### HSIC p-values ANM, PNL, RCD, BottomUpParceLiNGAM, GIN, and `indep_test="kci"` without conditions compute their HSIC p-values from the upper tail of the gamma approximation through its logarithm, as RESIT and CAM-UV already did. A p-value no longer rounds to 0 below about 1e-16; it reads 0 only below about 1e-308. A result can therefore change where both p-values were below 1e-16 and the HSIC statistics decided. On 970 pairs drawn as the ANM and PNL benchmarks draw theirs, from 200 to 16,000 rows, no direction changed. The [pairwise direction post](../../blog/20261015-andrey-0-4-0.md#reading-the-result) now describes 0.16.0's rule. ### FCI fits that are not PAGs FCI returns the same graphs as in 0.15.0, and its documentation now says what they can be. On finite data, test errors can give separating sets that no maximal ancestral graph (MAG) fits. The `pag` that FCI returns is then not the PAG of any MAG. The **Limitations** paragraph of {py:func}`~andrey.fci` gives how often: of 200 simulated datasets with 10 variables, mean degree 3, and 2,000 rows, 36 fits with `"fisherz"` and 36 with `"normal_scores"` returned such a graph. {py:func}`~andrey.bootstrap` already passes over such a fit, and its Notes now say so. ### J-PCMCI+ and a shared pulse J-PCMCI+ now tells you when a pulse that every dataset shares is left out of the time contexts. When a system variable has a copy among a test's conditions, `jpcmciplus` raises `SingularCorrelationError` with a note to list it as a time context. It previously dropped that variable's links without an error. [Hank Zhu](https://github.com/HouZhu2001) found and fixed it ([#841](https://github.com/Abel-ai-lab/andrey/pull/841)). Thank you, Hank. The [changelog](../changelog.md) lists every change.