0.14.0 release card: interventional data

Release card

Interventional data

0.14.0

This card measures the methods 0.14.0 makes supported against their authors’ code: GnIES, UT-IGSP, and IGSP on simulated data mixing observation and experiments, ICP on environments that leave one variable’s mechanism alone, CAM-UV on nonlinear data with hidden variables, and DYNOTEARS on vector autoregressions. Each result holds for the measured conditions; it depends on the data, settings, and hardware.

0.14.0 makes six methods supported, and we compared each with the code its authors released, on data simulated the way their papers do. GnIES, UT-IGSP, and IGSP learn from experiments mixed with observation, ICP finds the causes of one variable across environments, CAM-UV handles nonlinear data with hidden variables, and DYNOTEARS learns from time series.

Summary

Method	Reference	Comparisons	Better	Tie	Worse	Speed against the reference
GnIES	gnies 0.3.3	21 settings x 2 errors	3	39	0	34x faster
UT-IGSP	graphical-model-learning 0.1a8	56 settings x 2 errors	1	111	0	1.7x slower
ICP	InvariantCausalPrediction 0.8 (R)	3 settings x 2 tests	0	6	0	41x to 760x faster
CAM-UV	lingam 1.13.0, causal-learn 0.1.4.8	9 settings x 3 references	0	27	0	2.1x to 3.1x faster than lingam, 7.4x to 13.5x than causal-learn
DYNOTEARS	CausalNex 0.12.2	38 settings	5	32	1	1.1x to 3.4x faster in CPU time
IGSP	graphical-model-learning 0.1a8	87 settings, 114 comparisons	2	112	0	1.3x faster to 8.0x slower

Andrey matches every reference in almost every comparison, and in the few that differ it is more often better than worse. Here is how to read it:

• The speed is the median, over the datasets, of the reference’s fit time divided by Andrey’s on the same dataset, written as how many times faster or slower Andrey is.

• A comparison is better or worse when the 95% interval of the mean paired difference in errors lies entirely below or above 0.

• A better result has to hold twice. It counts only where a rerun on fresh seeds, which played no part in choosing Andrey’s defaults, agreed. A comparison better on the first seeds alone counts as a tie, and so do 17 UT-IGSP comparisons that were lower on the first seeds and on the seeds that chose its default, since those seeds are not fresh.

• Andrey must also beat doing nothing wherever the reference does, making fewer errors than the empty estimate: the empty graph, or for a target error no targets beyond those given. Where the reference does not beat it either, the floor is reported and not judged. The floor is judged on the first seeds; a rerun’s floor only confirms a failure there.

How we measured

Every dataset was simulated with andrey.data and given to both packages, so each comparison pairs Andrey’s error with the reference’s on the same data. D in the tables is the mean of those paired differences, Andrey’s error minus the reference’s, with its 95% bootstrap interval.

• GnIES, UT-IGSP, and IGSP ran one fit at a time on one core of an AMD EPYC 7713 with one BLAS thread.

• ICP, CAM-UV, and DYNOTEARS ran four fits at a time, each on one core of an Intel Core i7-7820X with one thread.

• GnIES and UT-IGSP run each package at its defaults, and Andrey’s defaults were chosen on seeds of their own before the comparison. The other references run at the comparison’s settings, given in each method’s section below.

GnIES

GnIES learns a graph from environments that change the noise of some variables, without being told which. Andrey is about 34 times faster than gnies 0.3.3 and as accurate: of the 8 comparisons better on seeds 0 to 19, 3 were better again on seeds 20 to 39, and the rest tie.

• The graphs are Erdos-Renyi DAGs on 10 variables, with linear equations and Gaussian noise.

• One environment is observational; each of the others changes the noise variance of its targets, and neither package is told the targets.

• Each environment has 10 to 1,000 rows, standardized on the pooled rows, with 20 datasets per setting.

• Both packages run at their defaults.

• We count two errors: the structural Hamming distance (SHD) of the returned interventional essential graph to the true one, and the target error, the targets missed plus the targets named wrongly.

Over the 420 datasets in the claim, Andrey took a median 0.030 times the reference’s fit time (quartiles 0.027 and 0.033). At 10 variables Andrey’s median fit took 0.29 to 1.33 s per setting, the reference’s 9.1 to 48.6 s.

GnIES against gnies 0.3.3, targets unknown

	SHD				Target error
	Andrey	gnies	Empty graph	Difference	Andrey	gnies	No targets	Difference
3 experiments, 1 target each
10 rows each	11.35	14.40	14.00	-3.05[-4.85, -1.20]better	1.85	2.45	3.00	-0.60[-1.00, -0.30]better
100 rows each	4.95	6.00	14.00	-1.05[-3.30, +0.80]	0.50	0.45	3.00	+0.05[-0.20, +0.35]
1,000 rows each	2.05	2.15	14.00	-0.10[-0.60, +0.30]	0.25	0.30	3.00	-0.05[-0.15, +0.00]
Denser graph, mean degree 4
10 rows each	20.00	21.95	20.00	-1.95[-3.55, -0.45]better	2.70	2.90	3.00	-0.20[-0.70, +0.25]
100 rows each	16.45	15.85	20.00	+0.60[-0.50, +1.80]	1.00	1.20	3.00	-0.20[-0.50, +0.05]
1,000 rows each	12.45	11.90	20.00	+0.55[-0.40, +1.95]	1.55	1.45	3.00	+0.10[+0.00, +0.30]
3 targets per experiment
10 rows each	11.30	12.75	14.00	-1.45[-2.90, -0.20]lower, first seeds only	4.35	4.70	9.00	-0.35[-0.90, +0.15]
100 rows each	3.00	4.05	14.00	-1.05[-2.55, -0.05]lower, first seeds only	0.65	0.45	9.00	+0.20[-0.10, +0.65]
1,000 rows each	1.80	1.70	14.00	+0.10[+0.00, +0.30]	0.30	0.35	9.00	-0.05[-0.15, +0.00]
Smaller change, variance x3 to x4
10 rows each	11.90	14.30	14.00	-2.40[-3.55, -1.35]lower, first seeds only	3.00	3.25	3.00	-0.25[-0.65, +0.15]
100 rows each	7.45	8.00	14.00	-0.55[-1.05, -0.15]lower, first seeds only	1.05	1.10	3.00	-0.05[-0.15, +0.00]
1,000 rows each	2.35	1.75	14.00	+0.60[-0.30, +2.10]	0.30	0.40	3.00	-0.10[-0.30, +0.00]
No observational environment
100 rows each	3.50	4.00	14.00	-0.50[-1.20, +0.15]	0.40	0.55	4.00	-0.15[-0.40, +0.00]
1,000 rows each	1.75	1.95	14.00	-0.20[-0.50, +0.00]	0.40	0.40	4.00	+0.00[+0.00, +0.00]
9 experiments
10 rows each	10.25	11.05	14.00	-0.80[-2.80, +1.35]	7.70	7.55	9.00	+0.15[-0.05, +0.35]
100 rows each	1.65	2.00	14.00	-0.35[-0.65, -0.10]lower, first seeds only	0.60	0.70	9.00	-0.10[-0.30, +0.00]
1,000 observed rows
experiments of 100 rows	2.65	3.25	14.00	-0.60[-1.65, +0.30]	0.45	0.35	3.00	+0.10[+0.00, +0.25]
Fewer experiments
1 experiment	2.45	2.55	14.00	-0.10[-0.25, +0.00]	0.30	0.30	1.00	+0.00[+0.00, +0.00]
2 experiments	3.70	2.70	14.00	+1.00[-0.30, +3.25]	0.55	0.30	2.00	+0.25[-0.05, +0.65]
Weights of either sign
100 rows each	3.50	3.70	14.00	-0.20[-1.00, +0.75]	0.25	0.35	3.00	-0.10[-0.25, +0.00]
1,000 rows each	1.40	1.40	14.00	+0.00[+0.00, +0.00]	0.30	0.30	3.00	+0.00[+0.00, +0.00]

Means over 20 datasets, seeds 0 to 19, on 10 variables; each experiment multiplies its targets' noise variance by 5 to 10 unless noted. The difference is Andrey's mean error minus gnies's, with its 95% interval, marked better only where seeds 20 to 39 agreed; unmarked is a tie. No targets is the target error of naming none: the number of true targets.

UT-IGSP

UT-IGSP learns from experiments whose targets it is told only in part, or not at all, and also learns those targets. At Andrey’s defaults it ties the authors’ library in 111 of 112 comparisons and is better in one: the SHD with perfect interventions and 1 off-target, lower on seeds 0 to 19 and again on fresh seeds 2000 to 2019. 17 more were lower on seeds 0 to 19 and on seeds 4000 to 4019, the seeds of the run that chose the default, which favor it, so they count as ties. Run at Andrey’s level, one over the observational rows, the library ties Andrey in 107 of the 112.

• The graphs are Erdos-Renyi DAGs on 20 variables with mean degree 1.5.

• An observational environment comes with 5 experiments. Each hits one designed target and 0 to 3 off-targets the fit is not told, in six groups of settings built from mean shifts, perfect interventions, variance changes, and weight changes.

• Each experiment has 1,000 to 5,000 rows, with 20 datasets per setting and 5 draws each.

• Both packages run at their defaults.

• We count the SHD of the returned interventional essential graph to the true one, and the target error.

UT-IGSP is slower than the library. At 20 variables its median fit took 0.24 s against the library’s 0.14 s, a median ratio of 1.68 per dataset (quartiles 1.47 and 1.93). At 50 variables it took 1.7 s against 0.9 s, and at 100 variables 15.2 s against 16.8 s. A later change in this release makes UT-IGSP prepare the rows about 1.2 times faster on large data (one core, 20 variables, 5 single-target experiments); these comparisons were not rerun after it.

UT-IGSP against the authors' library, at Andrey's defaults

	SHD				Target error
	Andrey	Library	Empty graph	Difference	Andrey	Library	Given targets	Difference
Mean shift of 1
0 off-targets1,000 rows each	0.06	1.08	15.00	-1.02[-1.63, -0.47]lower, first seeds and the default's seeds	0.14	0.20	0.00	-0.06[-0.20, +0.08]
0 off-targets2,000 rows each	0.09	0.14	15.00	-0.05[-0.22, +0.11]	0.07	0.05	0.00	+0.02[-0.09, +0.15]
0 off-targets3,000 rows each	0.10	0.06	15.00	+0.04[-0.09, +0.18]	0.10	0.00	0.00	+0.10[+0.00, +0.25]
0 off-targets4,000 rows each	0.02	0.04	15.00	-0.02[-0.12, +0.06]	0.05	0.00	0.00	+0.05[+0.00, +0.15]
0 off-targets5,000 rows each	0.02	0.04	15.00	-0.02[-0.12, +0.06]	0.00	0.00	0.00	+0.00[+0.00, +0.00]
1 off-target1,000 rows each	0.28	1.18	15.00	-0.90[-1.42, -0.42]lower, first seeds and the default's seeds	0.25	0.57	5.00	-0.32[-0.57, -0.07]lower, first seeds only
1 off-target2,000 rows each	0.30	0.44	15.00	-0.14[-0.40, +0.07]	0.17	0.18	5.00	-0.01[-0.08, +0.07]
1 off-target3,000 rows each	0.18	0.25	15.00	-0.07[-0.39, +0.17]	0.14	0.11	5.00	+0.03[-0.15, +0.20]
1 off-target4,000 rows each	0.01	0.08	15.00	-0.07[-0.20, +0.02]	0.05	0.03	5.00	+0.02[-0.07, +0.15]
1 off-target5,000 rows each	0.00	0.16	15.00	-0.16[-0.44, +0.00]	0.00	0.07	5.00	-0.07[-0.20, +0.00]
2 off-targets1,000 rows each	0.47	1.27	15.00	-0.80[-1.35, -0.26]lower, first seeds only	0.49	1.36	10.00	-0.87[-1.80, -0.15]lower, first seeds only
2 off-targets2,000 rows each	0.32	0.42	15.00	-0.10[-0.36, +0.15]	0.26	0.33	10.00	-0.07[-0.25, +0.12]
2 off-targets3,000 rows each	0.35	0.35	15.00	+0.00[-0.29, +0.30]	0.30	0.28	10.00	+0.02[-0.19, +0.24]
2 off-targets4,000 rows each	0.15	0.25	15.00	-0.10[-0.28, +0.00]	0.20	0.21	10.00	-0.01[-0.14, +0.13]
2 off-targets5,000 rows each	0.20	0.23	15.00	-0.03[-0.24, +0.15]	0.25	0.19	10.00	+0.06[-0.12, +0.30]
3 off-targets1,000 rows each	0.18	1.22	15.00	-1.04[-1.79, -0.40]lower, first seeds only	0.35	1.29	15.00	-0.94[-1.65, -0.35]lower, first seeds only
3 off-targets2,000 rows each	0.46	0.20	15.00	+0.26[-0.09, +0.69]	0.39	0.29	15.00	+0.10[-0.30, +0.48]
3 off-targets3,000 rows each	0.35	0.15	15.00	+0.20[+0.00, +0.43]	0.41	0.20	15.00	+0.21[+0.01, +0.45]higher, first seeds only
3 off-targets4,000 rows each	0.24	0.11	15.00	+0.13[-0.04, +0.36]	0.35	0.16	15.00	+0.19[+0.00, +0.43]
3 off-targets5,000 rows each	0.25	0.10	15.00	+0.15[-0.06, +0.48]	0.26	0.16	15.00	+0.10[-0.04, +0.32]
Perfect intervention
0 off-targets1,000 rows each	0.10	1.04	15.00	-0.94[-1.57, -0.37]lower, first seeds only	0.10	0.20	0.00	-0.10[-0.29, +0.06]
0 off-targets2,000 rows each	0.14	0.14	15.00	+0.00[-0.12, +0.14]	0.09	0.05	0.00	+0.04[-0.03, +0.15]
0 off-targets3,000 rows each	0.10	0.06	15.00	+0.04[-0.09, +0.18]	0.10	0.00	0.00	+0.10[+0.00, +0.25]
0 off-targets4,000 rows each	0.00	0.04	15.00	-0.04[-0.12, +0.00]	0.05	0.00	0.00	+0.05[+0.00, +0.15]
0 off-targets5,000 rows each	0.00	0.04	15.00	-0.04[-0.12, +0.00]	0.00	0.00	0.00	+0.00[+0.00, +0.00]
1 off-target1,000 rows each	0.10	0.89	15.00	-0.79[-1.35, -0.30]better	0.10	0.38	5.00	-0.28[-0.61, +0.00]
1 off-target2,000 rows each	0.12	0.14	15.00	-0.02[-0.15, +0.12]	0.08	0.06	5.00	+0.02[-0.07, +0.15]
1 off-target3,000 rows each	0.10	0.05	15.00	+0.05[-0.08, +0.20]	0.10	0.01	5.00	+0.09[-0.02, +0.25]
1 off-target4,000 rows each	0.01	0.04	15.00	-0.03[-0.12, +0.03]	0.05	0.01	5.00	+0.04[-0.03, +0.15]
1 off-target5,000 rows each	0.00	0.04	15.00	-0.04[-0.12, +0.00]	0.00	0.01	5.00	-0.01[-0.03, +0.00]
2 off-targets1,000 rows each	0.10	1.08	15.00	-0.98[-1.71, -0.39]lower, first seeds and the default's seeds	0.22	1.04	10.00	-0.82[-1.65, -0.16]lower, first seeds only
2 off-targets2,000 rows each	0.05	0.25	15.00	-0.20[-0.49, +0.05]	0.05	0.13	10.00	-0.08[-0.23, +0.05]
2 off-targets3,000 rows each	0.10	0.13	15.00	-0.03[-0.24, +0.14]	0.10	0.06	10.00	+0.04[-0.11, +0.21]
2 off-targets4,000 rows each	0.00	0.08	15.00	-0.08[-0.24, +0.00]	0.05	0.04	10.00	+0.01[-0.12, +0.15]
2 off-targets5,000 rows each	0.00	0.08	15.00	-0.08[-0.24, +0.00]	0.00	0.04	10.00	-0.04[-0.12, +0.00]
3 off-targets1,000 rows each	0.08	0.78	15.00	-0.70[-1.16, -0.33]lower, first seeds and the default's seeds	0.16	0.98	15.00	-0.82[-1.53, -0.27]lower, first seeds only
3 off-targets2,000 rows each	0.05	0.19	15.00	-0.14[-0.41, +0.07]	0.00	0.23	15.00	-0.23[-0.59, +0.00]
3 off-targets3,000 rows each	0.10	0.07	15.00	+0.03[-0.14, +0.19]	0.10	0.06	15.00	+0.04[-0.13, +0.20]
3 off-targets4,000 rows each	0.00	0.06	15.00	-0.06[-0.18, +0.00]	0.05	0.06	15.00	-0.01[-0.18, +0.15]
3 off-targets5,000 rows each	0.00	0.03	15.00	-0.03[-0.09, +0.00]	0.00	0.03	15.00	-0.03[-0.09, +0.00]
Mean shift of 2
0 off-targets1,000 rows each	0.00	0.84	15.00	-0.84[-1.41, -0.34]lower, first seeds only	0.05	0.32	0.00	-0.27[-0.56, -0.01]lower, first seeds and the default's seeds
3 off-targets1,000 rows each	0.20	0.98	15.00	-0.78[-1.38, -0.25]lower, first seeds only	0.28	1.56	15.00	-1.28[-2.34, -0.37]lower, first seeds only
Perfect, variance 0.01
0 off-targets1,000 rows each	0.10	1.04	15.00	-0.94[-1.55, -0.39]lower, first seeds and the default's seeds	0.19	0.20	0.00	-0.01[-0.11, +0.11]
3 off-targets1,000 rows each	0.08	0.78	15.00	-0.70[-1.16, -0.33]lower, first seeds and the default's seeds	0.14	1.00	15.00	-0.86[-1.58, -0.29]lower, first seeds only
Noise doubled
0 off-targets1,000 rows each	0.20	1.01	15.00	-0.81[-1.43, -0.27]lower, first seeds and the default's seeds	0.10	0.12	0.00	-0.02[-0.17, +0.13]
0 off-targets5,000 rows each	0.06	0.04	15.00	+0.02[+0.00, +0.05]	0.00	0.00	0.00	+0.00[+0.00, +0.00]
3 off-targets1,000 rows each	0.20	0.87	15.00	-0.67[-1.08, -0.32]lower, first seeds and the default's seeds	0.15	0.74	15.00	-0.59[-1.14, -0.14]lower, first seeds only
3 off-targets5,000 rows each	0.01	0.03	15.00	-0.02[-0.09, +0.03]	0.00	0.02	15.00	-0.02[-0.06, +0.00]
Mean shift of 1, no targets given
0 off-targets1,000 rows each	0.10	0.95	15.00	-0.85[-1.37, -0.37]lower, first seeds and the default's seeds	0.10	0.13	5.00	-0.03[-0.17, +0.10]
0 off-targets5,000 rows each	0.02	0.04	15.00	-0.02[-0.12, +0.06]	0.00	0.00	5.00	+0.00[+0.00, +0.00]
3 off-targets1,000 rows each	0.18	1.05	15.00	-0.87[-1.55, -0.34]lower, first seeds only	0.35	1.22	20.00	-0.87[-1.58, -0.29]lower, first seeds only
3 off-targets5,000 rows each	0.25	0.17	15.00	+0.08[-0.20, +0.45]	0.26	0.19	20.00	+0.07[-0.10, +0.31]
Parent weights halved
0 off-targets1,000 rows each	1.00	3.06	15.00	-2.06[-3.12, -1.18]lower, first seeds and the default's seeds	0.11	0.11	0.00	+0.00[-0.09, +0.13]
0 off-targets5,000 rows each	0.08	0.39	15.00	-0.31[-0.62, -0.03]lower, first seeds and the default's seeds	0.01	0.19	0.00	-0.18[-0.37, -0.04]lower, first seeds and the default's seeds
3 off-targets1,000 rows each	1.58	3.56	15.00	-1.98[-2.77, -1.26]lower, first seeds and the default's seeds	3.15	5.67	15.00	-2.52[-3.81, -1.39]lower, first seeds and the default's seeds
3 off-targets5,000 rows each	0.10	0.51	15.00	-0.41[-0.74, -0.11]lower, first seeds and the default's seeds	0.12	1.50	15.00	-1.38[-2.55, -0.38]lower, first seeds and the default's seeds

Means over 20 datasets, seeds 0 to 19, 5 draws each, on 20 variables with an observational environment and 5 experiments. Each experiment intervenes on one target it names and on off-targets it does not. The difference is Andrey's mean error minus the library's, with its 95% interval, marked better only where the rerun on fresh seeds, 2000 to 2019, agreed; one lower on the first seeds and on the seeds that chose the default, 4000 to 4019, is noted and counts as a tie; unmarked is a tie. Given targets is the target error of the named targets alone.

IGSP

IGSP learns from experiments whose targets are known. At Andrey’s defaults, on fresh seeds, it is worse than the authors’ library in none of 114 comparisons and lower in 17. Two of those count as better: at 1,000 rows the default level is 1e-3, so seeds 0 to 19 at that level measured the same comparison, and they agreed. With both packages at the same one of three fixed levels, no row of 684 is worse once rerun, and 13 are better.

• The graphs are the 7-variable DAG of Wang et al. (2017) and Erdos-Renyi DAGs on 10 or 20 variables with mean degree 1.5, with linear equations and Gaussian noise.

• An observational environment comes with settings whose targets are known, hit by perfect, inhibiting, or imperfect interventions, at the sizes of Wang et al. (2017) and Yang et al. (2018).

• Each setting has 1,000 to 100,000 rows, with 20 datasets per setting and 5 draws each.

• Andrey’s default level is one over the observational rows, as UT-IGSP’s. The library runs at its defaults but for the level.

• Where a setting has two or more targets, Andrey is also compared with a copy of the library whose I-contradicting test follows Definition 4.2 of Yang et al.; the library’s own test inverts that definition’s conditions.

• We count the SHD of the returned interventional essential graph to the true one.

IGSP is slower than the library in 84 of the 87 settings of its claim. Per setting its median fit took 0.74 to 8.0 times as long: a median of 1.8 times at 1,000 rows per setting, 4.3 at 10,000, and 3.5 at 100,000, and at most about 2 s. These times were measured at alpha 1e-3 before the faster row preparation in this release, and were not rerun after it. Out of claim, at 100 variables and 1,000 rows, the library finished 1 of 10 datasets within 3,600 s and used up to 72 GB of memory; Andrey took a median 33 s and at most 0.5 GB.

IGSP against the authors' library, at Andrey's defaults

	Andrey	Library		Library, Definition 4.2
		SHD	Difference	SHD	Difference
Wang et al., Figure 2: 1 setting
10 variables1,000 rows per setting	2.24	3.04	-0.80[-1.54, +0.00]	2.36	-0.12[-0.34, +0.08]
10 variables10,000 rows per setting	0.90	2.00	-1.10[-1.83, -0.38]lower, first seeds only	0.80	+0.10[-0.04, +0.26]
10 variables100,000 rows per setting	0.60	1.92	-1.32[-1.99, -0.64]lower, first seeds only	0.38	+0.22[+0.00, +0.56]
20 variables1,000 rows per setting	3.60	5.38	-1.78[-2.64, -0.86]better	3.41	+0.19[-0.21, +0.61]
20 variables10,000 rows per setting	0.96	4.31	-3.35[-4.15, -2.54]lower, first seeds only	1.74	-0.78[-1.32, -0.33]lower, first seeds only
20 variables100,000 rows per setting	0.35	3.59	-3.24[-4.11, -2.36]lower, first seeds only	0.57	-0.22[-0.67, +0.15]
Wang et al., Figure 2: 2 settings
10 variables1,000 rows per setting	1.89	2.71	-0.82[-1.70, +0.08]	1.95	-0.06[-0.23, +0.12]
10 variables10,000 rows per setting	0.75	2.00	-1.25[-2.03, -0.46]lower, first seeds only	0.62	+0.13[-0.02, +0.31]
10 variables100,000 rows per setting	0.30	1.68	-1.38[-2.09, -0.63]lower, first seeds only	0.32	-0.02[-0.06, +0.00]
20 variables1,000 rows per setting	3.58	5.69	-2.11[-2.85, -1.37]better	3.28	+0.30[-0.15, +0.77]
20 variables10,000 rows per setting	0.98	4.56	-3.58[-4.34, -2.82]lower, first seeds only	1.70	-0.72[-1.34, -0.23]lower, first seeds only
20 variables100,000 rows per setting	0.25	3.52	-3.27[-4.21, -2.33]lower, first seeds only	0.62	-0.37[-0.87, +0.00]
Wang et al., Figure 3: targets 4
weights from 0.11,000 rows per setting	1.20	1.15	+0.05[+0.00, +0.15]	-	-
weights from 0.110,000 rows per setting	0.30	0.25	+0.05[-0.15, +0.30]	-	-
weights from 0.1100,000 rows per setting	0.00	0.05	-0.05[-0.15, +0.00]	-	-
weights from 0.251,000 rows per setting	0.20	0.26	-0.06[-0.16, +0.00]	-	-
weights from 0.2510,000 rows per setting	0.00	0.05	-0.05[-0.15, +0.00]	-	-
weights from 0.25100,000 rows per setting	0.00	0.05	-0.05[-0.15, +0.00]	-	-
weights from 0.51,000 rows per setting	0.00	0.00	+0.00[+0.00, +0.00]	-	-
weights from 0.510,000 rows per setting	0.00	0.05	-0.05[-0.15, +0.00]	-	-
weights from 0.5100,000 rows per setting	0.00	0.05	-0.05[-0.15, +0.00]	-	-
Wang et al., Figure 3: targets 5
weights from 0.11,000 rows per setting	1.00	0.85	+0.15[+0.00, +0.45]	-	-
weights from 0.110,000 rows per setting	0.25	0.25	+0.00[-0.15, +0.15]	-	-
weights from 0.1100,000 rows per setting	0.00	0.05	-0.05[-0.15, +0.00]	-	-
weights from 0.251,000 rows per setting	0.15	0.21	-0.06[-0.16, +0.00]	-	-
weights from 0.2510,000 rows per setting	0.00	0.05	-0.05[-0.15, +0.00]	-	-
weights from 0.25100,000 rows per setting	0.00	0.05	-0.05[-0.15, +0.00]	-	-
weights from 0.51,000 rows per setting	0.00	0.00	+0.00[+0.00, +0.00]	-	-
weights from 0.510,000 rows per setting	0.00	0.05	-0.05[-0.15, +0.00]	-	-
weights from 0.5100,000 rows per setting	0.00	0.05	-0.05[-0.15, +0.00]	-	-
Wang et al., Figure 3: targets 4 and 5
weights from 0.11,000 rows per setting	0.90	1.00	-0.10[-0.30, +0.10]	0.85	+0.05[+0.00, +0.15]
weights from 0.110,000 rows per setting	0.25	0.25	+0.00[-0.15, +0.15]	0.25	+0.00[-0.15, +0.15]
weights from 0.1100,000 rows per setting	0.00	0.05	-0.05[-0.15, +0.00]	0.05	-0.05[-0.15, +0.00]
weights from 0.251,000 rows per setting	0.15	0.21	-0.06[-0.16, +0.00]	0.21	-0.06[-0.16, +0.00]
weights from 0.2510,000 rows per setting	0.00	0.05	-0.05[-0.15, +0.00]	0.05	-0.05[-0.15, +0.00]
weights from 0.25100,000 rows per setting	0.00	0.05	-0.05[-0.15, +0.00]	0.05	-0.05[-0.15, +0.00]
weights from 0.51,000 rows per setting	0.00	0.00	+0.00[+0.00, +0.00]	0.00	+0.00[+0.00, +0.00]
weights from 0.510,000 rows per setting	0.00	0.05	-0.05[-0.15, +0.00]	0.05	-0.05[-0.15, +0.00]
weights from 0.5100,000 rows per setting	0.00	0.05	-0.05[-0.15, +0.00]	0.05	-0.05[-0.15, +0.00]
Yang et al.: perfect interventions
1 single-target setting1,000 rows per setting	1.40	1.25	+0.15[-0.22, +0.58]	-	-
1 single-target setting10,000 rows per setting	0.30	0.30	+0.00[+0.00, +0.00]	-	-
2 single-target settings1,000 rows per setting	1.10	1.14	-0.04[-0.42, +0.34]	-	-
2 single-target settings10,000 rows per setting	0.40	0.21	+0.19[-0.18, +0.75]	-	-
4 single-target settings1,000 rows per setting	1.28	1.64	-0.36[-1.16, +0.35]	-	-
4 single-target settings10,000 rows per setting	0.15	0.45	-0.30[-0.65, -0.06]lower, first seeds only	-	-
6 single-target settings1,000 rows per setting	1.46	1.44	+0.02[-0.51, +0.51]	-	-
6 single-target settings10,000 rows per setting	0.53	0.60	-0.07[-0.26, +0.11]	-	-
8 single-target settings1,000 rows per setting	1.55	1.37	+0.18[-0.28, +0.67]	-	-
8 single-target settings10,000 rows per setting	0.80	0.79	+0.01[-0.20, +0.26]	-	-
10 single-target settings1,000 rows per setting	1.76	1.74	+0.02[-0.35, +0.34]	-	-
10 single-target settings10,000 rows per setting	1.30	1.20	+0.10[-0.13, +0.37]	-	-
Every variable, 20 variables1,000 rows per setting	3.19	3.51	-0.32[-1.07, +0.31]	-	-
Every variable, 20 variables10,000 rows per setting	1.88	2.37	-0.49[-1.14, +0.11]	-	-
Every pair of variables1,000 rows per setting	1.89	1.92	-0.03[-0.58, +0.47]	1.69	+0.20[-0.25, +0.71]
Every pair of variables10,000 rows per setting	1.20	1.60	-0.40[-0.80, -0.01]lower, first seeds only	1.13	+0.07[-0.20, +0.36]
Yang et al.: inhibiting interventions
1 single-target setting1,000 rows per setting	1.23	1.25	-0.02[-0.36, +0.32]	-	-
1 single-target setting10,000 rows per setting	0.30	0.30	+0.00[+0.00, +0.00]	-	-
2 single-target settings1,000 rows per setting	1.05	1.14	-0.09[-0.76, +0.48]	-	-
2 single-target settings10,000 rows per setting	0.40	0.21	+0.19[-0.18, +0.75]	-	-
4 single-target settings1,000 rows per setting	1.39	1.64	-0.25[-0.94, +0.38]	-	-
4 single-target settings10,000 rows per setting	0.45	0.37	+0.08[-0.10, +0.32]	-	-
6 single-target settings1,000 rows per setting	1.48	1.44	+0.04[-0.73, +0.73]	-	-
6 single-target settings10,000 rows per setting	0.70	0.61	+0.09[-0.13, +0.32]	-	-
8 single-target settings1,000 rows per setting	1.65	1.41	+0.24[-0.08, +0.55]	-	-
8 single-target settings10,000 rows per setting	0.89	0.78	+0.11[-0.25, +0.55]	-	-
10 single-target settings1,000 rows per setting	1.90	1.78	+0.12[-0.18, +0.45]	-	-
10 single-target settings10,000 rows per setting	1.16	1.22	-0.06[-0.32, +0.22]	-	-
Every variable, 20 variables1,000 rows per setting	3.31	3.50	-0.19[-1.15, +0.58]	-	-
Every variable, 20 variables10,000 rows per setting	1.70	2.45	-0.75[-1.32, -0.21]lower, first seeds only	-	-
Every pair of variables1,000 rows per setting	1.62	1.92	-0.30[-0.84, +0.17]	1.59	+0.03[-0.29, +0.37]
Every pair of variables10,000 rows per setting	1.05	1.53	-0.48[-0.92, -0.08]lower, first seeds only	1.13	-0.08[-0.30, +0.16]
Yang et al.: imperfect interventions
1 single-target setting1,000 rows per setting	1.15	1.25	-0.10[-0.46, +0.24]	-	-
1 single-target setting10,000 rows per setting	0.30	0.36	-0.06[-0.18, +0.00]	-	-
2 single-target settings1,000 rows per setting	1.55	1.42	+0.13[-0.20, +0.47]	-	-
2 single-target settings10,000 rows per setting	0.15	0.21	-0.06[-0.18, +0.00]	-	-
4 single-target settings1,000 rows per setting	1.88	1.72	+0.16[-0.45, +0.78]	-	-
4 single-target settings10,000 rows per setting	0.90	0.55	+0.35[-0.25, +0.98]	-	-
6 single-target settings1,000 rows per setting	1.25	1.43	-0.18[-0.84, +0.39]	-	-
6 single-target settings10,000 rows per setting	0.63	0.64	-0.01[-0.23, +0.23]	-	-
8 single-target settings1,000 rows per setting	1.44	1.44	+0.00[-0.25, +0.29]	-	-
8 single-target settings10,000 rows per setting	1.10	1.04	+0.06[-0.30, +0.47]	-	-
10 single-target settings1,000 rows per setting	1.72	1.84	-0.12[-0.67, +0.34]	-	-
10 single-target settings10,000 rows per setting	1.25	1.31	-0.06[-0.47, +0.36]	-	-
Every variable, 20 variables1,000 rows per setting	3.23	3.53	-0.30[-0.79, +0.19]	-	-
Every variable, 20 variables10,000 rows per setting	2.06	2.61	-0.55[-1.13, -0.03]lower, first seeds only	-	-
Every pair of variables1,000 rows per setting	1.73	1.80	-0.07[-0.59, +0.42]	1.80	-0.07[-0.59, +0.42]
Every pair of variables10,000 rows per setting	1.25	1.40	-0.15[-0.58, +0.24]	1.22	+0.03[-0.23, +0.27]

Mean SHD over 20 datasets, seeds 6000 to 6019, 5 draws each. Andrey runs at its defaults: 10 starts, at a level of one over the observational rows, 1e-3 at 1,000 rows, 1e-4 at 10,000, and 1e-5 at 100,000. The library runs 5 times at 1e-3. Definition 4.2 is a copy of the library whose I-contradicting test follows Yang et al.'s definition, run where a setting has two or more targets. The difference is Andrey's mean minus the library's, with its 95% interval, marked better only where seeds 0 to 19 agreed at 1,000 rows, the one size where both levels match; unmarked is a tie.

IGSP at three levels, both packages at the same one

	Against	Rows	Better	Tie	Worse
alpha 1e-4, 5 starts	Library	87	0	87	0
alpha 1e-4, 5 starts	Library, Definition 4.2	27	0	27	0
alpha 1e-4, 10 starts	Library	87	1	86	0
alpha 1e-4, 10 starts	Library, Definition 4.2	27	0	27	0
alpha 1e-3, 5 starts	Library	87	0	87	0
alpha 1e-3, 5 starts	Library, Definition 4.2	27	0	27	0
alpha 1e-3, 10 starts	Library	87	10	77	0
alpha 1e-3, 10 starts	Library, Definition 4.2	27	0	27	0
alpha 1e-2, 5 starts	Library	87	1	86	0
alpha 1e-2, 5 starts	Library, Definition 4.2	27	0	27	0
alpha 1e-2, 10 starts	Library	87	1	86	0
alpha 1e-2, 10 starts	Library, Definition 4.2	27	0	27	0

The 684 rows of the 87 settings above at three levels inside the authors' sweep, the library running 5 times, on seeds 0 to 19. Better and worse count only where a rerun on fresh seeds agreed.

ICP

ICP returns the causal parents of one target, so both packages fit the same datasets and we compare the fits one by one. Pooled over each setting’s sizes, every comparison ties, and Andrey is 41 to 760 times faster.

• The data follow linear Gaussian equations on 5 to 40 variables, as in the ICP paper’s Appendix G, in two environments of 100 to 500 rows.

• A second design applies do-interventions on 6 or 10 variables in 3 or 4 environments of 200 rows.

• Both packages run at alpha 0.05 and test every set of the other variables on every row.

• We count the target’s parents missed plus the variables named wrongly as parents.

With two environments the approximate test accepted the same sets as R on all 5,220 paired fits. Every other difference, 2,112 fits, traces to one of two choices, each rebuilt from the rows. With three or more environments Andrey corrects the approximate test by the number of environments, K, as the paper does, where R uses K - 1; and R’s exact test uses a larger residual variance.

Split by size, two rows differ. One approximate-test row is worse: with do-interventions on 6 variables, the K correction gives Andrey 0.024 more errors per fit (interval +0.002 to +0.046), all from parents it does not name. Andrey follows the paper’s K there, so the row is allowlisted as correct by the paper. One exact-test row is lower: on 9 variables with two environments, Andrey makes 0.012 fewer errors per fit (interval -0.022 to -0.003), on one set of seeds and not rerun.

ICP against InvariantCausalPrediction 0.8, the same fits

	Errors in the parents				FWER		Same accepted sets
	Andrey	R	Empty set	Difference	Andrey	R
Approximate test
5 to 12 variables2 environments	0.964	0.964	1.070	+0.00[+0.00, +0.00]	0.0080	0.0080	5,000 of 5,000
Do-interventions, 6 or 10 variables3 or 4 environments	1.006	1.001	1.250	+0.01[-0.01, +0.02]	0.0090	0.0150	584 of 1,000
16 or 21 variables, at most 3 parents2 environments	0.875	0.875	1.100	+0.00[+0.00, +0.00]	0.0100	0.0100	200 of 200
Exact test
5 to 12 variables2 environments	0.889	0.892	1.070	+0.00[-0.01, +0.00]	0.0064	0.0056	3,568 of 5,000
Do-interventions, 6 or 10 variables3 or 4 environments	1.175	1.177	1.250	+0.00[-0.01, +0.01]	0.0140	0.0110	813 of 1,000
16 or 21 variables, at most 3 parents2 environments	0.835	0.835	1.100	+0.00[+0.00, +0.00]	0.0100	0.0100	131 of 200

Means over the paired fits at alpha 0.05: 50 datasets of each of 100 scenarios with 5 to 12 variables, 50 of each of 20 with do-interventions, and 20 of each of 10 with 16 or 21 variables. The errors count the variables missing from or added to the target's true parents; the FWER is the share of fits that name a variable outside them. The difference is Andrey's mean errors minus R's, with its 95% interval; unmarked is a tie.

ICP's FWER at alpha 0.05, every fit

	Approximate test			Exact test
	FWER	95% interval	Success	FWER	95% interval	Success
In the claim
5 to 12 variables100,000 fits	0.0070	0.0065 to 0.0075	0.477	0.0063	0.0058 to 0.0068	0.494
Do-interventions, 6 or 10 variables4,000 fits	0.0095	0.0067 to 0.0130	0.498	0.0123	0.0091 to 0.0162	0.422
16 or 21 variables, at most 3 parents4,000 fits	0.0075	0.0051 to 0.0107	0.494	0.0065	0.0043 to 0.0095	0.544
40 variables, at most 3 parents1,000 fits	0.0030	0.0006 to 0.0087	0.497	0.0010	0.0000 to 0.0056	0.500
Out of the claim
Interventions that hit the target11,000 fits	0.1843	0.1771 to 0.1916	0.408	0.1146	0.1087 to 0.1207	0.433
A hidden parent of the target4,000 fits	0.0118	0.0086 to 0.0156	0.541	0.0182	0.0143 to 0.0229	0.565
A nonlinear parent of the target4,000 fits	0.0030	0.0016 to 0.0052	0.004	0.0100	0.0072 to 0.0136	0.002

The FWER is the share of fits that name a variable outside the target's true parents, with its 95% Clopper-Pearson interval; success is the share whose estimate equals them. In the claim the environments leave the target's own mechanism alone; out of it they do not, or a hidden or nonlinear parent breaks the linear model.

ICP’s guarantee holds in the claim, where the environments leave the target’s own mechanism alone: the FWER stays well below alpha in every setting, at most 0.0123, and no scenario’s FWER is above alpha beyond sampling error. Where an intervention can hit the target, a set holding a child of the target stays nearly invariant and is accepted, and the FWER rises to 0.18 and 0.11; R gives the same on the same fits (0.1936 and 0.1227). Given the targets, targets leaves the environments that hit the target out of the fit.

With the approximate test at 9 to 15 variables and 1,000 rows, Andrey’s median fit took 0.010 to 0.24 s against R’s 0.72 to 58 s, 72 to 241 times faster. With the exact test, which reads the same matrices in Andrey and solves a system per environment and set in R, it was 458 to 1,554 times faster. Over the paired fits the median ratio ran from 41 to 760.

CAM-UV

CAM-UV learns nonlinear graphs where some variables are hidden. Every setting ties each reference once rerun, lingam at its default set size included. The two settings of the paper’s simulation were better on seeds 0 to 19 and tied on seeds 20 to 39.

• The data have nonlinear additive effects on 4 to 50 observed variables, in the paper’s simulation, with one node in five or in ten hidden, or with none hidden.

• Each dataset has 500 to 2,000 rows, with 20 datasets per setting.

• The references search sets of up to 3 variables, the paper’s size and Andrey’s default; lingam also runs at its default of 2.

• We count the directed-edge errors against the edges CAM-UV can identify as direct, 2 for a reversed edge.

With one hidden variable in five, no implementation beats the empty graph. At 8 observed and 2 hidden variables, Andrey made 1.50 more errors per graph than the empty graph on seeds 0 to 19 (interval +0.30 to +2.75) and 1.20 more on seeds 20 to 40 (-0.20 to +2.60); lingam and causal-learn made 1.85 to 1.90 more and 1.20 to 1.30 more. The empty graph is a floor only for a setting where the reference’s own interval against it lies below 0, so these settings are judged against the references alone.

Andrey’s graph differs from lingam’s on 217 datasets. We traced 202 of them, every one below 50 observed variables, to five documented choices: the visiting order, the HSIC test, the regressions, phase 2, and the cycle drop. The other 15, at 50 observed variables, are not traced.

CAM-UV ran 2.1 to 3.1 times faster than lingam and 7.4 to 13.5 times faster than causal-learn, each searching sets of up to 3 variables, and 1.9 to 3.2 times faster than lingam at its default set size of 2. At 50 observed variables its median fit took 91 s against lingam’s 296 s and causal-learn’s 964 s.

CAM-UV against lingam and causal-learn

	Directed-edge errors				Andrey minus
	Andrey	lingam	causal-learn	Empty graph	lingam	causal-learn
One in five hidden
4 observed + 1 hidden2,000 rows	1.95	1.85	1.80	1.70	+0.10[-0.30, +0.55]	+0.15[-0.30, +0.60]
8 observed + 2 hidden2,000 rows	5.05	5.45	5.40	3.55	-0.40[-1.15, +0.25]	-0.35[-1.15, +0.35]
12 observed + 3 hidden2,000 rows	7.35	7.70	7.95	6.05	-0.35[-1.80, +1.00]	-0.60[-2.00, +0.70]
One in ten hidden
10 observed + 1 hidden1,000 rows	6.60	6.75	7.05	7.00	-0.15[-1.35, +1.05]	-0.45[-1.80, +0.85]
20 observed + 2 hidden1,000 rows	10.40	12.05	10.85	15.30	-1.65[-3.55, +0.20]	-0.45[-2.30, +1.45]
50 observed + 5 hidden1,000 rows	27.90	28.90	30.05	39.60	-1.00[-5.70, +3.80]	-2.15[-6.55, +2.40]
The paper's simulation
10 observed + 4 hidden500 rows	6.90	7.95	8.10	9.35	-1.05[-2.00, -0.25]lower, first seeds only	-1.20[-2.10, -0.40]lower, first seeds only
10 observed + 4 hidden1,000 rows	5.55	7.55	7.00	9.35	-2.00[-3.35, -0.80]lower, first seeds only	-1.45[-2.65, -0.45]lower, first seeds only
Nothing hidden
20 observed1,000 rows	11.55	10.25	10.85	20.00	+1.30[-0.95, +3.60]	+0.70[-1.70, +3.10]

Mean directed-edge errors over 20 datasets, seeds 0 to 19: a missing or extra edge costs 1 and a reversed one 2, against the edges CAM-UV can identify. lingam and causal-learn search sets of up to 3 variables, as Andrey does; lingam at its default set size of 2 ties Andrey in every setting too. The difference is Andrey's mean errors minus the reference's, with its 95% interval, marked better only where seeds 20 to 39 agreed; unmarked is a tie.

DYNOTEARS

DYNOTEARS learns the links within and across time steps of a time series. At 500 steps every setting whose noise has mean 0 ties CausalNex, with an F1 of at least 0.968 for both packages. With exponential noise of mean 1, as the paper’s simulations draw it, CausalNex cannot fit the series’ mean, and Andrey made fewer errors at every size at 500 steps and at 5 variables at 50 steps. On seeds 20 to 39, five of those six were better again; at 100 variables and 500 steps the rerun’s interval reaches 0, so that setting counts as a tie.

• The series are linear vector autoregressions with one lag, from the DYNOTEARS paper, on Erdos-Renyi or Barabasi-Albert graphs with Gaussian or exponential noise.

• They have 5 to 100 variables and 50 or 500 steps, with 20 series per setting.

• Both packages run at the paper’s settings for each length. CausalNex fits without a threshold and then keeps the weights above the paper’s thresholds.

• We count the SHD over the same-step and the lagged edges.

One setting is worse: exponential noise of mean 0 at 100 variables and 50 steps, by 3.20 errors per series on seeds 0 to 19 and 4.15 on seeds 20 to 39 (interval +1.30 to +7.10). It comes from centering. Andrey fits each column around its own mean, the same as an intercept, which the paper’s loss leaves out. Refitted without centering, Andrey ties CausalNex there once rerun. Centering costs errors on short series whose noise has mean 0, and removes 2.3 to 11.1 errors per series at 500 steps when the noise has mean 1.

DYNOTEARS used 1.1 to 3.4 times less CPU time than CausalNex, per setting the median ratio over the paired series, on one core of an Intel Core i7-7820X. At 100 variables and 500 steps its median fit took 299 s against CausalNex’s 393 s.

DYNOTEARS against CausalNex 0.12.2

	SHD over lags 0 and 1
	Andrey	CausalNex	Empty window	Difference
Gaussian noise, 500 steps
5 variables	0.15	0.15	13.5	+0.00[+0.00, +0.00]
10 variables	1.75	1.80	29.3	-0.05[-0.15, +0.00]
20 variables	1.35	1.30	57.6	+0.05[+0.00, +0.15]
50 variables	1.20	1.75	144.6	-0.55[-1.60, +0.05]
100 variables	3.15	3.35	283.2	-0.20[-0.75, +0.20]
Gaussian noise, 50 steps
5 variables	5.00	5.35	13.5	-0.35[-0.80, +0.00]
10 variables	12.35	12.20	29.3	+0.15[-0.30, +0.55]
20 variables	21.65	20.80	57.6	+0.85[+0.05, +1.65]higher, first seeds only
50 variables	59.60	59.00	144.6	+0.60[-0.15, +1.35]
100 variables	143.95	137.05	283.2	+6.90[+4.30, +9.55]higher, first seeds only
Exponential noise of mean 0, 500 steps
5 variables	0.15	0.15	13.5	+0.00[+0.00, +0.00]
10 variables	1.35	1.30	29.3	+0.05[+0.00, +0.15]
20 variables	1.35	1.35	57.6	+0.00[-0.20, +0.20]
50 variables	2.10	1.95	144.6	+0.15[-0.05, +0.35]
100 variables	6.25	6.30	283.2	-0.05[-0.85, +0.55]
Exponential noise of mean 0, 50 steps
5 variables	5.40	5.25	13.5	+0.15[-0.05, +0.35]
10 variables	12.95	12.75	29.3	+0.20[-0.25, +0.70]
20 variables	23.55	23.40	57.6	+0.15[-0.50, +0.75]
50 variables	59.45	58.80	144.6	+0.65[-0.75, +2.00]
100 variables	148.95	145.75	283.2	+3.20[+0.10, +6.00]worse
Barabasi-Albert and block-model graphs, 500 steps
5 variables	0.75	0.75	14.7	+0.00[+0.00, +0.00]
10 variables	0.20	0.25	31.4	-0.05[-0.15, +0.00]
20 variables	0.55	0.55	67.8	+0.00[+0.00, +0.00]
50 variables	1.85	1.95	176.6	-0.10[-0.30, +0.10]
Barabasi-Albert and block-model graphs, 50 steps
5 variables	6.10	5.85	14.7	+0.25[+0.00, +0.55]
10 variables	10.35	10.00	31.4	+0.35[+0.05, +0.65]higher, first seeds only
20 variables	24.20	23.40	67.8	+0.80[-0.65, +3.00]
50 variables	88.45	87.80	176.6	+0.65[-0.65, +2.00]
Exponential noise of mean 1, 500 steps
5 variables	0.15	4.85	13.5	-4.70[-6.40, -3.25]better
10 variables	1.35	12.45	29.3	-11.10[-16.00, -6.50]better
20 variables	1.35	8.50	57.6	-7.15[-10.25, -4.30]better
50 variables	2.15	6.90	144.6	-4.75[-8.15, -2.30]better
100 variables	6.20	8.70	283.2	-2.50[-4.75, -0.75]lower, first seeds only
Exponential noise of mean 1, 50 steps
5 variables	5.40	6.55	13.5	-1.15[-2.10, -0.20]better
10 variables	13.00	14.90	29.3	-1.90[-3.90, +0.10]
20 variables	23.55	23.50	57.6	+0.05[-1.70, +1.95]
50 variables	59.40	62.05	144.6	-2.65[-5.65, +0.20]
100 variables	148.70	147.45	283.2	+1.25[-3.10, +5.50]

Means over 20 series, seeds 0 to 19, at the paper's settings for 500 and for 50 steps. The empty window is the SHD of returning no edge. The difference is Andrey's mean SHD minus CausalNex's, with its 95% interval, marked better or worse only where a rerun on seeds 20 to 39 agreed; unmarked is a tie.

Other changes in 0.14.0

The changelog measures the other changes: the interventional simulator (SCM.do, hard, soft, and envs=), env and targets on every method that learns from environments (gies’s interventions and cdnod’s domain_index still work in this release, with a deprecation warning), the tie rank that ignores offsets, and the faster categorical tests.

Limits

Each result holds for the settings above. These are the places where the methods, or both packages, fall short:

• GnIES was measured on 10 variables, linear-Gaussian data, and environments that change only the noise variance of their targets. With 10 rows per environment neither package beats the empty graph in every setting: in B10 neither does, and in D10 Andrey does not on either seed set while the reference does on the rerun seeds only, so the floor, judged on the first seeds, does not apply there. Hard interventions, mean shifts, weight changes, non-Gaussian noise, nonlinear effects, and 20 to 100 variables lie outside the claim; the comparison page reports them without judging.

• UT-IGSP was measured on 20 variables and 5 experiments of 1,000 to 5,000 rows. At 50 variables with mean degree 5 and 3 experiments, neither package beats the empty graph. With 100 rows per experiment and no off-targets, Andrey names false targets more often than the library.

• IGSP was measured on linear Gaussian data with 7 to 20 variables, known targets, and 1,000 to 100,000 rows per setting. Out of claim, with 100 rows per setting on 10 variables, Andrey made 0.60 more errors than the library on one set of seeds, and at 400 variables it finished no dataset within 3,600 s. A target without causes keeps its distribution under these interventions, so its experiment changes nothing. The I-contradicting test still marks the true arcs out of it, and Andrey reverses 823 of those 3,960 arcs, the library 838, and GIES 16.

• ICP was measured on linear Gaussian data with 5 to 40 variables and 2 to 4 environments of 100 to 500 rows. Its guarantee needs environments that leave the target’s mechanism alone; where they do not, the FWER reaches 0.18, in R too, so pass the known targets. A hidden parent of the target raised the FWER to 0.018, and 0.145 in one scenario of 20. With at most 3 parents searched on 40 variables, it found almost none of them.

• CAM-UV was measured on 4 to 50 observed variables and 500 to 2,000 rows. With one variable in five hidden, no package makes fewer errors than the empty graph: at 8 observed and 2 hidden variables Andrey makes 1.50 more on seeds 0 to 19 (interval +0.30 to +2.75) and 1.20 more on seeds 20 to 40 (interval -0.20 to +2.60), and lingam and causal-learn 1.85 to 1.90 and 1.20 to 1.30 more. With nothing hidden every marked pair is false, and each package marks 3.2 to 3.6 pairs per graph.

• DYNOTEARS was measured on linear vector autoregressions with one lag, 5 to 100 variables, and 50 or 500 steps, at the paper’s settings for each length. With noise of mean 0, centering costs errors on short series: at 100 variables and 50 steps, about 3 per series more than CausalNex. At its defaults on 50 steps, both packages make more errors than the empty window from 20 variables on.

Reproduce

You can find environment setup, campaign drivers, and scores in the benchmark harness guide. Times will differ with the hardware and build; graphs should match closely.

Comparison	Lane	Sizes	Datasets x repeats	Wall cap (s)
GnIES against gnies 0.3.3	gnies_cpu.py	d 10; 10 to 1,000 rows per environment	20 x 1	1,800
UT-IGSP against graphical-model-learning 0.1a8	ut_igsp_cpu.py	d 20; 1,000 to 5,000 rows per setting	20 x 5	1,800
IGSP against graphical-model-learning 0.1a8	igsp_cpu.py	7 to 20 variables; 1,000 to 100,000 rows per setting	20 x 5	1,800
ICP against InvariantCausalPrediction 0.8	icp_cpu.py	5 to 40 variables; 2 to 4 environments of 100 to 500 rows	50 to 1,000 per scenario x 1	none
CAM-UV against lingam 1.13.0 and causal-learn 0.1.4.8	camuv_cpu.py	4 to 50 observed and 0 to 5 hidden variables; 500 to 2,000 rows	20 x 1	3,600
DYNOTEARS against CausalNex 0.12.2	dynotears_ts.py	5 to 100 variables; 50 and 500 steps	20 x 1	3,600