0.8.0 release card: nonlinear tests#
Release card
Nonlinear tests
0.8.0
We compare PC and FCI with the four nonlinear conditional-independence tests 0.8.0 adds, KCI, RCoT, CMIknn, and GPDC, with the packages that implement them: causal-learn in Python, pcalg with RCIT in R, and tigramite in Python. We also compare BOSS and GRaSP with the BDeu score with causal-learn on categorical data, PC and FCI with the batched chi-square and G-square tests with Andrey 0.7.0, and Andrey’s KCI and CMIknn on an NVIDIA A100 GPU with Andrey on the CPU. On an AMD EPYC 7713, PC with RCoT runs more than 10x faster than causal-learn at 50 variables, and BOSS and GRaSP with BDeu more than 130x faster than causal-learn on random categorical graphs; PC with KCI runs more than 2.5x faster than causal-learn at every size we timed, on each CPU.
Speed is not accuracy. On nonmonotone data, Fisher-Z’s errors pass the empty graph’s from 10 variables, and the nonlinear tests stay below it in every package. Each nonlinear test makes fewer errors than its reference package on average, 0.22 to 5.03 fewer errors per graph, with no size significantly worse; see Limits for the sizes where the reference is ahead. Every dataset lists its variables in a random order (see Method). You can find speedups and accuracy in the summary. These results apply to the measured workloads. Speed and accuracy depend on the data, settings, and hardware.
Summary#
We compare each method with every package that implements it with the same test or score. We report every paired speed ratio, regardless of accuracy. Each ratio compares the two packages on one CPU; the nonlinear and the BOSS and GRaSP rows ran on several CPUs, and each section names the CPU of every row. Some measurements are still running; their cells read “pending” (see Pending measurements).
Speed.
Family |
Method |
causal-learn |
pcalg |
tigramite |
Andrey 0.7.0 |
|---|---|---|---|---|---|
Nonlinear tests |
PC, KCI |
2.73-5.56x† |
- |
- |
- |
PC, RCoT |
5.92-35.7x |
9.51-29.2x |
- |
- |
|
PC, CMIknn |
- |
- |
2.40-10.3x |
- |
|
PC, GPDC |
- |
- |
1.38-1.89x |
- |
|
FCI, KCI |
2.76-5.23x |
- |
- |
- |
|
FCI, RCoT |
7.37-16.8x |
- |
- |
- |
|
Categorical scores |
BOSS, BDeu |
157-214x* |
- |
- |
- |
GRaSP, BDeu |
137-199x** |
- |
- |
- |
|
Categorical tests |
PC, chi-square |
- |
- |
- |
3.73-6.42x |
PC, G-square |
- |
- |
- |
3.70-6.26x |
|
FCI, chi-square |
- |
- |
- |
2.00-4.70x |
|
FCI, G-square |
- |
- |
- |
1.99-4.56x |
* BOSS, BDeu on diabetes: causal-learn finished none of 1 within the 2 h cap, so the speedup is at least 8.55x.
** GRaSP, BDeu on diabetes: causal-learn finished none of 1 within the 2 h cap, so the speedup is at least 18.0x.
† More sizes are pending; see Pending measurements.
Each cell is the range of Andrey’s speedup over that package across the method’s sizes, data, and CPUs; each method’s section below gives every size and its CPU. The nonlinear cells leave out the second machine’s rows, which each section lists apart. A dash marks a package we did not compare, and “not timed” a comparison with accuracy but no speed.
Accuracy.
Family |
Method |
causal-learn |
pcalg |
tigramite |
Andrey 0.7.0 |
|---|---|---|---|---|---|
Nonlinear tests |
KCI (PC, FCI) |
+0.22 [+0.10, +0.35] |
- |
- |
- |
RCoT (PC, FCI) |
+1.55 [+1.36, +1.74] |
+5.03 [+4.45, +5.63] |
- |
- |
|
CMIknn (PC) |
- |
- |
+0.70 [+0.49, +0.93] |
- |
|
GPDC (PC) |
- |
- |
+1.23 [+0.98, +1.50] |
- |
|
Categorical scores |
BOSS, BDeu |
-0.45 [-1.48, +0.55] |
- |
- |
- |
GRaSP, BDeu |
+0.22 [-0.78, +1.22] |
- |
- |
- |
|
Categorical tests |
PC, chi-square |
- |
- |
- |
0 |
PC, G-square |
- |
- |
- |
0 |
|
FCI, chi-square |
- |
- |
- |
0 |
|
FCI, G-square |
- |
- |
- |
0 |
Each cell is the mean number of errors Andrey saves per graph: the package’s errors minus Andrey’s, averaged over the datasets both finished, at every size with at least two of them, with its 95% percentile bootstrap interval (10,000 resamples). A positive number means Andrey makes fewer errors than the package, and a negative number more. A nonlinear row pools the datasets of PC and FCI with that test, and RCoT uses the mean of 4 draws per dataset; pcalg and tigramite are compared with PC only. No size is significantly worse: at no size is the interval of Andrey’s errors minus the package’s entirely above 0. The tables for each size below give that difference the other way, Andrey minus the package, so there a negative number means Andrey makes fewer errors. Against Andrey 0.7.0, both builds return the same graph on every dataset, so the difference is 0.
Method#
Data. The nonlinear tests run on two regimes over the same Erdos-Renyi DAGs with mean degree 2,
standardized, with Gaussian noise: additive data, x = sum w (p + sin p) + e, whose near-monotone
effects Fisher-Z still sees, and nonmonotone data, x = sum w cos(p) + e, whose even effects
partial correlation does not see. There are 20 datasets per size, one per seed, for accuracy; speed
is timed on the first 5. BOSS and GRaSP run on categorical data from an Erdos-Renyi DAG with mean
degree 2, 2 or 3 categories per variable, and Dirichlet(1) conditional tables, with 2,000 rows, and
on the diabetes network (413 variables) with 5,000 rows from its own tables. The categorical tests
run on the same random categorical graphs and on the alarm, hepar2, and andes networks with 1,000
and 5,000 rows. BOSS and GRaSP have 20 datasets per size on the random graphs and the categorical
tests 5; the networks give one dataset per size.
Shuffled columns. Every dataset lists its variables in an order drawn from its seed, with the true graph permuted the same way. The network timing rows against Andrey 0.7.0 keep the network’s column order; both builds see the same data.
Accuracy. PC’s errors are the SHD between CPDAGs: one for each pair of variables whose edge is missing, extra, or oriented differently, so the empty graph’s errors are the number of true edges. FCI’s errors are the SHD against the true PAG of the observed variables: one for each pair whose edge is missing, extra, or marked differently. Each nonlinear table also gives Andrey’s PC or FCI with Fisher-Z on the same datasets. Errors are medians over a size’s datasets. “Same graph” means the two packages return the same result. “Andrey minus” a package is the mean over the datasets both finished of Andrey’s errors minus the package’s, with a 95% percentile bootstrap interval (10,000 resamples); below 0, Andrey has fewer errors.
Timing. For the nonlinear tests, each timed dataset runs in 3 rounds on the same 4 cores: in each round, Andrey and then the other package fit every dataset, each fit in its own process after the row’s warm-up fit (none where a fit can pass 300 s). A dataset’s time is its median over the rounds; for each dataset, we divide the other package’s time by Andrey’s, and the speedup is the median of these ratios. A size gets a speedup only when both packages finished every timed dataset in all 3 rounds on one CPU type; otherwise its table says why. In the other lanes, times are seconds per fit, the median over a size’s datasets, and the speedup is the median of the paired ratios. A fit past its cap is coverage, never scored: the tables count the datasets both packages finished, with a bound that counts each unfinished fit at the cap. We show no speedup for a size at which both packages’ median times are under 0.1 s. Times and ratios have three significant figures.
Hardware. Each lane names its machines in the summary’s hardware:
Nonlinear tests, accuracy: server nodes shared with other jobs, almost all AMD EPYC 7713, each fit on its own 4 cores with 4 BLAS and OpenMP threads and a 30,000 MB memory cap. Graphs do not depend on the CPU; which fits pass the cap can. The wall cap was 1,800 s, and 3,600 s for CMIknn, for KCI at 5,000 rows, and for GPDC at 2,000 rows.
Nonlinear tests, speed: each row names its CPU, an AMD EPYC 7713, 9534, or 9555, or an Intel Xeon Gold 5418Y or 6542Y, with each fit on 4 pinned cores, most of them on a node shared with other jobs, and the same caps. A second machine, an Intel Core i7-7820X, timed some sizes on 2 physical cores and their 2 hyperthreads per fit, 3 sizes at once; its rows are listed apart and never compared with the servers’. Inside PC’s skeleton, Andrey runs RCoT, CMIknn, and GPDC on one thread, and KCI on one thread below 2,000 rows.
BOSS and GRaSP on categorical data: the random graphs on an AMD EPYC 7713, each fit on 4 pinned cores of a node shared with other jobs, 1 timed fit per dataset after 1 warm-up fit, and a 3,600 s wall cap; diabetes on an Intel Core i7-7820X, each fit on 4 pinned cores with nothing else running, and a 7,200 s cap with no warm-up.
Categorical tests against Andrey 0.7.0: an AMD EPYC 9534 exclusive node with every core visible to the fit and no core budget, each build in its own environment. The random-graph rows take 1 timed fit after 1 warm-up; the network rows take the faster of 2 fits.
FCI’s feasible sizes: an AMD EPYC 7713, each fit on its own 4 cores, one fit at a time per core group on an exclusive node, and a 30,000 MB memory cap.
Builds and versions#
Comparison |
Andrey build |
Measured package versions |
|---|---|---|
Nonlinear tests with KCI, CMIknn, and GPDC |
Pre-release 0.8.0, October 20, 2026 |
causal-learn 0.1.4.8, tigramite 5.2.10.1 |
Nonlinear tests with RCoT |
Pre-release 0.8.0, October 20, 2026, with quasi-Monte Carlo random Fourier features in RCoT |
causal-learn 0.1.4.8, pcalg 2.7.12 with RCIT 0.1.0 |
BOSS and GRaSP, random graphs |
Pre-release 0.8.0, October 22, 2026 |
causal-learn 0.1.4.8 |
BOSS and GRaSP, diabetes |
Pre-release 0.8.0, October 19, 2026, with the diabetes network |
causal-learn 0.1.4.8 |
Categorical tests |
Pre-release 0.8.0, October 18, 2026, with the batched tests |
Andrey 0.7.0 (pre-release build of October 18, 2026) |
FCI’s feasible sizes |
Pre-release 0.7.0, October 17, 2026 |
Tetrad (causal-cmd 1.12.0, tetrad-lib 7.6.3) |
Fisher-Z beside PC with RCoT comes from the October 20 build without the RCoT change, whose PC with
Fisher-Z returns the same graphs as the RCoT build’s on every dataset checked. Andrey ran on Python
3.13 with NumPy 2.5.4, SciPy 1.16.3, and torch 2.14.1 on the servers, and torch 2.13.0 on the second
machine; SciPy 1.18.1 for BOSS and GRaSP. tigramite ran in its own Python
environment, pcalg and RCIT in R 4.6.1 through rpy2 3.6.9. The 0.7.0 build’s categorical tests and
skeleton search are those of the released 0.7.0; its FCI’s possible-d-separation step is the
column-order form the released 0.7.0 replaced. The
summary contains one row per
comparison, data, and size, the second machine’s rows in entries of their own, the query lane under
tests, and FCI’s feasible sizes under boundary.
Nonlinear tests#
KCI, RCoT, CMIknn, and GPDC are settings of PC and FCI (indep_test). Both packages run the search
at alpha 0.05. The references differ by design: causal-learn’s KCI drops null eigenvalues below
1e-5, which Andrey keeps; causal-learn and RCIT draw RCoT’s random features from their own
generators; tigramite’s CMIknn reports b / B where Andrey reports (b + 1) / (B + 1); and
tigramite’s GPDC uses its own Gaussian process and an analytic null, where Andrey draws a
rank-permutation null of 1,000 draws inside every fit. Each section gives accuracy on 20 datasets
per size, then speed on 5 of them, one row per size and CPU.
PC with KCI#
Data |
d |
n |
Datasets |
Same graph |
Errors, Andrey |
Errors, causal-learn |
Errors, Fisher-Z |
Errors, empty graph |
Andrey ahead |
causal-learn ahead |
Andrey minus causal-learn |
Ahead in column order, Andrey / causal-learn |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
Additive |
5 |
500 |
20 of 20 |
19 of 20 |
0 |
0 |
1 |
5 |
1 |
0 |
-0.10 [-0.30, +0.00] |
0 / 0 |
Additive |
5 |
1,000 |
20 of 20 |
20 of 20 |
0 |
0 |
1 |
5 |
0 |
0 |
+0.00 [+0.00, +0.00] |
0 / 0 |
Additive |
5 |
2,000 |
20 of 20 |
20 of 20 |
0 |
0 |
1 |
5 |
0 |
0 |
+0.00 [+0.00, +0.00] |
0 / 0 |
Additive |
5 |
5,000 |
12 of 20 |
12 of 12 |
0 |
0 |
1.5 |
5 |
0 |
0 |
+0.00 [+0.00, +0.00] |
0 / 0 |
Additive |
10 |
500 |
20 of 20 |
18 of 20 |
2.5 |
2.5 |
4.5 |
10 |
0 |
1 |
+0.15 [+0.00, +0.45] |
0 / 0 |
Additive |
10 |
1,000 |
20 of 20 |
19 of 20 |
1 |
1 |
3.5 |
10 |
1 |
0 |
-0.05 [-0.15, +0.00] |
0 / 0 |
Additive |
10 |
2,000 |
20 of 20 |
16 of 20 |
1 |
1 |
4 |
10 |
3 |
1 |
-0.35 [-1.10, +0.30] |
0 / 0 |
Additive |
20 |
500 |
20 of 20 |
12 of 20 |
8 |
8.5 |
8 |
20 |
4 |
4 |
-0.05 [-0.90, +0.75] |
0 / 0 |
Additive |
20 |
1,000 |
20 of 20 |
5 of 20 |
5.5 |
7 |
8.5 |
20 |
8 |
3 |
-0.60 [-1.25, +0.05] |
0 / 0 |
Additive |
20 |
2,000 |
1 of 20 |
1 of 1 |
3 |
3 |
6 |
20 |
0 |
0 |
+0.00 |
0 / 0 |
Nonmonotone |
5 |
500 |
20 of 20 |
17 of 20 |
3 |
3 |
5 |
5 |
3 |
0 |
-0.35 [-0.75, +0.00] |
0 / 0 |
Nonmonotone |
5 |
1,000 |
20 of 20 |
18 of 20 |
2.5 |
2 |
5 |
5 |
1 |
1 |
-0.10 [-0.45, +0.15] |
0 / 0 |
Nonmonotone |
5 |
2,000 |
20 of 20 |
16 of 20 |
0.5 |
1.5 |
5.5 |
5 |
4 |
0 |
-0.50 [-1.05, -0.10] |
0 / 0 |
Nonmonotone |
5 |
5,000 |
15 of 20 |
12 of 15 |
0 |
0 |
6 |
5 |
3 |
0 |
-0.67 [-1.40, +0.00] |
0 / 0 |
Nonmonotone |
10 |
500 |
20 of 20 |
6 of 20 |
5 |
5 |
13 |
10 |
7 |
5 |
-0.10 [-0.95, +0.80] |
2 / 0 |
Nonmonotone |
10 |
1,000 |
20 of 20 |
5 of 20 |
3 |
4.5 |
14 |
10 |
9 |
5 |
-0.65 [-1.75, +0.45] |
0 / 1 |
Nonmonotone |
10 |
2,000 |
20 of 20 |
9 of 20 |
2 |
2.5 |
14 |
10 |
10 |
1 |
-1.20 [-2.00, -0.45] |
0 / 0 |
Nonmonotone |
20 |
500 |
20 of 20 |
1 of 20 |
10 |
10 |
28 |
20 |
7 |
8 |
+0.35 [-1.45, +2.25] |
0 / 0 |
Nonmonotone |
20 |
1,000 |
20 of 20 |
2 of 20 |
7 |
7 |
28.5 |
20 |
11 |
4 |
-1.05 [-2.05, -0.05] |
1 / 0 |
Nonmonotone |
20 |
2,000 |
14 of 20 |
3 of 14 |
5.5 |
5 |
29.5 |
20 |
8 |
2 |
-1.07 [-2.07, -0.07] |
0 / 1 |
causal-learn’s PC depends on the column order: its pc() orients the colliders in column order and
keeps the first of two that conflict (uc_rule=0, uc_priority=2). The last column compares it
with Andrey’s PC in its column-order mode (collider_order="column", with Meek’s rules one firing
at a time in column order) on the same datasets, like with like; the other columns are Andrey’s
order-free default.
Fits past the cap (coverage, never scored): Additive data, d = 5, n = 5,000: causal-learn 8 of 20. Additive data, d = 20, n = 2,000: Andrey 3 and causal-learn 19 of 20. Nonmonotone data, d = 5, n = 5,000: causal-learn 5 of 20. Nonmonotone data, d = 20, n = 2,000: Andrey 1 and causal-learn 6 of 20.
Data |
d |
n |
CPU |
Andrey (s) |
causal-learn (s) |
Speedup |
|---|---|---|---|---|---|---|
Additive |
5 |
500 |
AMD EPYC 7713 |
1.14 |
4.37 |
3.27x |
Additive |
5 |
1,000 |
AMD EPYC 7713 |
9.29 |
30.3 |
2.99x |
Additive |
5 |
2,000 |
AMD EPYC 7713 |
37.1 |
159 |
4.09x |
Additive |
5 |
5,000 |
AMD EPYC 7713 |
- |
- |
- |
Additive |
10 |
500 |
AMD EPYC 7713 |
5.53 |
22.4 |
3.88x |
Additive |
10 |
1,000 |
AMD EPYC 9534 |
42.4 |
140 |
3.55x |
Additive |
10 |
2,000 |
AMD EPYC 9534 |
121 |
598 |
4.65x |
Additive |
20 |
500 |
AMD EPYC 7713 |
19.3 |
85.5 |
4.43x |
Additive |
20 |
2,000 |
AMD EPYC 7713 |
- |
- |
- |
Nonmonotone |
5 |
500 |
AMD EPYC 7713 |
0.902 |
2.82 |
2.93x |
Nonmonotone |
5 |
1,000 |
AMD EPYC 7713 |
7.86 |
25.6 |
3.23x |
Nonmonotone |
5 |
2,000 |
AMD EPYC 9534 |
18.7 |
105 |
5.56x |
Nonmonotone |
5 |
5,000 |
AMD EPYC 7713 |
- |
- |
- |
Nonmonotone |
10 |
500 |
AMD EPYC 7713 |
4.36 |
14.9 |
3.20x |
Nonmonotone |
10 |
1,000 |
AMD EPYC 9555 |
10.9 |
30.5 |
2.73x |
Nonmonotone |
10 |
2,000 |
AMD EPYC 7713 |
133 |
630 |
4.16x |
Nonmonotone |
20 |
500 |
AMD EPYC 7713 |
13.5 |
51.0 |
3.78x |
Nonmonotone |
20 |
2,000 |
AMD EPYC 7713 |
- |
- |
- |
No speedup: Additive data, d = 5, n = 5,000, AMD EPYC 7713: causal-learn passed the 3,600 s cap on 1 of 5 timed datasets; with each unfinished fit at the cap, at least 2.42x. Additive data, d = 20, n = 2,000, AMD EPYC 7713: causal-learn passed the 1,800 s cap on 5 of 5 timed datasets; with each unfinished fit at the cap, at least 2.09x. Nonmonotone data, d = 5, n = 5,000, AMD EPYC 7713: causal-learn passed the 3,600 s cap on 2 of 5 timed datasets; with each unfinished fit at the cap, at least 2.07x. Nonmonotone data, d = 20, n = 2,000, AMD EPYC 7713: causal-learn passed the 1,800 s cap on 3 of 5 timed datasets; with each unfinished fit at the cap, at least 2.53x.
PC with RCoT#
Against causal-learn:
Data |
d |
n |
Datasets |
Same graph |
Errors, Andrey |
Errors, causal-learn |
Errors, Fisher-Z |
Errors, empty graph |
Andrey ahead |
causal-learn ahead |
Andrey minus causal-learn, mean of 4 draws |
Ahead in column order, Andrey / causal-learn |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
Additive |
5 |
500 |
20 of 20 |
13 of 20 |
1 |
0 |
1 |
5 |
2 |
5 |
-0.06 [-0.30, +0.16] |
2 / 5 |
Additive |
5 |
1,000 |
20 of 20 |
12 of 20 |
0 |
0 |
1 |
5 |
5 |
3 |
-0.47 [-1.00, -0.03] |
5 / 3 |
Additive |
5 |
2,000 |
20 of 20 |
17 of 20 |
0 |
0 |
1 |
5 |
3 |
0 |
-0.12 [-0.41, +0.16] |
3 / 0 |
Additive |
5 |
5,000 |
20 of 20 |
17 of 20 |
0 |
0 |
2 |
5 |
2 |
1 |
-0.09 [-0.41, +0.21] |
2 / 1 |
Additive |
10 |
500 |
20 of 20 |
7 of 20 |
2.5 |
3 |
4.5 |
10 |
7 |
3 |
-0.65 [-1.16, -0.19] |
6 / 3 |
Additive |
10 |
1,000 |
20 of 20 |
11 of 20 |
2 |
2 |
3.5 |
10 |
4 |
4 |
-0.50 [-1.01, -0.09] |
4 / 4 |
Additive |
10 |
2,000 |
20 of 20 |
10 of 20 |
1.5 |
1 |
4 |
10 |
5 |
5 |
-0.03 [-0.57, +0.54] |
4 / 5 |
Additive |
10 |
5,000 |
20 of 20 |
16 of 20 |
0 |
0 |
5.5 |
10 |
0 |
4 |
-0.16 [-0.49, +0.15] |
0 / 4 |
Additive |
20 |
500 |
20 of 20 |
3 of 20 |
7.5 |
7.5 |
8 |
20 |
8 |
8 |
-0.59 [-1.20, +0.03] |
10 / 6 |
Additive |
20 |
1,000 |
20 of 20 |
2 of 20 |
5.5 |
5 |
8.5 |
20 |
11 |
6 |
-0.56 [-0.96, -0.17] |
11 / 4 |
Additive |
20 |
2,000 |
20 of 20 |
3 of 20 |
3.5 |
5 |
9 |
20 |
11 |
5 |
-0.42 [-0.81, -0.06] |
8 / 8 |
Additive |
20 |
5,000 |
20 of 20 |
7 of 20 |
2 |
2.5 |
10 |
20 |
5 |
5 |
-0.60 [-1.35, +0.09] |
4 / 6 |
Additive |
50 |
500 |
20 of 20 |
0 of 20 |
18.5 |
20 |
23 |
50 |
12 |
4 |
-2.50 [-3.66, -1.36] |
11 / 7 |
Additive |
50 |
1,000 |
20 of 20 |
0 of 20 |
14.5 |
14 |
22 |
50 |
12 |
4 |
-1.91 [-2.95, -0.91] |
10 / 6 |
Additive |
50 |
2,000 |
20 of 20 |
0 of 20 |
7.5 |
10 |
25.5 |
50 |
14 |
3 |
-2.40 [-3.38, -1.54] |
9 / 7 |
Additive |
50 |
5,000 |
20 of 20 |
1 of 20 |
7 |
7.5 |
29 |
50 |
9 |
8 |
-1.48 [-2.42, -0.54] |
6 / 10 |
Nonmonotone |
5 |
500 |
20 of 20 |
3 of 20 |
3 |
3 |
5 |
5 |
9 |
4 |
-0.56 [-1.09, -0.05] |
8 / 6 |
Nonmonotone |
5 |
1,000 |
20 of 20 |
7 of 20 |
1.5 |
2.5 |
5 |
5 |
8 |
3 |
-0.65 [-1.13, -0.20] |
5 / 6 |
Nonmonotone |
5 |
2,000 |
20 of 20 |
9 of 20 |
0 |
2 |
5.5 |
5 |
9 |
2 |
-0.61 [-1.15, -0.10] |
8 / 2 |
Nonmonotone |
5 |
5,000 |
20 of 20 |
17 of 20 |
0 |
0 |
6 |
5 |
1 |
2 |
-0.33 [-0.64, -0.06] |
0 / 3 |
Nonmonotone |
10 |
500 |
20 of 20 |
1 of 20 |
5 |
5.5 |
13 |
10 |
10 |
5 |
-1.99 [-2.49, -1.50] |
7 / 5 |
Nonmonotone |
10 |
1,000 |
20 of 20 |
0 of 20 |
3.5 |
5.5 |
14 |
10 |
13 |
5 |
-1.79 [-2.46, -1.09] |
11 / 6 |
Nonmonotone |
10 |
2,000 |
20 of 20 |
5 of 20 |
1.5 |
3 |
14 |
10 |
13 |
1 |
-1.64 [-2.38, -0.86] |
12 / 3 |
Nonmonotone |
10 |
5,000 |
20 of 20 |
5 of 20 |
0.5 |
1.5 |
14 |
10 |
10 |
3 |
-1.25 [-1.81, -0.70] |
7 / 5 |
Nonmonotone |
20 |
500 |
20 of 20 |
0 of 20 |
11 |
13 |
28 |
20 |
9 |
8 |
-2.33 [-3.46, -1.27] |
10 / 6 |
Nonmonotone |
20 |
1,000 |
20 of 20 |
0 of 20 |
7 |
10.5 |
28.5 |
20 |
14 |
2 |
-3.59 [-4.79, -2.48] |
11 / 7 |
Nonmonotone |
20 |
2,000 |
20 of 20 |
0 of 20 |
4.5 |
7.5 |
30 |
20 |
12 |
6 |
-2.10 [-3.30, -0.93] |
11 / 6 |
Nonmonotone |
20 |
5,000 |
20 of 20 |
2 of 20 |
4 |
5 |
30.5 |
20 |
11 |
2 |
-1.81 [-2.85, -0.82] |
8 / 6 |
Nonmonotone |
50 |
500 |
20 of 20 |
0 of 20 |
25 |
37 |
87 |
50 |
19 |
1 |
-8.39 [-10.05, -6.85] |
17 / 2 |
Nonmonotone |
50 |
1,000 |
20 of 20 |
0 of 20 |
14.5 |
27 |
90 |
50 |
17 |
3 |
-8.18 [-9.56, -6.70] |
16 / 3 |
Nonmonotone |
50 |
2,000 |
20 of 20 |
0 of 20 |
13 |
20 |
95 |
50 |
19 |
0 |
-7.08 [-8.90, -5.30] |
15 / 3 |
Nonmonotone |
50 |
5,000 |
20 of 20 |
0 of 20 |
12 |
14 |
98 |
50 |
13 |
7 |
-3.30 [-4.84, -1.74] |
8 / 12 |
Both packages draw random features, so each dataset was also fit 4 times per package, with draw seeds 1000 x the dataset’s seed + k, k = 0 to 3. “Andrey minus causal-learn, mean of 4 draws” is the mean over the datasets of Andrey’s errors minus causal-learn’s, each averaged over the 4 draws, with its 95% bootstrap interval; the other columns count one fit per dataset.
causal-learn’s PC depends on the column order: its pc() orients the colliders in column order and
keeps the first of two that conflict (uc_rule=0, uc_priority=2). The last column compares it
with Andrey’s PC in its column-order mode (collider_order="column", with Meek’s rules one firing
at a time in column order) on the same datasets, like with like; the other columns are Andrey’s
order-free default.
Data |
d |
n |
CPU |
Andrey (s) |
causal-learn (s) |
Speedup |
|---|---|---|---|---|---|---|
Additive |
5 |
500 |
AMD EPYC 7713 |
0.0760 |
0.939 |
10.1x |
Additive |
5 |
1,000 |
AMD EPYC 7713 |
0.109 |
0.831 |
7.47x |
Additive |
5 |
2,000 |
AMD EPYC 7713 |
0.122 |
1.18 |
8.29x |
Additive |
5 |
5,000 |
AMD EPYC 7713 |
0.232 |
1.57 |
5.92x |
Additive |
10 |
500 |
AMD EPYC 7713 |
0.351 |
5.34 |
15.1x |
Additive |
10 |
1,000 |
AMD EPYC 7713 |
0.401 |
6.36 |
13.5x |
Additive |
10 |
2,000 |
AMD EPYC 7713 |
0.576 |
7.48 |
10.6x |
Additive |
10 |
5,000 |
AMD EPYC 7713 |
1.29 |
11.0 |
7.80x |
Additive |
20 |
500 |
AMD EPYC 7713 |
0.899 |
21.3 |
21.2x |
Additive |
20 |
1,000 |
AMD EPYC 7713 |
1.04 |
21.3 |
20.6x |
Additive |
20 |
2,000 |
AMD EPYC 7713 |
1.62 |
24.3 |
15.3x |
Additive |
20 |
5,000 |
AMD EPYC 7713 |
3.98 |
38.9 |
10.6x |
Additive |
50 |
500 |
AMD EPYC 7713 |
3.33 |
114 |
35.7x |
Additive |
50 |
1,000 |
AMD EPYC 7713 |
5.15 |
132 |
25.4x |
Additive |
50 |
2,000 |
AMD EPYC 7713 |
7.30 |
188 |
23.7x |
Additive |
50 |
5,000 |
AMD EPYC 7713 |
33.1 |
425 |
13.7x |
Nonmonotone |
5 |
500 |
AMD EPYC 7713 |
0.0738 |
0.496 |
6.43x |
Nonmonotone |
5 |
1,000 |
AMD EPYC 7713 |
0.0867 |
0.671 |
6.97x |
Nonmonotone |
5 |
2,000 |
AMD EPYC 7713 |
0.123 |
0.869 |
7.03x |
Nonmonotone |
5 |
5,000 |
AMD EPYC 7713 |
0.230 |
1.62 |
6.45x |
Nonmonotone |
10 |
500 |
AMD EPYC 7713 |
0.224 |
2.86 |
13.3x |
Nonmonotone |
10 |
1,000 |
AMD EPYC 7713 |
0.284 |
3.25 |
11.4x |
Nonmonotone |
10 |
2,000 |
AMD EPYC 7713 |
0.455 |
4.02 |
8.71x |
Nonmonotone |
10 |
5,000 |
AMD EPYC 7713 |
0.950 |
7.95 |
7.64x |
Nonmonotone |
20 |
500 |
AMD EPYC 7713 |
0.772 |
10.2 |
13.8x |
Nonmonotone |
20 |
1,000 |
AMD EPYC 7713 |
1.32 |
18.3 |
12.2x |
Nonmonotone |
20 |
2,000 |
AMD EPYC 7713 |
2.01 |
21.9 |
11.6x |
Nonmonotone |
20 |
5,000 |
AMD EPYC 7713 |
5.14 |
38.0 |
8.89x |
Nonmonotone |
50 |
500 |
AMD EPYC 7713 |
2.51 |
46.2 |
18.6x |
Nonmonotone |
50 |
1,000 |
AMD EPYC 7713 |
3.84 |
62.3 |
13.8x |
Nonmonotone |
50 |
2,000 |
AMD EPYC 7713 |
6.98 |
85.4 |
12.2x |
Nonmonotone |
50 |
5,000 |
AMD EPYC 7713 |
18.2 |
213 |
12.3x |
Against pcalg with RCIT’s RCoT:
Data |
d |
n |
Datasets |
Same graph |
Errors, Andrey |
Errors, pcalg |
Errors, Fisher-Z |
Errors, empty graph |
Andrey ahead |
pcalg ahead |
Andrey minus pcalg, mean of 4 draws |
Ahead in column order, Andrey / pcalg |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
Additive |
5 |
500 |
20 of 20 |
14 of 20 |
1 |
1 |
1 |
5 |
3 |
3 |
-0.49 [-1.01, -0.06] |
3 / 3 |
Additive |
5 |
1,000 |
20 of 20 |
14 of 20 |
0 |
0.5 |
1 |
5 |
5 |
1 |
-0.39 [-0.80, -0.04] |
5 / 1 |
Additive |
5 |
2,000 |
20 of 20 |
18 of 20 |
0 |
0 |
1 |
5 |
2 |
0 |
-0.30 [-0.85, +0.06] |
2 / 0 |
Additive |
5 |
5,000 |
20 of 20 |
16 of 20 |
0 |
0 |
2 |
5 |
3 |
1 |
-0.26 [-0.71, +0.11] |
3 / 1 |
Additive |
10 |
500 |
20 of 20 |
5 of 20 |
2.5 |
3 |
4.5 |
10 |
8 |
4 |
-0.59 [-1.11, -0.12] |
7 / 5 |
Additive |
10 |
1,000 |
20 of 20 |
7 of 20 |
2 |
2.5 |
3.5 |
10 |
8 |
4 |
-0.93 [-1.43, -0.46] |
8 / 4 |
Additive |
10 |
2,000 |
20 of 20 |
8 of 20 |
1.5 |
2.5 |
4 |
10 |
8 |
4 |
-0.47 [-1.04, +0.03] |
6 / 5 |
Additive |
10 |
5,000 |
20 of 20 |
11 of 20 |
0 |
0.5 |
5.5 |
10 |
6 |
2 |
-0.53 [-0.91, -0.21] |
6 / 2 |
Additive |
20 |
500 |
20 of 20 |
1 of 20 |
7.5 |
9 |
8 |
20 |
15 |
3 |
-2.10 [-3.26, -0.95] |
15 / 3 |
Additive |
20 |
1,000 |
20 of 20 |
0 of 20 |
5.5 |
9 |
8.5 |
20 |
17 |
3 |
-2.74 [-3.55, -1.86] |
18 / 2 |
Additive |
20 |
2,000 |
20 of 20 |
0 of 20 |
3.5 |
6 |
9 |
20 |
16 |
4 |
-2.38 [-3.40, -1.24] |
15 / 5 |
Additive |
20 |
5,000 |
20 of 20 |
4 of 20 |
2 |
6 |
10 |
20 |
13 |
2 |
-1.73 [-2.76, -0.66] |
11 / 1 |
Additive |
50 |
500 |
20 of 20 |
0 of 20 |
18.5 |
27 |
23 |
50 |
19 |
1 |
-7.55 [-9.96, -5.35] |
19 / 1 |
Additive |
50 |
1,000 |
20 of 20 |
0 of 20 |
14.5 |
23 |
22 |
50 |
19 |
1 |
-8.47 [-10.81, -6.19] |
18 / 2 |
Additive |
50 |
2,000 |
20 of 20 |
0 of 20 |
7.5 |
19 |
25.5 |
50 |
20 |
0 |
-8.97 [-10.81, -7.24] |
19 / 1 |
Additive |
50 |
5,000 |
20 of 20 |
0 of 20 |
7 |
16.5 |
29 |
50 |
20 |
0 |
-9.15 [-10.47, -7.74] |
19 / 1 |
Nonmonotone |
5 |
500 |
20 of 20 |
8 of 20 |
3 |
2.5 |
5 |
5 |
5 |
7 |
-0.26 [-0.78, +0.30] |
5 / 8 |
Nonmonotone |
5 |
1,000 |
20 of 20 |
6 of 20 |
1.5 |
3 |
5 |
5 |
8 |
4 |
-0.62 [-1.21, -0.10] |
7 / 7 |
Nonmonotone |
5 |
2,000 |
20 of 20 |
10 of 20 |
0 |
1 |
5.5 |
5 |
7 |
3 |
-0.64 [-1.07, -0.24] |
6 / 4 |
Nonmonotone |
5 |
5,000 |
20 of 20 |
13 of 20 |
0 |
0.5 |
6 |
5 |
5 |
2 |
-0.55 [-0.95, -0.19] |
5 / 3 |
Nonmonotone |
10 |
500 |
20 of 20 |
0 of 20 |
5 |
5.5 |
13 |
10 |
11 |
6 |
-1.65 [-2.58, -0.76] |
10 / 6 |
Nonmonotone |
10 |
1,000 |
20 of 20 |
2 of 20 |
3.5 |
4.5 |
14 |
10 |
11 |
7 |
-2.04 [-2.88, -1.09] |
8 / 9 |
Nonmonotone |
10 |
2,000 |
20 of 20 |
3 of 20 |
1.5 |
3 |
14 |
10 |
15 |
2 |
-1.60 [-2.40, -0.68] |
12 / 4 |
Nonmonotone |
10 |
5,000 |
20 of 20 |
3 of 20 |
0.5 |
3 |
14 |
10 |
13 |
3 |
-2.08 [-2.80, -1.38] |
9 / 6 |
Nonmonotone |
20 |
500 |
20 of 20 |
0 of 20 |
11 |
12 |
28 |
20 |
9 |
10 |
-2.88 [-3.98, -1.82] |
11 / 9 |
Nonmonotone |
20 |
1,000 |
20 of 20 |
0 of 20 |
7 |
12 |
28.5 |
20 |
18 |
2 |
-5.24 [-6.29, -4.20] |
13 / 3 |
Nonmonotone |
20 |
2,000 |
20 of 20 |
0 of 20 |
4.5 |
10 |
30 |
20 |
17 |
3 |
-4.97 [-6.40, -3.59] |
17 / 2 |
Nonmonotone |
20 |
5,000 |
20 of 20 |
1 of 20 |
4 |
7.5 |
30.5 |
20 |
14 |
2 |
-5.29 [-6.61, -4.12] |
13 / 3 |
Nonmonotone |
50 |
500 |
20 of 20 |
0 of 20 |
25 |
37.5 |
87 |
50 |
20 |
0 |
-15.76 [-17.98, -13.49] |
17 / 1 |
Nonmonotone |
50 |
1,000 |
20 of 20 |
0 of 20 |
14.5 |
36.5 |
90 |
50 |
20 |
0 |
-19.11 [-20.98, -17.27] |
19 / 1 |
Nonmonotone |
50 |
2,000 |
20 of 20 |
0 of 20 |
13 |
35.5 |
95 |
50 |
20 |
0 |
-23.95 [-26.35, -21.64] |
20 / 0 |
Nonmonotone |
50 |
5,000 |
20 of 20 |
0 of 20 |
12 |
37 |
98 |
50 |
20 |
0 |
-27.26 [-28.68, -25.62] |
20 / 0 |
Both packages draw random features, so each dataset was also fit 4 times per package, with draw seeds 1000 x the dataset’s seed + k, k = 0 to 3. “Andrey minus pcalg, mean of 4 draws” is the mean over the datasets of Andrey’s errors minus pcalg’s, each averaged over the 4 draws, with its 95% bootstrap interval; the other columns count one fit per dataset.
pcalg’s PC depends on the column order: its pc() runs with solve.confl = FALSE, its default,
where a later v-structure overwrites an earlier one in column order. The last column compares it
with Andrey’s PC in its column-order mode (collider_order="column", with Meek’s rules one firing
at a time in column order) on the same datasets, like with like; the other columns are Andrey’s
order-free default.
Data |
d |
n |
CPU |
Andrey (s) |
pcalg (s) |
Speedup |
|---|---|---|---|---|---|---|
Additive |
5 |
500 |
AMD EPYC 7713 |
0.0760 |
1.60 |
14.7x |
Additive |
5 |
1,000 |
AMD EPYC 7713 |
0.109 |
1.74 |
17.2x |
Additive |
5 |
2,000 |
AMD EPYC 7713 |
0.122 |
3.36 |
21.6x |
Additive |
5 |
5,000 |
AMD EPYC 7713 |
0.232 |
6.94 |
20.6x |
Additive |
10 |
500 |
AMD EPYC 7713 |
0.351 |
5.70 |
16.2x |
Additive |
10 |
1,000 |
AMD EPYC 7713 |
0.401 |
8.97 |
18.3x |
Additive |
10 |
2,000 |
AMD EPYC 7713 |
0.576 |
14.2 |
23.3x |
Additive |
10 |
5,000 |
AMD EPYC 7713 |
1.29 |
33.2 |
22.5x |
Additive |
20 |
500 |
AMD EPYC 7713 |
0.899 |
16.8 |
19.8x |
Additive |
20 |
1,000 |
AMD EPYC 7713 |
1.04 |
23.6 |
24.6x |
Additive |
20 |
2,000 |
AMD EPYC 7713 |
1.62 |
33.5 |
21.9x |
Additive |
20 |
5,000 |
AMD EPYC 7713 |
3.98 |
91.6 |
27.6x |
Additive |
50 |
500 |
AMD EPYC 7713 |
3.33 |
69.0 |
22.0x |
Additive |
50 |
1,000 |
AMD EPYC 7713 |
5.15 |
128 |
23.0x |
Additive |
50 |
2,000 |
AMD EPYC 7713 |
7.30 |
221 |
29.1x |
Additive |
50 |
5,000 |
AMD EPYC 7713 |
33.1 |
769 |
19.1x |
Nonmonotone |
5 |
500 |
AMD EPYC 7713 |
0.0738 |
0.696 |
9.51x |
Nonmonotone |
5 |
1,000 |
AMD EPYC 7713 |
0.0867 |
1.30 |
15.0x |
Nonmonotone |
5 |
2,000 |
AMD EPYC 7713 |
0.123 |
2.22 |
13.3x |
Nonmonotone |
5 |
5,000 |
AMD EPYC 7713 |
0.230 |
5.72 |
20.3x |
Nonmonotone |
10 |
500 |
AMD EPYC 7713 |
0.224 |
3.63 |
13.9x |
Nonmonotone |
10 |
1,000 |
AMD EPYC 7713 |
0.284 |
5.87 |
19.4x |
Nonmonotone |
10 |
2,000 |
AMD EPYC 7713 |
0.455 |
8.94 |
22.2x |
Nonmonotone |
10 |
5,000 |
AMD EPYC 7713 |
0.950 |
22.9 |
22.4x |
Nonmonotone |
20 |
500 |
AMD EPYC 7713 |
0.772 |
12.2 |
16.9x |
Nonmonotone |
20 |
1,000 |
AMD EPYC 7713 |
1.32 |
25.9 |
17.8x |
Nonmonotone |
20 |
2,000 |
AMD EPYC 7713 |
2.01 |
42.0 |
23.9x |
Nonmonotone |
20 |
5,000 |
AMD EPYC 7713 |
5.14 |
110 |
21.4x |
Nonmonotone |
50 |
500 |
AMD EPYC 7713 |
2.51 |
51.0 |
22.2x |
Nonmonotone |
50 |
1,000 |
AMD EPYC 7713 |
3.84 |
82.8 |
20.2x |
Nonmonotone |
50 |
2,000 |
AMD EPYC 7713 |
6.98 |
162 |
24.0x |
Nonmonotone |
50 |
5,000 |
AMD EPYC 7713 |
18.2 |
504 |
29.2x |
PC with CMIknn#
Andrey’s CMIknn runs much faster on CPUs with AVX-512, so its speedup over tigramite depends on the CPU: about 2.5x on an AMD EPYC 7713 (Zen 3, without AVX-512) and about 10x on the AVX-512 CPUs, an AMD EPYC 9534 and the Intel Xeon Gold 5418Y and 6542Y.
Data |
d |
n |
Datasets |
Same graph |
Errors, Andrey |
Errors, tigramite |
Errors, Fisher-Z |
Errors, empty graph |
Andrey ahead |
tigramite ahead |
Andrey minus tigramite |
Ahead in column order, Andrey / tigramite |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
Additive |
5 |
500 |
20 of 20 |
16 of 20 |
1 |
1 |
1 |
5 |
3 |
1 |
-0.20 [-0.60, +0.05] |
3 / 1 |
Additive |
5 |
1,000 |
20 of 20 |
20 of 20 |
0 |
0 |
1 |
5 |
0 |
0 |
+0.00 [+0.00, +0.00] |
0 / 0 |
Additive |
5 |
2,000 |
20 of 20 |
20 of 20 |
0 |
0 |
1 |
5 |
0 |
0 |
+0.00 [+0.00, +0.00] |
0 / 0 |
Additive |
10 |
500 |
20 of 20 |
8 of 20 |
4 |
5 |
4.5 |
10 |
8 |
1 |
-0.95 [-1.60, -0.35] |
7 / 1 |
Additive |
10 |
1,000 |
20 of 20 |
10 of 20 |
2.5 |
3.5 |
3.5 |
10 |
6 |
4 |
-0.45 [-1.15, +0.20] |
5 / 4 |
Additive |
20 |
500 |
20 of 20 |
3 of 20 |
8.5 |
10 |
8 |
20 |
11 |
1 |
-1.90 [-3.05, -0.85] |
12 / 1 |
Nonmonotone |
5 |
500 |
20 of 20 |
12 of 20 |
3 |
3 |
5 |
5 |
2 |
3 |
+0.05 [-0.25, +0.40] |
2 / 5 |
Nonmonotone |
5 |
1,000 |
20 of 20 |
13 of 20 |
1.5 |
2 |
5 |
5 |
2 |
4 |
+0.05 [-0.25, +0.30] |
1 / 6 |
Nonmonotone |
5 |
2,000 |
20 of 20 |
13 of 20 |
0.5 |
2 |
5.5 |
5 |
5 |
2 |
-0.50 [-1.25, +0.00] |
2 / 5 |
Nonmonotone |
10 |
500 |
20 of 20 |
4 of 20 |
6 |
6.5 |
13 |
10 |
9 |
6 |
-0.55 [-1.35, +0.20] |
10 / 4 |
Nonmonotone |
10 |
1,000 |
18 of 20 |
1 of 18 |
3.5 |
5.5 |
14 |
10 |
13 |
4 |
-1.94 [-3.17, -0.83] |
10 / 5 |
Nonmonotone |
20 |
500 |
20 of 20 |
0 of 20 |
10 |
13 |
28 |
20 |
17 |
1 |
-2.15 [-3.10, -1.25] |
16 / 2 |
tigramite’s PC depends on the column order: its PC runs with contemp_collider_rule="none", which
its documentation says is not order-independent. The last column compares it with Andrey’s PC in its
column-order mode (collider_order="column", with Meek’s rules one firing at a time in column
order) on the same datasets, like with like; the other columns are Andrey’s order-free default.
Fits past the cap (coverage, never scored): Nonmonotone data, d = 10, n = 1,000: tigramite 2 of 20.
Data |
d |
n |
CPU |
Andrey (s) |
tigramite (s) |
Speedup |
|---|---|---|---|---|---|---|
Additive |
5 |
500 |
Intel Xeon Gold 6542Y |
9.38 |
88.7 |
10.3x |
Additive |
5 |
1,000 |
Intel Xeon Gold 5418Y |
37.2 |
334 |
9.19x |
Additive |
5 |
2,000 |
Intel Core i7-7820X, second machine |
40.7 |
1,220 |
31.9x |
Additive |
10 |
500 |
AMD EPYC 7713 |
295 |
655 |
2.40x |
Additive |
10 |
1,000 |
Intel Core i7-7820X, second machine |
59.3 |
1,790 |
32.7x |
Additive |
20 |
500 |
Intel Core i7-7820X, second machine |
68.0 |
1,930 |
27.0x |
Nonmonotone |
5 |
500 |
AMD EPYC 9534 |
8.08 |
81.5 |
10.1x |
Nonmonotone |
5 |
1,000 |
AMD EPYC 7713 |
201 |
495 |
2.46x |
Nonmonotone |
5 |
2,000 |
Intel Core i7-7820X, second machine |
36.6 |
1,140 |
35.8x |
Nonmonotone |
10 |
500 |
AMD EPYC 7713 |
192 |
551 |
2.56x |
Nonmonotone |
10 |
1,000 |
Intel Core i7-7820X, second machine |
42.8 |
1,680 |
37.7x |
Nonmonotone |
20 |
500 |
Intel Core i7-7820X, second machine |
53.2 |
1,710 |
32.2x |
PC with GPDC#
Data |
d |
n |
Datasets |
Same graph |
Errors, Andrey |
Errors, tigramite |
Errors, Fisher-Z |
Errors, empty graph |
Andrey ahead |
tigramite ahead |
Andrey minus tigramite |
Ahead in column order, Andrey / tigramite |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
Additive |
5 |
500 |
20 of 20 |
19 of 20 |
0 |
0.5 |
1 |
5 |
1 |
0 |
-0.20 [-0.60, +0.00] |
1 / 0 |
Additive |
5 |
1,000 |
20 of 20 |
18 of 20 |
0 |
0 |
1 |
5 |
1 |
0 |
-0.05 [-0.15, +0.00] |
0 / 1 |
Additive |
5 |
2,000 |
20 of 20 |
18 of 20 |
0 |
0 |
1 |
5 |
2 |
0 |
-0.45 [-1.10, +0.00] |
2 / 0 |
Additive |
10 |
500 |
20 of 20 |
6 of 20 |
2 |
3 |
4.5 |
10 |
8 |
4 |
-1.00 [-2.00, -0.10] |
8 / 3 |
Additive |
10 |
1,000 |
20 of 20 |
6 of 20 |
1.5 |
3 |
3.5 |
10 |
12 |
1 |
-1.60 [-2.40, -0.80] |
10 / 2 |
Additive |
20 |
500 |
20 of 20 |
1 of 20 |
9 |
10 |
8 |
20 |
12 |
4 |
-1.70 [-2.75, -0.75] |
16 / 1 |
Additive |
20 |
1,000 |
10 of 20 |
0 of 10 |
3.5 |
6.5 |
6.5 |
20 |
8 |
1 |
-2.70 [-4.50, -1.20] |
8 / 0 |
Nonmonotone |
5 |
500 |
20 of 20 |
14 of 20 |
3 |
3 |
5 |
5 |
4 |
1 |
-0.40 [-0.85, +0.00] |
2 / 4 |
Nonmonotone |
5 |
1,000 |
20 of 20 |
14 of 20 |
2 |
2 |
5 |
5 |
5 |
0 |
-0.40 [-0.75, -0.10] |
3 / 3 |
Nonmonotone |
5 |
2,000 |
20 of 20 |
14 of 20 |
0 |
1 |
5.5 |
5 |
3 |
1 |
-0.25 [-0.65, +0.05] |
2 / 3 |
Nonmonotone |
10 |
500 |
20 of 20 |
2 of 20 |
5 |
6.5 |
13 |
10 |
13 |
3 |
-1.60 [-2.65, -0.55] |
11 / 3 |
Nonmonotone |
10 |
1,000 |
20 of 20 |
1 of 20 |
4.5 |
5.5 |
14 |
10 |
12 |
5 |
-1.20 [-2.20, -0.30] |
10 / 5 |
Nonmonotone |
20 |
500 |
20 of 20 |
0 of 20 |
13.5 |
16 |
28 |
20 |
17 |
2 |
-3.00 [-4.00, -2.00] |
14 / 5 |
Nonmonotone |
20 |
1,000 |
18 of 20 |
0 of 18 |
11.5 |
14 |
28.5 |
20 |
15 |
1 |
-3.67 [-5.33, -2.22] |
14 / 3 |
tigramite’s PC depends on the column order: its PC runs with contemp_collider_rule="none", which
its documentation says is not order-independent. The last column compares it with Andrey’s PC in its
column-order mode (collider_order="column", with Meek’s rules one firing at a time in column
order) on the same datasets, like with like; the other columns are Andrey’s order-free default.
Fits past the cap (coverage, never scored): Additive data, d = 20, n = 1,000: tigramite 10 of 20. Nonmonotone data, d = 20, n = 1,000: tigramite 2 of 20.
Data |
d |
n |
CPU |
Andrey (s) |
tigramite (s) |
Speedup |
|---|---|---|---|---|---|---|
Additive |
5 |
500 |
AMD EPYC 7713 |
13.8 |
25.9 |
1.87x |
Additive |
5 |
1,000 |
AMD EPYC 7713 |
49.5 |
80.1 |
1.63x |
Additive |
5 |
2,000 |
Intel Core i7-7820X, second machine |
303 |
415 |
1.21x |
Additive |
10 |
500 |
AMD EPYC 7713 |
36.6 |
76.3 |
1.85x |
Additive |
10 |
1,000 |
Intel Core i7-7820X, second machine |
296 |
431 |
1.46x |
Additive |
20 |
500 |
AMD EPYC 7713 |
146 |
292 |
1.76x |
Additive |
20 |
1,000 |
Intel Core i7-7820X, second machine |
- |
- |
- |
Nonmonotone |
5 |
500 |
AMD EPYC 9534 |
7.45 |
10.7 |
1.44x |
Nonmonotone |
5 |
1,000 |
AMD EPYC 7713 |
66.5 |
102 |
1.54x |
Nonmonotone |
5 |
2,000 |
Intel Core i7-7820X, second machine |
267 |
350 |
1.20x |
Nonmonotone |
10 |
500 |
AMD EPYC 7713 |
36.7 |
66.2 |
1.89x |
Nonmonotone |
10 |
1,000 |
AMD EPYC 7713 |
241 |
342 |
1.42x |
Nonmonotone |
10 |
1,000 |
Intel Core i7-7820X, second machine |
246 |
354 |
1.46x |
Nonmonotone |
20 |
500 |
AMD EPYC 7713 |
122 |
189 |
1.38x |
Nonmonotone |
20 |
1,000 |
Intel Core i7-7820X, second machine |
- |
- |
- |
No speedup: Additive data, d = 20, n = 1,000, Intel Core i7-7820X, second machine: tigramite passed the 1,800 s cap on 1 of 5 timed datasets; with each unfinished fit at the cap, at least 1.60x. Nonmonotone data, d = 20, n = 1,000, Intel Core i7-7820X, second machine: tigramite passed the 1,800 s cap on 1 of 5 timed datasets; with each unfinished fit at the cap, at least 1.24x.
FCI with KCI and RCoT#
With KCI, against causal-learn:
Data |
d |
n |
Datasets |
Same graph |
Errors, Andrey |
Errors, causal-learn |
Errors, Fisher-Z |
Errors, empty graph |
Andrey ahead |
causal-learn ahead |
Andrey minus causal-learn |
Ahead in column order, Andrey / causal-learn |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
Additive |
5 |
500 |
20 of 20 |
20 of 20 |
0 |
0 |
1 |
5 |
0 |
0 |
+0.00 [+0.00, +0.00] |
0 / 0 |
Additive |
5 |
1,000 |
20 of 20 |
20 of 20 |
0 |
0 |
1.5 |
5 |
0 |
0 |
+0.00 [+0.00, +0.00] |
0 / 0 |
Additive |
10 |
500 |
20 of 20 |
18 of 20 |
3 |
3 |
7 |
10 |
0 |
2 |
+0.10 [+0.00, +0.25] |
0 / 1 |
Additive |
10 |
1,000 |
20 of 20 |
20 of 20 |
1.5 |
1.5 |
7 |
10 |
0 |
0 |
+0.00 [+0.00, +0.00] |
0 / 0 |
Nonmonotone |
5 |
500 |
20 of 20 |
20 of 20 |
3 |
3 |
5 |
5 |
0 |
0 |
+0.00 [+0.00, +0.00] |
0 / 0 |
Nonmonotone |
5 |
1,000 |
20 of 20 |
20 of 20 |
2 |
2 |
6 |
5 |
0 |
0 |
+0.00 [+0.00, +0.00] |
0 / 0 |
Nonmonotone |
10 |
500 |
20 of 20 |
17 of 20 |
7 |
7 |
14 |
10 |
3 |
0 |
-0.15 [-0.30, +0.00] |
2 / 0 |
Nonmonotone |
10 |
1,000 |
20 of 20 |
16 of 20 |
6.5 |
6 |
14 |
10 |
2 |
2 |
+0.15 [-0.20, +0.65] |
1 / 2 |
causal-learn’s FCI depends on the column order: its fci() runs the possible-d-separation step and
the orientation rules in column order. The last column compares it with Andrey’s FCI in its
column-order mode (possible_dsep="fast", with the orientation rules one firing at a time in column
order) on the same datasets, like with like; the other columns are Andrey’s order-free default.
Data |
d |
n |
CPU |
Andrey (s) |
causal-learn (s) |
Speedup |
|---|---|---|---|---|---|---|
Additive |
5 |
500 |
AMD EPYC 9534 |
0.978 |
3.95 |
4.69x |
Additive |
5 |
1,000 |
AMD EPYC 9534 |
5.77 |
18.7 |
3.09x |
Additive |
10 |
500 |
AMD EPYC 9534 |
4.38 |
24.1 |
5.23x |
Additive |
10 |
1,000 |
AMD EPYC 9534 |
36.8 |
126 |
3.35x |
Nonmonotone |
5 |
500 |
Intel Xeon Gold 6542Y |
0.441 |
1.47 |
2.97x |
Nonmonotone |
5 |
1,000 |
Intel Xeon Gold 6542Y |
2.94 |
9.10 |
2.76x |
Nonmonotone |
10 |
500 |
Intel Xeon Gold 6542Y |
2.60 |
10.9 |
4.06x |
Nonmonotone |
10 |
1,000 |
Intel Xeon Gold 6542Y |
14.8 |
52.6 |
3.24x |
With RCoT, against causal-learn:
Data |
d |
n |
Datasets |
Same graph |
Errors, Andrey |
Errors, causal-learn |
Errors, Fisher-Z |
Errors, empty graph |
Andrey ahead |
causal-learn ahead |
Andrey minus causal-learn, mean of 4 draws |
Ahead in column order, Andrey / causal-learn |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
Additive |
5 |
500 |
20 of 20 |
16 of 20 |
1 |
0 |
1 |
5 |
0 |
4 |
-0.04 [-0.28, +0.21] |
0 / 4 |
Additive |
5 |
1,000 |
20 of 20 |
14 of 20 |
0 |
0 |
1.5 |
5 |
3 |
3 |
-0.44 [-0.96, +0.01] |
3 / 3 |
Additive |
10 |
500 |
20 of 20 |
5 of 20 |
4 |
5 |
7 |
10 |
6 |
5 |
-0.12 [-0.49, +0.24] |
6 / 5 |
Additive |
10 |
1,000 |
20 of 20 |
9 of 20 |
3.5 |
2 |
7 |
10 |
5 |
6 |
-0.54 [-1.19, +0.03] |
5 / 6 |
Nonmonotone |
5 |
500 |
20 of 20 |
7 of 20 |
3.5 |
3.5 |
5 |
5 |
5 |
6 |
-0.29 [-0.65, +0.09] |
6 / 7 |
Nonmonotone |
5 |
1,000 |
20 of 20 |
8 of 20 |
2 |
2.5 |
6 |
5 |
5 |
5 |
-0.33 [-0.71, +0.04] |
5 / 5 |
Nonmonotone |
10 |
500 |
20 of 20 |
0 of 20 |
7 |
8 |
14 |
10 |
9 |
4 |
-1.18 [-1.65, -0.71] |
8 / 4 |
Nonmonotone |
10 |
1,000 |
20 of 20 |
1 of 20 |
5 |
5.5 |
14 |
10 |
10 |
7 |
-0.86 [-1.29, -0.41] |
10 / 8 |
Both packages draw random features, so each dataset was also fit 4 times per package, with draw seeds 1000 x the dataset’s seed + k, k = 0 to 3. “Andrey minus causal-learn, mean of 4 draws” is the mean over the datasets of Andrey’s errors minus causal-learn’s, each averaged over the 4 draws, with its 95% bootstrap interval; the other columns count one fit per dataset.
causal-learn’s FCI depends on the column order: its fci() runs the possible-d-separation step and
the orientation rules in column order. The last column compares it with Andrey’s FCI in its
column-order mode (possible_dsep="fast", with the orientation rules one firing at a time in column
order) on the same datasets, like with like; the other columns are Andrey’s order-free default.
Data |
d |
n |
CPU |
Andrey (s) |
causal-learn (s) |
Speedup |
|---|---|---|---|---|---|---|
Additive |
5 |
500 |
AMD EPYC 9534 |
0.0693 |
0.701 |
10.6x |
Additive |
5 |
1,000 |
AMD EPYC 9534 |
0.0842 |
0.812 |
9.11x |
Additive |
10 |
500 |
AMD EPYC 9534 |
0.218 |
4.23 |
16.8x |
Additive |
10 |
1,000 |
AMD EPYC 9534 |
0.268 |
4.05 |
11.5x |
Nonmonotone |
5 |
500 |
AMD EPYC 9534 |
0.0693 |
0.472 |
7.37x |
Nonmonotone |
5 |
1,000 |
AMD EPYC 9534 |
0.0710 |
0.696 |
8.81x |
Nonmonotone |
10 |
500 |
AMD EPYC 9534 |
0.203 |
2.50 |
9.54x |
Nonmonotone |
10 |
1,000 |
AMD EPYC 9534 |
0.363 |
3.83 |
10.3x |
KCI and CMIknn on a GPU#
Andrey’s KCI and CMIknn also run on an NVIDIA GPU (ANDREY_BACKEND=cuda). We timed PC and FCI with
them on an NVIDIA A100-SXM4-80GB, with 4 cores of its AMD EPYC 7413 host, on the same 5 datasets
per size as the CPU rows, in 3 rounds. Each speedup divides Andrey’s time on the CPU the row names,
from that CPU’s own timing rounds, by its time on the GPU; a size timed on two CPUs has a row for
each. We publish a size only when every GPU fit returned the CPU’s graph on every dataset.
Method |
Data |
d |
n |
CPU |
Andrey on the CPU (s) |
Andrey on the GPU (s) |
GPU speedup |
Same graph |
|---|---|---|---|---|---|---|---|---|
PC, CMIknn |
Additive |
5 |
500 |
AMD EPYC 7713 |
45.0 |
4.26 |
10.1x |
5 of 5 |
PC, CMIknn |
Additive |
5 |
500 |
Intel Xeon Gold 6542Y |
9.38 |
4.26 |
2.07x |
5 of 5 |
PC, CMIknn |
Additive |
5 |
1,000 |
AMD EPYC 7713 |
183 |
12.5 |
14.2x |
5 of 5 |
PC, CMIknn |
Additive |
5 |
1,000 |
Intel Xeon Gold 5418Y |
37.2 |
12.5 |
2.97x |
5 of 5 |
PC, CMIknn |
Additive |
5 |
2,000 |
Intel Core i7-7820X, second machine |
40.7 |
45.3 |
0.931x |
5 of 5 |
PC, CMIknn |
Additive |
10 |
500 |
AMD EPYC 7713 |
295 |
23.6 |
12.6x |
5 of 5 |
PC, CMIknn |
Additive |
10 |
1,000 |
Intel Core i7-7820X, second machine |
59.3 |
67.5 |
0.915x |
5 of 5 |
PC, CMIknn |
Additive |
20 |
500 |
Intel Core i7-7820X, second machine |
68.0 |
106 |
0.644x |
5 of 5 |
PC, CMIknn |
Nonmonotone |
5 |
500 |
AMD EPYC 7713 |
36.3 |
3.93 |
9.05x |
5 of 5 |
PC, CMIknn |
Nonmonotone |
5 |
500 |
AMD EPYC 9534 |
8.08 |
3.93 |
2.06x |
5 of 5 |
PC, CMIknn |
Nonmonotone |
5 |
1,000 |
AMD EPYC 7713 |
201 |
11.5 |
17.2x |
5 of 5 |
PC, CMIknn |
Nonmonotone |
5 |
2,000 |
Intel Core i7-7820X, second machine |
36.6 |
38.6 |
0.946x |
5 of 5 |
PC, CMIknn |
Nonmonotone |
10 |
500 |
AMD EPYC 7713 |
192 |
15.0 |
12.8x |
5 of 5 |
PC, CMIknn |
Nonmonotone |
10 |
1,000 |
Intel Core i7-7820X, second machine |
42.8 |
46.4 |
0.923x |
5 of 5 |
PC, CMIknn |
Nonmonotone |
20 |
500 |
Intel Core i7-7820X, second machine |
53.2 |
76.8 |
0.690x |
5 of 5 |
PC, KCI |
Additive |
5 |
500 |
AMD EPYC 7713 |
1.14 |
0.387 |
3.46x |
5 of 5 |
PC, KCI |
Additive |
5 |
1,000 |
AMD EPYC 7713 |
9.29 |
1.56 |
6.78x |
5 of 5 |
PC, KCI |
Additive |
5 |
2,000 |
AMD EPYC 7713 |
37.1 |
9.06 |
4.09x |
5 of 5 |
PC, KCI |
Additive |
5 |
5,000 |
AMD EPYC 7713 |
923 |
58.3 |
15.8x |
5 of 5 |
PC, KCI |
Additive |
10 |
500 |
AMD EPYC 7713 |
5.53 |
2.06 |
2.77x |
5 of 5 |
PC, KCI |
Additive |
10 |
1,000 |
AMD EPYC 9534 |
42.4 |
8.62 |
4.58x |
5 of 5 |
PC, KCI |
Additive |
10 |
2,000 |
AMD EPYC 9534 |
122 |
39.9 |
3.15x |
5 of 5 |
PC, KCI |
Additive |
20 |
500 |
AMD EPYC 7713 |
19.3 |
6.81 |
2.70x |
5 of 5 |
PC, KCI |
Nonmonotone |
5 |
500 |
AMD EPYC 7713 |
0.902 |
0.276 |
3.30x |
5 of 5 |
PC, KCI |
Nonmonotone |
5 |
1,000 |
AMD EPYC 7713 |
7.86 |
1.31 |
6.08x |
5 of 5 |
PC, KCI |
Nonmonotone |
5 |
2,000 |
AMD EPYC 7713 |
28.6 |
6.33 |
4.77x |
5 of 5 |
PC, KCI |
Nonmonotone |
5 |
2,000 |
AMD EPYC 9534 |
18.7 |
6.33 |
3.14x |
5 of 5 |
PC, KCI |
Nonmonotone |
5 |
5,000 |
AMD EPYC 7713 |
1,480 |
58.3 |
25.4x |
5 of 5 |
PC, KCI |
Nonmonotone |
10 |
500 |
AMD EPYC 7713 |
4.36 |
1.33 |
3.52x |
5 of 5 |
PC, KCI |
Nonmonotone |
10 |
1,000 |
AMD EPYC 7713 |
28.1 |
5.58 |
5.12x |
5 of 5 |
PC, KCI |
Nonmonotone |
10 |
1,000 |
AMD EPYC 9555 |
10.9 |
5.58 |
1.98x |
5 of 5 |
PC, KCI |
Nonmonotone |
10 |
2,000 |
AMD EPYC 7713 |
133 |
28.0 |
5.29x |
5 of 5 |
PC, KCI |
Nonmonotone |
20 |
500 |
AMD EPYC 7713 |
13.5 |
4.74 |
2.88x |
5 of 5 |
PC, KCI |
Nonmonotone |
20 |
2,000 |
AMD EPYC 7713 |
712 |
119 |
6.03x |
5 of 5 |
FCI, CMIknn |
Additive |
5 |
500 |
AMD EPYC 9534 |
7.55 |
3.80 |
1.93x |
5 of 5 |
FCI, CMIknn |
Nonmonotone |
5 |
500 |
AMD EPYC 9534 |
6.96 |
3.71 |
1.82x |
5 of 5 |
FCI, KCI |
Additive |
5 |
500 |
AMD EPYC 9534 |
0.978 |
0.386 |
2.41x |
5 of 5 |
FCI, KCI |
Additive |
5 |
1,000 |
AMD EPYC 9534 |
5.77 |
1.61 |
4.16x |
5 of 5 |
FCI, KCI |
Additive |
10 |
500 |
AMD EPYC 9534 |
4.38 |
2.23 |
1.95x |
5 of 5 |
FCI, KCI |
Additive |
10 |
1,000 |
AMD EPYC 9534 |
36.8 |
9.85 |
3.74x |
5 of 5 |
FCI, KCI |
Nonmonotone |
5 |
500 |
Intel Xeon Gold 6542Y |
0.441 |
0.275 |
1.52x |
5 of 5 |
FCI, KCI |
Nonmonotone |
5 |
1,000 |
Intel Xeon Gold 6542Y |
2.94 |
1.31 |
2.39x |
5 of 5 |
FCI, KCI |
Nonmonotone |
10 |
500 |
Intel Xeon Gold 6542Y |
2.60 |
1.68 |
1.38x |
5 of 5 |
FCI, KCI |
Nonmonotone |
10 |
1,000 |
Intel Xeon Gold 6542Y |
14.8 |
7.42 |
2.10x |
5 of 5 |
On the GPU, KCI runs 2.70x to 25.4x faster than on an AMD EPYC 7713, the most at 5,000 rows, and 1.38x to 4.58x faster than on the AVX-512 servers. CMIknn runs 9.05x to 17.2x faster than on an AMD EPYC 7713 and 1.82x to 2.97x faster than on the AVX-512 servers, and slower than on the second machine. KCI at 20 variables has no GPU row at 1,000 rows, which has no CPU timing yet, or at 2,000 rows of additive data, where a CPU fit passed its cap (see Pending measurements).
The tests query by query#
Each test answers independent draws (level at alpha 0.05, and the KS distance of their p-values to
the uniform) and dependent draws (power), conditioning on k variables. “Same decision” is the
share of draws on which the reference decides as Andrey does. Seconds per query are the median over
3 rounds, on the CPU each row names.
Test |
n |
k |
Package |
Level |
Power |
KS distance |
Seconds per query |
Same decision as Andrey |
CPU |
|---|---|---|---|---|---|---|---|---|---|
CMIknn |
200 |
0 |
Andrey |
0.08 |
1.00 |
0.10 |
0.234 |
- |
AMD EPYC 7713 |
CMIknn |
200 |
0 |
tigramite |
0.08 |
1.00 |
0.11 |
0.584 |
100% |
AMD EPYC 7713 |
CMIknn |
200 |
1 |
Andrey |
0.04 |
1.00 |
0.14 |
0.298 |
- |
AMD EPYC 7713 |
CMIknn |
200 |
1 |
tigramite |
0.04 |
1.00 |
0.13 |
1.20 |
100% |
AMD EPYC 7713 |
CMIknn |
200 |
2 |
Andrey |
0.02 |
0.90 |
0.11 |
0.310 |
- |
AMD EPYC 7713 |
CMIknn |
200 |
2 |
tigramite |
0.02 |
0.95 |
0.16 |
1.72 |
99% |
AMD EPYC 7713 |
CMIknn |
200 |
3 |
Andrey |
0.12 |
0.85 |
0.26 |
0.312 |
- |
AMD EPYC 7713 |
CMIknn |
200 |
3 |
tigramite |
0.20 |
0.95 |
0.36 |
2.34 |
89% |
AMD EPYC 7713 |
CMIknn |
500 |
0 |
Andrey |
0.02 |
1.00 |
0.13 |
1.23 |
- |
AMD EPYC 7713 |
CMIknn |
500 |
0 |
tigramite |
0.02 |
1.00 |
0.15 |
2.28 |
100% |
AMD EPYC 7713 |
CMIknn |
500 |
1 |
Andrey |
0.02 |
1.00 |
0.17 |
1.39 |
- |
AMD EPYC 7713 |
CMIknn |
500 |
1 |
tigramite |
0.04 |
1.00 |
0.16 |
4.56 |
99% |
AMD EPYC 7713 |
CMIknn |
500 |
2 |
Andrey |
0.02 |
1.00 |
0.14 |
1.42 |
- |
AMD EPYC 7713 |
CMIknn |
500 |
2 |
tigramite |
0.10 |
1.00 |
0.16 |
6.80 |
94% |
AMD EPYC 7713 |
CMIknn |
500 |
3 |
Andrey |
0.16 |
1.00 |
0.33 |
1.45 |
- |
AMD EPYC 7713 |
CMIknn |
500 |
3 |
tigramite |
0.28 |
1.00 |
0.46 |
9.80 |
91% |
AMD EPYC 7713 |
CMIknn |
1,000 |
0 |
Andrey |
0.08 |
1.00 |
0.19 |
1.90 |
- |
AMD EPYC 9534 |
CMIknn |
1,000 |
0 |
tigramite |
0.08 |
1.00 |
0.16 |
5.87 |
100% |
AMD EPYC 9534 |
CMIknn |
1,000 |
1 |
Andrey |
0.04 |
1.00 |
0.13 |
2.15 |
- |
AMD EPYC 9534 |
CMIknn |
1,000 |
1 |
tigramite |
0.08 |
1.00 |
0.12 |
10.8 |
97% |
AMD EPYC 9534 |
CMIknn |
1,000 |
2 |
Andrey |
0.10 |
1.00 |
0.19 |
2.22 |
- |
AMD EPYC 9534 |
CMIknn |
1,000 |
2 |
tigramite |
0.12 |
1.00 |
0.22 |
16.6 |
96% |
AMD EPYC 9534 |
CMIknn |
1,000 |
3 |
Andrey |
0.14 |
1.00 |
0.25 |
2.18 |
- |
AMD EPYC 9534 |
CMIknn |
1,000 |
3 |
tigramite |
0.22 |
1.00 |
0.38 |
24.7 |
89% |
AMD EPYC 9534 |
GPDC |
200 |
0 |
Andrey |
0.04 |
1.00 |
0.17 |
0.229 |
- |
AMD EPYC 7713 |
GPDC |
200 |
0 |
tigramite |
0.08 |
1.00 |
0.15 |
0.000336 |
97% |
AMD EPYC 7713 |
GPDC |
200 |
1 |
Andrey |
0.02 |
1.00 |
0.10 |
0.290 |
- |
AMD EPYC 7713 |
GPDC |
200 |
1 |
tigramite |
0.04 |
1.00 |
0.13 |
0.0598 |
99% |
AMD EPYC 7713 |
GPDC |
200 |
2 |
Andrey |
0.06 |
1.00 |
0.12 |
0.291 |
- |
AMD EPYC 7713 |
GPDC |
200 |
2 |
tigramite |
0.08 |
1.00 |
0.15 |
0.0603 |
99% |
AMD EPYC 7713 |
GPDC |
200 |
3 |
Andrey |
0.00 |
1.00 |
0.22 |
0.292 |
- |
AMD EPYC 7713 |
GPDC |
200 |
3 |
tigramite |
0.04 |
1.00 |
0.21 |
0.0676 |
97% |
AMD EPYC 7713 |
GPDC |
500 |
0 |
Andrey |
0.02 |
1.00 |
0.11 |
1.37 |
- |
AMD EPYC 7713 |
GPDC |
500 |
0 |
tigramite |
0.02 |
1.00 |
0.14 |
0.000699 |
100% |
AMD EPYC 7713 |
GPDC |
500 |
1 |
Andrey |
0.02 |
1.00 |
0.11 |
2.01 |
- |
AMD EPYC 7713 |
GPDC |
500 |
1 |
tigramite |
0.04 |
1.00 |
0.09 |
0.682 |
99% |
AMD EPYC 7713 |
GPDC |
500 |
2 |
Andrey |
0.04 |
1.00 |
0.21 |
1.77 |
- |
AMD EPYC 7713 |
GPDC |
500 |
2 |
tigramite |
0.04 |
1.00 |
0.23 |
0.664 |
100% |
AMD EPYC 7713 |
GPDC |
500 |
3 |
Andrey |
0.08 |
1.00 |
0.09 |
1.86 |
- |
AMD EPYC 7713 |
GPDC |
500 |
3 |
tigramite |
0.08 |
1.00 |
0.11 |
0.678 |
100% |
AMD EPYC 7713 |
GPDC |
1,000 |
0 |
Andrey |
0.14 |
1.00 |
0.18 |
4.21 |
- |
AMD EPYC 9534 |
GPDC |
1,000 |
0 |
tigramite |
0.12 |
1.00 |
0.19 |
0.00107 |
99% |
AMD EPYC 9534 |
GPDC |
1,000 |
1 |
Andrey |
0.12 |
1.00 |
0.16 |
6.85 |
- |
AMD EPYC 9534 |
GPDC |
1,000 |
1 |
tigramite |
0.12 |
1.00 |
0.18 |
2.78 |
100% |
AMD EPYC 9534 |
GPDC |
1,000 |
2 |
Andrey |
0.08 |
1.00 |
0.09 |
7.09 |
- |
AMD EPYC 9534 |
GPDC |
1,000 |
2 |
tigramite |
0.08 |
1.00 |
0.14 |
2.82 |
100% |
AMD EPYC 9534 |
GPDC |
1,000 |
3 |
Andrey |
0.04 |
1.00 |
0.17 |
6.97 |
- |
AMD EPYC 9534 |
GPDC |
1,000 |
3 |
tigramite |
0.04 |
1.00 |
0.22 |
2.72 |
100% |
AMD EPYC 9534 |
KCI |
200 |
0 |
Andrey |
0.07 |
1.00 |
0.05 |
0.00223 |
- |
AMD EPYC 7713 |
KCI |
200 |
0 |
RCIT |
0.05 |
1.00 |
0.07 |
0.00550 |
92% |
AMD EPYC 7713 |
KCI |
200 |
0 |
causal-learn |
0.07 |
1.00 |
0.05 |
0.00397 |
100% |
AMD EPYC 7713 |
KCI |
200 |
0 |
causal-learn FastKCI |
0.14 |
1.00 |
0.20 |
0.645 |
87% |
AMD EPYC 7713 |
KCI |
200 |
1 |
Andrey |
0.07 |
1.00 |
0.09 |
0.00663 |
- |
AMD EPYC 7713 |
KCI |
200 |
1 |
RCIT |
0.17 |
1.00 |
0.16 |
0.0160 |
91% |
AMD EPYC 7713 |
KCI |
200 |
1 |
causal-learn |
0.07 |
1.00 |
0.09 |
0.0172 |
100% |
AMD EPYC 7713 |
KCI |
200 |
1 |
causal-learn FastKCI |
0.10 |
1.00 |
0.43 |
0.573 |
87% |
AMD EPYC 7713 |
KCI |
200 |
2 |
Andrey |
0.06 |
1.00 |
0.17 |
0.00685 |
- |
AMD EPYC 7713 |
KCI |
200 |
2 |
RCIT |
0.23 |
1.00 |
0.20 |
0.0180 |
84% |
AMD EPYC 7713 |
KCI |
200 |
2 |
causal-learn |
0.06 |
1.00 |
0.17 |
0.0195 |
100% |
AMD EPYC 7713 |
KCI |
200 |
2 |
causal-learn FastKCI |
0.29 |
1.00 |
0.54 |
0.493 |
73% |
AMD EPYC 7713 |
KCI |
200 |
3 |
Andrey |
0.10 |
1.00 |
0.25 |
0.00724 |
- |
AMD EPYC 7713 |
KCI |
200 |
3 |
RCIT |
0.28 |
1.00 |
0.24 |
0.0180 |
81% |
AMD EPYC 7713 |
KCI |
200 |
3 |
causal-learn |
0.10 |
1.00 |
0.25 |
0.0214 |
100% |
AMD EPYC 7713 |
KCI |
200 |
3 |
causal-learn FastKCI |
0.32 |
0.90 |
0.62 |
0.453 |
70% |
AMD EPYC 7713 |
KCI |
500 |
0 |
Andrey |
0.07 |
1.00 |
0.06 |
0.00919 |
- |
AMD EPYC 7713 |
KCI |
500 |
0 |
RCIT |
0.04 |
1.00 |
0.07 |
0.0560 |
94% |
AMD EPYC 7713 |
KCI |
500 |
0 |
causal-learn |
0.07 |
1.00 |
0.06 |
0.0177 |
100% |
AMD EPYC 7713 |
KCI |
500 |
0 |
causal-learn FastKCI |
0.07 |
1.00 |
0.09 |
0.875 |
91% |
AMD EPYC 7713 |
KCI |
500 |
1 |
Andrey |
0.07 |
1.00 |
0.07 |
0.0607 |
- |
AMD EPYC 7713 |
KCI |
500 |
1 |
RCIT |
0.04 |
1.00 |
0.04 |
0.154 |
96% |
AMD EPYC 7713 |
KCI |
500 |
1 |
causal-learn |
0.07 |
1.00 |
0.07 |
0.141 |
100% |
AMD EPYC 7713 |
KCI |
500 |
1 |
causal-learn FastKCI |
0.05 |
1.00 |
0.29 |
0.966 |
90% |
AMD EPYC 7713 |
KCI |
500 |
2 |
Andrey |
0.07 |
1.00 |
0.15 |
0.0613 |
- |
AMD EPYC 7713 |
KCI |
500 |
2 |
RCIT |
0.07 |
1.00 |
0.15 |
0.161 |
94% |
AMD EPYC 7713 |
KCI |
500 |
2 |
causal-learn |
0.07 |
1.00 |
0.16 |
0.146 |
100% |
AMD EPYC 7713 |
KCI |
500 |
2 |
causal-learn FastKCI |
0.14 |
1.00 |
0.39 |
0.927 |
81% |
AMD EPYC 7713 |
KCI |
500 |
3 |
Andrey |
0.08 |
1.00 |
0.16 |
0.0646 |
- |
AMD EPYC 7713 |
KCI |
500 |
3 |
RCIT |
0.08 |
1.00 |
0.15 |
0.169 |
93% |
AMD EPYC 7713 |
KCI |
500 |
3 |
causal-learn |
0.08 |
1.00 |
0.15 |
0.159 |
99% |
AMD EPYC 7713 |
KCI |
500 |
3 |
causal-learn FastKCI |
0.26 |
1.00 |
0.50 |
0.857 |
76% |
AMD EPYC 7713 |
KCI |
1,000 |
0 |
Andrey |
0.07 |
1.00 |
0.06 |
0.0520 |
- |
AMD EPYC 7713 |
KCI |
1,000 |
0 |
RCIT |
0.07 |
1.00 |
0.09 |
0.384 |
92% |
AMD EPYC 7713 |
KCI |
1,000 |
0 |
causal-learn |
0.07 |
1.00 |
0.06 |
0.0538 |
100% |
AMD EPYC 7713 |
KCI |
1,000 |
0 |
causal-learn FastKCI |
0.05 |
1.00 |
0.04 |
1.23 |
91% |
AMD EPYC 7713 |
KCI |
1,000 |
1 |
Andrey |
0.07 |
1.00 |
0.06 |
0.402 |
- |
AMD EPYC 7713 |
KCI |
1,000 |
1 |
RCIT |
0.06 |
1.00 |
0.10 |
0.992 |
96% |
AMD EPYC 7713 |
KCI |
1,000 |
1 |
causal-learn |
0.07 |
1.00 |
0.06 |
0.872 |
100% |
AMD EPYC 7713 |
KCI |
1,000 |
1 |
causal-learn FastKCI |
0.07 |
1.00 |
0.21 |
1.69 |
89% |
AMD EPYC 7713 |
KCI |
1,000 |
2 |
Andrey |
0.06 |
1.00 |
0.12 |
0.403 |
- |
AMD EPYC 7713 |
KCI |
1,000 |
2 |
RCIT |
0.07 |
1.00 |
0.16 |
1.01 |
93% |
AMD EPYC 7713 |
KCI |
1,000 |
2 |
causal-learn |
0.06 |
1.00 |
0.12 |
0.911 |
100% |
AMD EPYC 7713 |
KCI |
1,000 |
2 |
causal-learn FastKCI |
0.10 |
1.00 |
0.22 |
1.74 |
88% |
AMD EPYC 7713 |
KCI |
1,000 |
3 |
Andrey |
0.09 |
1.00 |
0.17 |
0.409 |
- |
AMD EPYC 7713 |
KCI |
1,000 |
3 |
RCIT |
0.10 |
1.00 |
0.26 |
1.04 |
93% |
AMD EPYC 7713 |
KCI |
1,000 |
3 |
causal-learn |
0.10 |
1.00 |
0.16 |
0.960 |
100% |
AMD EPYC 7713 |
KCI |
1,000 |
3 |
causal-learn FastKCI |
0.18 |
1.00 |
0.40 |
1.75 |
82% |
AMD EPYC 7713 |
KCI |
2,000 |
0 |
Andrey |
0.07 |
1.00 |
0.07 |
0.104 |
- |
AMD EPYC 9534 |
KCI |
2,000 |
0 |
RCIT |
0.04 |
1.00 |
0.09 |
2.23 |
92% |
AMD EPYC 9534 |
KCI |
2,000 |
0 |
causal-learn |
0.07 |
1.00 |
0.07 |
0.122 |
100% |
AMD EPYC 9534 |
KCI |
2,000 |
0 |
causal-learn FastKCI |
0.04 |
1.00 |
0.05 |
2.30 |
91% |
AMD EPYC 9534 |
KCI |
2,000 |
1 |
Andrey |
0.04 |
1.00 |
0.09 |
3.77 |
- |
AMD EPYC 7713 |
KCI |
2,000 |
1 |
RCIT |
0.04 |
1.00 |
0.06 |
7.17 |
95% |
AMD EPYC 7713 |
KCI |
2,000 |
1 |
causal-learn |
0.04 |
1.00 |
0.09 |
8.85 |
100% |
AMD EPYC 7713 |
KCI |
2,000 |
1 |
causal-learn FastKCI |
0.06 |
1.00 |
0.09 |
5.71 |
93% |
AMD EPYC 7713 |
KCI |
2,000 |
2 |
Andrey |
0.04 |
1.00 |
0.08 |
2.24 |
- |
AMD EPYC 9534 |
KCI |
2,000 |
2 |
RCIT |
0.12 |
1.00 |
0.24 |
5.77 |
88% |
AMD EPYC 9534 |
KCI |
2,000 |
2 |
causal-learn |
0.04 |
1.00 |
0.07 |
5.68 |
100% |
AMD EPYC 9534 |
KCI |
2,000 |
2 |
causal-learn FastKCI |
0.16 |
1.00 |
0.28 |
4.46 |
85% |
AMD EPYC 9534 |
KCI |
2,000 |
3 |
Andrey |
0.16 |
1.00 |
0.22 |
3.23 |
- |
AMD EPYC 7713 |
KCI |
2,000 |
3 |
RCIT |
0.23 |
1.00 |
0.39 |
7.91 |
81% |
AMD EPYC 7713 |
KCI |
2,000 |
3 |
causal-learn |
0.17 |
1.00 |
0.23 |
9.61 |
99% |
AMD EPYC 7713 |
KCI |
2,000 |
3 |
causal-learn FastKCI |
0.21 |
1.00 |
0.45 |
7.43 |
77% |
AMD EPYC 7713 |
RCoT |
200 |
0 |
Andrey |
0.05 |
1.00 |
0.07 |
0.00199 |
- |
AMD EPYC 7713 |
RCoT |
200 |
0 |
RCIT |
0.04 |
1.00 |
0.08 |
0.00300 |
95% |
AMD EPYC 7713 |
RCoT |
200 |
0 |
causal-learn |
0.07 |
1.00 |
0.09 |
0.00387 |
95% |
AMD EPYC 7713 |
RCoT |
200 |
1 |
Andrey |
0.04 |
1.00 |
0.07 |
0.00368 |
- |
AMD EPYC 7713 |
RCoT |
200 |
1 |
RCIT |
0.09 |
1.00 |
0.08 |
0.00900 |
93% |
AMD EPYC 7713 |
RCoT |
200 |
1 |
causal-learn |
0.07 |
1.00 |
0.06 |
0.00655 |
94% |
AMD EPYC 7713 |
RCoT |
200 |
2 |
Andrey |
0.06 |
1.00 |
0.13 |
0.00375 |
- |
AMD EPYC 7713 |
RCoT |
200 |
2 |
RCIT |
0.06 |
1.00 |
0.16 |
0.00900 |
94% |
AMD EPYC 7713 |
RCoT |
200 |
2 |
causal-learn |
0.09 |
1.00 |
0.17 |
0.00737 |
92% |
AMD EPYC 7713 |
RCoT |
200 |
3 |
Andrey |
0.16 |
1.00 |
0.25 |
0.00383 |
- |
AMD EPYC 7713 |
RCoT |
200 |
3 |
RCIT |
0.10 |
1.00 |
0.26 |
0.00900 |
85% |
AMD EPYC 7713 |
RCoT |
200 |
3 |
causal-learn |
0.12 |
1.00 |
0.23 |
0.00822 |
82% |
AMD EPYC 7713 |
RCoT |
500 |
0 |
Andrey |
0.04 |
1.00 |
0.07 |
0.00543 |
- |
AMD EPYC 7713 |
RCoT |
500 |
0 |
RCIT |
0.03 |
1.00 |
0.06 |
0.00800 |
95% |
AMD EPYC 7713 |
RCoT |
500 |
0 |
causal-learn |
0.04 |
1.00 |
0.09 |
0.0152 |
96% |
AMD EPYC 7713 |
RCoT |
500 |
1 |
Andrey |
0.04 |
1.00 |
0.06 |
0.00954 |
- |
AMD EPYC 7713 |
RCoT |
500 |
1 |
RCIT |
0.04 |
1.00 |
0.06 |
0.0240 |
95% |
AMD EPYC 7713 |
RCoT |
500 |
1 |
causal-learn |
0.04 |
1.00 |
0.08 |
0.0174 |
95% |
AMD EPYC 7713 |
RCoT |
500 |
2 |
Andrey |
0.12 |
1.00 |
0.27 |
0.00976 |
- |
AMD EPYC 7713 |
RCoT |
500 |
2 |
RCIT |
0.11 |
1.00 |
0.24 |
0.0260 |
90% |
AMD EPYC 7713 |
RCoT |
500 |
2 |
causal-learn |
0.08 |
1.00 |
0.16 |
0.0171 |
86% |
AMD EPYC 7713 |
RCoT |
500 |
3 |
Andrey |
0.14 |
1.00 |
0.26 |
0.00993 |
- |
AMD EPYC 7713 |
RCoT |
500 |
3 |
RCIT |
0.20 |
1.00 |
0.41 |
0.0260 |
80% |
AMD EPYC 7713 |
RCoT |
500 |
3 |
causal-learn |
0.23 |
1.00 |
0.35 |
0.0173 |
75% |
AMD EPYC 7713 |
RCoT |
1,000 |
0 |
Andrey |
0.04 |
1.00 |
0.09 |
0.00584 |
- |
AMD EPYC 7713 |
RCoT |
1,000 |
0 |
RCIT |
0.04 |
1.00 |
0.08 |
0.00800 |
97% |
AMD EPYC 7713 |
RCoT |
1,000 |
0 |
causal-learn |
0.04 |
1.00 |
0.09 |
0.0117 |
97% |
AMD EPYC 7713 |
RCoT |
1,000 |
1 |
Andrey |
0.09 |
1.00 |
0.18 |
0.0111 |
- |
AMD EPYC 7713 |
RCoT |
1,000 |
1 |
RCIT |
0.07 |
1.00 |
0.19 |
0.0370 |
88% |
AMD EPYC 7713 |
RCoT |
1,000 |
1 |
causal-learn |
0.07 |
1.00 |
0.12 |
0.0195 |
89% |
AMD EPYC 7713 |
RCoT |
1,000 |
2 |
Andrey |
0.20 |
1.00 |
0.25 |
0.0114 |
- |
AMD EPYC 7713 |
RCoT |
1,000 |
2 |
RCIT |
0.18 |
1.00 |
0.33 |
0.0380 |
86% |
AMD EPYC 7713 |
RCoT |
1,000 |
2 |
causal-learn |
0.20 |
1.00 |
0.31 |
0.0185 |
78% |
AMD EPYC 7713 |
RCoT |
1,000 |
3 |
Andrey |
0.34 |
1.00 |
0.47 |
0.0116 |
- |
AMD EPYC 7713 |
RCoT |
1,000 |
3 |
RCIT |
0.64 |
1.00 |
0.74 |
0.0380 |
64% |
AMD EPYC 7713 |
RCoT |
1,000 |
3 |
causal-learn |
0.47 |
1.00 |
0.52 |
0.0185 |
66% |
AMD EPYC 7713 |
RCoT |
2,000 |
0 |
Andrey |
0.04 |
1.00 |
0.08 |
0.00675 |
- |
AMD EPYC 7713 |
RCoT |
2,000 |
0 |
RCIT |
0.03 |
1.00 |
0.07 |
0.00900 |
97% |
AMD EPYC 7713 |
RCoT |
2,000 |
0 |
causal-learn |
0.06 |
1.00 |
0.09 |
0.0110 |
96% |
AMD EPYC 7713 |
RCoT |
2,000 |
1 |
Andrey |
0.07 |
1.00 |
0.09 |
0.0145 |
- |
AMD EPYC 7713 |
RCoT |
2,000 |
1 |
RCIT |
0.13 |
1.00 |
0.20 |
0.0600 |
88% |
AMD EPYC 7713 |
RCoT |
2,000 |
1 |
causal-learn |
0.06 |
1.00 |
0.16 |
0.0242 |
91% |
AMD EPYC 7713 |
RCoT |
2,000 |
2 |
Andrey |
0.36 |
1.00 |
0.49 |
0.0148 |
- |
AMD EPYC 7713 |
RCoT |
2,000 |
2 |
RCIT |
0.48 |
1.00 |
0.65 |
0.0610 |
70% |
AMD EPYC 7713 |
RCoT |
2,000 |
2 |
causal-learn |
0.41 |
1.00 |
0.52 |
0.0239 |
62% |
AMD EPYC 7713 |
RCoT |
2,000 |
3 |
Andrey |
0.71 |
1.00 |
0.73 |
0.0153 |
- |
AMD EPYC 7713 |
RCoT |
2,000 |
3 |
RCIT |
0.99 |
1.00 |
0.96 |
0.0630 |
75% |
AMD EPYC 7713 |
RCoT |
2,000 |
3 |
causal-learn |
0.79 |
1.00 |
0.75 |
0.0238 |
71% |
AMD EPYC 7713 |
RCoT |
5,000 |
0 |
Andrey |
0.02 |
1.00 |
0.07 |
0.00919 |
- |
AMD EPYC 7713 |
RCoT |
5,000 |
0 |
RCIT |
0.02 |
1.00 |
0.07 |
0.0120 |
97% |
AMD EPYC 7713 |
RCoT |
5,000 |
0 |
causal-learn |
0.04 |
1.00 |
0.08 |
0.0138 |
97% |
AMD EPYC 7713 |
RCoT |
5,000 |
1 |
Andrey |
0.15 |
1.00 |
0.26 |
0.0243 |
- |
AMD EPYC 7713 |
RCoT |
5,000 |
1 |
RCIT |
0.34 |
1.00 |
0.45 |
0.130 |
69% |
AMD EPYC 7713 |
RCoT |
5,000 |
1 |
causal-learn |
0.18 |
1.00 |
0.26 |
0.0397 |
78% |
AMD EPYC 7713 |
RCoT |
5,000 |
2 |
Andrey |
0.73 |
1.00 |
0.73 |
0.0259 |
- |
AMD EPYC 7713 |
RCoT |
5,000 |
2 |
RCIT |
0.98 |
1.00 |
0.94 |
0.133 |
75% |
AMD EPYC 7713 |
RCoT |
5,000 |
2 |
causal-learn |
0.73 |
1.00 |
0.71 |
0.0397 |
65% |
AMD EPYC 7713 |
RCoT |
5,000 |
3 |
Andrey |
0.94 |
1.00 |
0.90 |
0.0262 |
- |
AMD EPYC 7713 |
RCoT |
5,000 |
3 |
RCIT |
1.00 |
1.00 |
1.00 |
0.141 |
94% |
AMD EPYC 7713 |
RCoT |
5,000 |
3 |
causal-learn |
0.93 |
1.00 |
0.89 |
0.0398 |
90% |
AMD EPYC 7713 |
Categorical data#
BOSS and GRaSP with BDeu#
Both packages run BOSS and GRaSP with the BDeu score. The table adds Andrey’s categorical BIC
(score="bic_categorical") on the same datasets. The random graphs have 20 datasets per size, for
both accuracy and speed; at every size, the 95% interval of Andrey’s errors minus causal-learn’s
includes 0.
Data |
d |
n |
CPU |
Andrey (s) |
causal-learn (s) |
Speedup |
Datasets |
Same graph |
Errors, Andrey |
Errors, causal-learn |
Errors, categorical BIC |
Errors, empty graph |
Andrey ahead |
causal-learn ahead |
Andrey minus causal-learn |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Random graph |
10 |
2,000 |
AMD EPYC 7713 |
0.0217 |
3.43 |
157x |
20 of 20 |
12 of 20 |
2 |
1 |
2.5 |
10 |
2 |
6 |
+1.00 [-0.60, +2.50] |
Random graph |
20 |
2,000 |
AMD EPYC 7713 |
0.0920 |
17.9 |
182x |
20 of 20 |
9 of 20 |
6.5 |
6 |
4.5 |
20 |
5 |
6 |
+1.30 [-0.40, +3.25] |
Random graph |
40 |
2,000 |
AMD EPYC 7713 |
0.371 |
81.1 |
214x |
20 of 20 |
7 of 20 |
7 |
9 |
7.5 |
40 |
8 |
5 |
-0.95 [-2.70, +0.80] |
diabetes |
413 |
5,000 |
Intel Core i7-7820X |
842 |
timeout |
- |
0 of 1 |
- |
385 |
- |
671 |
602 |
0 |
0 |
- |
Unfinished fits (coverage, never scored): diabetes data, d = 413, n = 5,000: causal-learn 1 of 1 fits past the 7,200 s cap; with each unfinished fit at the cap, at least 8.55x.
Data |
d |
n |
CPU |
Andrey (s) |
causal-learn (s) |
Speedup |
Datasets |
Same graph |
Errors, Andrey |
Errors, causal-learn |
Errors, categorical BIC |
Errors, empty graph |
Andrey ahead |
causal-learn ahead |
Andrey minus causal-learn |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Random graph |
10 |
2,000 |
AMD EPYC 7713 |
0.0212 |
2.76 |
137x |
20 of 20 |
8 of 20 |
2.5 |
1 |
1 |
10 |
4 |
8 |
+0.55 [-0.60, +1.70] |
Random graph |
20 |
2,000 |
AMD EPYC 7713 |
0.0668 |
11.3 |
154x |
20 of 20 |
7 of 20 |
7 |
7.5 |
5 |
20 |
5 |
7 |
-0.85 [-2.80, +0.90] |
Random graph |
40 |
2,000 |
AMD EPYC 7713 |
0.376 |
77.0 |
199x |
20 of 20 |
3 of 20 |
10 |
9.5 |
8 |
40 |
8 |
7 |
-0.35 [-2.35, +1.70] |
diabetes |
413 |
5,000 |
Intel Core i7-7820X |
400 |
timeout |
- |
0 of 1 |
- |
476 |
- |
657 |
602 |
0 |
0 |
- |
Unfinished fits (coverage, never scored): diabetes data, d = 413, n = 5,000: causal-learn 1 of 1 fits past the 7,200 s cap; with each unfinished fit at the cap, at least 18.0x.
On diabetes, causal-learn finished neither search within the 2-hour cap. Andrey’s errors with BDeu are below the empty graph’s (385 and 476 against 602); with the categorical BIC they pass it (671 and 657).
The chi-square and G-square tests#
The two tests now answer every query of a skeleton level at once. Both builds return the same graph on every dataset, and every p-value of PC’s tests on alarm is equal bit for bit.
Method |
Data |
d |
n |
Andrey 0.8.0 (s) |
Andrey 0.7.0 (s) |
Speedup |
Same graph |
|---|---|---|---|---|---|---|---|
PC, chi-square |
Random graph |
10 |
2,000 |
0.00593 |
0.00959 |
- |
5 of 5 |
PC, chi-square |
Random graph |
20 |
2,000 |
0.0127 |
0.0334 |
- |
5 of 5 |
PC, chi-square |
Random graph |
40 |
2,000 |
0.0307 |
0.118 |
3.73x |
5 of 5 |
PC, chi-square |
alarm |
37 |
1,000 |
0.0619 |
0.266 |
4.30x |
1 of 1 |
PC, chi-square |
alarm |
37 |
5,000 |
0.167 |
0.832 |
4.98x |
1 of 1 |
PC, chi-square |
hepar2 |
70 |
1,000 |
0.0865 |
0.431 |
4.98x |
1 of 1 |
PC, chi-square |
hepar2 |
70 |
5,000 |
0.529 |
2.86 |
5.40x |
1 of 1 |
PC, chi-square |
andes |
223 |
1,000 |
0.824 |
5.30 |
6.42x |
1 of 1 |
PC, chi-square |
andes |
223 |
5,000 |
2.77 |
17.1 |
6.15x |
1 of 1 |
PC, G-square |
Random graph |
10 |
2,000 |
0.00584 |
0.00969 |
- |
5 of 5 |
PC, G-square |
Random graph |
20 |
2,000 |
0.0133 |
0.0337 |
- |
5 of 5 |
PC, G-square |
Random graph |
40 |
2,000 |
0.0312 |
0.118 |
3.70x |
5 of 5 |
PC, G-square |
alarm |
37 |
1,000 |
0.0635 |
0.273 |
4.31x |
1 of 1 |
PC, G-square |
alarm |
37 |
5,000 |
0.152 |
0.758 |
4.99x |
1 of 1 |
PC, G-square |
hepar2 |
70 |
1,000 |
0.0929 |
0.458 |
4.93x |
1 of 1 |
PC, G-square |
hepar2 |
70 |
5,000 |
0.672 |
3.45 |
5.14x |
1 of 1 |
PC, G-square |
andes |
223 |
1,000 |
0.896 |
5.61 |
6.26x |
1 of 1 |
PC, G-square |
andes |
223 |
5,000 |
2.78 |
17.1 |
6.18x |
1 of 1 |
FCI, chi-square |
alarm |
37 |
1,000 |
0.0694 |
0.273 |
3.93x |
1 of 1 |
FCI, chi-square |
alarm |
37 |
5,000 |
0.189 |
0.855 |
4.52x |
1 of 1 |
FCI, chi-square |
hepar2 |
70 |
1,000 |
0.117 |
0.461 |
3.93x |
1 of 1 |
FCI, chi-square |
hepar2 |
70 |
5,000 |
0.631 |
2.96 |
4.70x |
1 of 1 |
FCI, chi-square |
andes |
223 |
1,000 |
1.22 |
5.67 |
4.66x |
1 of 1 |
FCI, chi-square |
andes |
223 |
5,000 |
14.0 |
28.0 |
2.00x |
1 of 1 |
FCI, G-square |
alarm |
37 |
1,000 |
0.0708 |
0.281 |
3.97x |
1 of 1 |
FCI, G-square |
alarm |
37 |
5,000 |
0.175 |
0.779 |
4.46x |
1 of 1 |
FCI, G-square |
hepar2 |
70 |
1,000 |
0.119 |
0.482 |
4.05x |
1 of 1 |
FCI, G-square |
hepar2 |
70 |
5,000 |
0.892 |
3.67 |
4.12x |
1 of 1 |
FCI, G-square |
andes |
223 |
1,000 |
1.32 |
6.02 |
4.56x |
1 of 1 |
FCI, G-square |
andes |
223 |
5,000 |
14.2 |
28.3 |
1.99x |
1 of 1 |
FCI’s feasible sizes#
How far FCI and GFCI scale on latent-variable data: 8 regimes (mean degree 2 or 4, 10% or 20% of the
nodes hidden, n = 10 d or 50 d rows) up to 800 variables, 5 datasets each. Each cell gives the
fits finished within the cap and, over those, the median errors as a share of the empty graph’s
(above 1.00, more errors than the empty graph). The caps are 1,800 s up to 200 variables, 3,600 s at
400, and 7,200 s at 800. Andrey runs possible_dsep="fast" on the 0.7.0 build.
FCI:
Regime |
d = 20 |
d = 50 |
d = 100 |
d = 200 |
d = 400 |
d = 800 |
|---|---|---|---|---|---|---|
degree 2, 10% hidden, n = 10 d |
5/5, 0.48 |
5/5, 0.46 |
5/5, 0.54 |
5/5, 0.80 |
5/5, 0.90 |
5/5, 1.49 |
degree 2, 10% hidden, n = 50 d |
5/5, 0.40 |
5/5, 0.36 |
5/5, 0.52 |
5/5, 0.72 |
5/5, 0.84 |
5/5, 1.54 |
degree 2, 20% hidden, n = 10 d |
5/5, 0.52 |
5/5, 0.62 |
5/5, 0.68 |
5/5, 0.85 |
5/5, 0.92 |
5/5, 1.31 |
degree 2, 20% hidden, n = 50 d |
5/5, 0.41 |
5/5, 0.52 |
5/5, 0.59 |
5/5, 0.81 |
5/5, 0.84 |
3/5, 1.37 |
degree 4, 10% hidden, n = 10 d |
5/5, 0.87 |
5/5, 0.86 |
5/5, 0.82 |
5/5, 0.76 |
5/5, 0.67 |
5/5, 0.68 |
degree 4, 10% hidden, n = 50 d |
5/5, 0.87 |
5/5, 0.83 |
5/5, 0.78 |
5/5, 0.73 |
5/5, 0.64 |
5/5, 0.63 |
degree 4, 20% hidden, n = 10 d |
5/5, 0.88 |
5/5, 0.88 |
5/5, 0.87 |
5/5, 0.88 |
5/5, 0.78 |
5/5, 0.79 |
degree 4, 20% hidden, n = 50 d |
5/5, 0.84 |
5/5, 0.85 |
5/5, 0.80 |
5/5, 0.85 |
5/5, 0.75 |
5/5, 0.77 |
GFCI:
Regime |
d = 20 |
d = 50 |
d = 100 |
d = 200 |
d = 400 |
d = 800 |
|---|---|---|---|---|---|---|
degree 2, 10% hidden, n = 10 d |
5/5, 0.44 |
5/5, 0.30 |
5/5, 0.39 |
5/5, 0.44 |
5/5, 0.35 |
5/5, 0.41 |
degree 2, 10% hidden, n = 50 d |
5/5, 0.33 |
5/5, 0.20 |
5/5, 0.30 |
5/5, 0.28 |
5/5, 0.17 |
5/5, 0.26 |
degree 2, 20% hidden, n = 10 d |
5/5, 0.50 |
5/5, 0.55 |
5/5, 0.56 |
5/5, 0.62 |
5/5, 0.53 |
5/5, 0.56 |
degree 2, 20% hidden, n = 50 d |
5/5, 0.43 |
5/5, 0.46 |
5/5, 0.45 |
5/5, 0.51 |
5/5, 0.36 |
1/5, 0.33 |
degree 4, 10% hidden, n = 10 d |
5/5, 0.83 |
5/5, 0.79 |
5/5, 0.71 |
5/5, 0.60 |
5/5, 0.47 |
5/5, 0.44 |
degree 4, 10% hidden, n = 50 d |
5/5, 0.68 |
5/5, 0.67 |
5/5, 0.66 |
5/5, 0.55 |
5/5, 0.40 |
4/5, 0.37 |
degree 4, 20% hidden, n = 10 d |
5/5, 0.86 |
5/5, 0.87 |
5/5, 0.76 |
5/5, 0.81 |
5/5, 0.70 |
3/5, 0.72 |
degree 4, 20% hidden, n = 50 d |
5/5, 0.84 |
5/5, 0.80 |
5/5, 0.76 |
5/5, 0.79 |
2/5, 0.50 |
0/5 |
Tetrad’s GFCI, for context, with a cap of at most 1,800 s; a run stops Tetrad after its first fit past the cap, so later datasets are not run:
Regime |
d = 20 |
d = 50 |
d = 100 |
d = 200 |
d = 400 |
d = 800 |
|---|---|---|---|---|---|---|
degree 2, 10% hidden, n = 10 d |
5/5, 0.46 |
5/5, 0.51 |
5/5, 0.52 |
5/5, 0.73 |
4/5, 0.62 |
0/1 |
degree 2, 10% hidden, n = 50 d |
5/5, 0.35 |
5/5, 0.27 |
5/5, 0.33 |
1/2, 0.46 |
1/2, 0.28 |
0/1 |
degree 2, 20% hidden, n = 10 d |
5/5, 0.43 |
5/5, 0.62 |
5/5, 0.63 |
1/2, 0.54 |
0/1 |
0/1 |
degree 2, 20% hidden, n = 50 d |
5/5, 0.46 |
5/5, 0.46 |
0/1 |
not run |
0/1 |
0/1 |
degree 4, 10% hidden, n = 10 d |
5/5, 0.84 |
5/5, 0.85 |
1/2, 0.77 |
not run |
0/1 |
0/1 |
degree 4, 10% hidden, n = 50 d |
5/5, 0.74 |
0/1 |
not run |
not run |
0/1 |
0/1 |
degree 4, 20% hidden, n = 10 d |
5/5, 0.82 |
5/5, 0.86 |
0/1 |
not run |
0/1 |
0/1 |
degree 4, 20% hidden, n = 50 d |
5/5, 0.78 |
0/1 |
not run |
not run |
0/1 |
0/1 |
Other changes in 0.8.0#
The benchmark above covers the release’s theme; 0.8.0 also changes the following. The changelog lists every change.
KCI, RCoT, CMIknn, and GPDC are supported. Against each reference package, each test has no more errors than the reference on average over every paired dataset, and no size has its 95% interval of Andrey’s errors minus the reference’s entirely above 0. RCoT’s packages draw random features, so RCoT is judged on 4 draws per dataset, under a rule set before those fits ran.
Background knowledge for every search. PC, CD-NOD, FCI, GFCI, GES, GIES, XGES, hill climbing, tabu search, BOSS, GRaSP, exact search, and DirectLiNGAM take
knowledge=: the edges the graph must have and must not have, the only pairs that may be adjacent, and tiers in time order. PC, FCI, GFCI, and CD-NOD also takemax_depth.Tie breaks and column order.
andrey.config(tie_break=...)chooses how the searches break ties: by a rank computed from each column’s own data, the default, or by column position. The column order guide lists what each method keeps under a new column order and on another machine.Score defaults that follow the data (breaking). BOSS, GRaSP, GES, hill climbing, tabu search, and exact search pick BDeu for categorical data and the conditional Gaussian BIC for mixed data, and warn once; pass the linear-Gaussian BIC to keep the old default.
Faster categorical tests. The chi-square and G-square tests answer many queries at once: on alarm, hepar2, and andes, PC returns the same graphs 4.30x to 6.42x faster than 0.7.0, and FCI 1.99x to 4.70x (see the chi-square and G-square tests).
BDeu and the categorical BIC for BOSS and GRaSP.
score_func="local_score_BDeu"and"local_score_BIC_categorical", the scores the other score-based searches take for categorical data.BOSS and GRaSP decide every comparison as full scores would, in less time.
Clearer errors. Copied and constant columns are named before any test runs, the LiNGAM methods say how many rows their pruning step needs, and a PDAG without a consistent extension names a chordless cycle.
Order-free ties across machines. DirectLiNGAM, ICA-LiNGAM, exact search, and the score-based searches break ties by a rank computed from each column’s own data, so they return the same graph on every CPU and under any column order. In Temporal causal discovery, VARMA-LiNGAM and LongitudinalLiNGAM fit on rank-ordered columns.
FCI’s measured limits. FCI’s and GFCI’s documentation gives the sizes they finish within a cap (see FCI’s feasible sizes).
Column names. Labels are text and unique in every method, the CLI, and graph input and output.
pcalg’s graph conventions. The graph representation guide gives pcalg’s
amat.cpdagandamat.pagcodes and converts both to Andrey’s graphs.
Limits#
Pending measurements. See Pending measurements.
KCI. Measured up to 20 variables at 2,000 rows and 5 variables at 5,000 rows. causal-learn passed the 3,600 s cap on 8 of 20 additive and 5 of 20 nonmonotone datasets at 5 variables and 5,000 rows; at 20 variables and 2,000 rows, on nonmonotone data, Andrey passed the 1,800 s cap on 1 of 20 datasets and causal-learn on 6.
RCoT. Every fit finished at every size up to 50 variables and 5,000 rows. With one fit per dataset, causal-learn has fewer errors than Andrey on 4 of 20 datasets, and Andrey on none, with PC on additive data at 10 variables and 5,000 rows and with FCI on additive data at 5 variables and 500 rows; over 4 draws per dataset, Andrey minus causal-learn is -0.16 [-0.49, +0.15] and -0.04 [-0.28, +0.21] at those sizes.
CMIknn. Measured up to 20 variables at 500 rows, 10 at 1,000, and 5 at 2,000. A fit takes minutes at 10 variables and 500 rows (192 to 295 s on an AMD EPYC 7713). Andrey’s CMIknn runs much faster on CPUs with AVX-512, so the speedup over tigramite is 2.40x to 2.56x on an AMD EPYC 7713, 9.19x to 10.3x on the AVX-512 servers, and 27.0x to 37.7x on the second machine.
GPDC. Measured up to 20 variables at 1,000 rows and 5 at 2,000. The speedup over tigramite is 1.38x to 1.89x on the servers and 1.20x to 1.46x on the second machine. At 20 variables and 1,000 rows, tigramite passed the 1,800 s cap on 10 of 20 additive and 2 of 20 nonmonotone datasets, and Andrey on none.
FCI. Measured with KCI and RCoT at 5 and 10 variables, with 500 and 1,000 rows.
GPU. The GPU’s speedup depends on the CPU it replaces: CMIknn runs 9.05x to 17.2x faster than on an AMD EPYC 7713, 1.82x to 2.97x faster than on the AVX-512 servers, and slower than on the second machine (0.644x to 0.946x of its speed).
BOSS and GRaSP on small graphs. On the random graphs, causal-learn has fewer errors on average with BOSS at 10 and 20 variables (Andrey minus causal-learn +1.00 and +1.30) and with GRaSP at 10 (+0.55); Andrey has fewer at 40 with BOSS (-0.95) and at 20 and 40 with GRaSP (-0.85 and -0.35). Every 95% interval includes 0.
Categorical-test hardware. The rows against Andrey 0.7.0 ran with every core of the node visible to the fit, unlike the other lanes’ 4-core budget.
Hardware. Ratios are specific to the measured CPU, core budget, and settings; the same size can give different ratios on two CPUs.
Pending measurements#
These measurements were still running when the summary was built; the summary lists each under
incomplete, and its pending_fits counts the fits.
PC with the KCI test, causal-learn, speed on the second machine (Intel Core i7-7820X): additive, nonmonotone data, d = 10, n = 1,000; d = 20, n = 1,000
PC with the KCI test on the NVIDIA A100-SXM4-80GB, against Andrey’s CPU fits: additive, nonmonotone data, d = 20, n = 1,000 (no CPU timing of this size yet)
PC with the KCI test on the NVIDIA A100-SXM4-80GB, against Andrey’s CPU fits: additive data, d = 20, n = 2,000 (the CPU fit passed its cap on a timed dataset, so the GPU graph could not be checked on it)
Reproduce#
You can find environment setup, campaign drivers, and scores in the benchmark harness guide. To rerun the comparisons, install Andrey 0.8.0 (andrey-core), causal-learn 0.1.4.8, tigramite 5.2.10.1 in its own environment, and pcalg 2.7.12 with RCIT 0.1.0 in the harness’s R environment. Every driver shuffles the columns by default. Times will differ with the hardware and build; graphs should match closely.
Comparison |
Driver |
Sizes |
Datasets x repeats |
Wall cap (s) |
|---|---|---|---|---|
PC and FCI with the nonlinear tests |
|
d 5 to 50, n 500 to 5,000 |
20 x 1 (accuracy); 5 x 1, run 3 times (speed) |
1,800 or 3,600 |
KCI and CMIknn on a GPU |
|
d 5 to 20, n 500 to 5,000 |
5 x 1, run 3 times |
1,800 or 3,600 |
The tests query by query |
|
n 200 to 5,000, k 0 to 3 |
50 or 200 independent, 20 dependent draws |
1,800 or 3,600 |
BOSS and GRaSP on categorical data |
|
d 10, 20, 40; diabetes |
20 x 1; 1 x 1 |
3,600; 7,200 |
PC with the categorical tests |
|
d 10, 20, 40 |
5 x 1 |
1,800 |