---
name: benchmarks-0.8.0
description: The Andrey 0.8.0 release card - PC and FCI with the KCI, RCoT, CMIknn, and GPDC tests, BOSS and GRaSP on categorical data, and faster categorical tests.
meta:
  type: guide
  tags: [benchmarks, performance, accuracy, nonlinear]
---

# 0.8.0 release card: nonlinear tests

% The release's cover, as on the Release cards page, beside the opening (_static/benchmarks.css).
:::::{div} release-opening

:::{card}
:class-card: release-cover release-art release-art-0-8-0
:shadow: none

Release card

Nonlinear tests

0.8.0
:::

::::{div} release-opening-text

We compare PC and FCI with the four nonlinear conditional-independence tests 0.8.0 adds, KCI, RCoT,
CMIknn, and GPDC, with the packages that implement them: causal-learn in Python, pcalg with RCIT in
R, and tigramite in Python. We also compare BOSS and GRaSP with the BDeu score with causal-learn on
categorical data, PC and FCI with the batched chi-square and G-square tests with Andrey 0.7.0, and
Andrey's KCI and CMIknn on an NVIDIA A100 GPU with Andrey on the CPU. On an AMD EPYC 7713, PC with
RCoT runs more than 10x faster than causal-learn at 50 variables, and BOSS and GRaSP with BDeu more
than 130x faster than causal-learn on random categorical graphs; PC with KCI runs more than 2.5x
faster than causal-learn at every size we timed, on each CPU.

Speed is not accuracy. On nonmonotone data, Fisher-Z's errors pass the empty graph's from 10
variables, and the nonlinear tests stay below it in every package. Each nonlinear test makes fewer
errors than its reference package on average, 0.22 to 5.03 fewer errors per graph, with no size
significantly worse; see [Limits](#limits) for the sizes where the reference is ahead. Every dataset
lists its variables in a random order (see [Method](#method)). You can find speedups and accuracy in
the [summary](#summary). These results apply to the measured workloads. Speed and accuracy depend on
the data, settings, and hardware.

::::
:::::

## Summary

We compare each method with every package that implements it with the same test or score. We report
every paired speed ratio, regardless of accuracy. Each ratio compares the two packages on one CPU;
the nonlinear and the BOSS and GRaSP rows ran on several CPUs, and each section names the CPU of
every row. Some
measurements are still running; their cells read "pending" (see
[Pending measurements](#pending-measurements)).

**Speed.**

% generated: speed

| Family | Method | causal-learn | pcalg | tigramite | Andrey 0.7.0 |
|---|---|---:|---:|---:|---:|
| Nonlinear tests | PC, KCI | 2.73-5.56x† | - | - | - |
|  | PC, RCoT | 5.92-35.7x | 9.51-29.2x | - | - |
|  | PC, CMIknn | - | - | 2.40-10.3x | - |
|  | PC, GPDC | - | - | 1.38-1.89x | - |
|  | FCI, KCI | 2.76-5.23x | - | - | - |
|  | FCI, RCoT | 7.37-16.8x | - | - | - |
| Categorical scores | BOSS, BDeu | 157-214x\* | - | - | - |
|  | GRaSP, BDeu | 137-199x\*\* | - | - | - |
| Categorical tests | PC, chi-square | - | - | - | 3.73-6.42x |
|  | PC, G-square | - | - | - | 3.70-6.26x |
|  | FCI, chi-square | - | - | - | 2.00-4.70x |
|  | FCI, G-square | - | - | - | 1.99-4.56x |

\* BOSS, BDeu on diabetes: causal-learn finished none of 1 within the 2 h cap, so the speedup is at
least 8.55x.

\*\* GRaSP, BDeu on diabetes: causal-learn finished none of 1 within the 2 h cap, so the speedup is
at least 18.0x.

† More sizes are pending; see [Pending measurements](#pending-measurements).

% end generated

Each cell is the range of Andrey's speedup over that package across the method's sizes, data, and
CPUs; each method's section below gives every size and its CPU. The nonlinear cells leave out the
second machine's rows, which each section lists apart. A dash marks a package we did not compare,
and "not timed" a comparison with accuracy but no speed.

**Accuracy.**

% generated: accuracy

| Family | Method | causal-learn | pcalg | tigramite | Andrey 0.7.0 |
|---|---|---:|---:|---:|---:|
| Nonlinear tests | KCI (PC, FCI) | +0.22 [+0.10, +0.35] | - | - | - |
|  | RCoT (PC, FCI) | +1.55 [+1.36, +1.74] | +5.03 [+4.45, +5.63] | - | - |
|  | CMIknn (PC) | - | - | +0.70 [+0.49, +0.93] | - |
|  | GPDC (PC) | - | - | +1.23 [+0.98, +1.50] | - |
| Categorical scores | BOSS, BDeu | -0.45 [-1.48, +0.55] | - | - | - |
|  | GRaSP, BDeu | +0.22 [-0.78, +1.22] | - | - | - |
| Categorical tests | PC, chi-square | - | - | - | 0 |
|  | PC, G-square | - | - | - | 0 |
|  | FCI, chi-square | - | - | - | 0 |
|  | FCI, G-square | - | - | - | 0 |

% end generated

Each cell is the mean number of errors Andrey saves per graph: the package's errors minus Andrey's,
averaged over the datasets both finished, at every size with at least two of them, with its 95%
percentile bootstrap interval (10,000 resamples). A positive number means Andrey makes fewer errors
than the package, and a negative number more. A nonlinear row pools the datasets of PC and FCI with
that test, and RCoT uses the mean of 4 draws per dataset; pcalg and tigramite are compared with PC
only. No size is significantly worse: at no size is the interval of Andrey's errors minus the
package's entirely above 0. The tables for each size below give that difference the other way,
Andrey minus the package, so there a negative number means Andrey makes fewer errors. Against Andrey
0.7.0, both builds return the same graph on every dataset, so the difference is 0.

## Method

**Data.** The nonlinear tests run on two regimes over the same Erdos-Renyi DAGs with mean degree 2,
standardized, with Gaussian noise: additive data, `x = sum w (p + sin p) + e`, whose near-monotone
effects Fisher-Z still sees, and nonmonotone data, `x = sum w cos(p) + e`, whose even effects
partial correlation does not see. There are 20 datasets per size, one per seed, for accuracy; speed
is timed on the first 5. BOSS and GRaSP run on categorical data from an Erdos-Renyi DAG with mean
degree 2, 2 or 3 categories per variable, and Dirichlet(1) conditional tables, with 2,000 rows, and
on the diabetes network (413 variables) with 5,000 rows from its own tables. The categorical tests
run on the same random categorical graphs and on the alarm, hepar2, and andes networks with 1,000
and 5,000 rows. BOSS and GRaSP have 20 datasets per size on the random graphs and the categorical
tests 5; the networks give one dataset per size.

**Shuffled columns.** Every dataset lists its variables in an order drawn from its seed, with the
true graph permuted the same way. The network timing rows against Andrey 0.7.0 keep the network's
column order; both builds see the same data.

**Accuracy.** PC's errors are the SHD between CPDAGs: one for each pair of variables whose edge is
missing, extra, or oriented differently, so the empty graph's errors are the number of true edges.
FCI's errors are the SHD against the true PAG of the observed variables: one for each pair whose
edge is missing, extra, or marked differently. Each nonlinear table also gives Andrey's PC or FCI
with Fisher-Z on the same datasets. Errors are medians over a size's datasets. "Same graph" means
the two packages return the same result. "Andrey minus" a package is the mean over the datasets
both finished of Andrey's errors minus the package's, with a 95% percentile bootstrap interval
(10,000 resamples); below 0, Andrey has fewer errors.

**Timing.** For the nonlinear tests, each timed dataset runs in 3 rounds on the same 4 cores: in
each round, Andrey and then the other package fit every dataset, each fit in its own process after
the row's warm-up fit (none where a fit can pass 300 s). A dataset's time is its median over the
rounds; for each dataset, we divide the other package's time by Andrey's, and the speedup is the
median of these ratios. A size gets a speedup only when both packages finished every timed dataset
in all 3 rounds on one CPU type; otherwise its table says why. In the other lanes, times are seconds
per fit, the median over a size's datasets, and the speedup is the median of the paired ratios. A
fit past its cap is coverage, never scored: the tables count the datasets both packages finished,
with a bound that counts each unfinished fit at the cap. We show no speedup for a size at which both
packages' median times are under 0.1 s. Times and ratios have three significant figures.

**Hardware.** Each lane names its machines in the summary's `hardware`:

- Nonlinear tests, accuracy: server nodes shared with other jobs, almost all AMD EPYC 7713, each fit
  on its own 4 cores with 4 BLAS and OpenMP threads and a 30,000 MB memory cap. Graphs do not depend
  on the CPU; which fits pass the cap can. The wall cap was 1,800 s, and 3,600 s for CMIknn, for KCI
  at 5,000 rows, and for GPDC at 2,000 rows.
- Nonlinear tests, speed: each row names its CPU, an AMD EPYC 7713, 9534, or 9555, or an Intel Xeon
  Gold 5418Y or 6542Y, with each fit on 4 pinned cores, most of them on a node shared with other
  jobs, and the same caps. A second machine, an Intel Core i7-7820X, timed some sizes on 2 physical
  cores and their 2 hyperthreads per fit, 3 sizes at once; its rows are listed apart and never
  compared with the servers'. Inside PC's skeleton, Andrey runs RCoT, CMIknn, and GPDC on one
  thread, and KCI on one thread below 2,000 rows.
- BOSS and GRaSP on categorical data: the random graphs on an AMD EPYC 7713, each fit on 4 pinned
  cores of a node shared with other jobs, 1 timed fit per dataset after 1 warm-up fit, and a
  3,600 s wall cap; diabetes on an Intel Core i7-7820X, each fit on 4 pinned cores with nothing
  else running, and a 7,200 s cap with no warm-up.
- Categorical tests against Andrey 0.7.0: an AMD EPYC 9534 exclusive node with every core visible
  to the fit and no core budget, each build in its own environment. The random-graph rows take 1
  timed fit after 1 warm-up; the network rows take the faster of 2 fits.
- FCI's feasible sizes: an AMD EPYC 7713, each fit on its own 4 cores, one fit at a time per core
  group on an exclusive node, and a 30,000 MB memory cap.

## Builds and versions

| Comparison | Andrey build | Measured package versions |
|---|---|---|
| Nonlinear tests with KCI, CMIknn, and GPDC | Pre-release 0.8.0, October 20, 2026 | causal-learn 0.1.4.8, tigramite 5.2.10.1 |
| Nonlinear tests with RCoT | Pre-release 0.8.0, October 20, 2026, with quasi-Monte Carlo random Fourier features in RCoT | causal-learn 0.1.4.8, pcalg 2.7.12 with RCIT 0.1.0 |
| BOSS and GRaSP, random graphs | Pre-release 0.8.0, October 22, 2026 | causal-learn 0.1.4.8 |
| BOSS and GRaSP, diabetes | Pre-release 0.8.0, October 19, 2026, with the diabetes network | causal-learn 0.1.4.8 |
| Categorical tests | Pre-release 0.8.0, October 18, 2026, with the batched tests | Andrey 0.7.0 (pre-release build of October 18, 2026) |
| FCI's feasible sizes | Pre-release 0.7.0, October 17, 2026 | Tetrad (causal-cmd 1.12.0, tetrad-lib 7.6.3) |

Fisher-Z beside PC with RCoT comes from the October 20 build without the RCoT change, whose PC with
Fisher-Z returns the same graphs as the RCoT build's on every dataset checked. Andrey ran on Python
3.13 with NumPy 2.5.4, SciPy 1.16.3, and torch 2.14.1 on the servers, and torch 2.13.0 on the second
machine; SciPy 1.18.1 for BOSS and GRaSP. tigramite ran in its own Python
environment, pcalg and RCIT in R 4.6.1 through rpy2 3.6.9. The 0.7.0 build's categorical tests and
skeleton search are those of the released 0.7.0; its FCI's possible-d-separation step is the
column-order form the released 0.7.0 replaced. The
{download}`summary <../../../benchmarks/published/0.8.0/summary.json>` contains one row per
comparison, data, and size, the second machine's rows in entries of their own, the query lane under
`tests`, and FCI's feasible sizes under `boundary`.

## Nonlinear tests

KCI, RCoT, CMIknn, and GPDC are settings of PC and FCI (`indep_test`). Both packages run the search
at `alpha` 0.05. The references differ by design: causal-learn's KCI drops null eigenvalues below
1e-5, which Andrey keeps; causal-learn and RCIT draw RCoT's random features from their own
generators; tigramite's CMIknn reports `b / B` where Andrey reports `(b + 1) / (B + 1)`; and
tigramite's GPDC uses its own Gaussian process and an analytic null, where Andrey draws a
rank-permutation null of 1,000 draws inside every fit. Each section gives accuracy on 20 datasets
per size, then speed on 5 of them, one row per size and CPU.

### PC with KCI

% generated: PC, KCI vs causal-learn

| Data | d | n | Datasets | Same graph | Errors, Andrey | Errors, causal-learn | Errors, Fisher-Z | Errors, empty graph | Andrey ahead | causal-learn ahead | Andrey minus causal-learn | Ahead in column order, Andrey / causal-learn |
|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
| Additive | 5 | 500 | 20 of 20 | 19 of 20 | 0 | 0 | 1 | 5 | 1 | 0 | -0.10 [-0.30, +0.00] | 0 / 0 |
| Additive | 5 | 1,000 | 20 of 20 | 20 of 20 | 0 | 0 | 1 | 5 | 0 | 0 | +0.00 [+0.00, +0.00] | 0 / 0 |
| Additive | 5 | 2,000 | 20 of 20 | 20 of 20 | 0 | 0 | 1 | 5 | 0 | 0 | +0.00 [+0.00, +0.00] | 0 / 0 |
| Additive | 5 | 5,000 | 12 of 20 | 12 of 12 | 0 | 0 | 1.5 | 5 | 0 | 0 | +0.00 [+0.00, +0.00] | 0 / 0 |
| Additive | 10 | 500 | 20 of 20 | 18 of 20 | 2.5 | 2.5 | 4.5 | 10 | 0 | 1 | +0.15 [+0.00, +0.45] | 0 / 0 |
| Additive | 10 | 1,000 | 20 of 20 | 19 of 20 | 1 | 1 | 3.5 | 10 | 1 | 0 | -0.05 [-0.15, +0.00] | 0 / 0 |
| Additive | 10 | 2,000 | 20 of 20 | 16 of 20 | 1 | 1 | 4 | 10 | 3 | 1 | -0.35 [-1.10, +0.30] | 0 / 0 |
| Additive | 20 | 500 | 20 of 20 | 12 of 20 | 8 | 8.5 | 8 | 20 | 4 | 4 | -0.05 [-0.90, +0.75] | 0 / 0 |
| Additive | 20 | 1,000 | 20 of 20 | 5 of 20 | 5.5 | 7 | 8.5 | 20 | 8 | 3 | -0.60 [-1.25, +0.05] | 0 / 0 |
| Additive | 20 | 2,000 | 1 of 20 | 1 of 1 | 3 | 3 | 6 | 20 | 0 | 0 | +0.00 | 0 / 0 |
| Nonmonotone | 5 | 500 | 20 of 20 | 17 of 20 | 3 | 3 | 5 | 5 | 3 | 0 | -0.35 [-0.75, +0.00] | 0 / 0 |
| Nonmonotone | 5 | 1,000 | 20 of 20 | 18 of 20 | 2.5 | 2 | 5 | 5 | 1 | 1 | -0.10 [-0.45, +0.15] | 0 / 0 |
| Nonmonotone | 5 | 2,000 | 20 of 20 | 16 of 20 | 0.5 | 1.5 | 5.5 | 5 | 4 | 0 | -0.50 [-1.05, -0.10] | 0 / 0 |
| Nonmonotone | 5 | 5,000 | 15 of 20 | 12 of 15 | 0 | 0 | 6 | 5 | 3 | 0 | -0.67 [-1.40, +0.00] | 0 / 0 |
| Nonmonotone | 10 | 500 | 20 of 20 | 6 of 20 | 5 | 5 | 13 | 10 | 7 | 5 | -0.10 [-0.95, +0.80] | 2 / 0 |
| Nonmonotone | 10 | 1,000 | 20 of 20 | 5 of 20 | 3 | 4.5 | 14 | 10 | 9 | 5 | -0.65 [-1.75, +0.45] | 0 / 1 |
| Nonmonotone | 10 | 2,000 | 20 of 20 | 9 of 20 | 2 | 2.5 | 14 | 10 | 10 | 1 | -1.20 [-2.00, -0.45] | 0 / 0 |
| Nonmonotone | 20 | 500 | 20 of 20 | 1 of 20 | 10 | 10 | 28 | 20 | 7 | 8 | +0.35 [-1.45, +2.25] | 0 / 0 |
| Nonmonotone | 20 | 1,000 | 20 of 20 | 2 of 20 | 7 | 7 | 28.5 | 20 | 11 | 4 | -1.05 [-2.05, -0.05] | 1 / 0 |
| Nonmonotone | 20 | 2,000 | 14 of 20 | 3 of 14 | 5.5 | 5 | 29.5 | 20 | 8 | 2 | -1.07 [-2.07, -0.07] | 0 / 1 |

causal-learn's PC depends on the column order: its `pc()` orients the colliders in column order and
keeps the first of two that conflict (`uc_rule=0`, `uc_priority=2`). The last column compares it
with Andrey's PC in its column-order mode (`collider_order="column"`, with Meek's rules one firing
at a time in column order) on the same datasets, like with like; the other columns are Andrey's
order-free default.

Fits past the cap (coverage, never scored): Additive data, d = 5, n = 5,000: causal-learn 8 of 20.
Additive data, d = 20, n = 2,000: Andrey 3 and causal-learn 19 of 20. Nonmonotone data, d = 5, n =
5,000: causal-learn 5 of 20. Nonmonotone data, d = 20, n = 2,000: Andrey 1 and causal-learn 6 of 20.

| Data | d | n | CPU | Andrey (s) | causal-learn (s) | Speedup |
|---|---:|---:|---|---:|---:|---:|
| Additive | 5 | 500 | AMD EPYC 7713 | 1.14 | 4.37 | 3.27x |
| Additive | 5 | 1,000 | AMD EPYC 7713 | 9.29 | 30.3 | 2.99x |
| Additive | 5 | 2,000 | AMD EPYC 7713 | 37.1 | 159 | 4.09x |
| Additive | 5 | 5,000 | AMD EPYC 7713 | - | - | - |
| Additive | 10 | 500 | AMD EPYC 7713 | 5.53 | 22.4 | 3.88x |
| Additive | 10 | 1,000 | AMD EPYC 9534 | 42.4 | 140 | 3.55x |
| Additive | 10 | 2,000 | AMD EPYC 9534 | 121 | 598 | 4.65x |
| Additive | 20 | 500 | AMD EPYC 7713 | 19.3 | 85.5 | 4.43x |
| Additive | 20 | 2,000 | AMD EPYC 7713 | - | - | - |
| Nonmonotone | 5 | 500 | AMD EPYC 7713 | 0.902 | 2.82 | 2.93x |
| Nonmonotone | 5 | 1,000 | AMD EPYC 7713 | 7.86 | 25.6 | 3.23x |
| Nonmonotone | 5 | 2,000 | AMD EPYC 9534 | 18.7 | 105 | 5.56x |
| Nonmonotone | 5 | 5,000 | AMD EPYC 7713 | - | - | - |
| Nonmonotone | 10 | 500 | AMD EPYC 7713 | 4.36 | 14.9 | 3.20x |
| Nonmonotone | 10 | 1,000 | AMD EPYC 9555 | 10.9 | 30.5 | 2.73x |
| Nonmonotone | 10 | 2,000 | AMD EPYC 7713 | 133 | 630 | 4.16x |
| Nonmonotone | 20 | 500 | AMD EPYC 7713 | 13.5 | 51.0 | 3.78x |
| Nonmonotone | 20 | 2,000 | AMD EPYC 7713 | - | - | - |

No speedup: Additive data, d = 5, n = 5,000, AMD EPYC 7713: causal-learn passed the 3,600 s cap on 1
of 5 timed datasets; with each unfinished fit at the cap, at least 2.42x. Additive data, d = 20, n =
2,000, AMD EPYC 7713: causal-learn passed the 1,800 s cap on 5 of 5 timed datasets; with each
unfinished fit at the cap, at least 2.09x. Nonmonotone data, d = 5, n = 5,000, AMD EPYC 7713:
causal-learn passed the 3,600 s cap on 2 of 5 timed datasets; with each unfinished fit at the cap,
at least 2.07x. Nonmonotone data, d = 20, n = 2,000, AMD EPYC 7713: causal-learn passed the 1,800 s
cap on 3 of 5 timed datasets; with each unfinished fit at the cap, at least 2.53x.

% end generated

### PC with RCoT

Against causal-learn:

% generated: PC, RCoT vs causal-learn

| Data | d | n | Datasets | Same graph | Errors, Andrey | Errors, causal-learn | Errors, Fisher-Z | Errors, empty graph | Andrey ahead | causal-learn ahead | Andrey minus causal-learn, mean of 4 draws | Ahead in column order, Andrey / causal-learn |
|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
| Additive | 5 | 500 | 20 of 20 | 13 of 20 | 1 | 0 | 1 | 5 | 2 | 5 | -0.06 [-0.30, +0.16] | 2 / 5 |
| Additive | 5 | 1,000 | 20 of 20 | 12 of 20 | 0 | 0 | 1 | 5 | 5 | 3 | -0.47 [-1.00, -0.03] | 5 / 3 |
| Additive | 5 | 2,000 | 20 of 20 | 17 of 20 | 0 | 0 | 1 | 5 | 3 | 0 | -0.12 [-0.41, +0.16] | 3 / 0 |
| Additive | 5 | 5,000 | 20 of 20 | 17 of 20 | 0 | 0 | 2 | 5 | 2 | 1 | -0.09 [-0.41, +0.21] | 2 / 1 |
| Additive | 10 | 500 | 20 of 20 | 7 of 20 | 2.5 | 3 | 4.5 | 10 | 7 | 3 | -0.65 [-1.16, -0.19] | 6 / 3 |
| Additive | 10 | 1,000 | 20 of 20 | 11 of 20 | 2 | 2 | 3.5 | 10 | 4 | 4 | -0.50 [-1.01, -0.09] | 4 / 4 |
| Additive | 10 | 2,000 | 20 of 20 | 10 of 20 | 1.5 | 1 | 4 | 10 | 5 | 5 | -0.03 [-0.57, +0.54] | 4 / 5 |
| Additive | 10 | 5,000 | 20 of 20 | 16 of 20 | 0 | 0 | 5.5 | 10 | 0 | 4 | -0.16 [-0.49, +0.15] | 0 / 4 |
| Additive | 20 | 500 | 20 of 20 | 3 of 20 | 7.5 | 7.5 | 8 | 20 | 8 | 8 | -0.59 [-1.20, +0.03] | 10 / 6 |
| Additive | 20 | 1,000 | 20 of 20 | 2 of 20 | 5.5 | 5 | 8.5 | 20 | 11 | 6 | -0.56 [-0.96, -0.17] | 11 / 4 |
| Additive | 20 | 2,000 | 20 of 20 | 3 of 20 | 3.5 | 5 | 9 | 20 | 11 | 5 | -0.42 [-0.81, -0.06] | 8 / 8 |
| Additive | 20 | 5,000 | 20 of 20 | 7 of 20 | 2 | 2.5 | 10 | 20 | 5 | 5 | -0.60 [-1.35, +0.09] | 4 / 6 |
| Additive | 50 | 500 | 20 of 20 | 0 of 20 | 18.5 | 20 | 23 | 50 | 12 | 4 | -2.50 [-3.66, -1.36] | 11 / 7 |
| Additive | 50 | 1,000 | 20 of 20 | 0 of 20 | 14.5 | 14 | 22 | 50 | 12 | 4 | -1.91 [-2.95, -0.91] | 10 / 6 |
| Additive | 50 | 2,000 | 20 of 20 | 0 of 20 | 7.5 | 10 | 25.5 | 50 | 14 | 3 | -2.40 [-3.38, -1.54] | 9 / 7 |
| Additive | 50 | 5,000 | 20 of 20 | 1 of 20 | 7 | 7.5 | 29 | 50 | 9 | 8 | -1.48 [-2.42, -0.54] | 6 / 10 |
| Nonmonotone | 5 | 500 | 20 of 20 | 3 of 20 | 3 | 3 | 5 | 5 | 9 | 4 | -0.56 [-1.09, -0.05] | 8 / 6 |
| Nonmonotone | 5 | 1,000 | 20 of 20 | 7 of 20 | 1.5 | 2.5 | 5 | 5 | 8 | 3 | -0.65 [-1.13, -0.20] | 5 / 6 |
| Nonmonotone | 5 | 2,000 | 20 of 20 | 9 of 20 | 0 | 2 | 5.5 | 5 | 9 | 2 | -0.61 [-1.15, -0.10] | 8 / 2 |
| Nonmonotone | 5 | 5,000 | 20 of 20 | 17 of 20 | 0 | 0 | 6 | 5 | 1 | 2 | -0.33 [-0.64, -0.06] | 0 / 3 |
| Nonmonotone | 10 | 500 | 20 of 20 | 1 of 20 | 5 | 5.5 | 13 | 10 | 10 | 5 | -1.99 [-2.49, -1.50] | 7 / 5 |
| Nonmonotone | 10 | 1,000 | 20 of 20 | 0 of 20 | 3.5 | 5.5 | 14 | 10 | 13 | 5 | -1.79 [-2.46, -1.09] | 11 / 6 |
| Nonmonotone | 10 | 2,000 | 20 of 20 | 5 of 20 | 1.5 | 3 | 14 | 10 | 13 | 1 | -1.64 [-2.38, -0.86] | 12 / 3 |
| Nonmonotone | 10 | 5,000 | 20 of 20 | 5 of 20 | 0.5 | 1.5 | 14 | 10 | 10 | 3 | -1.25 [-1.81, -0.70] | 7 / 5 |
| Nonmonotone | 20 | 500 | 20 of 20 | 0 of 20 | 11 | 13 | 28 | 20 | 9 | 8 | -2.33 [-3.46, -1.27] | 10 / 6 |
| Nonmonotone | 20 | 1,000 | 20 of 20 | 0 of 20 | 7 | 10.5 | 28.5 | 20 | 14 | 2 | -3.59 [-4.79, -2.48] | 11 / 7 |
| Nonmonotone | 20 | 2,000 | 20 of 20 | 0 of 20 | 4.5 | 7.5 | 30 | 20 | 12 | 6 | -2.10 [-3.30, -0.93] | 11 / 6 |
| Nonmonotone | 20 | 5,000 | 20 of 20 | 2 of 20 | 4 | 5 | 30.5 | 20 | 11 | 2 | -1.81 [-2.85, -0.82] | 8 / 6 |
| Nonmonotone | 50 | 500 | 20 of 20 | 0 of 20 | 25 | 37 | 87 | 50 | 19 | 1 | -8.39 [-10.05, -6.85] | 17 / 2 |
| Nonmonotone | 50 | 1,000 | 20 of 20 | 0 of 20 | 14.5 | 27 | 90 | 50 | 17 | 3 | -8.18 [-9.56, -6.70] | 16 / 3 |
| Nonmonotone | 50 | 2,000 | 20 of 20 | 0 of 20 | 13 | 20 | 95 | 50 | 19 | 0 | -7.08 [-8.90, -5.30] | 15 / 3 |
| Nonmonotone | 50 | 5,000 | 20 of 20 | 0 of 20 | 12 | 14 | 98 | 50 | 13 | 7 | -3.30 [-4.84, -1.74] | 8 / 12 |

Both packages draw random features, so each dataset was also fit 4 times per package, with draw
seeds 1000 x the dataset's seed + k, k = 0 to 3. "Andrey minus causal-learn, mean of 4 draws" is the
mean over the datasets of Andrey's errors minus causal-learn's, each averaged over the 4 draws, with
its 95% bootstrap interval; the other columns count one fit per dataset.

causal-learn's PC depends on the column order: its `pc()` orients the colliders in column order and
keeps the first of two that conflict (`uc_rule=0`, `uc_priority=2`). The last column compares it
with Andrey's PC in its column-order mode (`collider_order="column"`, with Meek's rules one firing
at a time in column order) on the same datasets, like with like; the other columns are Andrey's
order-free default.

| Data | d | n | CPU | Andrey (s) | causal-learn (s) | Speedup |
|---|---:|---:|---|---:|---:|---:|
| Additive | 5 | 500 | AMD EPYC 7713 | 0.0760 | 0.939 | 10.1x |
| Additive | 5 | 1,000 | AMD EPYC 7713 | 0.109 | 0.831 | 7.47x |
| Additive | 5 | 2,000 | AMD EPYC 7713 | 0.122 | 1.18 | 8.29x |
| Additive | 5 | 5,000 | AMD EPYC 7713 | 0.232 | 1.57 | 5.92x |
| Additive | 10 | 500 | AMD EPYC 7713 | 0.351 | 5.34 | 15.1x |
| Additive | 10 | 1,000 | AMD EPYC 7713 | 0.401 | 6.36 | 13.5x |
| Additive | 10 | 2,000 | AMD EPYC 7713 | 0.576 | 7.48 | 10.6x |
| Additive | 10 | 5,000 | AMD EPYC 7713 | 1.29 | 11.0 | 7.80x |
| Additive | 20 | 500 | AMD EPYC 7713 | 0.899 | 21.3 | 21.2x |
| Additive | 20 | 1,000 | AMD EPYC 7713 | 1.04 | 21.3 | 20.6x |
| Additive | 20 | 2,000 | AMD EPYC 7713 | 1.62 | 24.3 | 15.3x |
| Additive | 20 | 5,000 | AMD EPYC 7713 | 3.98 | 38.9 | 10.6x |
| Additive | 50 | 500 | AMD EPYC 7713 | 3.33 | 114 | 35.7x |
| Additive | 50 | 1,000 | AMD EPYC 7713 | 5.15 | 132 | 25.4x |
| Additive | 50 | 2,000 | AMD EPYC 7713 | 7.30 | 188 | 23.7x |
| Additive | 50 | 5,000 | AMD EPYC 7713 | 33.1 | 425 | 13.7x |
| Nonmonotone | 5 | 500 | AMD EPYC 7713 | 0.0738 | 0.496 | 6.43x |
| Nonmonotone | 5 | 1,000 | AMD EPYC 7713 | 0.0867 | 0.671 | 6.97x |
| Nonmonotone | 5 | 2,000 | AMD EPYC 7713 | 0.123 | 0.869 | 7.03x |
| Nonmonotone | 5 | 5,000 | AMD EPYC 7713 | 0.230 | 1.62 | 6.45x |
| Nonmonotone | 10 | 500 | AMD EPYC 7713 | 0.224 | 2.86 | 13.3x |
| Nonmonotone | 10 | 1,000 | AMD EPYC 7713 | 0.284 | 3.25 | 11.4x |
| Nonmonotone | 10 | 2,000 | AMD EPYC 7713 | 0.455 | 4.02 | 8.71x |
| Nonmonotone | 10 | 5,000 | AMD EPYC 7713 | 0.950 | 7.95 | 7.64x |
| Nonmonotone | 20 | 500 | AMD EPYC 7713 | 0.772 | 10.2 | 13.8x |
| Nonmonotone | 20 | 1,000 | AMD EPYC 7713 | 1.32 | 18.3 | 12.2x |
| Nonmonotone | 20 | 2,000 | AMD EPYC 7713 | 2.01 | 21.9 | 11.6x |
| Nonmonotone | 20 | 5,000 | AMD EPYC 7713 | 5.14 | 38.0 | 8.89x |
| Nonmonotone | 50 | 500 | AMD EPYC 7713 | 2.51 | 46.2 | 18.6x |
| Nonmonotone | 50 | 1,000 | AMD EPYC 7713 | 3.84 | 62.3 | 13.8x |
| Nonmonotone | 50 | 2,000 | AMD EPYC 7713 | 6.98 | 85.4 | 12.2x |
| Nonmonotone | 50 | 5,000 | AMD EPYC 7713 | 18.2 | 213 | 12.3x |

% end generated

Against pcalg with RCIT's RCoT:

% generated: PC, RCoT vs pcalg

| Data | d | n | Datasets | Same graph | Errors, Andrey | Errors, pcalg | Errors, Fisher-Z | Errors, empty graph | Andrey ahead | pcalg ahead | Andrey minus pcalg, mean of 4 draws | Ahead in column order, Andrey / pcalg |
|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
| Additive | 5 | 500 | 20 of 20 | 14 of 20 | 1 | 1 | 1 | 5 | 3 | 3 | -0.49 [-1.01, -0.06] | 3 / 3 |
| Additive | 5 | 1,000 | 20 of 20 | 14 of 20 | 0 | 0.5 | 1 | 5 | 5 | 1 | -0.39 [-0.80, -0.04] | 5 / 1 |
| Additive | 5 | 2,000 | 20 of 20 | 18 of 20 | 0 | 0 | 1 | 5 | 2 | 0 | -0.30 [-0.85, +0.06] | 2 / 0 |
| Additive | 5 | 5,000 | 20 of 20 | 16 of 20 | 0 | 0 | 2 | 5 | 3 | 1 | -0.26 [-0.71, +0.11] | 3 / 1 |
| Additive | 10 | 500 | 20 of 20 | 5 of 20 | 2.5 | 3 | 4.5 | 10 | 8 | 4 | -0.59 [-1.11, -0.12] | 7 / 5 |
| Additive | 10 | 1,000 | 20 of 20 | 7 of 20 | 2 | 2.5 | 3.5 | 10 | 8 | 4 | -0.93 [-1.43, -0.46] | 8 / 4 |
| Additive | 10 | 2,000 | 20 of 20 | 8 of 20 | 1.5 | 2.5 | 4 | 10 | 8 | 4 | -0.47 [-1.04, +0.03] | 6 / 5 |
| Additive | 10 | 5,000 | 20 of 20 | 11 of 20 | 0 | 0.5 | 5.5 | 10 | 6 | 2 | -0.53 [-0.91, -0.21] | 6 / 2 |
| Additive | 20 | 500 | 20 of 20 | 1 of 20 | 7.5 | 9 | 8 | 20 | 15 | 3 | -2.10 [-3.26, -0.95] | 15 / 3 |
| Additive | 20 | 1,000 | 20 of 20 | 0 of 20 | 5.5 | 9 | 8.5 | 20 | 17 | 3 | -2.74 [-3.55, -1.86] | 18 / 2 |
| Additive | 20 | 2,000 | 20 of 20 | 0 of 20 | 3.5 | 6 | 9 | 20 | 16 | 4 | -2.38 [-3.40, -1.24] | 15 / 5 |
| Additive | 20 | 5,000 | 20 of 20 | 4 of 20 | 2 | 6 | 10 | 20 | 13 | 2 | -1.73 [-2.76, -0.66] | 11 / 1 |
| Additive | 50 | 500 | 20 of 20 | 0 of 20 | 18.5 | 27 | 23 | 50 | 19 | 1 | -7.55 [-9.96, -5.35] | 19 / 1 |
| Additive | 50 | 1,000 | 20 of 20 | 0 of 20 | 14.5 | 23 | 22 | 50 | 19 | 1 | -8.47 [-10.81, -6.19] | 18 / 2 |
| Additive | 50 | 2,000 | 20 of 20 | 0 of 20 | 7.5 | 19 | 25.5 | 50 | 20 | 0 | -8.97 [-10.81, -7.24] | 19 / 1 |
| Additive | 50 | 5,000 | 20 of 20 | 0 of 20 | 7 | 16.5 | 29 | 50 | 20 | 0 | -9.15 [-10.47, -7.74] | 19 / 1 |
| Nonmonotone | 5 | 500 | 20 of 20 | 8 of 20 | 3 | 2.5 | 5 | 5 | 5 | 7 | -0.26 [-0.78, +0.30] | 5 / 8 |
| Nonmonotone | 5 | 1,000 | 20 of 20 | 6 of 20 | 1.5 | 3 | 5 | 5 | 8 | 4 | -0.62 [-1.21, -0.10] | 7 / 7 |
| Nonmonotone | 5 | 2,000 | 20 of 20 | 10 of 20 | 0 | 1 | 5.5 | 5 | 7 | 3 | -0.64 [-1.07, -0.24] | 6 / 4 |
| Nonmonotone | 5 | 5,000 | 20 of 20 | 13 of 20 | 0 | 0.5 | 6 | 5 | 5 | 2 | -0.55 [-0.95, -0.19] | 5 / 3 |
| Nonmonotone | 10 | 500 | 20 of 20 | 0 of 20 | 5 | 5.5 | 13 | 10 | 11 | 6 | -1.65 [-2.58, -0.76] | 10 / 6 |
| Nonmonotone | 10 | 1,000 | 20 of 20 | 2 of 20 | 3.5 | 4.5 | 14 | 10 | 11 | 7 | -2.04 [-2.88, -1.09] | 8 / 9 |
| Nonmonotone | 10 | 2,000 | 20 of 20 | 3 of 20 | 1.5 | 3 | 14 | 10 | 15 | 2 | -1.60 [-2.40, -0.68] | 12 / 4 |
| Nonmonotone | 10 | 5,000 | 20 of 20 | 3 of 20 | 0.5 | 3 | 14 | 10 | 13 | 3 | -2.08 [-2.80, -1.38] | 9 / 6 |
| Nonmonotone | 20 | 500 | 20 of 20 | 0 of 20 | 11 | 12 | 28 | 20 | 9 | 10 | -2.88 [-3.98, -1.82] | 11 / 9 |
| Nonmonotone | 20 | 1,000 | 20 of 20 | 0 of 20 | 7 | 12 | 28.5 | 20 | 18 | 2 | -5.24 [-6.29, -4.20] | 13 / 3 |
| Nonmonotone | 20 | 2,000 | 20 of 20 | 0 of 20 | 4.5 | 10 | 30 | 20 | 17 | 3 | -4.97 [-6.40, -3.59] | 17 / 2 |
| Nonmonotone | 20 | 5,000 | 20 of 20 | 1 of 20 | 4 | 7.5 | 30.5 | 20 | 14 | 2 | -5.29 [-6.61, -4.12] | 13 / 3 |
| Nonmonotone | 50 | 500 | 20 of 20 | 0 of 20 | 25 | 37.5 | 87 | 50 | 20 | 0 | -15.76 [-17.98, -13.49] | 17 / 1 |
| Nonmonotone | 50 | 1,000 | 20 of 20 | 0 of 20 | 14.5 | 36.5 | 90 | 50 | 20 | 0 | -19.11 [-20.98, -17.27] | 19 / 1 |
| Nonmonotone | 50 | 2,000 | 20 of 20 | 0 of 20 | 13 | 35.5 | 95 | 50 | 20 | 0 | -23.95 [-26.35, -21.64] | 20 / 0 |
| Nonmonotone | 50 | 5,000 | 20 of 20 | 0 of 20 | 12 | 37 | 98 | 50 | 20 | 0 | -27.26 [-28.68, -25.62] | 20 / 0 |

Both packages draw random features, so each dataset was also fit 4 times per package, with draw
seeds 1000 x the dataset's seed + k, k = 0 to 3. "Andrey minus pcalg, mean of 4 draws" is the mean
over the datasets of Andrey's errors minus pcalg's, each averaged over the 4 draws, with its 95%
bootstrap interval; the other columns count one fit per dataset.

pcalg's PC depends on the column order: its `pc()` runs with `solve.confl = FALSE`, its default,
where a later v-structure overwrites an earlier one in column order. The last column compares it
with Andrey's PC in its column-order mode (`collider_order="column"`, with Meek's rules one firing
at a time in column order) on the same datasets, like with like; the other columns are Andrey's
order-free default.

| Data | d | n | CPU | Andrey (s) | pcalg (s) | Speedup |
|---|---:|---:|---|---:|---:|---:|
| Additive | 5 | 500 | AMD EPYC 7713 | 0.0760 | 1.60 | 14.7x |
| Additive | 5 | 1,000 | AMD EPYC 7713 | 0.109 | 1.74 | 17.2x |
| Additive | 5 | 2,000 | AMD EPYC 7713 | 0.122 | 3.36 | 21.6x |
| Additive | 5 | 5,000 | AMD EPYC 7713 | 0.232 | 6.94 | 20.6x |
| Additive | 10 | 500 | AMD EPYC 7713 | 0.351 | 5.70 | 16.2x |
| Additive | 10 | 1,000 | AMD EPYC 7713 | 0.401 | 8.97 | 18.3x |
| Additive | 10 | 2,000 | AMD EPYC 7713 | 0.576 | 14.2 | 23.3x |
| Additive | 10 | 5,000 | AMD EPYC 7713 | 1.29 | 33.2 | 22.5x |
| Additive | 20 | 500 | AMD EPYC 7713 | 0.899 | 16.8 | 19.8x |
| Additive | 20 | 1,000 | AMD EPYC 7713 | 1.04 | 23.6 | 24.6x |
| Additive | 20 | 2,000 | AMD EPYC 7713 | 1.62 | 33.5 | 21.9x |
| Additive | 20 | 5,000 | AMD EPYC 7713 | 3.98 | 91.6 | 27.6x |
| Additive | 50 | 500 | AMD EPYC 7713 | 3.33 | 69.0 | 22.0x |
| Additive | 50 | 1,000 | AMD EPYC 7713 | 5.15 | 128 | 23.0x |
| Additive | 50 | 2,000 | AMD EPYC 7713 | 7.30 | 221 | 29.1x |
| Additive | 50 | 5,000 | AMD EPYC 7713 | 33.1 | 769 | 19.1x |
| Nonmonotone | 5 | 500 | AMD EPYC 7713 | 0.0738 | 0.696 | 9.51x |
| Nonmonotone | 5 | 1,000 | AMD EPYC 7713 | 0.0867 | 1.30 | 15.0x |
| Nonmonotone | 5 | 2,000 | AMD EPYC 7713 | 0.123 | 2.22 | 13.3x |
| Nonmonotone | 5 | 5,000 | AMD EPYC 7713 | 0.230 | 5.72 | 20.3x |
| Nonmonotone | 10 | 500 | AMD EPYC 7713 | 0.224 | 3.63 | 13.9x |
| Nonmonotone | 10 | 1,000 | AMD EPYC 7713 | 0.284 | 5.87 | 19.4x |
| Nonmonotone | 10 | 2,000 | AMD EPYC 7713 | 0.455 | 8.94 | 22.2x |
| Nonmonotone | 10 | 5,000 | AMD EPYC 7713 | 0.950 | 22.9 | 22.4x |
| Nonmonotone | 20 | 500 | AMD EPYC 7713 | 0.772 | 12.2 | 16.9x |
| Nonmonotone | 20 | 1,000 | AMD EPYC 7713 | 1.32 | 25.9 | 17.8x |
| Nonmonotone | 20 | 2,000 | AMD EPYC 7713 | 2.01 | 42.0 | 23.9x |
| Nonmonotone | 20 | 5,000 | AMD EPYC 7713 | 5.14 | 110 | 21.4x |
| Nonmonotone | 50 | 500 | AMD EPYC 7713 | 2.51 | 51.0 | 22.2x |
| Nonmonotone | 50 | 1,000 | AMD EPYC 7713 | 3.84 | 82.8 | 20.2x |
| Nonmonotone | 50 | 2,000 | AMD EPYC 7713 | 6.98 | 162 | 24.0x |
| Nonmonotone | 50 | 5,000 | AMD EPYC 7713 | 18.2 | 504 | 29.2x |

% end generated

### PC with CMIknn

Andrey's CMIknn runs much faster on CPUs with AVX-512, so its speedup over tigramite depends on the
CPU: about 2.5x on an AMD EPYC 7713 (Zen 3, without AVX-512) and about 10x on the AVX-512 CPUs, an
AMD EPYC 9534 and the Intel Xeon Gold 5418Y and 6542Y.

% generated: PC, CMIknn vs tigramite

| Data | d | n | Datasets | Same graph | Errors, Andrey | Errors, tigramite | Errors, Fisher-Z | Errors, empty graph | Andrey ahead | tigramite ahead | Andrey minus tigramite | Ahead in column order, Andrey / tigramite |
|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
| Additive | 5 | 500 | 20 of 20 | 16 of 20 | 1 | 1 | 1 | 5 | 3 | 1 | -0.20 [-0.60, +0.05] | 3 / 1 |
| Additive | 5 | 1,000 | 20 of 20 | 20 of 20 | 0 | 0 | 1 | 5 | 0 | 0 | +0.00 [+0.00, +0.00] | 0 / 0 |
| Additive | 5 | 2,000 | 20 of 20 | 20 of 20 | 0 | 0 | 1 | 5 | 0 | 0 | +0.00 [+0.00, +0.00] | 0 / 0 |
| Additive | 10 | 500 | 20 of 20 | 8 of 20 | 4 | 5 | 4.5 | 10 | 8 | 1 | -0.95 [-1.60, -0.35] | 7 / 1 |
| Additive | 10 | 1,000 | 20 of 20 | 10 of 20 | 2.5 | 3.5 | 3.5 | 10 | 6 | 4 | -0.45 [-1.15, +0.20] | 5 / 4 |
| Additive | 20 | 500 | 20 of 20 | 3 of 20 | 8.5 | 10 | 8 | 20 | 11 | 1 | -1.90 [-3.05, -0.85] | 12 / 1 |
| Nonmonotone | 5 | 500 | 20 of 20 | 12 of 20 | 3 | 3 | 5 | 5 | 2 | 3 | +0.05 [-0.25, +0.40] | 2 / 5 |
| Nonmonotone | 5 | 1,000 | 20 of 20 | 13 of 20 | 1.5 | 2 | 5 | 5 | 2 | 4 | +0.05 [-0.25, +0.30] | 1 / 6 |
| Nonmonotone | 5 | 2,000 | 20 of 20 | 13 of 20 | 0.5 | 2 | 5.5 | 5 | 5 | 2 | -0.50 [-1.25, +0.00] | 2 / 5 |
| Nonmonotone | 10 | 500 | 20 of 20 | 4 of 20 | 6 | 6.5 | 13 | 10 | 9 | 6 | -0.55 [-1.35, +0.20] | 10 / 4 |
| Nonmonotone | 10 | 1,000 | 18 of 20 | 1 of 18 | 3.5 | 5.5 | 14 | 10 | 13 | 4 | -1.94 [-3.17, -0.83] | 10 / 5 |
| Nonmonotone | 20 | 500 | 20 of 20 | 0 of 20 | 10 | 13 | 28 | 20 | 17 | 1 | -2.15 [-3.10, -1.25] | 16 / 2 |

tigramite's PC depends on the column order: its PC runs with `contemp_collider_rule="none"`, which
its documentation says is not order-independent. The last column compares it with Andrey's PC in its
column-order mode (`collider_order="column"`, with Meek's rules one firing at a time in column
order) on the same datasets, like with like; the other columns are Andrey's order-free default.

Fits past the cap (coverage, never scored): Nonmonotone data, d = 10, n = 1,000: tigramite 2 of 20.

| Data | d | n | CPU | Andrey (s) | tigramite (s) | Speedup |
|---|---:|---:|---|---:|---:|---:|
| Additive | 5 | 500 | Intel Xeon Gold 6542Y | 9.38 | 88.7 | 10.3x |
| Additive | 5 | 1,000 | Intel Xeon Gold 5418Y | 37.2 | 334 | 9.19x |
| Additive | 5 | 2,000 | Intel Core i7-7820X, second machine | 40.7 | 1,220 | 31.9x |
| Additive | 10 | 500 | AMD EPYC 7713 | 295 | 655 | 2.40x |
| Additive | 10 | 1,000 | Intel Core i7-7820X, second machine | 59.3 | 1,790 | 32.7x |
| Additive | 20 | 500 | Intel Core i7-7820X, second machine | 68.0 | 1,930 | 27.0x |
| Nonmonotone | 5 | 500 | AMD EPYC 9534 | 8.08 | 81.5 | 10.1x |
| Nonmonotone | 5 | 1,000 | AMD EPYC 7713 | 201 | 495 | 2.46x |
| Nonmonotone | 5 | 2,000 | Intel Core i7-7820X, second machine | 36.6 | 1,140 | 35.8x |
| Nonmonotone | 10 | 500 | AMD EPYC 7713 | 192 | 551 | 2.56x |
| Nonmonotone | 10 | 1,000 | Intel Core i7-7820X, second machine | 42.8 | 1,680 | 37.7x |
| Nonmonotone | 20 | 500 | Intel Core i7-7820X, second machine | 53.2 | 1,710 | 32.2x |

% end generated

### PC with GPDC

% generated: PC, GPDC vs tigramite

| Data | d | n | Datasets | Same graph | Errors, Andrey | Errors, tigramite | Errors, Fisher-Z | Errors, empty graph | Andrey ahead | tigramite ahead | Andrey minus tigramite | Ahead in column order, Andrey / tigramite |
|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
| Additive | 5 | 500 | 20 of 20 | 19 of 20 | 0 | 0.5 | 1 | 5 | 1 | 0 | -0.20 [-0.60, +0.00] | 1 / 0 |
| Additive | 5 | 1,000 | 20 of 20 | 18 of 20 | 0 | 0 | 1 | 5 | 1 | 0 | -0.05 [-0.15, +0.00] | 0 / 1 |
| Additive | 5 | 2,000 | 20 of 20 | 18 of 20 | 0 | 0 | 1 | 5 | 2 | 0 | -0.45 [-1.10, +0.00] | 2 / 0 |
| Additive | 10 | 500 | 20 of 20 | 6 of 20 | 2 | 3 | 4.5 | 10 | 8 | 4 | -1.00 [-2.00, -0.10] | 8 / 3 |
| Additive | 10 | 1,000 | 20 of 20 | 6 of 20 | 1.5 | 3 | 3.5 | 10 | 12 | 1 | -1.60 [-2.40, -0.80] | 10 / 2 |
| Additive | 20 | 500 | 20 of 20 | 1 of 20 | 9 | 10 | 8 | 20 | 12 | 4 | -1.70 [-2.75, -0.75] | 16 / 1 |
| Additive | 20 | 1,000 | 10 of 20 | 0 of 10 | 3.5 | 6.5 | 6.5 | 20 | 8 | 1 | -2.70 [-4.50, -1.20] | 8 / 0 |
| Nonmonotone | 5 | 500 | 20 of 20 | 14 of 20 | 3 | 3 | 5 | 5 | 4 | 1 | -0.40 [-0.85, +0.00] | 2 / 4 |
| Nonmonotone | 5 | 1,000 | 20 of 20 | 14 of 20 | 2 | 2 | 5 | 5 | 5 | 0 | -0.40 [-0.75, -0.10] | 3 / 3 |
| Nonmonotone | 5 | 2,000 | 20 of 20 | 14 of 20 | 0 | 1 | 5.5 | 5 | 3 | 1 | -0.25 [-0.65, +0.05] | 2 / 3 |
| Nonmonotone | 10 | 500 | 20 of 20 | 2 of 20 | 5 | 6.5 | 13 | 10 | 13 | 3 | -1.60 [-2.65, -0.55] | 11 / 3 |
| Nonmonotone | 10 | 1,000 | 20 of 20 | 1 of 20 | 4.5 | 5.5 | 14 | 10 | 12 | 5 | -1.20 [-2.20, -0.30] | 10 / 5 |
| Nonmonotone | 20 | 500 | 20 of 20 | 0 of 20 | 13.5 | 16 | 28 | 20 | 17 | 2 | -3.00 [-4.00, -2.00] | 14 / 5 |
| Nonmonotone | 20 | 1,000 | 18 of 20 | 0 of 18 | 11.5 | 14 | 28.5 | 20 | 15 | 1 | -3.67 [-5.33, -2.22] | 14 / 3 |

tigramite's PC depends on the column order: its PC runs with `contemp_collider_rule="none"`, which
its documentation says is not order-independent. The last column compares it with Andrey's PC in its
column-order mode (`collider_order="column"`, with Meek's rules one firing at a time in column
order) on the same datasets, like with like; the other columns are Andrey's order-free default.

Fits past the cap (coverage, never scored): Additive data, d = 20, n = 1,000: tigramite 10 of 20.
Nonmonotone data, d = 20, n = 1,000: tigramite 2 of 20.

| Data | d | n | CPU | Andrey (s) | tigramite (s) | Speedup |
|---|---:|---:|---|---:|---:|---:|
| Additive | 5 | 500 | AMD EPYC 7713 | 13.8 | 25.9 | 1.87x |
| Additive | 5 | 1,000 | AMD EPYC 7713 | 49.5 | 80.1 | 1.63x |
| Additive | 5 | 2,000 | Intel Core i7-7820X, second machine | 303 | 415 | 1.21x |
| Additive | 10 | 500 | AMD EPYC 7713 | 36.6 | 76.3 | 1.85x |
| Additive | 10 | 1,000 | Intel Core i7-7820X, second machine | 296 | 431 | 1.46x |
| Additive | 20 | 500 | AMD EPYC 7713 | 146 | 292 | 1.76x |
| Additive | 20 | 1,000 | Intel Core i7-7820X, second machine | - | - | - |
| Nonmonotone | 5 | 500 | AMD EPYC 9534 | 7.45 | 10.7 | 1.44x |
| Nonmonotone | 5 | 1,000 | AMD EPYC 7713 | 66.5 | 102 | 1.54x |
| Nonmonotone | 5 | 2,000 | Intel Core i7-7820X, second machine | 267 | 350 | 1.20x |
| Nonmonotone | 10 | 500 | AMD EPYC 7713 | 36.7 | 66.2 | 1.89x |
| Nonmonotone | 10 | 1,000 | AMD EPYC 7713 | 241 | 342 | 1.42x |
| Nonmonotone | 10 | 1,000 | Intel Core i7-7820X, second machine | 246 | 354 | 1.46x |
| Nonmonotone | 20 | 500 | AMD EPYC 7713 | 122 | 189 | 1.38x |
| Nonmonotone | 20 | 1,000 | Intel Core i7-7820X, second machine | - | - | - |

No speedup: Additive data, d = 20, n = 1,000, Intel Core i7-7820X, second machine: tigramite passed
the 1,800 s cap on 1 of 5 timed datasets; with each unfinished fit at the cap, at least 1.60x.
Nonmonotone data, d = 20, n = 1,000, Intel Core i7-7820X, second machine: tigramite passed the 1,800
s cap on 1 of 5 timed datasets; with each unfinished fit at the cap, at least 1.24x.

% end generated

### FCI with KCI and RCoT

With KCI, against causal-learn:

% generated: FCI, KCI vs causal-learn

| Data | d | n | Datasets | Same graph | Errors, Andrey | Errors, causal-learn | Errors, Fisher-Z | Errors, empty graph | Andrey ahead | causal-learn ahead | Andrey minus causal-learn | Ahead in column order, Andrey / causal-learn |
|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
| Additive | 5 | 500 | 20 of 20 | 20 of 20 | 0 | 0 | 1 | 5 | 0 | 0 | +0.00 [+0.00, +0.00] | 0 / 0 |
| Additive | 5 | 1,000 | 20 of 20 | 20 of 20 | 0 | 0 | 1.5 | 5 | 0 | 0 | +0.00 [+0.00, +0.00] | 0 / 0 |
| Additive | 10 | 500 | 20 of 20 | 18 of 20 | 3 | 3 | 7 | 10 | 0 | 2 | +0.10 [+0.00, +0.25] | 0 / 1 |
| Additive | 10 | 1,000 | 20 of 20 | 20 of 20 | 1.5 | 1.5 | 7 | 10 | 0 | 0 | +0.00 [+0.00, +0.00] | 0 / 0 |
| Nonmonotone | 5 | 500 | 20 of 20 | 20 of 20 | 3 | 3 | 5 | 5 | 0 | 0 | +0.00 [+0.00, +0.00] | 0 / 0 |
| Nonmonotone | 5 | 1,000 | 20 of 20 | 20 of 20 | 2 | 2 | 6 | 5 | 0 | 0 | +0.00 [+0.00, +0.00] | 0 / 0 |
| Nonmonotone | 10 | 500 | 20 of 20 | 17 of 20 | 7 | 7 | 14 | 10 | 3 | 0 | -0.15 [-0.30, +0.00] | 2 / 0 |
| Nonmonotone | 10 | 1,000 | 20 of 20 | 16 of 20 | 6.5 | 6 | 14 | 10 | 2 | 2 | +0.15 [-0.20, +0.65] | 1 / 2 |

causal-learn's FCI depends on the column order: its `fci()` runs the possible-d-separation step and
the orientation rules in column order. The last column compares it with Andrey's FCI in its
column-order mode (`possible_dsep="fast"`, with the orientation rules one firing at a time in column
order) on the same datasets, like with like; the other columns are Andrey's order-free default.

| Data | d | n | CPU | Andrey (s) | causal-learn (s) | Speedup |
|---|---:|---:|---|---:|---:|---:|
| Additive | 5 | 500 | AMD EPYC 9534 | 0.978 | 3.95 | 4.69x |
| Additive | 5 | 1,000 | AMD EPYC 9534 | 5.77 | 18.7 | 3.09x |
| Additive | 10 | 500 | AMD EPYC 9534 | 4.38 | 24.1 | 5.23x |
| Additive | 10 | 1,000 | AMD EPYC 9534 | 36.8 | 126 | 3.35x |
| Nonmonotone | 5 | 500 | Intel Xeon Gold 6542Y | 0.441 | 1.47 | 2.97x |
| Nonmonotone | 5 | 1,000 | Intel Xeon Gold 6542Y | 2.94 | 9.10 | 2.76x |
| Nonmonotone | 10 | 500 | Intel Xeon Gold 6542Y | 2.60 | 10.9 | 4.06x |
| Nonmonotone | 10 | 1,000 | Intel Xeon Gold 6542Y | 14.8 | 52.6 | 3.24x |

% end generated

With RCoT, against causal-learn:

% generated: FCI, RCoT vs causal-learn

| Data | d | n | Datasets | Same graph | Errors, Andrey | Errors, causal-learn | Errors, Fisher-Z | Errors, empty graph | Andrey ahead | causal-learn ahead | Andrey minus causal-learn, mean of 4 draws | Ahead in column order, Andrey / causal-learn |
|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
| Additive | 5 | 500 | 20 of 20 | 16 of 20 | 1 | 0 | 1 | 5 | 0 | 4 | -0.04 [-0.28, +0.21] | 0 / 4 |
| Additive | 5 | 1,000 | 20 of 20 | 14 of 20 | 0 | 0 | 1.5 | 5 | 3 | 3 | -0.44 [-0.96, +0.01] | 3 / 3 |
| Additive | 10 | 500 | 20 of 20 | 5 of 20 | 4 | 5 | 7 | 10 | 6 | 5 | -0.12 [-0.49, +0.24] | 6 / 5 |
| Additive | 10 | 1,000 | 20 of 20 | 9 of 20 | 3.5 | 2 | 7 | 10 | 5 | 6 | -0.54 [-1.19, +0.03] | 5 / 6 |
| Nonmonotone | 5 | 500 | 20 of 20 | 7 of 20 | 3.5 | 3.5 | 5 | 5 | 5 | 6 | -0.29 [-0.65, +0.09] | 6 / 7 |
| Nonmonotone | 5 | 1,000 | 20 of 20 | 8 of 20 | 2 | 2.5 | 6 | 5 | 5 | 5 | -0.33 [-0.71, +0.04] | 5 / 5 |
| Nonmonotone | 10 | 500 | 20 of 20 | 0 of 20 | 7 | 8 | 14 | 10 | 9 | 4 | -1.18 [-1.65, -0.71] | 8 / 4 |
| Nonmonotone | 10 | 1,000 | 20 of 20 | 1 of 20 | 5 | 5.5 | 14 | 10 | 10 | 7 | -0.86 [-1.29, -0.41] | 10 / 8 |

Both packages draw random features, so each dataset was also fit 4 times per package, with draw
seeds 1000 x the dataset's seed + k, k = 0 to 3. "Andrey minus causal-learn, mean of 4 draws" is the
mean over the datasets of Andrey's errors minus causal-learn's, each averaged over the 4 draws, with
its 95% bootstrap interval; the other columns count one fit per dataset.

causal-learn's FCI depends on the column order: its `fci()` runs the possible-d-separation step and
the orientation rules in column order. The last column compares it with Andrey's FCI in its
column-order mode (`possible_dsep="fast"`, with the orientation rules one firing at a time in column
order) on the same datasets, like with like; the other columns are Andrey's order-free default.

| Data | d | n | CPU | Andrey (s) | causal-learn (s) | Speedup |
|---|---:|---:|---|---:|---:|---:|
| Additive | 5 | 500 | AMD EPYC 9534 | 0.0693 | 0.701 | 10.6x |
| Additive | 5 | 1,000 | AMD EPYC 9534 | 0.0842 | 0.812 | 9.11x |
| Additive | 10 | 500 | AMD EPYC 9534 | 0.218 | 4.23 | 16.8x |
| Additive | 10 | 1,000 | AMD EPYC 9534 | 0.268 | 4.05 | 11.5x |
| Nonmonotone | 5 | 500 | AMD EPYC 9534 | 0.0693 | 0.472 | 7.37x |
| Nonmonotone | 5 | 1,000 | AMD EPYC 9534 | 0.0710 | 0.696 | 8.81x |
| Nonmonotone | 10 | 500 | AMD EPYC 9534 | 0.203 | 2.50 | 9.54x |
| Nonmonotone | 10 | 1,000 | AMD EPYC 9534 | 0.363 | 3.83 | 10.3x |

% end generated

### KCI and CMIknn on a GPU

Andrey's KCI and CMIknn also run on an NVIDIA GPU (`ANDREY_BACKEND=cuda`). We timed PC and FCI with
them on an NVIDIA A100-SXM4-80GB, with 4 cores of its AMD EPYC 7413 host, on the same 5 datasets
per size as the CPU rows, in 3 rounds. Each speedup divides Andrey's time on the CPU the row names,
from that CPU's own timing rounds, by its time on the GPU; a size timed on two CPUs has a row for
each. We publish a size only when every GPU fit returned the CPU's graph on every dataset.

% generated: gpu

| Method | Data | d | n | CPU | Andrey on the CPU (s) | Andrey on the GPU (s) | GPU speedup | Same graph |
|---|---|---:|---:|---|---:|---:|---:|---:|
| PC, CMIknn | Additive | 5 | 500 | AMD EPYC 7713 | 45.0 | 4.26 | 10.1x | 5 of 5 |
| PC, CMIknn | Additive | 5 | 500 | Intel Xeon Gold 6542Y | 9.38 | 4.26 | 2.07x | 5 of 5 |
| PC, CMIknn | Additive | 5 | 1,000 | AMD EPYC 7713 | 183 | 12.5 | 14.2x | 5 of 5 |
| PC, CMIknn | Additive | 5 | 1,000 | Intel Xeon Gold 5418Y | 37.2 | 12.5 | 2.97x | 5 of 5 |
| PC, CMIknn | Additive | 5 | 2,000 | Intel Core i7-7820X, second machine | 40.7 | 45.3 | 0.931x | 5 of 5 |
| PC, CMIknn | Additive | 10 | 500 | AMD EPYC 7713 | 295 | 23.6 | 12.6x | 5 of 5 |
| PC, CMIknn | Additive | 10 | 1,000 | Intel Core i7-7820X, second machine | 59.3 | 67.5 | 0.915x | 5 of 5 |
| PC, CMIknn | Additive | 20 | 500 | Intel Core i7-7820X, second machine | 68.0 | 106 | 0.644x | 5 of 5 |
| PC, CMIknn | Nonmonotone | 5 | 500 | AMD EPYC 7713 | 36.3 | 3.93 | 9.05x | 5 of 5 |
| PC, CMIknn | Nonmonotone | 5 | 500 | AMD EPYC 9534 | 8.08 | 3.93 | 2.06x | 5 of 5 |
| PC, CMIknn | Nonmonotone | 5 | 1,000 | AMD EPYC 7713 | 201 | 11.5 | 17.2x | 5 of 5 |
| PC, CMIknn | Nonmonotone | 5 | 2,000 | Intel Core i7-7820X, second machine | 36.6 | 38.6 | 0.946x | 5 of 5 |
| PC, CMIknn | Nonmonotone | 10 | 500 | AMD EPYC 7713 | 192 | 15.0 | 12.8x | 5 of 5 |
| PC, CMIknn | Nonmonotone | 10 | 1,000 | Intel Core i7-7820X, second machine | 42.8 | 46.4 | 0.923x | 5 of 5 |
| PC, CMIknn | Nonmonotone | 20 | 500 | Intel Core i7-7820X, second machine | 53.2 | 76.8 | 0.690x | 5 of 5 |
| PC, KCI | Additive | 5 | 500 | AMD EPYC 7713 | 1.14 | 0.387 | 3.46x | 5 of 5 |
| PC, KCI | Additive | 5 | 1,000 | AMD EPYC 7713 | 9.29 | 1.56 | 6.78x | 5 of 5 |
| PC, KCI | Additive | 5 | 2,000 | AMD EPYC 7713 | 37.1 | 9.06 | 4.09x | 5 of 5 |
| PC, KCI | Additive | 5 | 5,000 | AMD EPYC 7713 | 923 | 58.3 | 15.8x | 5 of 5 |
| PC, KCI | Additive | 10 | 500 | AMD EPYC 7713 | 5.53 | 2.06 | 2.77x | 5 of 5 |
| PC, KCI | Additive | 10 | 1,000 | AMD EPYC 9534 | 42.4 | 8.62 | 4.58x | 5 of 5 |
| PC, KCI | Additive | 10 | 2,000 | AMD EPYC 9534 | 122 | 39.9 | 3.15x | 5 of 5 |
| PC, KCI | Additive | 20 | 500 | AMD EPYC 7713 | 19.3 | 6.81 | 2.70x | 5 of 5 |
| PC, KCI | Nonmonotone | 5 | 500 | AMD EPYC 7713 | 0.902 | 0.276 | 3.30x | 5 of 5 |
| PC, KCI | Nonmonotone | 5 | 1,000 | AMD EPYC 7713 | 7.86 | 1.31 | 6.08x | 5 of 5 |
| PC, KCI | Nonmonotone | 5 | 2,000 | AMD EPYC 7713 | 28.6 | 6.33 | 4.77x | 5 of 5 |
| PC, KCI | Nonmonotone | 5 | 2,000 | AMD EPYC 9534 | 18.7 | 6.33 | 3.14x | 5 of 5 |
| PC, KCI | Nonmonotone | 5 | 5,000 | AMD EPYC 7713 | 1,480 | 58.3 | 25.4x | 5 of 5 |
| PC, KCI | Nonmonotone | 10 | 500 | AMD EPYC 7713 | 4.36 | 1.33 | 3.52x | 5 of 5 |
| PC, KCI | Nonmonotone | 10 | 1,000 | AMD EPYC 7713 | 28.1 | 5.58 | 5.12x | 5 of 5 |
| PC, KCI | Nonmonotone | 10 | 1,000 | AMD EPYC 9555 | 10.9 | 5.58 | 1.98x | 5 of 5 |
| PC, KCI | Nonmonotone | 10 | 2,000 | AMD EPYC 7713 | 133 | 28.0 | 5.29x | 5 of 5 |
| PC, KCI | Nonmonotone | 20 | 500 | AMD EPYC 7713 | 13.5 | 4.74 | 2.88x | 5 of 5 |
| PC, KCI | Nonmonotone | 20 | 2,000 | AMD EPYC 7713 | 712 | 119 | 6.03x | 5 of 5 |
| FCI, CMIknn | Additive | 5 | 500 | AMD EPYC 9534 | 7.55 | 3.80 | 1.93x | 5 of 5 |
| FCI, CMIknn | Nonmonotone | 5 | 500 | AMD EPYC 9534 | 6.96 | 3.71 | 1.82x | 5 of 5 |
| FCI, KCI | Additive | 5 | 500 | AMD EPYC 9534 | 0.978 | 0.386 | 2.41x | 5 of 5 |
| FCI, KCI | Additive | 5 | 1,000 | AMD EPYC 9534 | 5.77 | 1.61 | 4.16x | 5 of 5 |
| FCI, KCI | Additive | 10 | 500 | AMD EPYC 9534 | 4.38 | 2.23 | 1.95x | 5 of 5 |
| FCI, KCI | Additive | 10 | 1,000 | AMD EPYC 9534 | 36.8 | 9.85 | 3.74x | 5 of 5 |
| FCI, KCI | Nonmonotone | 5 | 500 | Intel Xeon Gold 6542Y | 0.441 | 0.275 | 1.52x | 5 of 5 |
| FCI, KCI | Nonmonotone | 5 | 1,000 | Intel Xeon Gold 6542Y | 2.94 | 1.31 | 2.39x | 5 of 5 |
| FCI, KCI | Nonmonotone | 10 | 500 | Intel Xeon Gold 6542Y | 2.60 | 1.68 | 1.38x | 5 of 5 |
| FCI, KCI | Nonmonotone | 10 | 1,000 | Intel Xeon Gold 6542Y | 14.8 | 7.42 | 2.10x | 5 of 5 |

% end generated

On the GPU, KCI runs 2.70x to 25.4x faster than on an AMD EPYC 7713, the most at 5,000 rows, and
1.38x to 4.58x faster than on the AVX-512 servers. CMIknn runs 9.05x to 17.2x faster than on an AMD
EPYC 7713 and 1.82x to 2.97x faster than on the AVX-512 servers, and slower than on the second
machine. KCI at 20 variables has no GPU row at 1,000 rows, which has no CPU timing yet, or at 2,000
rows of additive data, where a CPU fit passed its cap (see
[Pending measurements](#pending-measurements)).

### The tests query by query

Each test answers independent draws (level at `alpha` 0.05, and the KS distance of their p-values to
the uniform) and dependent draws (power), conditioning on `k` variables. "Same decision" is the
share of draws on which the reference decides as Andrey does. Seconds per query are the median over
3 rounds, on the CPU each row names.

% generated: queries

| Test | n | k | Package | Level | Power | KS distance | Seconds per query | Same decision as Andrey | CPU |
|---|---:|---:|---|---:|---:|---:|---:|---:|---|
| CMIknn | 200 | 0 | Andrey | 0.08 | 1.00 | 0.10 | 0.234 | - | AMD EPYC 7713 |
| CMIknn | 200 | 0 | tigramite | 0.08 | 1.00 | 0.11 | 0.584 | 100% | AMD EPYC 7713 |
| CMIknn | 200 | 1 | Andrey | 0.04 | 1.00 | 0.14 | 0.298 | - | AMD EPYC 7713 |
| CMIknn | 200 | 1 | tigramite | 0.04 | 1.00 | 0.13 | 1.20 | 100% | AMD EPYC 7713 |
| CMIknn | 200 | 2 | Andrey | 0.02 | 0.90 | 0.11 | 0.310 | - | AMD EPYC 7713 |
| CMIknn | 200 | 2 | tigramite | 0.02 | 0.95 | 0.16 | 1.72 | 99% | AMD EPYC 7713 |
| CMIknn | 200 | 3 | Andrey | 0.12 | 0.85 | 0.26 | 0.312 | - | AMD EPYC 7713 |
| CMIknn | 200 | 3 | tigramite | 0.20 | 0.95 | 0.36 | 2.34 | 89% | AMD EPYC 7713 |
| CMIknn | 500 | 0 | Andrey | 0.02 | 1.00 | 0.13 | 1.23 | - | AMD EPYC 7713 |
| CMIknn | 500 | 0 | tigramite | 0.02 | 1.00 | 0.15 | 2.28 | 100% | AMD EPYC 7713 |
| CMIknn | 500 | 1 | Andrey | 0.02 | 1.00 | 0.17 | 1.39 | - | AMD EPYC 7713 |
| CMIknn | 500 | 1 | tigramite | 0.04 | 1.00 | 0.16 | 4.56 | 99% | AMD EPYC 7713 |
| CMIknn | 500 | 2 | Andrey | 0.02 | 1.00 | 0.14 | 1.42 | - | AMD EPYC 7713 |
| CMIknn | 500 | 2 | tigramite | 0.10 | 1.00 | 0.16 | 6.80 | 94% | AMD EPYC 7713 |
| CMIknn | 500 | 3 | Andrey | 0.16 | 1.00 | 0.33 | 1.45 | - | AMD EPYC 7713 |
| CMIknn | 500 | 3 | tigramite | 0.28 | 1.00 | 0.46 | 9.80 | 91% | AMD EPYC 7713 |
| CMIknn | 1,000 | 0 | Andrey | 0.08 | 1.00 | 0.19 | 1.90 | - | AMD EPYC 9534 |
| CMIknn | 1,000 | 0 | tigramite | 0.08 | 1.00 | 0.16 | 5.87 | 100% | AMD EPYC 9534 |
| CMIknn | 1,000 | 1 | Andrey | 0.04 | 1.00 | 0.13 | 2.15 | - | AMD EPYC 9534 |
| CMIknn | 1,000 | 1 | tigramite | 0.08 | 1.00 | 0.12 | 10.8 | 97% | AMD EPYC 9534 |
| CMIknn | 1,000 | 2 | Andrey | 0.10 | 1.00 | 0.19 | 2.22 | - | AMD EPYC 9534 |
| CMIknn | 1,000 | 2 | tigramite | 0.12 | 1.00 | 0.22 | 16.6 | 96% | AMD EPYC 9534 |
| CMIknn | 1,000 | 3 | Andrey | 0.14 | 1.00 | 0.25 | 2.18 | - | AMD EPYC 9534 |
| CMIknn | 1,000 | 3 | tigramite | 0.22 | 1.00 | 0.38 | 24.7 | 89% | AMD EPYC 9534 |
| GPDC | 200 | 0 | Andrey | 0.04 | 1.00 | 0.17 | 0.229 | - | AMD EPYC 7713 |
| GPDC | 200 | 0 | tigramite | 0.08 | 1.00 | 0.15 | 0.000336 | 97% | AMD EPYC 7713 |
| GPDC | 200 | 1 | Andrey | 0.02 | 1.00 | 0.10 | 0.290 | - | AMD EPYC 7713 |
| GPDC | 200 | 1 | tigramite | 0.04 | 1.00 | 0.13 | 0.0598 | 99% | AMD EPYC 7713 |
| GPDC | 200 | 2 | Andrey | 0.06 | 1.00 | 0.12 | 0.291 | - | AMD EPYC 7713 |
| GPDC | 200 | 2 | tigramite | 0.08 | 1.00 | 0.15 | 0.0603 | 99% | AMD EPYC 7713 |
| GPDC | 200 | 3 | Andrey | 0.00 | 1.00 | 0.22 | 0.292 | - | AMD EPYC 7713 |
| GPDC | 200 | 3 | tigramite | 0.04 | 1.00 | 0.21 | 0.0676 | 97% | AMD EPYC 7713 |
| GPDC | 500 | 0 | Andrey | 0.02 | 1.00 | 0.11 | 1.37 | - | AMD EPYC 7713 |
| GPDC | 500 | 0 | tigramite | 0.02 | 1.00 | 0.14 | 0.000699 | 100% | AMD EPYC 7713 |
| GPDC | 500 | 1 | Andrey | 0.02 | 1.00 | 0.11 | 2.01 | - | AMD EPYC 7713 |
| GPDC | 500 | 1 | tigramite | 0.04 | 1.00 | 0.09 | 0.682 | 99% | AMD EPYC 7713 |
| GPDC | 500 | 2 | Andrey | 0.04 | 1.00 | 0.21 | 1.77 | - | AMD EPYC 7713 |
| GPDC | 500 | 2 | tigramite | 0.04 | 1.00 | 0.23 | 0.664 | 100% | AMD EPYC 7713 |
| GPDC | 500 | 3 | Andrey | 0.08 | 1.00 | 0.09 | 1.86 | - | AMD EPYC 7713 |
| GPDC | 500 | 3 | tigramite | 0.08 | 1.00 | 0.11 | 0.678 | 100% | AMD EPYC 7713 |
| GPDC | 1,000 | 0 | Andrey | 0.14 | 1.00 | 0.18 | 4.21 | - | AMD EPYC 9534 |
| GPDC | 1,000 | 0 | tigramite | 0.12 | 1.00 | 0.19 | 0.00107 | 99% | AMD EPYC 9534 |
| GPDC | 1,000 | 1 | Andrey | 0.12 | 1.00 | 0.16 | 6.85 | - | AMD EPYC 9534 |
| GPDC | 1,000 | 1 | tigramite | 0.12 | 1.00 | 0.18 | 2.78 | 100% | AMD EPYC 9534 |
| GPDC | 1,000 | 2 | Andrey | 0.08 | 1.00 | 0.09 | 7.09 | - | AMD EPYC 9534 |
| GPDC | 1,000 | 2 | tigramite | 0.08 | 1.00 | 0.14 | 2.82 | 100% | AMD EPYC 9534 |
| GPDC | 1,000 | 3 | Andrey | 0.04 | 1.00 | 0.17 | 6.97 | - | AMD EPYC 9534 |
| GPDC | 1,000 | 3 | tigramite | 0.04 | 1.00 | 0.22 | 2.72 | 100% | AMD EPYC 9534 |
| KCI | 200 | 0 | Andrey | 0.07 | 1.00 | 0.05 | 0.00223 | - | AMD EPYC 7713 |
| KCI | 200 | 0 | RCIT | 0.05 | 1.00 | 0.07 | 0.00550 | 92% | AMD EPYC 7713 |
| KCI | 200 | 0 | causal-learn | 0.07 | 1.00 | 0.05 | 0.00397 | 100% | AMD EPYC 7713 |
| KCI | 200 | 0 | causal-learn FastKCI | 0.14 | 1.00 | 0.20 | 0.645 | 87% | AMD EPYC 7713 |
| KCI | 200 | 1 | Andrey | 0.07 | 1.00 | 0.09 | 0.00663 | - | AMD EPYC 7713 |
| KCI | 200 | 1 | RCIT | 0.17 | 1.00 | 0.16 | 0.0160 | 91% | AMD EPYC 7713 |
| KCI | 200 | 1 | causal-learn | 0.07 | 1.00 | 0.09 | 0.0172 | 100% | AMD EPYC 7713 |
| KCI | 200 | 1 | causal-learn FastKCI | 0.10 | 1.00 | 0.43 | 0.573 | 87% | AMD EPYC 7713 |
| KCI | 200 | 2 | Andrey | 0.06 | 1.00 | 0.17 | 0.00685 | - | AMD EPYC 7713 |
| KCI | 200 | 2 | RCIT | 0.23 | 1.00 | 0.20 | 0.0180 | 84% | AMD EPYC 7713 |
| KCI | 200 | 2 | causal-learn | 0.06 | 1.00 | 0.17 | 0.0195 | 100% | AMD EPYC 7713 |
| KCI | 200 | 2 | causal-learn FastKCI | 0.29 | 1.00 | 0.54 | 0.493 | 73% | AMD EPYC 7713 |
| KCI | 200 | 3 | Andrey | 0.10 | 1.00 | 0.25 | 0.00724 | - | AMD EPYC 7713 |
| KCI | 200 | 3 | RCIT | 0.28 | 1.00 | 0.24 | 0.0180 | 81% | AMD EPYC 7713 |
| KCI | 200 | 3 | causal-learn | 0.10 | 1.00 | 0.25 | 0.0214 | 100% | AMD EPYC 7713 |
| KCI | 200 | 3 | causal-learn FastKCI | 0.32 | 0.90 | 0.62 | 0.453 | 70% | AMD EPYC 7713 |
| KCI | 500 | 0 | Andrey | 0.07 | 1.00 | 0.06 | 0.00919 | - | AMD EPYC 7713 |
| KCI | 500 | 0 | RCIT | 0.04 | 1.00 | 0.07 | 0.0560 | 94% | AMD EPYC 7713 |
| KCI | 500 | 0 | causal-learn | 0.07 | 1.00 | 0.06 | 0.0177 | 100% | AMD EPYC 7713 |
| KCI | 500 | 0 | causal-learn FastKCI | 0.07 | 1.00 | 0.09 | 0.875 | 91% | AMD EPYC 7713 |
| KCI | 500 | 1 | Andrey | 0.07 | 1.00 | 0.07 | 0.0607 | - | AMD EPYC 7713 |
| KCI | 500 | 1 | RCIT | 0.04 | 1.00 | 0.04 | 0.154 | 96% | AMD EPYC 7713 |
| KCI | 500 | 1 | causal-learn | 0.07 | 1.00 | 0.07 | 0.141 | 100% | AMD EPYC 7713 |
| KCI | 500 | 1 | causal-learn FastKCI | 0.05 | 1.00 | 0.29 | 0.966 | 90% | AMD EPYC 7713 |
| KCI | 500 | 2 | Andrey | 0.07 | 1.00 | 0.15 | 0.0613 | - | AMD EPYC 7713 |
| KCI | 500 | 2 | RCIT | 0.07 | 1.00 | 0.15 | 0.161 | 94% | AMD EPYC 7713 |
| KCI | 500 | 2 | causal-learn | 0.07 | 1.00 | 0.16 | 0.146 | 100% | AMD EPYC 7713 |
| KCI | 500 | 2 | causal-learn FastKCI | 0.14 | 1.00 | 0.39 | 0.927 | 81% | AMD EPYC 7713 |
| KCI | 500 | 3 | Andrey | 0.08 | 1.00 | 0.16 | 0.0646 | - | AMD EPYC 7713 |
| KCI | 500 | 3 | RCIT | 0.08 | 1.00 | 0.15 | 0.169 | 93% | AMD EPYC 7713 |
| KCI | 500 | 3 | causal-learn | 0.08 | 1.00 | 0.15 | 0.159 | 99% | AMD EPYC 7713 |
| KCI | 500 | 3 | causal-learn FastKCI | 0.26 | 1.00 | 0.50 | 0.857 | 76% | AMD EPYC 7713 |
| KCI | 1,000 | 0 | Andrey | 0.07 | 1.00 | 0.06 | 0.0520 | - | AMD EPYC 7713 |
| KCI | 1,000 | 0 | RCIT | 0.07 | 1.00 | 0.09 | 0.384 | 92% | AMD EPYC 7713 |
| KCI | 1,000 | 0 | causal-learn | 0.07 | 1.00 | 0.06 | 0.0538 | 100% | AMD EPYC 7713 |
| KCI | 1,000 | 0 | causal-learn FastKCI | 0.05 | 1.00 | 0.04 | 1.23 | 91% | AMD EPYC 7713 |
| KCI | 1,000 | 1 | Andrey | 0.07 | 1.00 | 0.06 | 0.402 | - | AMD EPYC 7713 |
| KCI | 1,000 | 1 | RCIT | 0.06 | 1.00 | 0.10 | 0.992 | 96% | AMD EPYC 7713 |
| KCI | 1,000 | 1 | causal-learn | 0.07 | 1.00 | 0.06 | 0.872 | 100% | AMD EPYC 7713 |
| KCI | 1,000 | 1 | causal-learn FastKCI | 0.07 | 1.00 | 0.21 | 1.69 | 89% | AMD EPYC 7713 |
| KCI | 1,000 | 2 | Andrey | 0.06 | 1.00 | 0.12 | 0.403 | - | AMD EPYC 7713 |
| KCI | 1,000 | 2 | RCIT | 0.07 | 1.00 | 0.16 | 1.01 | 93% | AMD EPYC 7713 |
| KCI | 1,000 | 2 | causal-learn | 0.06 | 1.00 | 0.12 | 0.911 | 100% | AMD EPYC 7713 |
| KCI | 1,000 | 2 | causal-learn FastKCI | 0.10 | 1.00 | 0.22 | 1.74 | 88% | AMD EPYC 7713 |
| KCI | 1,000 | 3 | Andrey | 0.09 | 1.00 | 0.17 | 0.409 | - | AMD EPYC 7713 |
| KCI | 1,000 | 3 | RCIT | 0.10 | 1.00 | 0.26 | 1.04 | 93% | AMD EPYC 7713 |
| KCI | 1,000 | 3 | causal-learn | 0.10 | 1.00 | 0.16 | 0.960 | 100% | AMD EPYC 7713 |
| KCI | 1,000 | 3 | causal-learn FastKCI | 0.18 | 1.00 | 0.40 | 1.75 | 82% | AMD EPYC 7713 |
| KCI | 2,000 | 0 | Andrey | 0.07 | 1.00 | 0.07 | 0.104 | - | AMD EPYC 9534 |
| KCI | 2,000 | 0 | RCIT | 0.04 | 1.00 | 0.09 | 2.23 | 92% | AMD EPYC 9534 |
| KCI | 2,000 | 0 | causal-learn | 0.07 | 1.00 | 0.07 | 0.122 | 100% | AMD EPYC 9534 |
| KCI | 2,000 | 0 | causal-learn FastKCI | 0.04 | 1.00 | 0.05 | 2.30 | 91% | AMD EPYC 9534 |
| KCI | 2,000 | 1 | Andrey | 0.04 | 1.00 | 0.09 | 3.77 | - | AMD EPYC 7713 |
| KCI | 2,000 | 1 | RCIT | 0.04 | 1.00 | 0.06 | 7.17 | 95% | AMD EPYC 7713 |
| KCI | 2,000 | 1 | causal-learn | 0.04 | 1.00 | 0.09 | 8.85 | 100% | AMD EPYC 7713 |
| KCI | 2,000 | 1 | causal-learn FastKCI | 0.06 | 1.00 | 0.09 | 5.71 | 93% | AMD EPYC 7713 |
| KCI | 2,000 | 2 | Andrey | 0.04 | 1.00 | 0.08 | 2.24 | - | AMD EPYC 9534 |
| KCI | 2,000 | 2 | RCIT | 0.12 | 1.00 | 0.24 | 5.77 | 88% | AMD EPYC 9534 |
| KCI | 2,000 | 2 | causal-learn | 0.04 | 1.00 | 0.07 | 5.68 | 100% | AMD EPYC 9534 |
| KCI | 2,000 | 2 | causal-learn FastKCI | 0.16 | 1.00 | 0.28 | 4.46 | 85% | AMD EPYC 9534 |
| KCI | 2,000 | 3 | Andrey | 0.16 | 1.00 | 0.22 | 3.23 | - | AMD EPYC 7713 |
| KCI | 2,000 | 3 | RCIT | 0.23 | 1.00 | 0.39 | 7.91 | 81% | AMD EPYC 7713 |
| KCI | 2,000 | 3 | causal-learn | 0.17 | 1.00 | 0.23 | 9.61 | 99% | AMD EPYC 7713 |
| KCI | 2,000 | 3 | causal-learn FastKCI | 0.21 | 1.00 | 0.45 | 7.43 | 77% | AMD EPYC 7713 |
| RCoT | 200 | 0 | Andrey | 0.05 | 1.00 | 0.07 | 0.00199 | - | AMD EPYC 7713 |
| RCoT | 200 | 0 | RCIT | 0.04 | 1.00 | 0.08 | 0.00300 | 95% | AMD EPYC 7713 |
| RCoT | 200 | 0 | causal-learn | 0.07 | 1.00 | 0.09 | 0.00387 | 95% | AMD EPYC 7713 |
| RCoT | 200 | 1 | Andrey | 0.04 | 1.00 | 0.07 | 0.00368 | - | AMD EPYC 7713 |
| RCoT | 200 | 1 | RCIT | 0.09 | 1.00 | 0.08 | 0.00900 | 93% | AMD EPYC 7713 |
| RCoT | 200 | 1 | causal-learn | 0.07 | 1.00 | 0.06 | 0.00655 | 94% | AMD EPYC 7713 |
| RCoT | 200 | 2 | Andrey | 0.06 | 1.00 | 0.13 | 0.00375 | - | AMD EPYC 7713 |
| RCoT | 200 | 2 | RCIT | 0.06 | 1.00 | 0.16 | 0.00900 | 94% | AMD EPYC 7713 |
| RCoT | 200 | 2 | causal-learn | 0.09 | 1.00 | 0.17 | 0.00737 | 92% | AMD EPYC 7713 |
| RCoT | 200 | 3 | Andrey | 0.16 | 1.00 | 0.25 | 0.00383 | - | AMD EPYC 7713 |
| RCoT | 200 | 3 | RCIT | 0.10 | 1.00 | 0.26 | 0.00900 | 85% | AMD EPYC 7713 |
| RCoT | 200 | 3 | causal-learn | 0.12 | 1.00 | 0.23 | 0.00822 | 82% | AMD EPYC 7713 |
| RCoT | 500 | 0 | Andrey | 0.04 | 1.00 | 0.07 | 0.00543 | - | AMD EPYC 7713 |
| RCoT | 500 | 0 | RCIT | 0.03 | 1.00 | 0.06 | 0.00800 | 95% | AMD EPYC 7713 |
| RCoT | 500 | 0 | causal-learn | 0.04 | 1.00 | 0.09 | 0.0152 | 96% | AMD EPYC 7713 |
| RCoT | 500 | 1 | Andrey | 0.04 | 1.00 | 0.06 | 0.00954 | - | AMD EPYC 7713 |
| RCoT | 500 | 1 | RCIT | 0.04 | 1.00 | 0.06 | 0.0240 | 95% | AMD EPYC 7713 |
| RCoT | 500 | 1 | causal-learn | 0.04 | 1.00 | 0.08 | 0.0174 | 95% | AMD EPYC 7713 |
| RCoT | 500 | 2 | Andrey | 0.12 | 1.00 | 0.27 | 0.00976 | - | AMD EPYC 7713 |
| RCoT | 500 | 2 | RCIT | 0.11 | 1.00 | 0.24 | 0.0260 | 90% | AMD EPYC 7713 |
| RCoT | 500 | 2 | causal-learn | 0.08 | 1.00 | 0.16 | 0.0171 | 86% | AMD EPYC 7713 |
| RCoT | 500 | 3 | Andrey | 0.14 | 1.00 | 0.26 | 0.00993 | - | AMD EPYC 7713 |
| RCoT | 500 | 3 | RCIT | 0.20 | 1.00 | 0.41 | 0.0260 | 80% | AMD EPYC 7713 |
| RCoT | 500 | 3 | causal-learn | 0.23 | 1.00 | 0.35 | 0.0173 | 75% | AMD EPYC 7713 |
| RCoT | 1,000 | 0 | Andrey | 0.04 | 1.00 | 0.09 | 0.00584 | - | AMD EPYC 7713 |
| RCoT | 1,000 | 0 | RCIT | 0.04 | 1.00 | 0.08 | 0.00800 | 97% | AMD EPYC 7713 |
| RCoT | 1,000 | 0 | causal-learn | 0.04 | 1.00 | 0.09 | 0.0117 | 97% | AMD EPYC 7713 |
| RCoT | 1,000 | 1 | Andrey | 0.09 | 1.00 | 0.18 | 0.0111 | - | AMD EPYC 7713 |
| RCoT | 1,000 | 1 | RCIT | 0.07 | 1.00 | 0.19 | 0.0370 | 88% | AMD EPYC 7713 |
| RCoT | 1,000 | 1 | causal-learn | 0.07 | 1.00 | 0.12 | 0.0195 | 89% | AMD EPYC 7713 |
| RCoT | 1,000 | 2 | Andrey | 0.20 | 1.00 | 0.25 | 0.0114 | - | AMD EPYC 7713 |
| RCoT | 1,000 | 2 | RCIT | 0.18 | 1.00 | 0.33 | 0.0380 | 86% | AMD EPYC 7713 |
| RCoT | 1,000 | 2 | causal-learn | 0.20 | 1.00 | 0.31 | 0.0185 | 78% | AMD EPYC 7713 |
| RCoT | 1,000 | 3 | Andrey | 0.34 | 1.00 | 0.47 | 0.0116 | - | AMD EPYC 7713 |
| RCoT | 1,000 | 3 | RCIT | 0.64 | 1.00 | 0.74 | 0.0380 | 64% | AMD EPYC 7713 |
| RCoT | 1,000 | 3 | causal-learn | 0.47 | 1.00 | 0.52 | 0.0185 | 66% | AMD EPYC 7713 |
| RCoT | 2,000 | 0 | Andrey | 0.04 | 1.00 | 0.08 | 0.00675 | - | AMD EPYC 7713 |
| RCoT | 2,000 | 0 | RCIT | 0.03 | 1.00 | 0.07 | 0.00900 | 97% | AMD EPYC 7713 |
| RCoT | 2,000 | 0 | causal-learn | 0.06 | 1.00 | 0.09 | 0.0110 | 96% | AMD EPYC 7713 |
| RCoT | 2,000 | 1 | Andrey | 0.07 | 1.00 | 0.09 | 0.0145 | - | AMD EPYC 7713 |
| RCoT | 2,000 | 1 | RCIT | 0.13 | 1.00 | 0.20 | 0.0600 | 88% | AMD EPYC 7713 |
| RCoT | 2,000 | 1 | causal-learn | 0.06 | 1.00 | 0.16 | 0.0242 | 91% | AMD EPYC 7713 |
| RCoT | 2,000 | 2 | Andrey | 0.36 | 1.00 | 0.49 | 0.0148 | - | AMD EPYC 7713 |
| RCoT | 2,000 | 2 | RCIT | 0.48 | 1.00 | 0.65 | 0.0610 | 70% | AMD EPYC 7713 |
| RCoT | 2,000 | 2 | causal-learn | 0.41 | 1.00 | 0.52 | 0.0239 | 62% | AMD EPYC 7713 |
| RCoT | 2,000 | 3 | Andrey | 0.71 | 1.00 | 0.73 | 0.0153 | - | AMD EPYC 7713 |
| RCoT | 2,000 | 3 | RCIT | 0.99 | 1.00 | 0.96 | 0.0630 | 75% | AMD EPYC 7713 |
| RCoT | 2,000 | 3 | causal-learn | 0.79 | 1.00 | 0.75 | 0.0238 | 71% | AMD EPYC 7713 |
| RCoT | 5,000 | 0 | Andrey | 0.02 | 1.00 | 0.07 | 0.00919 | - | AMD EPYC 7713 |
| RCoT | 5,000 | 0 | RCIT | 0.02 | 1.00 | 0.07 | 0.0120 | 97% | AMD EPYC 7713 |
| RCoT | 5,000 | 0 | causal-learn | 0.04 | 1.00 | 0.08 | 0.0138 | 97% | AMD EPYC 7713 |
| RCoT | 5,000 | 1 | Andrey | 0.15 | 1.00 | 0.26 | 0.0243 | - | AMD EPYC 7713 |
| RCoT | 5,000 | 1 | RCIT | 0.34 | 1.00 | 0.45 | 0.130 | 69% | AMD EPYC 7713 |
| RCoT | 5,000 | 1 | causal-learn | 0.18 | 1.00 | 0.26 | 0.0397 | 78% | AMD EPYC 7713 |
| RCoT | 5,000 | 2 | Andrey | 0.73 | 1.00 | 0.73 | 0.0259 | - | AMD EPYC 7713 |
| RCoT | 5,000 | 2 | RCIT | 0.98 | 1.00 | 0.94 | 0.133 | 75% | AMD EPYC 7713 |
| RCoT | 5,000 | 2 | causal-learn | 0.73 | 1.00 | 0.71 | 0.0397 | 65% | AMD EPYC 7713 |
| RCoT | 5,000 | 3 | Andrey | 0.94 | 1.00 | 0.90 | 0.0262 | - | AMD EPYC 7713 |
| RCoT | 5,000 | 3 | RCIT | 1.00 | 1.00 | 1.00 | 0.141 | 94% | AMD EPYC 7713 |
| RCoT | 5,000 | 3 | causal-learn | 0.93 | 1.00 | 0.89 | 0.0398 | 90% | AMD EPYC 7713 |

% end generated

## Categorical data

### BOSS and GRaSP with BDeu

Both packages run BOSS and GRaSP with the BDeu score. The table adds Andrey's categorical BIC
(`score="bic_categorical"`) on the same datasets. The random graphs have 20 datasets per size, for
both accuracy and speed; at every size, the 95% interval of Andrey's errors minus causal-learn's
includes 0.

% generated: BOSS, BDeu vs causal-learn

| Data | d | n | CPU | Andrey (s) | causal-learn (s) | Speedup | Datasets | Same graph | Errors, Andrey | Errors, causal-learn | Errors, categorical BIC | Errors, empty graph | Andrey ahead | causal-learn ahead | Andrey minus causal-learn |
|---|---:|---:|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
| Random graph | 10 | 2,000 | AMD EPYC 7713 | 0.0217 | 3.43 | 157x | 20 of 20 | 12 of 20 | 2 | 1 | 2.5 | 10 | 2 | 6 | +1.00 [-0.60, +2.50] |
| Random graph | 20 | 2,000 | AMD EPYC 7713 | 0.0920 | 17.9 | 182x | 20 of 20 | 9 of 20 | 6.5 | 6 | 4.5 | 20 | 5 | 6 | +1.30 [-0.40, +3.25] |
| Random graph | 40 | 2,000 | AMD EPYC 7713 | 0.371 | 81.1 | 214x | 20 of 20 | 7 of 20 | 7 | 9 | 7.5 | 40 | 8 | 5 | -0.95 [-2.70, +0.80] |
| diabetes | 413 | 5,000 | Intel Core i7-7820X | 842 | timeout | - | 0 of 1 | - | 385 | - | 671 | 602 | 0 | 0 | - |

Unfinished fits (coverage, never scored): diabetes data, d = 413, n = 5,000: causal-learn 1 of 1
fits past the 7,200 s cap; with each unfinished fit at the cap, at least 8.55x.

% end generated

% generated: GRaSP, BDeu vs causal-learn

| Data | d | n | CPU | Andrey (s) | causal-learn (s) | Speedup | Datasets | Same graph | Errors, Andrey | Errors, causal-learn | Errors, categorical BIC | Errors, empty graph | Andrey ahead | causal-learn ahead | Andrey minus causal-learn |
|---|---:|---:|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
| Random graph | 10 | 2,000 | AMD EPYC 7713 | 0.0212 | 2.76 | 137x | 20 of 20 | 8 of 20 | 2.5 | 1 | 1 | 10 | 4 | 8 | +0.55 [-0.60, +1.70] |
| Random graph | 20 | 2,000 | AMD EPYC 7713 | 0.0668 | 11.3 | 154x | 20 of 20 | 7 of 20 | 7 | 7.5 | 5 | 20 | 5 | 7 | -0.85 [-2.80, +0.90] |
| Random graph | 40 | 2,000 | AMD EPYC 7713 | 0.376 | 77.0 | 199x | 20 of 20 | 3 of 20 | 10 | 9.5 | 8 | 40 | 8 | 7 | -0.35 [-2.35, +1.70] |
| diabetes | 413 | 5,000 | Intel Core i7-7820X | 400 | timeout | - | 0 of 1 | - | 476 | - | 657 | 602 | 0 | 0 | - |

Unfinished fits (coverage, never scored): diabetes data, d = 413, n = 5,000: causal-learn 1 of 1
fits past the 7,200 s cap; with each unfinished fit at the cap, at least 18.0x.

% end generated

On diabetes, causal-learn finished neither search within the 2-hour cap. Andrey's errors with BDeu
are below the empty graph's (385 and 476 against 602); with the categorical BIC they pass it (671
and 657).

### The chi-square and G-square tests

The two tests now answer every query of a skeleton level at once. Both builds return the same graph
on every dataset, and every p-value of PC's tests on alarm is equal bit for bit.

% generated: earlier

| Method | Data | d | n | Andrey 0.8.0 (s) | Andrey 0.7.0 (s) | Speedup | Same graph |
|---|---|---:|---:|---:|---:|---:|---:|
| PC, chi-square | Random graph | 10 | 2,000 | 0.00593 | 0.00959 | - | 5 of 5 |
| PC, chi-square | Random graph | 20 | 2,000 | 0.0127 | 0.0334 | - | 5 of 5 |
| PC, chi-square | Random graph | 40 | 2,000 | 0.0307 | 0.118 | 3.73x | 5 of 5 |
| PC, chi-square | alarm | 37 | 1,000 | 0.0619 | 0.266 | 4.30x | 1 of 1 |
| PC, chi-square | alarm | 37 | 5,000 | 0.167 | 0.832 | 4.98x | 1 of 1 |
| PC, chi-square | hepar2 | 70 | 1,000 | 0.0865 | 0.431 | 4.98x | 1 of 1 |
| PC, chi-square | hepar2 | 70 | 5,000 | 0.529 | 2.86 | 5.40x | 1 of 1 |
| PC, chi-square | andes | 223 | 1,000 | 0.824 | 5.30 | 6.42x | 1 of 1 |
| PC, chi-square | andes | 223 | 5,000 | 2.77 | 17.1 | 6.15x | 1 of 1 |
| PC, G-square | Random graph | 10 | 2,000 | 0.00584 | 0.00969 | - | 5 of 5 |
| PC, G-square | Random graph | 20 | 2,000 | 0.0133 | 0.0337 | - | 5 of 5 |
| PC, G-square | Random graph | 40 | 2,000 | 0.0312 | 0.118 | 3.70x | 5 of 5 |
| PC, G-square | alarm | 37 | 1,000 | 0.0635 | 0.273 | 4.31x | 1 of 1 |
| PC, G-square | alarm | 37 | 5,000 | 0.152 | 0.758 | 4.99x | 1 of 1 |
| PC, G-square | hepar2 | 70 | 1,000 | 0.0929 | 0.458 | 4.93x | 1 of 1 |
| PC, G-square | hepar2 | 70 | 5,000 | 0.672 | 3.45 | 5.14x | 1 of 1 |
| PC, G-square | andes | 223 | 1,000 | 0.896 | 5.61 | 6.26x | 1 of 1 |
| PC, G-square | andes | 223 | 5,000 | 2.78 | 17.1 | 6.18x | 1 of 1 |
| FCI, chi-square | alarm | 37 | 1,000 | 0.0694 | 0.273 | 3.93x | 1 of 1 |
| FCI, chi-square | alarm | 37 | 5,000 | 0.189 | 0.855 | 4.52x | 1 of 1 |
| FCI, chi-square | hepar2 | 70 | 1,000 | 0.117 | 0.461 | 3.93x | 1 of 1 |
| FCI, chi-square | hepar2 | 70 | 5,000 | 0.631 | 2.96 | 4.70x | 1 of 1 |
| FCI, chi-square | andes | 223 | 1,000 | 1.22 | 5.67 | 4.66x | 1 of 1 |
| FCI, chi-square | andes | 223 | 5,000 | 14.0 | 28.0 | 2.00x | 1 of 1 |
| FCI, G-square | alarm | 37 | 1,000 | 0.0708 | 0.281 | 3.97x | 1 of 1 |
| FCI, G-square | alarm | 37 | 5,000 | 0.175 | 0.779 | 4.46x | 1 of 1 |
| FCI, G-square | hepar2 | 70 | 1,000 | 0.119 | 0.482 | 4.05x | 1 of 1 |
| FCI, G-square | hepar2 | 70 | 5,000 | 0.892 | 3.67 | 4.12x | 1 of 1 |
| FCI, G-square | andes | 223 | 1,000 | 1.32 | 6.02 | 4.56x | 1 of 1 |
| FCI, G-square | andes | 223 | 5,000 | 14.2 | 28.3 | 1.99x | 1 of 1 |

% end generated

## FCI's feasible sizes

How far FCI and GFCI scale on latent-variable data: 8 regimes (mean degree 2 or 4, 10% or 20% of the
nodes hidden, `n = 10 d` or `50 d` rows) up to 800 variables, 5 datasets each. Each cell gives the
fits finished within the cap and, over those, the median errors as a share of the empty graph's
(above 1.00, more errors than the empty graph). The caps are 1,800 s up to 200 variables, 3,600 s at
400, and 7,200 s at 800. Andrey runs `possible_dsep="fast"` on the 0.7.0 build.

FCI:

% generated: boundary fci

| Regime | d = 20 | d = 50 | d = 100 | d = 200 | d = 400 | d = 800 |
|---|---:|---:|---:|---:|---:|---:|
| degree 2, 10% hidden, n = 10 d | 5/5, 0.48 | 5/5, 0.46 | 5/5, 0.54 | 5/5, 0.80 | 5/5, 0.90 | 5/5, 1.49 |
| degree 2, 10% hidden, n = 50 d | 5/5, 0.40 | 5/5, 0.36 | 5/5, 0.52 | 5/5, 0.72 | 5/5, 0.84 | 5/5, 1.54 |
| degree 2, 20% hidden, n = 10 d | 5/5, 0.52 | 5/5, 0.62 | 5/5, 0.68 | 5/5, 0.85 | 5/5, 0.92 | 5/5, 1.31 |
| degree 2, 20% hidden, n = 50 d | 5/5, 0.41 | 5/5, 0.52 | 5/5, 0.59 | 5/5, 0.81 | 5/5, 0.84 | 3/5, 1.37 |
| degree 4, 10% hidden, n = 10 d | 5/5, 0.87 | 5/5, 0.86 | 5/5, 0.82 | 5/5, 0.76 | 5/5, 0.67 | 5/5, 0.68 |
| degree 4, 10% hidden, n = 50 d | 5/5, 0.87 | 5/5, 0.83 | 5/5, 0.78 | 5/5, 0.73 | 5/5, 0.64 | 5/5, 0.63 |
| degree 4, 20% hidden, n = 10 d | 5/5, 0.88 | 5/5, 0.88 | 5/5, 0.87 | 5/5, 0.88 | 5/5, 0.78 | 5/5, 0.79 |
| degree 4, 20% hidden, n = 50 d | 5/5, 0.84 | 5/5, 0.85 | 5/5, 0.80 | 5/5, 0.85 | 5/5, 0.75 | 5/5, 0.77 |

% end generated

GFCI:

% generated: boundary gfci

| Regime | d = 20 | d = 50 | d = 100 | d = 200 | d = 400 | d = 800 |
|---|---:|---:|---:|---:|---:|---:|
| degree 2, 10% hidden, n = 10 d | 5/5, 0.44 | 5/5, 0.30 | 5/5, 0.39 | 5/5, 0.44 | 5/5, 0.35 | 5/5, 0.41 |
| degree 2, 10% hidden, n = 50 d | 5/5, 0.33 | 5/5, 0.20 | 5/5, 0.30 | 5/5, 0.28 | 5/5, 0.17 | 5/5, 0.26 |
| degree 2, 20% hidden, n = 10 d | 5/5, 0.50 | 5/5, 0.55 | 5/5, 0.56 | 5/5, 0.62 | 5/5, 0.53 | 5/5, 0.56 |
| degree 2, 20% hidden, n = 50 d | 5/5, 0.43 | 5/5, 0.46 | 5/5, 0.45 | 5/5, 0.51 | 5/5, 0.36 | 1/5, 0.33 |
| degree 4, 10% hidden, n = 10 d | 5/5, 0.83 | 5/5, 0.79 | 5/5, 0.71 | 5/5, 0.60 | 5/5, 0.47 | 5/5, 0.44 |
| degree 4, 10% hidden, n = 50 d | 5/5, 0.68 | 5/5, 0.67 | 5/5, 0.66 | 5/5, 0.55 | 5/5, 0.40 | 4/5, 0.37 |
| degree 4, 20% hidden, n = 10 d | 5/5, 0.86 | 5/5, 0.87 | 5/5, 0.76 | 5/5, 0.81 | 5/5, 0.70 | 3/5, 0.72 |
| degree 4, 20% hidden, n = 50 d | 5/5, 0.84 | 5/5, 0.80 | 5/5, 0.76 | 5/5, 0.79 | 2/5, 0.50 | 0/5 |

% end generated

Tetrad's GFCI, for context, with a cap of at most 1,800 s; a run stops Tetrad after its first fit
past the cap, so later datasets are not run:

% generated: boundary tetrad_gfci

| Regime | d = 20 | d = 50 | d = 100 | d = 200 | d = 400 | d = 800 |
|---|---:|---:|---:|---:|---:|---:|
| degree 2, 10% hidden, n = 10 d | 5/5, 0.46 | 5/5, 0.51 | 5/5, 0.52 | 5/5, 0.73 | 4/5, 0.62 | 0/1 |
| degree 2, 10% hidden, n = 50 d | 5/5, 0.35 | 5/5, 0.27 | 5/5, 0.33 | 1/2, 0.46 | 1/2, 0.28 | 0/1 |
| degree 2, 20% hidden, n = 10 d | 5/5, 0.43 | 5/5, 0.62 | 5/5, 0.63 | 1/2, 0.54 | 0/1 | 0/1 |
| degree 2, 20% hidden, n = 50 d | 5/5, 0.46 | 5/5, 0.46 | 0/1 | not run | 0/1 | 0/1 |
| degree 4, 10% hidden, n = 10 d | 5/5, 0.84 | 5/5, 0.85 | 1/2, 0.77 | not run | 0/1 | 0/1 |
| degree 4, 10% hidden, n = 50 d | 5/5, 0.74 | 0/1 | not run | not run | 0/1 | 0/1 |
| degree 4, 20% hidden, n = 10 d | 5/5, 0.82 | 5/5, 0.86 | 0/1 | not run | 0/1 | 0/1 |
| degree 4, 20% hidden, n = 50 d | 5/5, 0.78 | 0/1 | not run | not run | 0/1 | 0/1 |

% end generated

## Other changes in 0.8.0

The benchmark above covers the release's theme; 0.8.0 also changes the following. The
[changelog](../changelog.md) lists every change.

- **KCI, RCoT, CMIknn, and GPDC are supported.** Against each reference package, each test has no
  more errors than the reference on average over every paired dataset, and no size has its 95%
  interval of Andrey's errors minus the reference's entirely above 0. RCoT's packages draw random
  features, so RCoT is judged on 4 draws per dataset, under a rule set before those fits ran.
- **Background knowledge for every search.** PC, CD-NOD, FCI, GFCI, GES, GIES, XGES, hill climbing,
  tabu search, BOSS, GRaSP, exact search, and DirectLiNGAM take `knowledge=`: the edges the graph
  must have and must not have, the only pairs that may be adjacent, and tiers in time order. PC,
  FCI, GFCI, and CD-NOD also take `max_depth`.
- **Tie breaks and column order.** `andrey.config(tie_break=...)` chooses how the searches break
  ties: by a rank computed from each column's own data, the default, or by column position. The
  [column order guide](../guides/column-order.md) lists what each method keeps under a new column
  order and on another machine.
- **Score defaults that follow the data (breaking).** BOSS, GRaSP, GES, hill climbing, tabu search,
  and exact search pick BDeu for categorical data and the conditional Gaussian BIC for mixed data,
  and warn once; pass the linear-Gaussian BIC to keep the old default.
- **Faster categorical tests.** The chi-square and G-square tests answer many queries at once: on
  alarm, hepar2, and andes, PC returns the same graphs 4.30x to 6.42x faster than 0.7.0, and FCI
  1.99x to 4.70x (see [the chi-square and G-square tests](#the-chi-square-and-g-square-tests)).
- **BDeu and the categorical BIC for BOSS and GRaSP.** `score_func="local_score_BDeu"` and
  `"local_score_BIC_categorical"`, the scores the other score-based searches take for categorical
  data.
- **BOSS and GRaSP decide every comparison as full scores would**, in less time.
- **Clearer errors.** Copied and constant columns are named before any test runs, the LiNGAM methods
  say how many rows their pruning step needs, and a PDAG without a consistent extension names a
  chordless cycle.
- **Order-free ties across machines.** DirectLiNGAM, ICA-LiNGAM, exact search, and the score-based
  searches break ties by a rank computed from each column's own data, so they return the same graph
  on every CPU and under any column order. In Temporal causal discovery, VARMA-LiNGAM and
  LongitudinalLiNGAM fit on rank-ordered columns.
- **FCI's measured limits.** FCI's and GFCI's documentation gives the sizes they finish within a cap
  (see [FCI's feasible sizes](#fcis-feasible-sizes)).
- **Column names.** Labels are text and unique in every method, the CLI, and graph input and output.
- **pcalg's graph conventions.** The [graph representation guide](../guides/graph-representation.md)
  gives pcalg's `amat.cpdag` and `amat.pag` codes and converts both to Andrey's graphs.

## Limits

- **Pending measurements.** See [Pending measurements](#pending-measurements).
- **KCI.** Measured up to 20 variables at 2,000 rows and 5 variables at 5,000 rows. causal-learn
  passed the 3,600 s cap on 8 of 20 additive and 5 of 20 nonmonotone datasets at 5 variables and
  5,000 rows; at 20 variables and 2,000 rows, on nonmonotone data, Andrey passed the 1,800 s cap on
  1 of 20 datasets and causal-learn on 6.
- **RCoT.** Every fit finished at every size up to 50 variables and 5,000 rows. With one fit per
  dataset, causal-learn has fewer errors than Andrey on 4 of 20 datasets, and Andrey on none, with
  PC on additive data at 10 variables and 5,000 rows and with FCI on additive data at 5 variables
  and 500 rows; over 4 draws per dataset, Andrey minus causal-learn is -0.16 [-0.49, +0.15] and
  -0.04 [-0.28, +0.21] at those sizes.
- **CMIknn.** Measured up to 20 variables at 500 rows, 10 at 1,000, and 5 at 2,000. A fit takes
  minutes at 10 variables and 500 rows (192 to 295 s on an AMD EPYC 7713). Andrey's CMIknn runs
  much faster on CPUs with AVX-512, so the speedup over tigramite is 2.40x to 2.56x on an AMD EPYC
  7713, 9.19x to 10.3x on the AVX-512 servers, and 27.0x to 37.7x on the second machine.
- **GPDC.** Measured up to 20 variables at 1,000 rows and 5 at 2,000. The speedup over tigramite is
  1.38x to 1.89x on the servers and 1.20x to 1.46x on the second machine. At 20 variables and 1,000
  rows, tigramite passed the 1,800 s cap on 10 of 20 additive and 2 of 20 nonmonotone datasets, and
  Andrey on none.
- **FCI.** Measured with KCI and RCoT at 5 and 10 variables, with 500 and 1,000 rows.
- **GPU.** The GPU's speedup depends on the CPU it replaces: CMIknn runs 9.05x to 17.2x faster
  than on an AMD EPYC 7713, 1.82x to 2.97x faster than on the AVX-512 servers, and slower than on
  the second machine (0.644x to 0.946x of its speed).
- **BOSS and GRaSP on small graphs.** On the random graphs, causal-learn has fewer errors on
  average with BOSS at 10 and 20 variables (Andrey minus causal-learn +1.00 and +1.30) and with
  GRaSP at 10 (+0.55); Andrey has fewer at 40 with BOSS (-0.95) and at 20 and 40 with GRaSP (-0.85
  and -0.35). Every 95% interval includes 0.
- **Categorical-test hardware.** The rows against Andrey 0.7.0 ran with every core of the node
  visible to the fit, unlike the other lanes' 4-core budget.
- **Hardware.** Ratios are specific to the measured CPU, core budget, and settings; the same size
  can give different ratios on two CPUs.

## Pending measurements

% generated: pending

These measurements were still running when the summary was built; the summary lists each under
`incomplete`, and its `pending_fits` counts the fits.

- PC with the KCI test, causal-learn, speed on the second machine (Intel Core i7-7820X): additive,
  nonmonotone data, d = 10, n = 1,000; d = 20, n = 1,000
- PC with the KCI test on the NVIDIA A100-SXM4-80GB, against Andrey's CPU fits: additive,
  nonmonotone data, d = 20, n = 1,000 (no CPU timing of this size yet)
- PC with the KCI test on the NVIDIA A100-SXM4-80GB, against Andrey's CPU fits: additive data, d =
  20, n = 2,000 (the CPU fit passed its cap on a timed dataset, so the GPU graph could not be
  checked on it)

% end generated

## Reproduce

You can find environment setup, campaign drivers, and scores in the [benchmark harness
guide][harness]. To rerun the comparisons, install Andrey 0.8.0 (andrey-core), causal-learn
0.1.4.8, tigramite 5.2.10.1 in its own environment, and pcalg 2.7.12 with RCIT 0.1.0 in the
harness's R environment. Every driver shuffles the columns by default. Times will differ with the
hardware and build; graphs should match closely.

| Comparison | Driver | Sizes | Datasets x repeats | Wall cap (s) |
|---|---|---|---|---:|
| PC and FCI with the nonlinear tests | `nonlinear_ci_cpu.py` | d 5 to 50, n 500 to 5,000 | 20 x 1 (accuracy); 5 x 1, run 3 times (speed) | 1,800 or 3,600 |
| KCI and CMIknn on a GPU | `nonlinear_ci_cpu.py` with `ANDREY_BACKEND=cuda` | d 5 to 20, n 500 to 5,000 | 5 x 1, run 3 times | 1,800 or 3,600 |
| The tests query by query | `nonlinear_ci_queries.py` | n 200 to 5,000, k 0 to 3 | 50 or 200 independent, 20 dependent draws | 1,800 or 3,600 |
| BOSS and GRaSP on categorical data | `discrete_cpu.py` | d 10, 20, 40; diabetes | 20 x 1; 1 x 1 | 3,600; 7,200 |
| PC with the categorical tests | `discrete_cpu.py` | d 10, 20, 40 | 5 x 1 | 1,800 |

[harness]: https://github.com/Abel-ai-lab/andrey/blob/main/benchmarks/README.md