---
name: benchmarks-0.13.0
description: The Andrey 0.13.0 benchmarks - IAMB and MMPC against bnlearn, CAM and RESIT against their authors' code, and the searches on five discrete networks.
meta:
  type: guide
  tags: [benchmarks, performance, accuracy, constraint-based, functional]
---

# 0.13.0 release card: functional causal models

:::::{div} release-opening

:::{card}
:class-card: release-cover release-art release-art-0-13-0
:shadow: none

Release card

Functional causal models

0.13.0
:::

::::{div} release-opening-text

This card measures the four methods 0.13.0 makes supported: CAM and RESIT against their authors'
code, and IAMB and MMPC against bnlearn, on linear-Gaussian data and on five discrete networks.
Each result holds for the measured conditions; it depends on the data, settings, and hardware.

::::
:::::

IAMB, MMPC, CAM, and RESIT are supported methods. We compare IAMB and MMPC with bnlearn 5.2.1 on the
graph each returns, scored against that graph's own truth: the CPDAG for `structure="cpdag"`, the
skeleton for `structure="skeleton"`, the default of MMPC, and the moral graph for
`structure="moral"`, which bnlearn's `learn.mb` blankets joined by the AND rule give. Three grids of
linear-Gaussian data measure them: standardized columns, columns left in their own units, and 200
variables. The same two methods, Inter-IAMB, MMHC, and RSMAX2 are then compared on five discrete
networks (asia, child, insurance, alarm, and hepar2). There each package runs its own tests:
Andrey's calibrated chi-square and G-square tests (`indep_test="chisq"` and `"gsq"`) against
bnlearn's `x2-adf` and `mi-adf`. The hybrids score with BDeu, equivalent sample size 1, in both
packages. CAM is compared with CAM 1.0, the R package of its authors, on nonlinear additive-noise
data, the model it assumes, and then on two kinds of data outside that model. RESIT is compared with
the authors' R code and with lingam's implementation on the same kinds of data, at 5 to 20
variables.

## Summary

Linear-Gaussian data:

| Grid | Method | Graph | Settings | Better | Tie | Worse | Not judged | Worst setting: D, mean [95%] | Median time ratio |
|---|---|---|---:|---:|---:|---:|---:|---|---:|
| Standardized data, 20 to 100 variables | IAMB | CPDAG | 12 | 0 | 10 | 1 | 1 | +0.63 [-0.42, +1.74] (d 50, n 5,000, degree 4) | 4.6 |
| Standardized data, 20 to 100 variables | IAMB | skeleton | 12 | 1 | 11 | 0 | 0 | +0.10 [-0.25, +0.45] (d 50, n 1,000, degree 4) | 3.9 |
| Standardized data, 20 to 100 variables | IAMB | moral graph | 12 | 1 | 11 | 0 | 0 | +2.25 [-0.55, +5.50] (d 100, n 5,000, degree 4) | 1.6 |
| Standardized data, 20 to 100 variables | MMPC | skeleton | 12 | 0 | 12 | 0 | 0 | +0.60 [-0.40, +1.65] (d 100, n 1,000, degree 4) | 7.6 |
| Standardized data, 20 to 100 variables | MMPC | CPDAG | 12 | 0 | 9 | 1 | 2 | +0.70 [+0.10, +1.55] (d 100, n 1,000, degree 2) | 18.8 |
| Unstandardized data, 20 to 100 variables | IAMB | CPDAG | 12 | 1 | 11 | 0 | 0 | +0.90 [+0.40, +1.50] (d 20, n 5,000, degree 4) | 2.7 |
| Unstandardized data, 20 to 100 variables | IAMB | skeleton | 12 | 1 | 11 | 0 | 0 | +0.00 [+0.00, +0.00] (d 20, n 1,000, degree 2) | 2.7 |
| Unstandardized data, 20 to 100 variables | IAMB | moral graph | 12 | 2 | 10 | 0 | 0 | +0.00 [+0.00, +0.00] (d 20, n 1,000, degree 2) | 1.7 |
| Unstandardized data, 20 to 100 variables | MMPC | skeleton | 12 | 1 | 11 | 0 | 0 | +0.30 [-0.70, +1.20] (d 20, n 5,000, degree 4) | 5.6 |
| Unstandardized data, 20 to 100 variables | MMPC | CPDAG | 12 | 0 | 6 | 0 | 6 | +1.50 [-0.30, +3.50] (d 20, n 5,000, degree 4) | 20.7 |
| 200 variables | IAMB | CPDAG | 3 | 0 | 2 | 0 | 1 | +0.00 [+0.00, +0.00] (d 200, n 1,000, degree 2) | 2.2 |
| 200 variables | IAMB | skeleton | 3 | 0 | 3 | 0 | 0 | +0.70 [-0.40, +2.30] (d 200, n 1,000, degree 4) | 2.0 |
| 200 variables | IAMB | moral graph | 4 | 1 | 3 | 0 | 0 | +0.40 [-2.00, +3.10] (d 200, n 1,000, degree 4) | 2.9 |
| 200 variables | MMPC | skeleton | 4 | 0 | 3 | 1 | 0 | +1.60 [+0.10, +3.10] (d 200, n 5,000, degree 2) | 16.1 |
| 200 variables | MMPC | CPDAG | 1 | 0 | 0 | 0 | 1 | - | 29.1 |

Discrete networks, 10 settings per row (5 networks, 1,000 and 5,000 rows):

| Method | Graph | Test | Settings | Better | Tie | Worse | Not judged | Worst setting: D, mean [95%] | Median time ratio | hepar2 time ratio, 1,000 / 5,000 rows |
|---|---|---|---:|---:|---:|---:|---:|---|---:|---|
| IAMB | CPDAG | chi-square | 10 | 6 | 4 | 0 | 0 | +0.05 [-0.25, +0.35] (asia, n 5,000) | 0.4 | 0.28 / 0.35 |
| IAMB | CPDAG | G-square | 10 | 7 | 3 | 0 | 0 | +1.00 [-1.35, +3.65] (alarm, n 5,000) | 0.1 | 0.10 / 0.24 |
| IAMB | skeleton | chi-square | 10 | 7 | 3 | 0 | 0 | +0.05 [-0.25, +0.35] (asia, n 5,000) | 0.4 | 0.27 / 0.38 |
| IAMB | skeleton | G-square | 10 | 8 | 2 | 0 | 0 | +0.20 [-0.10, +0.50] (asia, n 5,000) | 0.1 | 0.10 / 0.24 |
| IAMB | moral graph | chi-square | 10 | 4 | 4 | 2 | 0 | +5.81 [+1.81, +9.62] (hepar2, n 5,000) | 0.4 | 0.41 / 0.41 |
| IAMB | moral graph | G-square | 10 | 8 | 2 | 0 | 0 | +0.20 [-0.15, +0.55] (asia, n 5,000) | 0.2 | 0.15 / 0.28 |
| Inter-IAMB | CPDAG | chi-square | 10 | 7 | 3 | 0 | 0 | +0.05 [-0.15, +0.30] (asia, n 5,000) | 0.4 | 0.29 / 0.38 |
| Inter-IAMB | CPDAG | G-square | 10 | 6 | 4 | 0 | 0 | +2.50 [+1.10, +3.95] (alarm, n 5,000) | 0.1 | 0.09 / 0.22 |
| Inter-IAMB | skeleton | chi-square | 10 | 7 | 3 | 0 | 0 | +0.05 [-0.15, +0.30] (asia, n 5,000) | 0.4 | 0.25 / 0.38 |
| Inter-IAMB | skeleton | G-square | 10 | 8 | 2 | 0 | 0 | +0.15 [-0.10, +0.40] (asia, n 5,000) | 0.1 | 0.09 / 0.23 |
| Inter-IAMB | moral graph | chi-square | 10 | 6 | 3 | 1 | 0 | +4.67 [+1.47, +8.33] (hepar2, n 1,000) | 0.4 | 0.38 / 0.46 |
| Inter-IAMB | moral graph | G-square | 10 | 7 | 3 | 0 | 0 | +0.10 [-0.20, +0.40] (asia, n 5,000) | 0.1 | 0.14 / 0.27 |
| MMPC | skeleton | chi-square | 10 | 8 | 2 | 0 | 0 | +0.00 [+0.00, +0.00] (asia, n 1,000) | 0.7 | 0.83 / 0.74 |
| MMPC | skeleton | G-square | 10 | 8 | 2 | 0 | 0 | +0.00 [+0.00, +0.00] (asia, n 5,000) | 0.2 | 0.29 / 0.29 |
| MMPC | CPDAG | chi-square | 10 | 7 | 3 | 0 | 0 | +0.00 [+0.00, +0.00] (asia, n 1,000) | 0.5 | 0.80 / 0.16 |
| MMPC | CPDAG | G-square | 10 | 7 | 3 | 0 | 0 | +0.00 [+0.00, +0.00] (asia, n 5,000) | 0.1 | 0.24 / 0.06 |
| MMHC | CPDAG | chi-square | 10 | 6 | 4 | 0 | 0 | +0.25 [-0.65, +1.10] (asia, n 1,000) | 0.7 | 0.83 / 0.77 |
| MMHC | CPDAG | G-square | 10 | 5 | 5 | 0 | 0 | +0.40 [-1.40, +2.20] (asia, n 5,000) | 0.2 | 0.31 / 0.28 |
| RSMAX2 | CPDAG | chi-square | 10 | 5 | 4 | 0 | 1 | +0.20 [-1.00, +1.40] (asia, n 1,000) | 0.7 | 0.86 / 0.74 |
| RSMAX2 | CPDAG | G-square | 10 | 5 | 5 | 0 | 0 | +0.40 [-1.40, +2.20] (asia, n 5,000) | 0.2 | 0.31 / 0.26 |
| RSMAX2, IAMB restrict | CPDAG | chi-square | 10 | 5 | 4 | 0 | 1 | +0.30 [-0.60, +1.30] (asia, n 1,000) | 0.4 | 0.27 / 0.43 |
| RSMAX2, IAMB restrict | CPDAG | G-square | 10 | 4 | 6 | 0 | 0 | +0.30 [-0.40, +1.00] (asia, n 1,000) | 0.2 | 0.12 / 0.26 |

A setting is one size, sample size, and mean degree, or one network and sample size. D is Andrey's
error minus bnlearn's on the same dataset. A setting with at least 10 paired datasets is worse when
the paired 95% bootstrap interval of the mean D lies wholly above 0, better when wholly below, and a
tie otherwise; a setting with fewer is listed, not judged. A worse setting counts as a tie only if a
rerun on fresh seeds with at least 10 paired datasets is not worse; one whose rerun has fewer pairs
counts as worse. The time ratio is bnlearn's fit time over Andrey's, the median of the settings'
median ratios, so a ratio below 1 means Andrey is slower. The worst setting is the one with the
largest D from the first run.

Nonlinear additive-noise data, CAM against CAM 1.0 (the DAG each returns, scored against the true
DAG):

| Grid | Method | Graph | Settings | Better | Tie | Worse | Not judged | Worst setting: D, mean [95%] | Median time ratio |
|---|---|---|---:|---:|---:|---:|---:|---|---:|
| Additive noise, Gaussian and Gumbel, 10 to 100 variables | CAM | DAG | 16 | 2 | 14 | 0 | 0 | +0.15 [+0.00, +0.40] (d 50, Gaussian noise, standardized) | 9.7 |

A CAM setting is one size, noise, and column scale (standardized or in their own units) at 1,000
rows and 20 datasets, with the same verdict rule. The two better settings are Gumbel noise at 100
variables, on both scales. Only the settings of the model CAM assumes carry a verdict. The 16
settings of location-scale noise and of linear-Gaussian data, which the model does not cover, are
reported in the tables and not judged: none has CAM 1.0 making fewer errors, and one, linear-Gaussian
data in their own units at 100 variables, has Andrey making fewer. CAM returns the same graph as CAM
1.0 on 278 of the 320 in-model datasets. CAM 1.0 caps a variable's parents at the smaller of the
variables minus one and the rows over 20, 50 at 1,000 rows, which binds at 100 variables; Andrey
sets no cap by default.

Nonlinear additive-noise data, RESIT against the authors' code and against lingam (the DAG each
returns, scored against the true DAG):

| Grid | Method | Graph | Settings | Better | Tie | Worse | Not judged | Worst setting: D, mean [95%] | Median time ratio |
|---|---|---|---:|---:|---:|---:|---:|---|---:|
| Additive noise, Gaussian and Gumbel, 5 to 20 variables | RESIT, authors' code | DAG | 12 | 0 | 12 | 0 | 0 | +1.50 [-3.30, +6.30] (d 20, Gumbel noise, own units) | 3.7 |
| Additive noise, Gaussian and Gumbel, 5 to 20 variables | RESIT, lingam | DAG | 12 | 11 | 1 | 0 | 0 | -2.65 [-4.15, -1.20] (d 5, Gaussian noise, standardized) | 2.4 |

A RESIT setting is one size, noise, and column scale (standardized or in their own units) at 1,000
rows and 20 datasets (10 at 20 variables in their own units), with the same verdict rule. Against
the authors' code no setting is better and none is worse: 12 tie, one of them after a rerun, because
Gaussian noise at 10 variables on standardized columns was worse on its first 20 datasets and is not
on the next 20 (described below). Orientation is better in 2 settings against the authors' code and
in all 12 against lingam. Against lingam 11 settings are better and one ties. Only the settings of
the model RESIT assumes carry a verdict. The 24 settings of location-scale noise and of
linear-Gaussian data, which the model does not cover, are reported in the tables and not judged. In
them the authors' code makes fewer errors in 3 (location-scale noise: +0.65 [+0.20, +1.20] at 10
variables standardized, +0.35 [+0.05, +0.75] at 5 variables and +15.80 [+5.00, +27.70] at 20
variables in their own units, the last on 10 datasets), lingam in 2 (location-scale noise in their
own units: +3.65 [+1.45, +5.85] at 10 variables and +21.00 [+4.90, +35.80] at 20 variables on 10
datasets), and RESIT makes fewer than lingam in 1 (location-scale noise, standardized, 20 variables:
-10.90 [-16.50, -5.60]). RESIT returns the authors' graph on 87 of the 220 in-model datasets and
lingam's on 19. Andrey's RESIT returns the same graph on standardized columns and on columns in
their own units on all 200 datasets of the four data types, the authors' code on 55, and lingam on
43.

## Where bnlearn makes fewer errors

10 of the 355 settings are worse in their first run:

| Method | Graph | Grid | Setting | Andrey / bnlearn (mean errors) | D, first run [95%] | Rerun on fresh seeds: D [95%] | Counts as |
|---|---|---|---|---|---|---|---|
| IAMB | CPDAG | standardized | d 50, n 5,000, degree 2 | 9.15 / 8.65 | +0.50 [+0.05, +1.05] | not rerun; cause traced below | worse |
| MMPC | CPDAG | standardized | d 100, n 1,000, degree 2 | 37.05 / 36.35 | +0.70 [+0.10, +1.55] | +0.56, 9 datasets, seeds 20-29: not judged | worse, rerun not judged |
| IAMB | CPDAG | unstandardized | d 20, n 1,000, degree 4 | 22.10 / 21.50 | +0.60 [+0.10, +1.10] | -0.10 [-0.30, +0.00] (10 datasets, seeds 20-29) | tie |
| IAMB | CPDAG | unstandardized | d 20, n 5,000, degree 4 | 17.10 / 16.20 | +0.90 [+0.40, +1.50] | -0.10 [-0.30, +0.00] (10 datasets, seeds 20-29) | tie |
| MMPC | skeleton | 200 variables | d 200, n 5,000, degree 2 | 75.50 / 73.90 | +1.60 [+0.10, +3.10] | +2.70 [+1.20, +4.40] (10 datasets, seeds 20-29) | worse |
| IAMB | moral graph | networks, chi-square | alarm, n 1,000 | 35.85 / 34.55 | +1.30 [+0.15, +2.55] | +2.10 [-0.30, +4.30] (10 datasets, seeds 20-29) | tie |
| IAMB | moral graph | networks, chi-square | hepar2, n 1,000 | 142.00 / 136.27 | +5.73 [+3.27, +8.40] | +4.11, 9 datasets, seeds 20-29: not judged | worse, rerun not judged |
| IAMB | moral graph | networks, chi-square | hepar2, n 5,000 | 121.94 / 116.12 | +5.81 [+1.81, +9.62] | +10.30 [+5.20, +15.20] (10 datasets, seeds 20-29) | worse |
| Inter-IAMB | CPDAG | networks, G-square | alarm, n 5,000 | 17.10 / 14.60 | +2.50 [+1.10, +3.95] | -2.00 [-4.50, +0.30] (10 datasets, seeds 20-29) | tie |
| Inter-IAMB | moral graph | networks, chi-square | hepar2, n 1,000 | 141.47 / 136.80 | +4.67 [+1.47, +8.33] | +2.00, 9 datasets, seeds 20-29: not judged | worse, rerun not judged |

**IAMB on linear-Gaussian data.** The two unstandardized settings at 20 variables and mean degree 4
do not replicate: the rerun's D is -0.10 on both, and 7 and 9 of its 10 graphs are identical. The
standardized setting with 50 variables, 5,000 rows, and mean degree 2 is worse (16 of its 20 graphs
are identical). Most of its gap, +7 pairs of the +10 over its 20 datasets, sits in 2 datasets whose
skeletons differ, and the skeleton difference starts at one node's first grow step, where the
p-values of several candidates underflow to exactly 0 in bnlearn, which takes the first column among
them. Andrey takes the candidate with the strongest test statistic, the association measure of
Tsamardinos, Aliferis, and Statnikov (2003), so its result does not depend on the column order. The
choice bnlearn makes, applied in Andrey's search, reproduces bnlearn's skeleton in all 4 traced
datasets. The other +3 are orientation differences in 2 datasets with equal skeletons: Meek's rules
on a cyclic partially directed graph, and a collider that bnlearn skips for closing a directed
cycle. The skeleton of the same cell is a tie, and so is its CPDAG on unstandardized data (-0.20
[-0.60, +0.00]).

**MMPC, skeleton at 200 variables.** The skeleton at 200 variables, 5,000 rows, and mean degree 2 is
worse (D +1.60 [+0.10, +3.10] on seeds 0-9 and +2.70 [+1.20, +4.40] on seeds 20-29). At this size
MMPC's forward step often faces candidates whose dependence is too strong to rank by p-value, and
Andrey admits the one with the strongest test statistic, the association measure of Tsamardinos,
Brown, and Aliferis (2006), which is a true neighbor in 2,376 of 2,384 first picks (99.7%); the
first column, bnlearn's pick, is one in 49%. The extra errors are false edges between unrelated
pairs whose statistic is just past the cutoff (z 2.0 to 3.1, the cutoff is 1.96); an arbitrary pick
happens to admit a strongly correlated non-neighbor that removes some of them. Strongest-first keeps
44 such edges that the other order removes, about 2 per graph.

**MMPC, CPDAG at 100 variables.** The setting with 100 variables, 1,000 rows, and mean degree 2 is
worse on seeds 0-19 (D +0.70 [+0.10, +1.55]). Its rerun on seeds 20-29 has 9 paired datasets, since
bnlearn did not finish one in 600 s, so it is not judged: the mean D is +0.56 (an interval of -0.22
to +1.78), 6 of the 9 graphs are identical, and the whole difference sits in 3 datasets (-1, +1,
+5). The setting counts as worse.

**IAMB on hepar2.** The moral graph of hepar2 at 5,000 rows is worse on both runs (D +5.81 [+1.81,
+9.63] on seeds 0-19 and +10.30 [+5.20, +15.20] on seeds 20-29). The two packages run different
discrete tests: Andrey's chi-square test is calibrated by the exact permutation mean of the
statistic as its degrees of freedom (Tsamardinos and Borboudakis 2010), and bnlearn's x2-adf adjusts
the degrees of freedom by a rule for sparse tables. Their graphs are not equal on any of the 3,183
datasets of child, insurance, alarm, and hepar2 (595 of 790 on asia), and the two tests agree on
97.7% of random triples.

**Reruns that cannot judge.** The moral graph of hepar2 at 1,000 rows is worse in its first run for
IAMB (+5.73) and Inter-IAMB (+4.67) with chi-square tests. Each rerun on seeds 20-29 has 9 paired
datasets, since the datasets on which both packages return an error (a constant column at 1,000
rows) are left out, so it is reported, not judged. Both reruns keep a positive mean D (+4.11 and
+2.00). We have not traced these two settings separately; they involve the same pair of chi-square
tests that differ at 5,000 rows. The two alarm settings that were worse in their first run (IAMB
moral graph with chi-square at 1,000 rows, Inter-IAMB CPDAG with G-square at 5,000 rows) tie on
their reruns.

## Where the authors' RESIT code makes fewer errors

One of the 12 settings of the model is worse in its first run: Gaussian additive noise at 10
variables on standardized columns, where Andrey makes 6.80 mean errors and the authors' code 5.60 (D
+1.20 [+0.05, +2.50], seeds 0-19). The rerun on 20 fresh datasets (seeds 20-39) has the interval
[-0.60, +2.55] around D +0.95 (8.00 and 7.05 mean errors), which includes 0, so the setting counts
as a tie and both runs are in the summary. The orientation ties in both runs. The 40 datasets
together, which was not planned, give +1.08 [+0.13, +2.10].

**RESIT, Gaussian noise, 10 variables.** We traced the gap on the 40 datasets. It comes from the
order in which RESIT tries to remove a variable's parents, which the published algorithm does not
fix: the authors' code takes the columns in their order in the file, and Andrey takes a rank
computed from the data that does not depend on the column order. Several predecessors can fit nearly
as well, so two orders keep different parents. With the authors' column order Andrey returns the
authors' graph on 38 of the 40 datasets and the gap falls to +0.20 [+0.00, +0.50]; 48 random parent
orders moved the mean errors from 0.8 below to 0.8 above the column order's. Whether Andrey's order
is systematically worse than a random one was then tested on 40 datasets not used before (seeds
40-79) with the rule written down in advance: its mean errors are +0.16 [-0.49, +0.85] from those of
16 random orders; 5 of the 16 random orders do as badly or worse (P = 0.35), so it is no worse than
a random order. On those datasets Andrey is +0.68 [-0.53, +1.88] from the authors' code, and a
random order +0.51 [-0.28, +1.26]. What is left of the gap comes from near ties in the choice of the
causal order, where Andrey's smoothing parameters and the authors' GCV minimum differ slightly, as
in CAM. The order is arbitrary in the algorithm, so we do not choose it by its score on these data.

## Method

**Data.** Linear-Gaussian data from Erdos-Renyi graphs with 20, 50, and 100 variables, 1,000 and
5,000 rows, and mean degree 2 and 4, as standardized columns and as columns left in their own units,
and with 200 variables, 1,000 and 5,000 rows, and mean degree 2 and 4, on the outputs whose fits at
100 variables all finished. Each setting has up to 20 datasets (10 at 200 variables), one per seed,
with the columns in a shuffled order. The CPU budget stopped some settings at 10 to 16 datasets. A
setting keeps at least 10, and the ones that bnlearn could not finish keep fewer pairs. The discrete
data are samples of 1,000 and 5,000 rows, up to 20 per setting, from the published networks asia (8
variables), child (20), insurance (27), alarm (37), and hepar2 (70), loaded with
`andrey.data.load_dataset`. A dataset with a constant column is left out when both packages return
an error on it.

**Fits.** One fit at a time on one core, a 600 s cap per fit, and one untimed warm-up fit except at
5,000 rows, mean degree 4, and 50 or more variables, where there is none. Linear-Gaussian data uses
Fisher-z tests at alpha 0.05 (bnlearn `test="zf"`); discrete data uses chi-square and G-square tests
at alpha 0.05 (bnlearn `x2-adf` and `mi-adf`). Andrey's chi-square and G-square tests take as their
degrees of freedom the exact permutation mean of the statistic (Tsamardinos and Borboudakis, 2010);
bnlearn's adjusted tests change the degrees of freedom by a rule for sparse tables. A fit that
reaches the cap is a timeout. It is counted as coverage, left out of the pairs, and never scored as
an empty graph. No discrete-network fit reached the cap. The discrete-network fit times were
measured again with the faster discrete tests of this release, on the same datasets and seeds with
the same protocol; the graphs and errors are those of the first campaign, and the 8,350 comparisons
of the two sets of graphs found no difference.

**Errors.** The structural Hamming distance of the returned graph to its own truth. The tables give
the median over the paired datasets, for Andrey, bnlearn, and the empty graph; D is the mean of the
per-dataset differences, the statistic of the verdict. A DAG that bnlearn returns is compared
through its CPDAG.

**CAM data.** Erdos-Renyi graphs of mean degree 2 with 10, 20, 50, and 100 variables and 1,000 rows,
20 datasets per setting, drawn by `andrey.data.sample_scm` in eight regimes: additive noise with
Gaussian or Gumbel noise, which CAM assumes; location-scale noise, whose scale depends on the
parents; and linear-Gaussian data, which identify only an equivalence class. Each is standardized
(the default) and left in its own units. The columns are in a shuffled order.

**CAM fits.** One fit at a time on one core, a 1,800 s cap per fit, and a short warm-up fit in each
process. CAM 1.0 runs with its preselection and pruning on (`variableSel` and `pruning`, tests at
0.001), which are Andrey's settings. Two differences are named, not tuned away: CAM 1.0's parent cap
above, and smoothing parameters that come from different GCV optimizers, so two edges of nearly
equal gain can be added in either order. Errors are the structural Hamming distance of the returned
DAG, where a reversed edge counts 1, and the share of true edges returned with the right direction.
No fit reached the cap. The reference fits are those of a campaign on the pre-release build; Andrey's
fits were run again on this release's code, on the same datasets and seeds, and the 685 graphs of
CAM from 10 to 100 variables are the campaign's graphs.

**RESIT data.** Erdos-Renyi graphs of mean degree 2 with 5, 10, and 20 variables and 1,000 rows, 20
datasets per setting (10 at 20 variables in their own units, where the CPU budget guard skipped the
rest), drawn by `andrey.data.sample_scm` in the eight regimes of the CAM data, standardized and in
their own units, with the columns in a shuffled order.

**RESIT fits.** One fit at a time on one core, a 1,800 s cap per fit, and a short warm-up fit in
each process. Andrey's `resit` is compared with the code of the authors (Peters, Mooij, Janzing, and
Scholkopf, 2014; R, additive models from `mgcv::gam` with GCV smoothing, an HSIC independence test,
alpha 0.05) and with lingam 1.12.2 (`lingam.RESIT` with a random forest of depth 4, alpha 0.01).
Each package runs its documented settings, and the differences are named, not tuned away. lingam
regresses with a random forest, tests at alpha 0.01, and takes as the next sink the variable with
the smallest HSIC statistic, where Andrey and the authors' code take the largest p-value. The
authors' code removes a variable's parents in the order of the columns, where Andrey removes them in
a rank computed from the data, and it compares p-values themselves, so p-values below the smallest
double tie at 0. The algorithm of the paper fixes neither order. Errors are the structural Hamming
distance of the returned DAG, where a reversed edge counts 1, and the share of true edges returned
with the right direction. No fit reached the cap. The reference fits are those of a campaign on a
pre-release build; Andrey's fits were run again on this release's code, on the same datasets and
seeds.

**Rerun.** A worse setting is rerun once on fresh seeds (20 to 29 for the other methods and 20 to 39
for RESIT, which has 20 datasets per setting), with the same build and arms. The verdict of the
rerun is its own interval.

## Speed

On linear-Gaussian data Andrey is faster in the median on every grid and for both methods: IAMB 3.8x
on standardized data, 2.1x on unstandardized data, and 2.1x at 200 variables; MMPC 13.1x, 6.6x, and
23.4x, each the median over the grid's settings and graphs. The ratio of a setting is the median of
the per-dataset ratios of the two fit times on one core. 4 of the 135 settings are slower than
bnlearn, none by more than 15%: IAMB moral graph at d 20, n 1,000, degree 2 (0.88, standardized);
IAMB moral graph at d 50, n 1,000, degree 2 (0.98, standardized); IAMB CPDAG at d 100, n 1,000,
degree 2 (0.93, unstandardized); IAMB moral graph at d 20, n 1,000, degree 4 (0.85, unstandardized).

**On the discrete networks Andrey is slower than bnlearn.** 186 of the 220 settings have a time
ratio below 1, including all 44 settings on hepar2, the largest network (70 variables): the ratios
there are 0.09 to 0.86 at 1,000 rows and 0.06 to 0.77 at 5,000 rows, and the G-square lanes are the
slowest. The faster discrete tests of this release cut Andrey's fit time on hepar2 by 1.0x to 6.0x
between the two measurements (about, since they ran on different machines; the CPDAG of MMPC at
5,000 rows did not change, because most of its time goes to the collider tests, which still run one
at a time). The ratios at hepar2 were 0.01 to 0.37 before and are 0.06 to 0.86 now; they stay
below 1. The median ratio over the lanes of each network:

| Network | Variables | Chi-square | G-square |
|---|---:|---:|---:|
| asia | 8 | 2.22 | 1.04 |
| child | 20 | 0.51 | 0.15 |
| insurance | 27 | 0.38 | 0.11 |
| alarm | 37 | 0.43 | 0.16 |
| hepar2 | 70 | 0.40 | 0.24 |

Fit times at hepar2, in seconds for Andrey (the first campaign, then with the faster tests) and as
the ratio of bnlearn's time to Andrey's, before -> after. The two measurements ran on different
machines, so seconds do not compare across them; the ratio pairs the two packages in one process
slot.

| Method | Graph | Test | 1,000 rows: Andrey seconds | 1,000 rows: bnlearn time / Andrey time | 5,000 rows: Andrey seconds | 5,000 rows: bnlearn time / Andrey time |
|---|---|---|---:|---:|---:|---:|
| IAMB | CPDAG | chi-square | 6.26 -> 1.29 | 0.07 -> 0.28 | 15.29 -> 4.77 | 0.06 -> 0.35 |
| IAMB | CPDAG | G-square | 19.81 -> 3.71 | 0.02 -> 0.10 | 35.09 -> 7.16 | 0.04 -> 0.24 |
| IAMB | skeleton | chi-square | 6.25 -> 1.21 | 0.06 -> 0.27 | 15.12 -> 4.50 | 0.07 -> 0.38 |
| IAMB | skeleton | G-square | 21.65 -> 3.59 | 0.01 -> 0.10 | 33.10 -> 7.13 | 0.03 -> 0.24 |
| IAMB | moral graph | chi-square | 6.51 -> 1.30 | 0.09 -> 0.41 | 15.32 -> 4.65 | 0.08 -> 0.41 |
| IAMB | moral graph | G-square | 19.53 -> 3.54 | 0.02 -> 0.15 | 33.65 -> 7.09 | 0.04 -> 0.28 |
| Inter-IAMB | CPDAG | chi-square | 6.52 -> 1.63 | 0.07 -> 0.29 | 15.05 -> 4.91 | 0.07 -> 0.38 |
| Inter-IAMB | CPDAG | G-square | 19.31 -> 4.40 | 0.02 -> 0.09 | 36.44 -> 9.20 | 0.03 -> 0.22 |
| Inter-IAMB | skeleton | chi-square | 6.37 -> 1.67 | 0.06 -> 0.25 | 14.68 -> 4.86 | 0.08 -> 0.38 |
| Inter-IAMB | skeleton | G-square | 20.21 -> 4.54 | 0.02 -> 0.09 | 33.96 -> 8.44 | 0.03 -> 0.23 |
| Inter-IAMB | moral graph | chi-square | 6.72 -> 1.63 | 0.09 -> 0.38 | 14.83 -> 4.63 | 0.09 -> 0.46 |
| Inter-IAMB | moral graph | G-square | 21.56 -> 4.64 | 0.02 -> 0.14 | 33.22 -> 8.21 | 0.04 -> 0.27 |
| MMPC | skeleton | chi-square | 0.59 -> 0.33 | 0.34 -> 0.83 | 1.95 -> 1.15 | 0.37 -> 0.74 |
| MMPC | skeleton | G-square | 2.18 -> 0.90 | 0.09 -> 0.29 | 5.21 -> 2.54 | 0.13 -> 0.29 |
| MMPC | CPDAG | chi-square | 0.68 -> 0.39 | 0.33 -> 0.80 | 6.20 -> 6.47 | 0.14 -> 0.16 |
| MMPC | CPDAG | G-square | 2.59 -> 1.15 | 0.09 -> 0.24 | 15.65 -> 14.00 | 0.05 -> 0.06 |
| MMHC | CPDAG | chi-square | 0.92 -> 0.33 | 0.28 -> 0.83 | 2.14 -> 0.97 | 0.33 -> 0.77 |
| MMHC | CPDAG | G-square | 2.79 -> 0.90 | 0.10 -> 0.31 | 6.12 -> 2.94 | 0.12 -> 0.28 |
| RSMAX2 | CPDAG | chi-square | 0.93 -> 0.32 | 0.36 -> 0.86 | 1.76 -> 1.02 | 0.28 -> 0.74 |
| RSMAX2 | CPDAG | G-square | 2.30 -> 0.95 | 0.10 -> 0.31 | 5.21 -> 3.13 | 0.10 -> 0.26 |
| RSMAX2, IAMB restrict | CPDAG | chi-square | 6.41 -> 1.34 | 0.05 -> 0.27 | 13.54 -> 3.91 | 0.07 -> 0.43 |
| RSMAX2, IAMB restrict | CPDAG | G-square | 16.45 -> 3.02 | 0.02 -> 0.12 | 30.63 -> 6.98 | 0.04 -> 0.26 |

**CAM is faster than CAM 1.0 in every setting.** In the 16 settings of its model, the time ratio of
the fit times on one core is 6.5x to 14.7x, from 6.5x to 8.9x at 10 and 20 variables up to 14.0x to
14.7x at 100, with a median of 9.7x; outside the model it is 7.4x to 14.9x. Andrey's CPU time packed
with other jobs, against alone, differed by at most 4%.

**RESIT is faster than both references in every setting.** In the 12 settings of its model, the time
ratio of the fit times on one core is 3.2x to 4.6x against the authors' code, with a median of 3.7x,
and 2.1x to 2.9x against lingam, with a median of 2.4x; outside the model it is 3.5x to 5.6x and
2.3x to 3.1x. A fit takes 1.9 s at 5 variables, 8 s at 10, and 53 s at 20. Packed with other jobs,
against alone (10 datasets at 10 variables), the CPU time of Andrey's fits was 1.00 times its time
alone, lingam's 1.00, and the authors' code's 1.08, so the ratios against the authors' code may be
up to 8% higher than on an idle machine.

## Tables

### IAMB

#### Standardized data, 20 to 100 variables

CPDAG, `structure="cpdag"`, against bnlearn `iamb()`:

| Degree | d | n | Datasets | Andrey | bnlearn | Empty graph | D, mean [95%] | Verdict | bnlearn time / Andrey time |
|---:|---:|---:|---:|---:|---:|---:|---|---|---:|
| 2 | 20 | 1,000 | 10 | 2.5 | 2.5 | 20.0 | +0.00 [+0.00, +0.00] | tie | 1.8 |
| 4 | 20 | 1,000 | 10 | 24.0 | 23.5 | 40.0 | +0.20 [-0.20, +0.70] | tie | 2.5 |
| 2 | 20 | 5,000 | 10 | 2.0 | 2.0 | 20.0 | +0.00 [+0.00, +0.00] | tie | 4.6 |
| 4 | 20 | 5,000 | 20 | 18.0 | 18.0 | 40.0 | +0.20 [-0.40, +0.75] | tie | 15.1 |
| 2 | 50 | 1,000 | 20 | 7.5 | 7.5 | 50.0 | +0.00 [+0.00, +0.00] | tie | 1.1 |
| 4 | 50 | 1,000 | 20 | 50.5 | 51.0 | 100.0 | +0.05 [-0.55, +0.65] | tie | 4.5 |
| 2 | 50 | 5,000 | 20 | 8.5 | 8.5 | 50.0 | +0.50 [+0.05, +1.05] | worse | 5.4 |
| 4 | 50 | 5,000 | 19 | 42.0 | 41.0 | 100.0 | +0.63 [-0.42, +1.74] | tie | 17.6 |
| 2 | 100 | 1,000 | 20 | 32.0 | 32.5 | 100.0 | -0.20 [-0.75, +0.25] | tie | 1.2 |
| 4 | 100 | 1,000 | 17 | 97.0 | 97.0 | 200.0 | -0.12 [-0.82, +0.59] | tie | 4.2 |
| 2 | 100 | 5,000 | 14 | 25.5 | 25.5 | 100.0 | +0.36 [-0.14, +1.14] | tie | 4.6 |
| 4 | 100 | 5,000 | 9 | 71.0 | 74.0 | 200.0 | +0.00, under 10 datasets | not judged | 35.7 |

Skeleton, `structure="skeleton"`, against bnlearn `iamb(undirected=TRUE)`:

| Degree | d | n | Datasets | Andrey | bnlearn | Empty graph | D, mean [95%] | Verdict | bnlearn time / Andrey time |
|---:|---:|---:|---:|---:|---:|---:|---|---|---:|
| 2 | 20 | 1,000 | 10 | 1.5 | 1.5 | 20.0 | +0.00 [+0.00, +0.00] | tie | 1.3 |
| 4 | 20 | 1,000 | 10 | 15.0 | 15.0 | 40.0 | +0.00 [+0.00, +0.00] | tie | 1.9 |
| 2 | 20 | 5,000 | 10 | 1.0 | 1.0 | 20.0 | +0.00 [+0.00, +0.00] | tie | 4.2 |
| 4 | 20 | 5,000 | 20 | 10.5 | 10.0 | 40.0 | +0.05 [-0.25, +0.30] | tie | 11.4 |
| 2 | 50 | 1,000 | 20 | 6.0 | 6.0 | 50.0 | +0.00 [+0.00, +0.00] | tie | 1.1 |
| 4 | 50 | 1,000 | 20 | 34.0 | 34.0 | 100.0 | +0.10 [-0.25, +0.45] | tie | 3.8 |
| 2 | 50 | 5,000 | 20 | 5.0 | 5.0 | 50.0 | +0.10 [+0.00, +0.25] | tie | 4.4 |
| 4 | 50 | 5,000 | 19 | 24.0 | 26.0 | 100.0 | -0.26 [-0.79, +0.11] | tie | 11.0 |
| 2 | 100 | 1,000 | 20 | 23.0 | 24.0 | 100.0 | -0.20 [-0.55, +0.00] | tie | 1.1 |
| 4 | 100 | 1,000 | 17 | 63.0 | 63.0 | 200.0 | -0.18 [-0.65, +0.24] | tie | 3.5 |
| 2 | 100 | 5,000 | 14 | 19.5 | 19.0 | 100.0 | +0.07 [+0.00, +0.21] | tie | 4.0 |
| 4 | 100 | 5,000 | 14 | 45.5 | 46.5 | 200.0 | -0.64 [-1.21, -0.14] | better | 29.5 |

Moral graph, `structure="moral"`, against bnlearn `learn.mb` joined by the AND rule:

| Degree | d | n | Datasets | Andrey | bnlearn | Empty graph | D, mean [95%] | Verdict | bnlearn time / Andrey time |
|---:|---:|---:|---:|---:|---:|---:|---|---|---:|
| 2 | 20 | 1,000 | 10 | 1.5 | 1.5 | 31.5 | +0.00 [+0.00, +0.00] | tie | 0.88 |
| 4 | 20 | 1,000 | 10 | 10.5 | 10.5 | 74.0 | +0.00 [+0.00, +0.00] | tie | 1.1 |
| 2 | 20 | 5,000 | 10 | 1.0 | 1.0 | 31.5 | -0.10 [-0.30, +0.00] | tie | 2.9 |
| 4 | 20 | 5,000 | 20 | 1.5 | 2.0 | 71.0 | -0.45 [-0.85, -0.05] | better | 4.5 |
| 2 | 50 | 1,000 | 20 | 8.0 | 8.0 | 80.0 | -0.15 [-0.40, +0.00] | tie | 0.98 |
| 4 | 50 | 1,000 | 20 | 38.5 | 37.0 | 208.5 | +0.40 [-0.60, +1.50] | tie | 1.5 |
| 2 | 50 | 5,000 | 20 | 6.5 | 7.0 | 80.0 | -0.20 [-0.70, +0.20] | tie | 4.3 |
| 4 | 50 | 5,000 | 20 | 7.5 | 7.0 | 208.5 | +0.35 [-1.00, +1.65] | tie | 1.8 |
| 2 | 100 | 1,000 | 20 | 33.0 | 33.0 | 168.5 | -0.20 [-0.55, +0.00] | tie | 1.0 |
| 4 | 100 | 1,000 | 17 | 125.0 | 126.0 | 445.0 | -0.88 [-1.71, +0.00] | tie | 1.4 |
| 2 | 100 | 5,000 | 14 | 22.0 | 23.0 | 170.5 | -0.36 [-0.93, +0.07] | tie | 3.7 |
| 4 | 100 | 5,000 | 20 | 32.0 | 33.0 | 445.0 | +2.25 [-0.55, +5.50] | tie | 4.5 |

#### Unstandardized data, 20 to 100 variables

CPDAG, `structure="cpdag"`, against bnlearn `iamb()`:

| Degree | d | n | Datasets | Andrey | bnlearn | Empty graph | D, mean [95%] | Verdict | bnlearn time / Andrey time |
|---:|---:|---:|---:|---:|---:|---:|---|---|---:|
| 2 | 20 | 1,000 | 10 | 8.0 | 8.0 | 20.0 | +0.10 [+0.00, +0.30] | tie | 2.5 |
| 4 | 20 | 1,000 | 10 | 23.0 | 22.0 | 40.0 | +0.60 [+0.10, +1.10] | worse, tie on rerun | 1.5 |
| 2 | 20 | 5,000 | 10 | 5.0 | 5.0 | 20.0 | +0.00 [+0.00, +0.00] | tie | 2.9 |
| 4 | 20 | 5,000 | 10 | 17.5 | 17.0 | 40.0 | +0.90 [+0.40, +1.50] | worse, tie on rerun | 10.3 |
| 2 | 50 | 1,000 | 10 | 26.5 | 26.0 | 50.0 | +0.30 [+0.00, +0.60] | tie | 1.7 |
| 4 | 50 | 1,000 | 10 | 47.5 | 47.0 | 100.0 | -0.30 [-1.20, +0.50] | tie | 1.9 |
| 2 | 50 | 5,000 | 10 | 24.5 | 24.5 | 50.0 | -0.20 [-0.60, +0.00] | tie | 4.3 |
| 4 | 50 | 5,000 | 10 | 34.0 | 34.0 | 100.0 | -0.40 [-1.50, +0.50] | tie | 9.3 |
| 2 | 100 | 1,000 | 10 | 78.5 | 79.5 | 100.0 | -0.60 [-1.10, -0.20] | better | 0.93 |
| 4 | 100 | 1,000 | 10 | 95.5 | 95.0 | 200.0 | +0.60 [-0.60, +1.90] | tie | 1.8 |
| 2 | 100 | 5,000 | 10 | 68.0 | 67.5 | 100.0 | -0.10 [-0.90, +0.50] | tie | 4.6 |
| 4 | 100 | 5,000 | 16 | 65.5 | 67.5 | 200.0 | -0.69 [-1.88, +0.31] | tie | 21.5 |

Skeleton, `structure="skeleton"`, against bnlearn `iamb(undirected=TRUE)`:

| Degree | d | n | Datasets | Andrey | bnlearn | Empty graph | D, mean [95%] | Verdict | bnlearn time / Andrey time |
|---:|---:|---:|---:|---:|---:|---:|---|---|---:|
| 2 | 20 | 1,000 | 10 | 5.0 | 5.0 | 20.0 | +0.00 [+0.00, +0.00] | tie | 2.2 |
| 4 | 20 | 1,000 | 10 | 14.5 | 14.5 | 40.0 | +0.00 [+0.00, +0.00] | tie | 1.4 |
| 2 | 20 | 5,000 | 10 | 3.5 | 3.5 | 20.0 | +0.00 [+0.00, +0.00] | tie | 4.1 |
| 4 | 20 | 5,000 | 10 | 11.0 | 11.0 | 40.0 | +0.00 [+0.00, +0.00] | tie | 9.5 |
| 2 | 50 | 1,000 | 10 | 18.0 | 18.0 | 50.0 | +0.00 [+0.00, +0.00] | tie | 1.6 |
| 4 | 50 | 1,000 | 10 | 30.5 | 30.5 | 100.0 | +0.00 [+0.00, +0.00] | tie | 1.6 |
| 2 | 50 | 5,000 | 10 | 15.5 | 15.5 | 50.0 | +0.00 [+0.00, +0.00] | tie | 4.7 |
| 4 | 50 | 5,000 | 10 | 20.0 | 20.0 | 100.0 | -0.20 [-0.50, +0.00] | tie | 7.4 |
| 2 | 100 | 1,000 | 10 | 59.0 | 59.0 | 100.0 | +0.00 [+0.00, +0.00] | tie | 1.1 |
| 4 | 100 | 1,000 | 10 | 68.0 | 68.0 | 200.0 | +0.00 [+0.00, +0.00] | tie | 1.8 |
| 2 | 100 | 5,000 | 10 | 49.5 | 49.5 | 100.0 | -0.10 [-0.30, +0.00] | tie | 3.2 |
| 4 | 100 | 5,000 | 16 | 43.5 | 44.0 | 200.0 | -0.44 [-0.75, -0.19] | better | 13.4 |

Moral graph, `structure="moral"`, against bnlearn `learn.mb` joined by the AND rule:

| Degree | d | n | Datasets | Andrey | bnlearn | Empty graph | D, mean [95%] | Verdict | bnlearn time / Andrey time |
|---:|---:|---:|---:|---:|---:|---:|---|---|---:|
| 2 | 20 | 1,000 | 10 | 8.0 | 8.0 | 31.5 | +0.00 [+0.00, +0.00] | tie | 1.0 |
| 4 | 20 | 1,000 | 10 | 31.0 | 31.0 | 74.0 | +0.00 [+0.00, +0.00] | tie | 0.85 |
| 2 | 20 | 5,000 | 10 | 6.0 | 6.0 | 31.5 | +0.00 [+0.00, +0.00] | tie | 2.1 |
| 4 | 20 | 5,000 | 10 | 15.5 | 15.5 | 74.0 | -0.10 [-0.30, +0.00] | tie | 3.9 |
| 2 | 50 | 1,000 | 10 | 35.0 | 35.0 | 81.0 | +0.00 [+0.00, +0.00] | tie | 1.4 |
| 4 | 50 | 1,000 | 10 | 96.0 | 95.5 | 215.0 | -0.10 [-0.40, +0.20] | tie | 1.1 |
| 2 | 50 | 5,000 | 10 | 30.0 | 30.0 | 81.0 | -0.30 [-0.70, +0.00] | tie | 3.3 |
| 4 | 50 | 5,000 | 10 | 58.0 | 59.0 | 215.0 | -0.60 [-1.00, -0.20] | better | 1.9 |
| 2 | 100 | 1,000 | 10 | 106.0 | 106.0 | 172.0 | +0.00 [+0.00, +0.00] | tie | 1.1 |
| 4 | 100 | 1,000 | 10 | 226.5 | 227.0 | 454.5 | -0.10 [-0.30, +0.00] | tie | 1.5 |
| 2 | 100 | 5,000 | 10 | 84.0 | 84.0 | 172.0 | -0.40 [-0.90, +0.00] | tie | 4.2 |
| 4 | 100 | 5,000 | 16 | 150.0 | 151.0 | 449.5 | -1.00 [-1.88, -0.25] | better | 3.4 |

#### 200 variables

CPDAG, `structure="cpdag"`, against bnlearn `iamb()`:

| Degree | d | n | Datasets | Andrey | bnlearn | Empty graph | D, mean [95%] | Verdict | bnlearn time / Andrey time |
|---:|---:|---:|---:|---:|---:|---:|---|---|---:|
| 2 | 200 | 1,000 | 10 | 92.5 | 92.5 | 200.0 | +0.00 [+0.00, +0.00] | tie | 1.3 |
| 4 | 200 | 1,000 | 9 | 175.0 | 172.0 | 400.0 | +1.67, under 10 datasets | not judged | 2.2 |
| 2 | 200 | 5,000 | 10 | 87.5 | 87.5 | 200.0 | -0.20 [-1.20, +0.60] | tie | 4.2 |

Skeleton, `structure="skeleton"`, against bnlearn `iamb(undirected=TRUE)`:

| Degree | d | n | Datasets | Andrey | bnlearn | Empty graph | D, mean [95%] | Verdict | bnlearn time / Andrey time |
|---:|---:|---:|---:|---:|---:|---:|---|---|---:|
| 2 | 200 | 1,000 | 10 | 63.0 | 63.0 | 200.0 | +0.00 [+0.00, +0.00] | tie | 1.1 |
| 4 | 200 | 1,000 | 10 | 126.5 | 125.5 | 400.0 | +0.70 [-0.40, +2.30] | tie | 2.0 |
| 2 | 200 | 5,000 | 10 | 60.5 | 60.5 | 200.0 | -0.20 [-0.50, +0.00] | tie | 4.3 |

Moral graph, `structure="moral"`, against bnlearn `learn.mb` joined by the AND rule:

| Degree | d | n | Datasets | Andrey | bnlearn | Empty graph | D, mean [95%] | Verdict | bnlearn time / Andrey time |
|---:|---:|---:|---:|---:|---:|---:|---|---|---:|
| 2 | 200 | 1,000 | 10 | 113.5 | 113.5 | 325.5 | +0.00 [-0.30, +0.30] | tie | 1.1 |
| 4 | 200 | 1,000 | 10 | 327.5 | 331.0 | 914.0 | +0.40 [-2.00, +3.10] | tie | 1.5 |
| 2 | 200 | 5,000 | 10 | 94.0 | 95.5 | 325.5 | -0.90 [-1.60, -0.10] | better | 4.2 |
| 4 | 200 | 5,000 | 10 | 116.5 | 122.5 | 914.0 | -1.30 [-6.20, +3.40] | tie | 5.0 |

#### Discrete networks

CPDAG, `structure="cpdag"`, against bnlearn `iamb()`, with Andrey's chisq and gsq against bnlearn's
x2-adf and mi-adf:

| Network | d | n | Test | Datasets | Andrey | bnlearn | Empty graph | D, mean [95%] | Verdict | bnlearn time / Andrey time |
|---|---:|---:|---|---:|---:|---:|---:|---|---|---:|
| asia | 8 | 1,000 | chi-square | 20 | 6.0 | 6.0 | 8.0 | +0.00 [+0.00, +0.00] | tie | 2.5 |
| asia | 8 | 1,000 | G-square | 20 | 5.0 | 5.5 | 8.0 | -0.10 [-0.30, +0.10] | tie | 1.1 |
| asia | 8 | 5,000 | chi-square | 20 | 6.0 | 6.0 | 8.0 | +0.05 [-0.25, +0.35] | tie | 2.3 |
| asia | 8 | 5,000 | G-square | 20 | 6.0 | 6.0 | 8.0 | +0.20 [-0.10, +0.50] | tie | 1.1 |
| child | 20 | 1,000 | chi-square | 20 | 15.0 | 17.0 | 25.0 | -2.10 [-3.35, -0.95] | better | 0.38 |
| child | 20 | 1,000 | G-square | 20 | 14.5 | 16.5 | 25.0 | -3.20 [-4.30, -2.15] | better | 0.10 |
| child | 20 | 5,000 | chi-square | 20 | 7.0 | 13.0 | 25.0 | -5.95 [-8.10, -3.85] | better | 0.52 |
| child | 20 | 5,000 | G-square | 20 | 6.0 | 12.5 | 25.0 | -5.55 [-8.10, -2.90] | better | 0.15 |
| insurance | 27 | 1,000 | chi-square | 17 | 43.0 | 44.0 | 52.0 | -2.06 [-4.06, -0.12] | better | 0.29 |
| insurance | 27 | 1,000 | G-square | 17 | 40.0 | 46.0 | 52.0 | -6.88 [-8.88, -4.82] | better | 0.07 |
| insurance | 27 | 5,000 | chi-square | 20 | 32.0 | 40.5 | 52.0 | -6.65 [-9.30, -3.85] | better | 0.42 |
| insurance | 27 | 5,000 | G-square | 20 | 34.0 | 39.0 | 52.0 | -6.70 [-9.75, -3.80] | better | 0.12 |
| alarm | 37 | 1,000 | chi-square | 20 | 29.0 | 29.5 | 46.0 | -0.85 [-3.20, +1.45] | tie | 0.30 |
| alarm | 37 | 1,000 | G-square | 20 | 28.0 | 31.0 | 46.0 | -2.85 [-4.55, -1.10] | better | 0.10 |
| alarm | 37 | 5,000 | chi-square | 20 | 20.5 | 23.0 | 46.0 | -2.75 [-5.70, -0.15] | better | 0.43 |
| alarm | 37 | 5,000 | G-square | 20 | 18.5 | 18.0 | 46.0 | +1.00 [-1.35, +3.65] | tie | 0.15 |
| hepar2 | 70 | 1,000 | chi-square | 15 | 117.0 | 117.0 | 123.0 | -2.27 [-4.73, +0.20] | tie | 0.28 |
| hepar2 | 70 | 1,000 | G-square | 14 | 108.0 | 114.5 | 123.0 | -4.36 [-7.43, -1.36] | better | 0.10 |
| hepar2 | 70 | 5,000 | chi-square | 16 | 94.0 | 97.5 | 123.0 | -4.31 [-7.56, -1.12] | better | 0.35 |
| hepar2 | 70 | 5,000 | G-square | 17 | 89.0 | 94.0 | 123.0 | -6.24 [-8.76, -3.41] | better | 0.24 |

Skeleton, `structure="skeleton"`, against bnlearn `iamb(undirected=TRUE)`, with Andrey's chisq and
gsq against bnlearn's x2-adf and mi-adf:

| Network | d | n | Test | Datasets | Andrey | bnlearn | Empty graph | D, mean [95%] | Verdict | bnlearn time / Andrey time |
|---|---:|---:|---|---:|---:|---:|---:|---|---|---:|
| asia | 8 | 1,000 | chi-square | 20 | 4.0 | 4.0 | 8.0 | +0.00 [+0.00, +0.00] | tie | 2.5 |
| asia | 8 | 1,000 | G-square | 20 | 4.0 | 4.0 | 8.0 | -0.15 [-0.35, +0.05] | tie | 0.90 |
| asia | 8 | 5,000 | chi-square | 20 | 4.0 | 4.0 | 8.0 | +0.05 [-0.25, +0.35] | tie | 2.3 |
| asia | 8 | 5,000 | G-square | 20 | 4.0 | 4.0 | 8.0 | +0.20 [-0.10, +0.50] | tie | 1.2 |
| child | 20 | 1,000 | chi-square | 20 | 6.0 | 10.0 | 25.0 | -4.00 [-4.60, -3.45] | better | 0.40 |
| child | 20 | 1,000 | G-square | 20 | 5.5 | 10.0 | 25.0 | -3.90 [-4.55, -3.20] | better | 0.11 |
| child | 20 | 5,000 | chi-square | 20 | 2.0 | 7.0 | 25.0 | -5.30 [-6.25, -4.40] | better | 0.54 |
| child | 20 | 5,000 | G-square | 20 | 2.0 | 6.0 | 25.0 | -4.55 [-5.65, -3.50] | better | 0.14 |
| insurance | 27 | 1,000 | chi-square | 17 | 26.0 | 32.0 | 52.0 | -5.94 [-7.00, -4.82] | better | 0.28 |
| insurance | 27 | 1,000 | G-square | 17 | 27.0 | 34.0 | 52.0 | -7.29 [-8.24, -6.47] | better | 0.07 |
| insurance | 27 | 5,000 | chi-square | 20 | 18.0 | 26.0 | 52.0 | -8.50 [-10.05, -6.95] | better | 0.44 |
| insurance | 27 | 5,000 | G-square | 20 | 18.0 | 27.0 | 52.0 | -8.95 [-10.15, -7.80] | better | 0.12 |
| alarm | 37 | 1,000 | chi-square | 20 | 16.0 | 17.0 | 46.0 | -1.25 [-2.10, -0.35] | better | 0.30 |
| alarm | 37 | 1,000 | G-square | 20 | 14.0 | 17.0 | 46.0 | -2.60 [-3.75, -1.50] | better | 0.09 |
| alarm | 37 | 5,000 | chi-square | 20 | 12.0 | 14.0 | 46.0 | -3.05 [-4.70, -1.40] | better | 0.47 |
| alarm | 37 | 5,000 | G-square | 20 | 8.0 | 10.0 | 46.0 | -1.95 [-2.80, -1.10] | better | 0.16 |
| hepar2 | 70 | 1,000 | chi-square | 15 | 96.0 | 98.0 | 123.0 | -1.33 [-3.13, +0.47] | tie | 0.27 |
| hepar2 | 70 | 1,000 | G-square | 14 | 92.5 | 95.0 | 123.0 | -3.07 [-4.50, -1.71] | better | 0.10 |
| hepar2 | 70 | 5,000 | chi-square | 16 | 74.5 | 77.0 | 123.0 | -3.25 [-5.25, -1.44] | better | 0.38 |
| hepar2 | 70 | 5,000 | G-square | 17 | 67.0 | 73.0 | 123.0 | -5.41 [-7.47, -3.35] | better | 0.24 |

Moral graph, `structure="moral"`, against bnlearn `learn.mb` joined by the AND rule, with Andrey's
chisq and gsq against bnlearn's x2-adf and mi-adf:

| Network | d | n | Test | Datasets | Andrey | bnlearn | Empty graph | D, mean [95%] | Verdict | bnlearn time / Andrey time |
|---|---:|---:|---|---:|---:|---:|---:|---|---|---:|
| asia | 8 | 1,000 | chi-square | 20 | 5.0 | 5.0 | 10.0 | +0.00 [+0.00, +0.00] | tie | 1.1 |
| asia | 8 | 1,000 | G-square | 20 | 5.0 | 5.0 | 10.0 | -0.15 [-0.35, +0.05] | tie | 0.43 |
| asia | 8 | 5,000 | chi-square | 20 | 5.0 | 5.0 | 10.0 | +0.10 [-0.20, +0.40] | tie | 1.7 |
| asia | 8 | 5,000 | G-square | 20 | 5.0 | 5.0 | 10.0 | +0.20 [-0.15, +0.55] | tie | 0.77 |
| child | 20 | 1,000 | chi-square | 20 | 11.0 | 15.5 | 30.0 | -4.45 [-5.20, -3.65] | better | 0.36 |
| child | 20 | 1,000 | G-square | 20 | 10.0 | 15.0 | 30.0 | -5.05 [-5.70, -4.35] | better | 0.10 |
| child | 20 | 5,000 | chi-square | 20 | 5.5 | 11.5 | 30.0 | -5.90 [-7.15, -4.60] | better | 0.57 |
| child | 20 | 5,000 | G-square | 20 | 5.0 | 11.0 | 30.0 | -5.65 [-6.70, -4.60] | better | 0.16 |
| insurance | 27 | 1,000 | chi-square | 17 | 38.0 | 47.0 | 70.0 | -9.88 [-10.88, -8.94] | better | 0.32 |
| insurance | 27 | 1,000 | G-square | 17 | 38.0 | 48.0 | 70.0 | -10.29 [-11.18, -9.41] | better | 0.09 |
| insurance | 27 | 5,000 | chi-square | 20 | 28.5 | 42.0 | 70.0 | -13.30 [-14.65, -11.95] | better | 0.47 |
| insurance | 27 | 5,000 | G-square | 20 | 29.0 | 42.5 | 70.0 | -13.25 [-14.40, -12.05] | better | 0.13 |
| alarm | 37 | 1,000 | chi-square | 20 | 36.5 | 33.5 | 65.0 | +1.30 [+0.15, +2.55] | worse, tie on rerun | 0.35 |
| alarm | 37 | 1,000 | G-square | 20 | 30.0 | 31.5 | 65.0 | -2.25 [-3.65, -0.80] | better | 0.11 |
| alarm | 37 | 5,000 | chi-square | 20 | 32.0 | 30.0 | 65.0 | +2.15 [-0.30, +4.60] | tie | 0.48 |
| alarm | 37 | 5,000 | G-square | 20 | 20.0 | 23.5 | 65.0 | -2.45 [-4.05, -0.90] | better | 0.18 |
| hepar2 | 70 | 1,000 | chi-square | 15 | 142.0 | 135.0 | 158.0 | +5.73 [+3.27, +8.40] | worse, rerun has 9 pairs | 0.41 |
| hepar2 | 70 | 1,000 | G-square | 14 | 129.5 | 130.5 | 158.0 | -2.14 [-3.64, -0.71] | better | 0.15 |
| hepar2 | 70 | 5,000 | chi-square | 16 | 122.0 | 116.5 | 158.0 | +5.81 [+1.81, +9.62] | worse, worse on rerun | 0.41 |
| hepar2 | 70 | 5,000 | G-square | 17 | 101.0 | 107.0 | 158.0 | -8.35 [-11.82, -5.06] | better | 0.28 |

### MMPC

#### Standardized data, 20 to 100 variables

Skeleton, `structure="skeleton"`, the default, against bnlearn `mmpc(undirected=TRUE)`:

| Degree | d | n | Datasets | Andrey | bnlearn | Empty graph | D, mean [95%] | Verdict | bnlearn time / Andrey time |
|---:|---:|---:|---:|---:|---:|---:|---|---|---:|
| 2 | 20 | 1,000 | 20 | 2.5 | 2.5 | 20.0 | +0.05 [-0.10, +0.20] | tie | 4.8 |
| 4 | 20 | 1,000 | 10 | 20.5 | 21.5 | 40.0 | -0.50 [-1.10, +0.00] | tie | 4.1 |
| 2 | 20 | 5,000 | 20 | 1.0 | 1.0 | 20.0 | +0.00 [-0.25, +0.25] | tie | 13.1 |
| 4 | 20 | 5,000 | 20 | 20.5 | 20.0 | 40.0 | -0.75 [-2.05, +0.60] | tie | 22.2 |
| 2 | 50 | 1,000 | 20 | 8.5 | 8.0 | 50.0 | +0.10 [-0.10, +0.35] | tie | 6.7 |
| 4 | 50 | 1,000 | 20 | 50.5 | 51.0 | 100.0 | +0.35 [-0.40, +1.15] | tie | 5.3 |
| 2 | 50 | 5,000 | 20 | 6.5 | 7.0 | 50.0 | -0.10 [-0.40, +0.25] | tie | 14.2 |
| 4 | 50 | 5,000 | 20 | 47.0 | 47.5 | 100.0 | -0.15 [-1.55, +1.30] | tie | 5.8 |
| 2 | 100 | 1,000 | 20 | 27.0 | 27.0 | 100.0 | +0.20 [-0.10, +0.50] | tie | 6.4 |
| 4 | 100 | 1,000 | 20 | 91.5 | 92.5 | 200.0 | +0.60 [-0.40, +1.65] | tie | 8.5 |
| 2 | 100 | 5,000 | 16 | 23.0 | 23.5 | 100.0 | +0.31 [-0.25, +0.81] | tie | 22.6 |
| 4 | 100 | 5,000 | 20 | 82.5 | 84.5 | 200.0 | -0.30 [-2.30, +1.55] | tie | 29.4 |

CPDAG, `structure="cpdag"`, against bnlearn `mmpc(undirected=FALSE)`:

| Degree | d | n | Datasets | Andrey | bnlearn | Empty graph | D, mean [95%] | Verdict | bnlearn time / Andrey time |
|---:|---:|---:|---:|---:|---:|---:|---|---|---:|
| 2 | 20 | 1,000 | 20 | 5.5 | 6.5 | 20.0 | -0.10 [-0.55, +0.20] | tie | 4.2 |
| 4 | 20 | 1,000 | 10 | 31.5 | 30.5 | 40.0 | +0.10 [-1.60, +1.60] | tie | 4.9 |
| 2 | 20 | 5,000 | 20 | 2.5 | 2.5 | 20.0 | +0.20 [-0.45, +0.85] | tie | 12.3 |
| 4 | 20 | 5,000 | 20 | 29.0 | 29.5 | 40.0 | -0.90 [-2.60, +0.70] | tie | 22.0 |
| 2 | 50 | 1,000 | 20 | 13.5 | 13.0 | 50.0 | +0.25 [-0.30, +1.00] | tie | 7.2 |
| 4 | 50 | 1,000 | 19 | 66.0 | 67.0 | 100.0 | -0.16 [-1.79, +1.68] | tie | 18.8 |
| 2 | 50 | 5,000 | 20 | 7.0 | 7.5 | 50.0 | +0.25 [-0.30, +0.95] | tie | 25.0 |
| 4 | 50 | 5,000 | 10 | 66.0 | 68.0 | 100.0 | -1.40 [-5.90, +3.10] | tie | 26.3 |
| 2 | 100 | 1,000 | 20 | 37.0 | 36.0 | 100.0 | +0.70 [+0.10, +1.55] | worse, rerun has 9 pairs | 17.9 |
| 4 | 100 | 1,000 | 2 | 127.5 | 118.0 | 200.0 | +9.50, under 10 datasets | not judged | 27.1 |
| 2 | 100 | 5,000 | 14 | 27.0 | 28.5 | 100.0 | +0.57 [-0.50, +1.79] | tie | 98.7 |
| 4 | 100 | 5,000 | 0 | - | - | - | no pairs | not judged | - |

#### Unstandardized data, 20 to 100 variables

Skeleton, `structure="skeleton"`, the default, against bnlearn `mmpc(undirected=TRUE)`:

| Degree | d | n | Datasets | Andrey | bnlearn | Empty graph | D, mean [95%] | Verdict | bnlearn time / Andrey time |
|---:|---:|---:|---:|---:|---:|---:|---|---|---:|
| 2 | 20 | 1,000 | 10 | 4.5 | 4.5 | 20.0 | +0.00 [+0.00, +0.00] | tie | 3.8 |
| 4 | 20 | 1,000 | 10 | 16.5 | 16.5 | 40.0 | +0.00 [+0.00, +0.00] | tie | 3.2 |
| 2 | 20 | 5,000 | 10 | 3.0 | 3.0 | 20.0 | -0.20 [-0.50, +0.00] | tie | 5.8 |
| 4 | 20 | 5,000 | 10 | 12.0 | 13.0 | 40.0 | +0.30 [-0.70, +1.20] | tie | 10.4 |
| 2 | 50 | 1,000 | 10 | 16.0 | 16.0 | 50.0 | +0.00 [+0.00, +0.00] | tie | 5.9 |
| 4 | 50 | 1,000 | 10 | 31.0 | 31.0 | 100.0 | +0.20 [+0.00, +0.60] | tie | 4.9 |
| 2 | 50 | 5,000 | 10 | 15.0 | 15.0 | 50.0 | +0.30 [+0.00, +0.80] | tie | 6.6 |
| 4 | 50 | 5,000 | 14 | 22.5 | 23.5 | 100.0 | -0.50 [-1.00, -0.14] | better | 2.4 |
| 2 | 100 | 1,000 | 10 | 57.5 | 57.5 | 100.0 | +0.00 [+0.00, +0.00] | tie | 5.3 |
| 4 | 100 | 1,000 | 14 | 65.0 | 65.0 | 200.0 | +0.00 [+0.00, +0.00] | tie | 4.4 |
| 2 | 100 | 5,000 | 14 | 49.0 | 50.5 | 100.0 | -0.07 [-0.57, +0.29] | tie | 12.0 |
| 4 | 100 | 5,000 | 14 | 43.0 | 42.5 | 200.0 | +0.29 [-0.14, +0.71] | tie | 6.2 |

CPDAG, `structure="cpdag"`, against bnlearn `mmpc(undirected=FALSE)`:

| Degree | d | n | Datasets | Andrey | bnlearn | Empty graph | D, mean [95%] | Verdict | bnlearn time / Andrey time |
|---:|---:|---:|---:|---:|---:|---:|---|---|---:|
| 2 | 20 | 1,000 | 10 | 6.5 | 6.0 | 20.0 | +0.10 [-0.30, +0.60] | tie | 4.7 |
| 4 | 20 | 1,000 | 10 | 23.0 | 23.5 | 40.0 | +0.00 [-0.30, +0.30] | tie | 7.4 |
| 2 | 20 | 5,000 | 10 | 6.0 | 6.5 | 20.0 | -0.20 [-0.50, +0.00] | tie | 6.8 |
| 4 | 20 | 5,000 | 10 | 21.0 | 18.5 | 40.0 | +1.50 [-0.30, +3.50] | tie | 57.5 |
| 2 | 50 | 1,000 | 10 | 25.0 | 25.0 | 50.0 | +0.00 [+0.00, +0.00] | tie | 6.6 |
| 4 | 50 | 1,000 | 9 | 53.0 | 50.0 | 100.0 | +0.44, under 10 datasets | not judged | 26.9 |
| 2 | 50 | 5,000 | 10 | 26.0 | 25.5 | 50.0 | +0.70 [-0.20, +2.20] | tie | 27.2 |
| 4 | 50 | 5,000 | 0 | - | - | - | no pairs | not judged | - |
| 2 | 100 | 1,000 | 8 | 73.5 | 73.5 | 100.0 | +0.12, under 10 datasets | not judged | 20.7 |
| 4 | 100 | 1,000 | 0 | - | - | - | no pairs | not judged | - |
| 2 | 100 | 5,000 | 3 | 66.0 | 70.0 | 100.0 | -2.00, under 10 datasets | not judged | 111.0 |
| 4 | 100 | 5,000 | 0 | - | - | - | no pairs | not judged | - |

#### 200 variables

Skeleton, `structure="skeleton"`, the default, against bnlearn `mmpc(undirected=TRUE)`:

| Degree | d | n | Datasets | Andrey | bnlearn | Empty graph | D, mean [95%] | Verdict | bnlearn time / Andrey time |
|---:|---:|---:|---:|---:|---:|---:|---|---|---:|
| 2 | 200 | 1,000 | 10 | 68.5 | 68.5 | 200.0 | +0.40 [+0.00, +0.80] | tie | 8.8 |
| 4 | 200 | 1,000 | 10 | 162.5 | 164.0 | 400.0 | -1.70 [-3.90, +0.20] | tie | 6.4 |
| 2 | 200 | 5,000 | 10 | 70.5 | 67.0 | 200.0 | +1.60 [+0.10, +3.10] | worse, worse on rerun | 23.4 |
| 4 | 200 | 5,000 | 10 | 136.5 | 140.0 | 400.0 | -2.70 [-7.00, +1.50] | tie | 52.9 |

CPDAG, `structure="cpdag"`, against bnlearn `mmpc(undirected=FALSE)`:

| Degree | d | n | Datasets | Andrey | bnlearn | Empty graph | D, mean [95%] | Verdict | bnlearn time / Andrey time |
|---:|---:|---:|---:|---:|---:|---:|---|---|---:|
| 2 | 200 | 1,000 | 5 | 89.0 | 88.0 | 200.0 | +0.00, under 10 datasets | not judged | 29.1 |

#### Discrete networks

Skeleton, `structure="skeleton"`, the default, against bnlearn `mmpc(undirected=TRUE)`, with
Andrey's chisq and gsq against bnlearn's x2-adf and mi-adf:

| Network | d | n | Test | Datasets | Andrey | bnlearn | Empty graph | D, mean [95%] | Verdict | bnlearn time / Andrey time |
|---|---:|---:|---|---:|---:|---:|---:|---|---|---:|
| asia | 8 | 1,000 | chi-square | 20 | 3.0 | 3.0 | 8.0 | +0.00 [+0.00, +0.00] | tie | 3.0 |
| asia | 8 | 1,000 | G-square | 20 | 3.0 | 3.0 | 8.0 | -0.20 [-0.50, +0.05] | tie | 1.6 |
| asia | 8 | 5,000 | chi-square | 20 | 2.0 | 2.0 | 8.0 | +0.00 [+0.00, +0.00] | tie | 3.4 |
| asia | 8 | 5,000 | G-square | 20 | 2.5 | 2.5 | 8.0 | +0.00 [+0.00, +0.00] | tie | 2.0 |
| child | 20 | 1,000 | chi-square | 10 | 5.0 | 9.5 | 25.0 | -4.80 [-5.20, -4.40] | better | 0.66 |
| child | 20 | 1,000 | G-square | 20 | 5.0 | 10.0 | 25.0 | -4.90 [-5.30, -4.45] | better | 0.15 |
| child | 20 | 5,000 | chi-square | 20 | 1.0 | 5.0 | 25.0 | -4.45 [-5.25, -3.70] | better | 0.68 |
| child | 20 | 5,000 | G-square | 20 | 1.0 | 5.0 | 25.0 | -3.70 [-4.45, -3.05] | better | 0.19 |
| insurance | 27 | 1,000 | chi-square | 17 | 24.0 | 30.0 | 52.0 | -6.29 [-7.06, -5.53] | better | 0.58 |
| insurance | 27 | 1,000 | G-square | 17 | 23.0 | 29.0 | 52.0 | -7.18 [-8.18, -6.18] | better | 0.14 |
| insurance | 27 | 5,000 | chi-square | 20 | 15.0 | 24.0 | 52.0 | -9.10 [-9.90, -8.30] | better | 0.55 |
| insurance | 27 | 5,000 | G-square | 20 | 15.0 | 22.5 | 52.0 | -7.85 [-8.70, -6.95] | better | 0.16 |
| alarm | 37 | 1,000 | chi-square | 20 | 11.0 | 16.0 | 46.0 | -5.05 [-5.55, -4.60] | better | 0.68 |
| alarm | 37 | 1,000 | G-square | 20 | 10.0 | 15.0 | 46.0 | -5.05 [-5.50, -4.60] | better | 0.20 |
| alarm | 37 | 5,000 | chi-square | 20 | 5.0 | 11.0 | 46.0 | -6.25 [-7.20, -5.35] | better | 0.89 |
| alarm | 37 | 5,000 | G-square | 20 | 5.0 | 8.0 | 46.0 | -3.40 [-4.00, -2.80] | better | 0.29 |
| hepar2 | 70 | 1,000 | chi-square | 15 | 96.0 | 99.0 | 123.0 | -1.47 [-2.20, -0.80] | better | 0.83 |
| hepar2 | 70 | 1,000 | G-square | 15 | 93.0 | 97.0 | 123.0 | -1.47 [-2.60, -0.40] | better | 0.29 |
| hepar2 | 70 | 5,000 | chi-square | 20 | 70.5 | 73.5 | 123.0 | -3.70 [-4.35, -3.05] | better | 0.74 |
| hepar2 | 70 | 5,000 | G-square | 20 | 68.0 | 72.0 | 123.0 | -3.60 [-4.55, -2.65] | better | 0.29 |

CPDAG, `structure="cpdag"`, against bnlearn `mmpc(undirected=FALSE)`, with Andrey's chisq and gsq
against bnlearn's x2-adf and mi-adf:

| Network | d | n | Test | Datasets | Andrey | bnlearn | Empty graph | D, mean [95%] | Verdict | bnlearn time / Andrey time |
|---|---:|---:|---|---:|---:|---:|---:|---|---|---:|
| asia | 8 | 1,000 | chi-square | 20 | 4.0 | 4.0 | 8.0 | +0.00 [+0.00, +0.00] | tie | 3.0 |
| asia | 8 | 1,000 | G-square | 20 | 4.0 | 4.0 | 8.0 | -0.20 [-0.50, +0.05] | tie | 1.4 |
| asia | 8 | 5,000 | chi-square | 20 | 3.0 | 3.0 | 8.0 | +0.00 [+0.00, +0.00] | tie | 3.1 |
| asia | 8 | 5,000 | G-square | 20 | 3.5 | 3.5 | 8.0 | +0.00 [+0.00, +0.00] | tie | 1.9 |
| child | 20 | 1,000 | chi-square | 10 | 13.5 | 19.0 | 25.0 | -5.40 [-6.60, -4.00] | better | 0.47 |
| child | 20 | 1,000 | G-square | 20 | 14.0 | 19.0 | 25.0 | -5.45 [-6.25, -4.60] | better | 0.11 |
| child | 20 | 5,000 | chi-square | 20 | 9.5 | 17.0 | 25.0 | -7.80 [-8.70, -6.95] | better | 0.33 |
| child | 20 | 5,000 | G-square | 20 | 10.5 | 16.0 | 25.0 | -6.85 [-7.70, -6.10] | better | 0.10 |
| insurance | 27 | 1,000 | chi-square | 17 | 34.0 | 47.0 | 52.0 | -11.82 [-13.47, -10.18] | better | 0.26 |
| insurance | 27 | 1,000 | G-square | 17 | 34.0 | 46.0 | 52.0 | -12.88 [-14.24, -11.53] | better | 0.06 |
| insurance | 27 | 5,000 | chi-square | 20 | 27.0 | 44.0 | 52.0 | -17.35 [-19.70, -15.00] | better | 0.08 |
| insurance | 27 | 5,000 | G-square | 20 | 23.5 | 43.0 | 52.0 | -18.35 [-20.30, -16.30] | better | 0.02 |
| alarm | 37 | 1,000 | chi-square | 20 | 28.0 | 38.0 | 46.0 | -11.55 [-14.00, -9.30] | better | 0.49 |
| alarm | 37 | 1,000 | G-square | 20 | 26.5 | 38.0 | 46.0 | -11.95 [-13.40, -10.45] | better | 0.16 |
| alarm | 37 | 5,000 | chi-square | 20 | 11.5 | 30.0 | 46.0 | -18.10 [-20.35, -16.05] | better | 0.48 |
| alarm | 37 | 5,000 | G-square | 20 | 11.5 | 26.0 | 46.0 | -15.40 [-18.10, -12.75] | better | 0.18 |
| hepar2 | 70 | 1,000 | chi-square | 15 | 126.0 | 128.0 | 123.0 | -2.80 [-6.87, +1.00] | tie | 0.80 |
| hepar2 | 70 | 1,000 | G-square | 15 | 119.0 | 128.0 | 123.0 | -3.60 [-11.07, +3.80] | tie | 0.24 |
| hepar2 | 70 | 5,000 | chi-square | 20 | 94.5 | 98.0 | 123.0 | -9.45 [-14.75, -4.80] | better | 0.16 |
| hepar2 | 70 | 5,000 | G-square | 20 | 92.0 | 95.0 | 123.0 | -3.60 [-5.10, -2.20] | better | 0.06 |

### Inter-IAMB

#### Discrete networks

CPDAG, `structure="cpdag"`, against bnlearn `inter.iamb()`, with Andrey's chisq and gsq against
bnlearn's x2-adf and mi-adf:

| Network | d | n | Test | Datasets | Andrey | bnlearn | Empty graph | D, mean [95%] | Verdict | bnlearn time / Andrey time |
|---|---:|---:|---|---:|---:|---:|---:|---|---|---:|
| asia | 8 | 1,000 | chi-square | 20 | 6.0 | 6.0 | 8.0 | +0.00 [+0.00, +0.00] | tie | 2.4 |
| asia | 8 | 1,000 | G-square | 20 | 5.0 | 5.5 | 8.0 | -0.10 [-0.30, +0.10] | tie | 0.83 |
| asia | 8 | 5,000 | chi-square | 20 | 6.0 | 6.0 | 8.0 | +0.05 [-0.15, +0.30] | tie | 1.9 |
| asia | 8 | 5,000 | G-square | 20 | 6.0 | 6.0 | 8.0 | +0.15 [-0.10, +0.40] | tie | 0.88 |
| child | 20 | 1,000 | chi-square | 20 | 15.5 | 17.0 | 25.0 | -2.65 [-4.30, -1.15] | better | 0.36 |
| child | 20 | 1,000 | G-square | 20 | 14.0 | 16.5 | 25.0 | -3.60 [-4.75, -2.50] | better | 0.09 |
| child | 20 | 5,000 | chi-square | 20 | 5.0 | 13.0 | 25.0 | -6.90 [-9.55, -4.20] | better | 0.43 |
| child | 20 | 5,000 | G-square | 20 | 6.0 | 12.5 | 25.0 | -5.60 [-7.50, -3.50] | better | 0.15 |
| insurance | 27 | 1,000 | chi-square | 17 | 40.0 | 44.0 | 52.0 | -3.65 [-5.47, -2.00] | better | 0.27 |
| insurance | 27 | 1,000 | G-square | 17 | 39.0 | 46.0 | 52.0 | -6.94 [-8.59, -5.24] | better | 0.07 |
| insurance | 27 | 5,000 | chi-square | 20 | 31.5 | 39.0 | 52.0 | -7.35 [-9.45, -5.25] | better | 0.36 |
| insurance | 27 | 5,000 | G-square | 20 | 30.0 | 38.0 | 52.0 | -8.05 [-10.90, -5.45] | better | 0.10 |
| alarm | 37 | 1,000 | chi-square | 20 | 28.5 | 29.5 | 46.0 | -2.15 [-4.90, +0.45] | tie | 0.27 |
| alarm | 37 | 1,000 | G-square | 20 | 28.0 | 31.0 | 46.0 | -4.15 [-6.15, -2.15] | better | 0.09 |
| alarm | 37 | 5,000 | chi-square | 20 | 18.0 | 21.0 | 46.0 | -2.70 [-4.75, -0.80] | better | 0.38 |
| alarm | 37 | 5,000 | G-square | 20 | 17.0 | 14.0 | 46.0 | +2.50 [+1.10, +3.95] | worse, tie on rerun | 0.15 |
| hepar2 | 70 | 1,000 | chi-square | 15 | 116.0 | 117.0 | 123.0 | -2.60 [-5.40, -0.07] | better | 0.29 |
| hepar2 | 70 | 1,000 | G-square | 12 | 111.5 | 113.5 | 123.0 | -1.92 [-5.67, +1.75] | tie | 0.09 |
| hepar2 | 70 | 5,000 | chi-square | 15 | 89.0 | 97.0 | 123.0 | -7.73 [-10.73, -4.47] | better | 0.38 |
| hepar2 | 70 | 5,000 | G-square | 15 | 89.0 | 95.0 | 123.0 | -6.20 [-8.93, -3.73] | better | 0.22 |

Skeleton, `structure="skeleton"`, against bnlearn `inter.iamb(undirected=TRUE)`, with Andrey's chisq
and gsq against bnlearn's x2-adf and mi-adf:

| Network | d | n | Test | Datasets | Andrey | bnlearn | Empty graph | D, mean [95%] | Verdict | bnlearn time / Andrey time |
|---|---:|---:|---|---:|---:|---:|---:|---|---|---:|
| asia | 8 | 1,000 | chi-square | 20 | 4.0 | 4.0 | 8.0 | +0.00 [+0.00, +0.00] | tie | 2.1 |
| asia | 8 | 1,000 | G-square | 20 | 4.0 | 4.0 | 8.0 | -0.15 [-0.35, +0.05] | tie | 0.88 |
| asia | 8 | 5,000 | chi-square | 20 | 4.0 | 4.0 | 8.0 | +0.05 [-0.15, +0.30] | tie | 1.9 |
| asia | 8 | 5,000 | G-square | 20 | 4.0 | 4.0 | 8.0 | +0.15 [-0.10, +0.40] | tie | 0.96 |
| child | 20 | 1,000 | chi-square | 20 | 6.0 | 10.0 | 25.0 | -4.70 [-5.30, -4.15] | better | 0.30 |
| child | 20 | 1,000 | G-square | 20 | 5.0 | 10.0 | 25.0 | -4.40 [-4.95, -3.80] | better | 0.09 |
| child | 20 | 5,000 | chi-square | 20 | 1.5 | 6.0 | 25.0 | -5.80 [-6.65, -5.00] | better | 0.51 |
| child | 20 | 5,000 | G-square | 20 | 1.0 | 6.0 | 25.0 | -5.25 [-6.10, -4.45] | better | 0.14 |
| insurance | 27 | 1,000 | chi-square | 17 | 25.0 | 32.0 | 52.0 | -6.88 [-7.82, -5.88] | better | 0.28 |
| insurance | 27 | 1,000 | G-square | 17 | 26.0 | 34.0 | 52.0 | -8.47 [-9.12, -7.76] | better | 0.07 |
| insurance | 27 | 5,000 | chi-square | 20 | 16.0 | 24.0 | 52.0 | -8.65 [-9.75, -7.55] | better | 0.35 |
| insurance | 27 | 5,000 | G-square | 20 | 14.5 | 24.0 | 52.0 | -9.30 [-10.45, -8.05] | better | 0.10 |
| alarm | 37 | 1,000 | chi-square | 20 | 14.0 | 17.0 | 46.0 | -2.75 [-3.80, -1.75] | better | 0.25 |
| alarm | 37 | 1,000 | G-square | 20 | 13.5 | 17.0 | 46.0 | -3.30 [-4.65, -2.00] | better | 0.10 |
| alarm | 37 | 5,000 | chi-square | 20 | 9.5 | 13.0 | 46.0 | -3.35 [-4.80, -1.95] | better | 0.39 |
| alarm | 37 | 5,000 | G-square | 20 | 7.0 | 8.0 | 46.0 | -1.05 [-1.60, -0.50] | better | 0.15 |
| hepar2 | 70 | 1,000 | chi-square | 15 | 97.0 | 98.0 | 123.0 | -1.27 [-3.27, +1.00] | tie | 0.25 |
| hepar2 | 70 | 1,000 | G-square | 12 | 92.5 | 95.0 | 123.0 | -2.83 [-4.75, -1.00] | better | 0.09 |
| hepar2 | 70 | 5,000 | chi-square | 15 | 72.0 | 77.0 | 123.0 | -6.07 [-7.60, -4.40] | better | 0.38 |
| hepar2 | 70 | 5,000 | G-square | 15 | 67.0 | 74.0 | 123.0 | -5.33 [-7.00, -3.67] | better | 0.23 |

Moral graph, `structure="moral"`, against bnlearn `learn.mb` with `inter.iamb`, joined by the AND
rule, with Andrey's chisq and gsq against bnlearn's x2-adf and mi-adf:

| Network | d | n | Test | Datasets | Andrey | bnlearn | Empty graph | D, mean [95%] | Verdict | bnlearn time / Andrey time |
|---|---:|---:|---|---:|---:|---:|---:|---|---|---:|
| asia | 8 | 1,000 | chi-square | 20 | 5.0 | 5.0 | 10.0 | +0.00 [+0.00, +0.00] | tie | 1.0 |
| asia | 8 | 1,000 | G-square | 20 | 5.0 | 5.0 | 10.0 | -0.10 [-0.30, +0.10] | tie | 0.37 |
| asia | 8 | 5,000 | chi-square | 20 | 5.0 | 5.0 | 10.0 | +0.05 [-0.15, +0.30] | tie | 1.3 |
| asia | 8 | 5,000 | G-square | 20 | 5.0 | 5.0 | 10.0 | +0.10 [-0.20, +0.40] | tie | 0.56 |
| child | 20 | 1,000 | chi-square | 20 | 10.0 | 15.5 | 30.0 | -5.60 [-6.35, -4.90] | better | 0.38 |
| child | 20 | 1,000 | G-square | 20 | 9.0 | 15.0 | 30.0 | -5.65 [-6.45, -4.90] | better | 0.10 |
| child | 20 | 5,000 | chi-square | 20 | 5.0 | 11.0 | 30.0 | -6.05 [-7.05, -5.10] | better | 0.54 |
| child | 20 | 5,000 | G-square | 20 | 4.5 | 10.5 | 30.0 | -6.20 [-7.10, -5.30] | better | 0.15 |
| insurance | 27 | 1,000 | chi-square | 17 | 36.0 | 47.0 | 70.0 | -11.12 [-11.94, -10.24] | better | 0.30 |
| insurance | 27 | 1,000 | G-square | 17 | 37.0 | 49.0 | 70.0 | -11.94 [-12.47, -11.35] | better | 0.07 |
| insurance | 27 | 5,000 | chi-square | 20 | 28.0 | 40.0 | 70.0 | -12.80 [-13.80, -11.80] | better | 0.40 |
| insurance | 27 | 5,000 | G-square | 20 | 25.0 | 39.5 | 70.0 | -14.00 [-15.30, -12.55] | better | 0.11 |
| alarm | 37 | 1,000 | chi-square | 20 | 33.0 | 33.5 | 65.0 | -2.10 [-3.60, -0.60] | better | 0.31 |
| alarm | 37 | 1,000 | G-square | 20 | 28.0 | 31.5 | 65.0 | -3.70 [-5.20, -2.20] | better | 0.11 |
| alarm | 37 | 5,000 | chi-square | 20 | 25.0 | 27.0 | 65.0 | -1.90 [-3.75, -0.20] | better | 0.43 |
| alarm | 37 | 5,000 | G-square | 20 | 18.5 | 21.0 | 65.0 | -3.10 [-3.85, -2.35] | better | 0.17 |
| hepar2 | 70 | 1,000 | chi-square | 15 | 140.0 | 137.0 | 158.0 | +4.67 [+1.47, +8.33] | worse, rerun has 9 pairs | 0.38 |
| hepar2 | 70 | 1,000 | G-square | 12 | 129.0 | 130.5 | 158.0 | -1.67 [-3.33, +0.08] | tie | 0.14 |
| hepar2 | 70 | 5,000 | chi-square | 15 | 116.0 | 117.0 | 158.0 | -0.53 [-3.80, +2.87] | tie | 0.46 |
| hepar2 | 70 | 5,000 | G-square | 15 | 100.0 | 109.0 | 158.0 | -9.40 [-11.67, -7.20] | better | 0.27 |

### MMHC

#### Discrete networks

CPDAG with the defaults, against bnlearn `mmhc()`, with Andrey's chisq and gsq against bnlearn's
x2-adf and mi-adf:

| Network | d | n | Test | Datasets | Andrey | bnlearn | Empty graph | D, mean [95%] | Verdict | bnlearn time / Andrey time |
|---|---:|---:|---|---:|---:|---:|---:|---|---|---:|
| asia | 8 | 1,000 | chi-square | 20 | 4.5 | 4.0 | 8.0 | +0.25 [-0.65, +1.10] | tie | 2.2 |
| asia | 8 | 1,000 | G-square | 10 | 5.5 | 5.5 | 8.0 | +0.20 [-1.00, +1.40] | tie | 1.2 |
| asia | 8 | 5,000 | chi-square | 10 | 4.0 | 4.0 | 8.0 | +0.20 [-1.40, +1.80] | tie | 2.4 |
| asia | 8 | 5,000 | G-square | 10 | 4.0 | 4.0 | 8.0 | +0.40 [-1.40, +2.20] | tie | 1.7 |
| child | 20 | 1,000 | chi-square | 20 | 17.5 | 19.0 | 25.0 | -2.30 [-4.00, -0.80] | better | 0.63 |
| child | 20 | 1,000 | G-square | 20 | 17.0 | 19.0 | 25.0 | -2.45 [-3.95, -1.10] | better | 0.17 |
| child | 20 | 5,000 | chi-square | 10 | 12.0 | 11.0 | 25.0 | -2.30 [-6.60, +1.70] | tie | 0.70 |
| child | 20 | 5,000 | G-square | 10 | 12.5 | 10.0 | 25.0 | -0.20 [-4.40, +3.50] | tie | 0.20 |
| insurance | 27 | 1,000 | chi-square | 17 | 37.0 | 40.0 | 52.0 | -4.35 [-7.00, -1.82] | better | 0.52 |
| insurance | 27 | 1,000 | G-square | 17 | 36.0 | 41.0 | 52.0 | -5.29 [-7.29, -3.29] | better | 0.16 |
| insurance | 27 | 5,000 | chi-square | 20 | 30.5 | 34.0 | 52.0 | -3.75 [-6.65, -0.65] | better | 0.60 |
| insurance | 27 | 5,000 | G-square | 10 | 32.0 | 35.5 | 52.0 | -4.40 [-6.70, -2.40] | better | 0.17 |
| alarm | 37 | 1,000 | chi-square | 10 | 23.5 | 29.0 | 46.0 | -5.00 [-6.80, -3.10] | better | 0.61 |
| alarm | 37 | 1,000 | G-square | 20 | 23.5 | 26.0 | 46.0 | -2.60 [-4.20, -1.05] | better | 0.20 |
| alarm | 37 | 5,000 | chi-square | 20 | 19.5 | 22.0 | 46.0 | -3.00 [-5.55, -0.60] | better | 0.78 |
| alarm | 37 | 5,000 | G-square | 20 | 20.0 | 22.0 | 46.0 | -0.80 [-3.00, +1.35] | tie | 0.31 |
| hepar2 | 70 | 1,000 | chi-square | 15 | 110.0 | 112.0 | 123.0 | -1.20 [-3.60, +1.07] | tie | 0.83 |
| hepar2 | 70 | 1,000 | G-square | 15 | 112.0 | 111.0 | 123.0 | +0.13 [-2.33, +2.40] | tie | 0.31 |
| hepar2 | 70 | 5,000 | chi-square | 20 | 90.0 | 92.5 | 123.0 | -3.60 [-6.70, -1.00] | better | 0.77 |
| hepar2 | 70 | 5,000 | G-square | 20 | 90.0 | 92.0 | 123.0 | -3.60 [-6.30, -1.50] | better | 0.28 |

### RSMAX2

#### Discrete networks

CPDAG with `restrict="mmpc"` and `maximize="hc"`, the defaults, against bnlearn `rsmax2()` with the
same phases, with Andrey's chisq and gsq against bnlearn's x2-adf and mi-adf:

| Network | d | n | Test | Datasets | Andrey | bnlearn | Empty graph | D, mean [95%] | Verdict | bnlearn time / Andrey time |
|---|---:|---:|---|---:|---:|---:|---:|---|---|---:|
| asia | 8 | 1,000 | chi-square | 10 | 5.0 | 5.0 | 8.0 | +0.20 [-1.00, +1.40] | tie | 2.1 |
| asia | 8 | 1,000 | G-square | 20 | 5.0 | 4.0 | 8.0 | +0.30 [-0.60, +1.15] | tie | 1.3 |
| asia | 8 | 5,000 | chi-square | 10 | 4.0 | 4.0 | 8.0 | +0.20 [-1.40, +1.80] | tie | 2.4 |
| asia | 8 | 5,000 | G-square | 10 | 4.0 | 4.0 | 8.0 | +0.40 [-1.40, +2.20] | tie | 1.3 |
| child | 20 | 1,000 | chi-square | 20 | 17.5 | 19.0 | 25.0 | -2.30 [-4.05, -0.75] | better | 0.63 |
| child | 20 | 1,000 | G-square | 10 | 17.0 | 19.0 | 25.0 | -2.30 [-4.40, -0.60] | better | 0.19 |
| child | 20 | 5,000 | chi-square | 20 | 12.0 | 13.5 | 25.0 | -2.10 [-4.55, +0.10] | tie | 0.69 |
| child | 20 | 5,000 | G-square | 20 | 12.0 | 10.5 | 25.0 | -0.40 [-2.85, +1.85] | tie | 0.20 |
| insurance | 27 | 1,000 | chi-square | 8 | 37.0 | 40.5 | 52.0 | -5.75, under 10 datasets | not judged | 0.55 |
| insurance | 27 | 1,000 | G-square | 17 | 36.0 | 41.0 | 52.0 | -5.29 [-7.35, -3.29] | better | 0.18 |
| insurance | 27 | 5,000 | chi-square | 20 | 30.5 | 34.0 | 52.0 | -3.75 [-6.65, -0.65] | better | 0.58 |
| insurance | 27 | 5,000 | G-square | 20 | 31.5 | 34.5 | 52.0 | -3.40 [-6.05, -0.60] | better | 0.17 |
| alarm | 37 | 1,000 | chi-square | 20 | 23.5 | 29.0 | 46.0 | -4.75 [-6.45, -2.85] | better | 0.74 |
| alarm | 37 | 1,000 | G-square | 20 | 23.5 | 26.0 | 46.0 | -2.60 [-4.15, -1.05] | better | 0.20 |
| alarm | 37 | 5,000 | chi-square | 20 | 19.5 | 22.0 | 46.0 | -3.00 [-5.50, -0.60] | better | 0.82 |
| alarm | 37 | 5,000 | G-square | 20 | 20.0 | 22.0 | 46.0 | -0.80 [-3.00, +1.30] | tie | 0.30 |
| hepar2 | 70 | 1,000 | chi-square | 15 | 110.0 | 112.0 | 123.0 | -1.20 [-3.47, +1.13] | tie | 0.86 |
| hepar2 | 70 | 1,000 | G-square | 15 | 112.0 | 111.0 | 123.0 | +0.13 [-2.33, +2.47] | tie | 0.31 |
| hepar2 | 70 | 5,000 | chi-square | 20 | 90.0 | 92.5 | 123.0 | -3.60 [-6.70, -1.00] | better | 0.74 |
| hepar2 | 70 | 5,000 | G-square | 20 | 90.0 | 92.0 | 123.0 | -3.60 [-6.25, -1.50] | better | 0.26 |

CPDAG with `restrict="iamb"` and `maximize="hc"`, against bnlearn `rsmax2(restrict="iamb")`, with
Andrey's chisq and gsq against bnlearn's x2-adf and mi-adf:

| Network | d | n | Test | Datasets | Andrey | bnlearn | Empty graph | D, mean [95%] | Verdict | bnlearn time / Andrey time |
|---|---:|---:|---|---:|---:|---:|---:|---|---|---:|
| asia | 8 | 1,000 | chi-square | 10 | 6.0 | 5.0 | 8.0 | +0.30 [-0.60, +1.30] | tie | 1.9 |
| asia | 8 | 1,000 | G-square | 20 | 5.0 | 5.0 | 8.0 | +0.30 [-0.40, +1.00] | tie | 0.87 |
| asia | 8 | 5,000 | chi-square | 10 | 5.5 | 5.0 | 8.0 | -0.10 [-1.30, +1.00] | tie | 1.8 |
| asia | 8 | 5,000 | G-square | 10 | 6.0 | 5.0 | 8.0 | +0.20 [-1.20, +1.50] | tie | 1.0 |
| child | 20 | 1,000 | chi-square | 20 | 17.5 | 19.0 | 25.0 | -2.05 [-3.50, -0.85] | better | 0.40 |
| child | 20 | 1,000 | G-square | 10 | 17.0 | 19.0 | 25.0 | -2.50 [-4.50, -0.90] | better | 0.12 |
| child | 20 | 5,000 | chi-square | 20 | 13.0 | 15.0 | 25.0 | -2.40 [-4.85, -0.25] | better | 0.51 |
| child | 20 | 5,000 | G-square | 20 | 14.0 | 12.5 | 25.0 | -0.60 [-3.35, +1.90] | tie | 0.18 |
| insurance | 27 | 1,000 | chi-square | 8 | 41.5 | 46.5 | 52.0 | -5.75, under 10 datasets | not judged | 0.35 |
| insurance | 27 | 1,000 | G-square | 17 | 43.0 | 48.0 | 52.0 | -5.82 [-8.18, -3.35] | better | 0.09 |
| insurance | 27 | 5,000 | chi-square | 20 | 35.5 | 40.0 | 52.0 | -6.20 [-9.65, -2.60] | better | 0.39 |
| insurance | 27 | 5,000 | G-square | 20 | 35.0 | 41.5 | 52.0 | -6.40 [-10.15, -2.80] | better | 0.13 |
| alarm | 37 | 1,000 | chi-square | 20 | 29.0 | 30.5 | 46.0 | -1.10 [-3.20, +1.05] | tie | 0.30 |
| alarm | 37 | 1,000 | G-square | 20 | 27.5 | 30.0 | 46.0 | -1.45 [-3.80, +0.70] | tie | 0.10 |
| alarm | 37 | 5,000 | chi-square | 20 | 21.5 | 25.5 | 46.0 | -3.40 [-5.90, -0.85] | better | 0.42 |
| alarm | 37 | 5,000 | G-square | 20 | 20.5 | 21.0 | 46.0 | -0.20 [-2.15, +1.85] | tie | 0.18 |
| hepar2 | 70 | 1,000 | chi-square | 15 | 112.0 | 112.0 | 123.0 | -1.33 [-3.87, +1.20] | tie | 0.27 |
| hepar2 | 70 | 1,000 | G-square | 15 | 112.0 | 112.0 | 123.0 | -2.07 [-4.53, +0.47] | tie | 0.12 |
| hepar2 | 70 | 5,000 | chi-square | 20 | 93.0 | 95.0 | 123.0 | -2.80 [-5.30, -0.55] | better | 0.43 |
| hepar2 | 70 | 5,000 | G-square | 20 | 90.0 | 94.5 | 123.0 | -4.95 [-7.90, -2.50] | better | 0.26 |

### CAM

Andrey's CAM against CAM 1.0 at 1,000 rows and 20 datasets per setting. D is Andrey's SHD minus CAM
1.0's on the same dataset. Orientation is the share of true edges returned with the right direction.
The settings outside the model carry no verdict.

#### Standardized columns, in the claim

Additive noise, Gaussian:

| d | Pairs | SHD: Andrey / CAM 1.0 / empty | D, mean [95%] | Verdict | Orientation: Andrey / CAM 1.0 | Same graph | CPU s: Andrey / CAM 1.0 | Time ratio [95%] |
|---:|---:|---|---|---|---|---:|---|---|
| 10 | 20 | 1.35 / 1.35 / 10.00 | +0.00 [+0.00, +0.00] | tie | 0.915 / 0.915 | 20/20 | 0.677 / 5.2 | 7.2x [6.6, 7.6] |
| 20 | 20 | 4.25 / 4.25 / 20.00 | +0.00 [+0.00, +0.00] | tie | 0.897 / 0.897 | 19/20 | 3.11 / 24.1 | 8.0x [7.4, 8.9] |
| 50 | 20 | 10.85 / 10.70 / 50.00 | +0.15 [+0.00, +0.40] | tie | 0.910 / 0.911 | 18/20 | 8.32 / 90 | 10.5x [9.4, 11.3] |
| 100 | 20 | 20.55 / 20.80 / 100.00 | -0.25 [-0.65, +0.05] | tie | 0.917 / 0.916 | 16/20 | 19.8 / 281 | 14.1x [13.4, 14.5] |

Additive noise, Gumbel:

| d | Pairs | SHD: Andrey / CAM 1.0 / empty | D, mean [95%] | Verdict | Orientation: Andrey / CAM 1.0 | Same graph | CPU s: Andrey / CAM 1.0 | Time ratio [95%] |
|---:|---:|---|---|---|---|---:|---|---|
| 10 | 20 | 4.65 / 4.70 / 10.00 | -0.05 [-0.15, +0.00] | tie | 0.745 / 0.745 | 19/20 | 0.697 / 5.29 | 7.6x [6.8, 7.8] |
| 20 | 20 | 8.65 / 8.60 / 20.00 | +0.05 [+0.00, +0.15] | tie | 0.762 / 0.762 | 19/20 | 3.29 / 24 | 8.0x [7.4, 9.0] |
| 50 | 20 | 18.15 / 18.30 / 50.00 | -0.15 [-0.30, +0.00] | tie | 0.817 / 0.816 | 17/20 | 7.56 / 92.9 | 11.8x [11.3, 12.7] |
| 100 | 20 | 31.65 / 32.35 / 100.00 | -0.70 [-1.40, -0.20] | better | 0.839 / 0.837 | 10/20 | 18.3 / 280 | 14.7x [14.0, 15.7] |

#### Columns in their own units, in the claim

Additive noise, Gaussian:

| d | Pairs | SHD: Andrey / CAM 1.0 / empty | D, mean [95%] | Verdict | Orientation: Andrey / CAM 1.0 | Same graph | CPU s: Andrey / CAM 1.0 | Time ratio [95%] |
|---:|---:|---|---|---|---|---:|---|---|
| 10 | 20 | 1.35 / 1.35 / 10.00 | +0.00 [+0.00, +0.00] | tie | 0.915 / 0.915 | 20/20 | 0.755 / 5.31 | 6.5x [5.8, 7.7] |
| 20 | 20 | 4.25 / 4.25 / 20.00 | +0.00 [+0.00, +0.00] | tie | 0.897 / 0.897 | 19/20 | 3 / 26.6 | 8.9x [8.5, 10.2] |
| 50 | 20 | 10.85 / 10.80 / 50.00 | +0.05 [+0.00, +0.15] | tie | 0.910 / 0.910 | 19/20 | 8.65 / 98 | 12.3x [11.0, 12.7] |
| 100 | 20 | 20.55 / 21.45 / 100.00 | -0.90 [-2.60, +0.05] | tie | 0.917 / 0.914 | 16/20 | 19.6 / 272 | 14.0x [13.4, 14.5] |

Additive noise, Gumbel:

| d | Pairs | SHD: Andrey / CAM 1.0 / empty | D, mean [95%] | Verdict | Orientation: Andrey / CAM 1.0 | Same graph | CPU s: Andrey / CAM 1.0 | Time ratio [95%] |
|---:|---:|---|---|---|---|---:|---|---|
| 10 | 20 | 4.65 / 4.70 / 10.00 | -0.05 [-0.15, +0.00] | tie | 0.745 / 0.745 | 19/20 | 0.728 / 5.05 | 6.8x [6.5, 7.3] |
| 20 | 20 | 8.65 / 8.60 / 20.00 | +0.05 [+0.00, +0.15] | tie | 0.762 / 0.762 | 19/20 | 2.68 / 23.6 | 8.6x [8.0, 10.1] |
| 50 | 20 | 18.15 / 18.30 / 50.00 | -0.15 [-0.30, +0.00] | tie | 0.817 / 0.816 | 17/20 | 7.43 / 87.8 | 11.6x [10.8, 12.1] |
| 100 | 20 | 31.65 / 32.50 / 100.00 | -0.85 [-1.65, -0.20] | better | 0.839 / 0.836 | 11/20 | 18.7 / 274 | 14.4x [13.9, 15.0] |

#### Outside the claim, standardized columns

Location-scale noise:

| d | Pairs | SHD: Andrey / CAM 1.0 / empty | D, mean [95%] | Orientation: Andrey / CAM 1.0 | Same graph | CPU s: Andrey / CAM 1.0 | Time ratio [95%] |
|---:|---:|---|---|---|---:|---|---|
| 10 | 20 | 2.70 / 2.70 / 10.00 | +0.00 [+0.00, +0.00] | 0.800 / 0.800 | 20/20 | 0.682 / 4.63 | 7.4x [6.2, 8.3] |
| 20 | 20 | 8.85 / 8.85 / 20.00 | +0.00 [+0.00, +0.00] | 0.765 / 0.765 | 20/20 | 2.5 / 22.8 | 9.0x [8.1, 9.5] |
| 50 | 20 | 18.30 / 18.40 / 50.00 | -0.10 [-0.30, +0.00] | 0.798 / 0.798 | 18/20 | 8.22 / 89 | 10.8x [9.8, 11.6] |
| 100 | 20 | 34.80 / 35.00 / 100.00 | -0.20 [-0.55, +0.00] | 0.810 / 0.810 | 18/20 | 18.1 / 255 | 14.9x [13.5, 15.2] |

Linear Gaussian:

| d | Pairs | SHD: Andrey / CAM 1.0 / empty | D, mean [95%] | Orientation: Andrey / CAM 1.0 | Same graph | CPU s: Andrey / CAM 1.0 | Time ratio [95%] |
|---:|---:|---|---|---|---:|---|---|
| 10 | 20 | 5.50 / 5.70 / 10.00 | -0.20 [-0.90, +0.20] | 0.640 / 0.630 | 16/20 | 0.904 / 7.45 | 8.8x [8.4, 9.2] |
| 20 | 20 | 12.95 / 12.05 / 20.00 | +0.90 [-0.05, +2.25] | 0.608 / 0.620 | 15/20 | 3.31 / 27.7 | 8.4x [7.9, 9.5] |
| 50 | 20 | 33.85 / 33.70 / 50.00 | +0.15 [-1.40, +1.65] | 0.617 / 0.618 | 7/20 | 9.22 / 110 | 11.4x [11.1, 12.6] |
| 100 | 20 | 67.50 / 69.55 / 100.00 | -2.05 [-4.55, +0.30] | 0.627 / 0.620 | 5/20 | 20.9 / 266 | 12.6x [11.7, 13.7] |

#### Outside the claim, columns in their own units

Location-scale noise:

| d | Pairs | SHD: Andrey / CAM 1.0 / empty | D, mean [95%] | Orientation: Andrey / CAM 1.0 | Same graph | CPU s: Andrey / CAM 1.0 | Time ratio [95%] |
|---:|---:|---|---|---|---:|---|---|
| 10 | 20 | 2.70 / 2.70 / 10.00 | +0.00 [+0.00, +0.00] | 0.800 / 0.800 | 20/20 | 0.67 / 5.12 | 8.0x [6.7, 8.5] |
| 20 | 20 | 8.85 / 8.85 / 20.00 | +0.00 [+0.00, +0.00] | 0.765 / 0.765 | 20/20 | 2.42 / 23 | 8.8x [8.4, 10.2] |
| 50 | 20 | 18.30 / 18.40 / 50.00 | -0.10 [-0.30, +0.00] | 0.798 / 0.798 | 18/20 | 8.07 / 92.3 | 10.8x [10.0, 11.9] |
| 100 | 20 | 34.80 / 34.85 / 100.00 | -0.05 [-0.15, +0.00] | 0.810 / 0.810 | 18/20 | 18.8 / 262 | 14.1x [13.2, 15.1] |

Linear Gaussian:

| d | Pairs | SHD: Andrey / CAM 1.0 / empty | D, mean [95%] | Orientation: Andrey / CAM 1.0 | Same graph | CPU s: Andrey / CAM 1.0 | Time ratio [95%] |
|---:|---:|---|---|---|---:|---|---|
| 10 | 20 | 5.50 / 6.00 / 10.00 | -0.50 [-1.25, +0.10] | 0.640 / 0.610 | 12/20 | 0.912 / 7.08 | 8.5x [8.1, 8.7] |
| 20 | 20 | 12.95 / 13.00 / 20.00 | -0.05 [-1.50, +1.55] | 0.608 / 0.588 | 13/20 | 3.4 / 27.4 | 8.1x [7.4, 8.7] |
| 50 | 20 | 33.85 / 34.60 / 50.00 | -0.75 [-1.75, +0.30] | 0.617 / 0.604 | 7/20 | 8.84 / 104 | 11.5x [11.2, 12.2] |
| 100 | 20 | 67.50 / 72.75 / 100.00 | -5.25 [-7.70, -2.90] | 0.627 / 0.595 | 1/20 | 20.9 / 286 | 14.1x [12.7, 14.9] |

### RESIT

Andrey's RESIT against the authors' code and lingam at 1,000 rows and 20 datasets per setting (10 at
20 variables in their own units). D is Andrey's SHD minus the reference's on the same dataset.
Orientation is the share of true edges returned with the right direction. The settings outside the
model carry no verdict. The setting marked "tie after rerun" is described above.

#### Standardized columns, in the claim

Additive noise, Gaussian:

| d | Reference | Pairs | SHD: Andrey / reference / empty | D, mean [95%] | Verdict | Orientation: Andrey / reference | Same graph | CPU s: Andrey / reference | Time ratio [95%] |
|---:|---|---:|---|---|---|---|---:|---|---|
| 5 | authors' code | 20 | 0.85 / 0.85 / 5.00 | +0.00 [+0.00, +0.00] | tie | 0.910 / 0.910 | 20/20 | 1.86 / 6.76 | 3.6x [3.5, 3.7] |
| 10 | authors' code | 20 | 6.80 / 5.60 / 10.00 | +1.20 [+0.05, +2.50] | tie after rerun | 0.815 / 0.810 | 9/20 | 8.18 / 29.3 | 3.6x [3.4, 3.7] |
| 20 | authors' code | 20 | 19.40 / 20.20 / 20.00 | -0.80 [-3.20, +1.60] | tie | 0.685 / 0.688 | 0/20 | 53.2 / 232 | 4.4x [3.8, 4.6] |
| 5 | authors' code | 20 | 0.85 / 3.50 / 5.00 | -2.65 [-4.15, -1.20] | better | 0.910 / 0.680 | 5/20 | 1.86 / 4.33 | 2.3x [2.2, 2.4] |
| 10 | authors' code | 20 | 6.80 / 12.25 / 10.00 | -5.45 [-8.45, -2.40] | better | 0.815 / 0.610 | 0/20 | 8.18 / 22.2 | 2.7x [2.6, 2.8] |
| 20 | authors' code | 20 | 19.40 / 44.80 / 20.00 | -25.40 [-31.90, -19.15] | better | 0.685 / 0.458 | 0/20 | 53.2 / 126 | 2.4x [2.1, 2.5] |
| 5 | lingam | 20 | 0.85 / 3.50 / 5.00 | -2.65 [-4.15, -1.20] | better | 0.910 / 0.680 | 5/20 | 1.86 / 4.33 | 2.3x [2.2, 2.4] |
| 10 | lingam | 20 | 6.80 / 12.25 / 10.00 | -5.45 [-8.45, -2.40] | better | 0.815 / 0.610 | 0/20 | 8.18 / 22.2 | 2.7x [2.6, 2.8] |
| 20 | lingam | 20 | 19.40 / 44.80 / 20.00 | -25.40 [-31.90, -19.15] | better | 0.685 / 0.458 | 0/20 | 53.2 / 126 | 2.4x [2.1, 2.5] |

Additive noise, Gumbel:

| d | Reference | Pairs | SHD: Andrey / reference / empty | D, mean [95%] | Verdict | Orientation: Andrey / reference | Same graph | CPU s: Andrey / reference | Time ratio [95%] |
|---:|---|---:|---|---|---|---|---:|---|---|
| 5 | authors' code | 20 | 0.15 / 0.25 / 5.00 | -0.10 [-0.60, +0.30] | tie | 0.980 / 0.980 | 18/20 | 1.94 / 6.79 | 3.5x [3.4, 3.6] |
| 10 | authors' code | 20 | 3.60 / 3.50 / 10.00 | +0.10 [-0.90, +1.05] | tie | 0.905 / 0.905 | 9/20 | 7.9 / 29.1 | 3.7x [3.5, 3.9] |
| 20 | authors' code | 20 | 15.55 / 14.45 / 20.00 | +1.10 [-1.70, +3.80] | tie | 0.765 / 0.773 | 0/20 | 52.7 / 235 | 4.6x [4.4, 4.8] |
| 5 | authors' code | 20 | 0.15 / 2.85 / 5.00 | -2.70 [-3.95, -1.60] | better | 0.980 / 0.800 | 6/20 | 1.94 / 4.45 | 2.3x [2.3, 2.4] |
| 10 | authors' code | 20 | 3.60 / 12.75 / 10.00 | -9.15 [-12.50, -5.70] | better | 0.905 / 0.700 | 0/20 | 7.9 / 22.5 | 2.9x [2.8, 2.9] |
| 20 | authors' code | 20 | 15.55 / 46.10 / 20.00 | -30.55 [-36.80, -24.15] | better | 0.765 / 0.607 | 0/20 | 52.7 / 132 | 2.5x [2.4, 2.6] |
| 5 | lingam | 20 | 0.15 / 2.85 / 5.00 | -2.70 [-3.95, -1.60] | better | 0.980 / 0.800 | 6/20 | 1.94 / 4.45 | 2.3x [2.3, 2.4] |
| 10 | lingam | 20 | 3.60 / 12.75 / 10.00 | -9.15 [-12.50, -5.70] | better | 0.905 / 0.700 | 0/20 | 7.9 / 22.5 | 2.9x [2.8, 2.9] |
| 20 | lingam | 20 | 15.55 / 46.10 / 20.00 | -30.55 [-36.80, -24.15] | better | 0.765 / 0.607 | 0/20 | 52.7 / 132 | 2.5x [2.4, 2.6] |

#### Columns in their own units, in the claim

Additive noise, Gaussian:

| d | Reference | Pairs | SHD: Andrey / reference / empty | D, mean [95%] | Verdict | Orientation: Andrey / reference | Same graph | CPU s: Andrey / reference | Time ratio [95%] |
|---:|---|---:|---|---|---|---|---:|---|---|
| 5 | authors' code | 20 | 0.85 / 1.15 / 5.00 | -0.30 [-0.70, +0.10] | tie | 0.910 / 0.870 | 13/20 | 1.94 / 6.73 | 3.6x [3.3, 3.7] |
| 10 | authors' code | 20 | 6.80 / 7.30 / 10.00 | -0.50 [-3.00, +2.15] | tie | 0.815 / 0.710 | 1/20 | 8.48 / 30 | 3.4x [3.3, 3.6] |
| 20 | authors' code | 10 | 19.80 / 18.90 / 20.00 | +0.90 [-4.10, +5.80] | tie | 0.685 / 0.610 | 0/10 | 53.5 / 224 | 4.4x [3.7, 4.5] |
| 5 | authors' code | 20 | 0.85 / 3.55 / 5.00 | -2.70 [-4.05, -1.40] | better | 0.910 / 0.620 | 3/20 | 1.94 / 4.42 | 2.4x [2.1, 2.5] |
| 10 | authors' code | 20 | 6.80 / 9.80 / 10.00 | -3.00 [-5.65, -0.20] | better | 0.815 / 0.565 | 0/20 | 8.48 / 23.2 | 2.7x [2.6, 2.8] |
| 20 | authors' code | 10 | 19.80 / 22.70 / 20.00 | -2.90 [-7.40, +1.70] | tie | 0.685 / 0.525 | 0/10 | 53.5 / 121 | 2.3x [2.0, 2.4] |
| 5 | lingam | 20 | 0.85 / 3.55 / 5.00 | -2.70 [-4.05, -1.40] | better | 0.910 / 0.620 | 3/20 | 1.94 / 4.42 | 2.4x [2.1, 2.5] |
| 10 | lingam | 20 | 6.80 / 9.80 / 10.00 | -3.00 [-5.65, -0.20] | better | 0.815 / 0.565 | 0/20 | 8.48 / 23.2 | 2.7x [2.6, 2.8] |
| 20 | lingam | 10 | 19.80 / 22.70 / 20.00 | -2.90 [-7.40, +1.70] | tie | 0.685 / 0.525 | 0/10 | 53.5 / 121 | 2.3x [2.0, 2.4] |

Additive noise, Gumbel:

| d | Reference | Pairs | SHD: Andrey / reference / empty | D, mean [95%] | Verdict | Orientation: Andrey / reference | Same graph | CPU s: Andrey / reference | Time ratio [95%] |
|---:|---|---:|---|---|---|---|---:|---|---|
| 5 | authors' code | 20 | 0.15 / 0.40 / 5.00 | -0.25 [-0.60, +0.00] | tie | 0.980 / 0.940 | 15/20 | 2.26 / 7.16 | 3.2x [3.1, 3.5] |
| 10 | authors' code | 20 | 3.60 / 4.15 / 10.00 | -0.55 [-2.35, +1.25] | tie | 0.905 / 0.865 | 2/20 | 7.91 / 30.5 | 3.8x [3.7, 3.9] |
| 20 | authors' code | 10 | 16.00 / 14.50 / 20.00 | +1.50 [-3.30, +6.30] | tie | 0.780 / 0.725 | 0/10 | 54.6 / 233 | 4.3x [4.2, 4.7] |
| 5 | authors' code | 20 | 0.15 / 3.10 / 5.00 | -2.95 [-4.10, -1.85] | better | 0.980 / 0.770 | 5/20 | 2.26 / 4.59 | 2.1x [2.0, 2.2] |
| 10 | authors' code | 20 | 3.60 / 11.10 / 10.00 | -7.50 [-10.90, -4.20] | better | 0.905 / 0.675 | 0/20 | 7.91 / 23.2 | 2.9x [2.8, 3.0] |
| 20 | authors' code | 10 | 16.00 / 31.20 / 20.00 | -15.20 [-23.50, -6.40] | better | 0.780 / 0.540 | 0/10 | 54.6 / 124 | 2.3x [2.2, 2.5] |
| 5 | lingam | 20 | 0.15 / 3.10 / 5.00 | -2.95 [-4.10, -1.85] | better | 0.980 / 0.770 | 5/20 | 2.26 / 4.59 | 2.1x [2.0, 2.2] |
| 10 | lingam | 20 | 3.60 / 11.10 / 10.00 | -7.50 [-10.90, -4.20] | better | 0.905 / 0.675 | 0/20 | 7.91 / 23.2 | 2.9x [2.8, 3.0] |
| 20 | lingam | 10 | 16.00 / 31.20 / 20.00 | -15.20 [-23.50, -6.40] | better | 0.780 / 0.540 | 0/10 | 54.6 / 124 | 2.3x [2.2, 2.5] |

#### Outside the claim, standardized columns

Location-scale noise:

| d | Reference | Pairs | SHD: Andrey / reference / empty | D, mean [95%] | Orientation: Andrey / reference | Same graph | CPU s: Andrey / reference | Time ratio [95%] |
|---:|---|---:|---|---|---|---:|---|---|
| 5 | authors' code | 20 | 7.75 / 7.75 / 5.00 | +0.00 [+0.00, +0.00] | 0.330 / 0.330 | 20/20 | 1.84 / 7.85 | 4.2x [3.8, 4.8] |
| 10 | authors' code | 20 | 28.95 / 28.30 / 10.00 | +0.65 [+0.20, +1.20] | 0.255 / 0.260 | 13/20 | 8.17 / 29.7 | 3.6x [3.5, 3.7] |
| 20 | authors' code | 20 | 87.70 / 86.45 / 20.00 | +1.25 [-0.35, +2.95] | 0.300 / 0.297 | 0/20 | 54.4 / 240 | 4.4x [4.1, 4.6] |
| 5 | authors' code | 20 | 7.75 / 7.60 / 5.00 | +0.15 [-0.30, +0.60] | 0.330 / 0.360 | 6/20 | 1.84 / 4.69 | 2.5x [2.4, 2.6] |
| 10 | authors' code | 20 | 28.95 / 29.50 / 10.00 | -0.55 [-2.90, +1.55] | 0.255 / 0.280 | 1/20 | 8.17 / 23.3 | 2.8x [2.7, 2.9] |
| 20 | authors' code | 20 | 87.70 / 98.60 / 20.00 | -10.90 [-16.50, -5.60] | 0.300 / 0.245 | 0/20 | 54.4 / 139 | 2.5x [2.4, 2.8] |
| 5 | lingam | 20 | 7.75 / 7.60 / 5.00 | +0.15 [-0.30, +0.60] | 0.330 / 0.360 | 6/20 | 1.84 / 4.69 | 2.5x [2.4, 2.6] |
| 10 | lingam | 20 | 28.95 / 29.50 / 10.00 | -0.55 [-2.90, +1.55] | 0.255 / 0.280 | 1/20 | 8.17 / 23.3 | 2.8x [2.7, 2.9] |
| 20 | lingam | 20 | 87.70 / 98.60 / 20.00 | -10.90 [-16.50, -5.60] | 0.300 / 0.245 | 0/20 | 54.4 / 139 | 2.5x [2.4, 2.8] |

Linear Gaussian:

| d | Reference | Pairs | SHD: Andrey / reference / empty | D, mean [95%] | Orientation: Andrey / reference | Same graph | CPU s: Andrey / reference | Time ratio [95%] |
|---:|---|---:|---|---|---|---:|---|---|
| 5 | authors' code | 20 | 4.95 / 5.30 / 5.00 | -0.35 [-0.95, +0.00] | 0.450 / 0.420 | 18/20 | 1.87 / 6.8 | 3.6x [3.5, 3.7] |
| 10 | authors' code | 20 | 13.30 / 13.35 / 10.00 | -0.05 [-0.95, +0.85] | 0.415 / 0.420 | 4/20 | 8.03 / 28.5 | 3.5x [3.3, 3.7] |
| 20 | authors' code | 20 | 32.20 / 31.80 / 20.00 | +0.40 [-3.25, +4.05] | 0.420 / 0.427 | 0/20 | 42.7 / 230 | 5.6x [5.3, 6.2] |
| 5 | authors' code | 20 | 4.95 / 3.90 / 5.00 | +1.05 [-0.80, +2.75] | 0.450 / 0.480 | 1/20 | 1.87 / 4.45 | 2.4x [2.3, 2.4] |
| 10 | authors' code | 20 | 13.30 / 11.40 / 10.00 | +1.90 [-0.70, +4.40] | 0.415 / 0.470 | 0/20 | 8.03 / 22.1 | 2.8x [2.6, 2.9] |
| 20 | authors' code | 20 | 32.20 / 34.10 / 20.00 | -1.90 [-6.85, +2.90] | 0.420 / 0.450 | 0/20 | 42.7 / 135 | 3.1x [2.9, 3.3] |
| 5 | lingam | 20 | 4.95 / 3.90 / 5.00 | +1.05 [-0.80, +2.75] | 0.450 / 0.480 | 1/20 | 1.87 / 4.45 | 2.4x [2.3, 2.4] |
| 10 | lingam | 20 | 13.30 / 11.40 / 10.00 | +1.90 [-0.70, +4.40] | 0.415 / 0.470 | 0/20 | 8.03 / 22.1 | 2.8x [2.6, 2.9] |
| 20 | lingam | 20 | 32.20 / 34.10 / 20.00 | -1.90 [-6.85, +2.90] | 0.420 / 0.450 | 0/20 | 42.7 / 135 | 3.1x [2.9, 3.3] |

#### Outside the claim, columns in their own units

Location-scale noise:

| d | Reference | Pairs | SHD: Andrey / reference / empty | D, mean [95%] | Orientation: Andrey / reference | Same graph | CPU s: Andrey / reference | Time ratio [95%] |
|---:|---|---:|---|---|---|---:|---|---|
| 5 | authors' code | 20 | 7.75 / 7.40 / 5.00 | +0.35 [+0.05, +0.75] | 0.330 / 0.360 | 10/20 | 2.05 / 7.41 | 3.6x [3.4, 3.9] |
| 10 | authors' code | 20 | 28.95 / 27.55 / 10.00 | +1.40 [-0.10, +3.05] | 0.255 / 0.250 | 2/20 | 8.05 / 32 | 3.9x [3.9, 4.0] |
| 20 | authors' code | 10 | 95.80 / 80.00 / 20.00 | +15.80 [+5.00, +27.70] | 0.265 / 0.310 | 0/10 | 53.4 / 246 | 4.6x [3.8, 4.9] |
| 5 | authors' code | 20 | 7.75 / 7.80 / 5.00 | -0.05 [-0.60, +0.40] | 0.330 / 0.330 | 8/20 | 2.05 / 4.68 | 2.3x [2.2, 2.4] |
| 10 | authors' code | 20 | 28.95 / 25.30 / 10.00 | +3.65 [+1.45, +5.85] | 0.255 / 0.315 | 1/20 | 8.05 / 24.2 | 3.0x [2.9, 3.1] |
| 20 | authors' code | 10 | 95.80 / 74.80 / 20.00 | +21.00 [+4.90, +35.80] | 0.265 / 0.345 | 0/10 | 53.4 / 132 | 2.5x [2.1, 2.6] |
| 5 | lingam | 20 | 7.75 / 7.80 / 5.00 | -0.05 [-0.60, +0.40] | 0.330 / 0.330 | 8/20 | 2.05 / 4.68 | 2.3x [2.2, 2.4] |
| 10 | lingam | 20 | 28.95 / 25.30 / 10.00 | +3.65 [+1.45, +5.85] | 0.255 / 0.315 | 1/20 | 8.05 / 24.2 | 3.0x [2.9, 3.1] |
| 20 | lingam | 10 | 95.80 / 74.80 / 20.00 | +21.00 [+4.90, +35.80] | 0.265 / 0.345 | 0/10 | 53.4 / 132 | 2.5x [2.1, 2.6] |

Linear Gaussian:

| d | Reference | Pairs | SHD: Andrey / reference / empty | D, mean [95%] | Orientation: Andrey / reference | Same graph | CPU s: Andrey / reference | Time ratio [95%] |
|---:|---|---:|---|---|---|---:|---|---|
| 5 | authors' code | 20 | 4.95 / 5.55 / 5.00 | -0.60 [-1.75, +0.45] | 0.450 / 0.410 | 6/20 | 1.9 / 7.2 | 3.8x [3.7, 3.9] |
| 10 | authors' code | 20 | 13.30 / 12.85 / 10.00 | +0.45 [-1.70, +2.45] | 0.415 / 0.420 | 1/20 | 7.76 / 36.6 | 4.7x [4.4, 4.9] |
| 20 | authors' code | 10 | 32.40 / 33.80 / 20.00 | -1.40 [-8.90, +6.20] | 0.460 / 0.340 | 0/10 | 42.5 / 237 | 5.5x [5.1, 5.8] |
| 5 | authors' code | 20 | 4.95 / 4.00 / 5.00 | +0.95 [-0.85, +2.70] | 0.450 / 0.470 | 2/20 | 1.9 / 4.55 | 2.4x [2.3, 2.5] |
| 10 | authors' code | 20 | 13.30 / 11.15 / 10.00 | +2.15 [-0.20, +4.60] | 0.415 / 0.415 | 0/20 | 7.76 / 23.6 | 3.0x [2.9, 3.2] |
| 20 | authors' code | 10 | 32.40 / 24.80 / 20.00 | +7.60 [-0.90, +16.10] | 0.460 / 0.425 | 0/10 | 42.5 / 131 | 3.1x [2.8, 3.2] |
| 5 | lingam | 20 | 4.95 / 4.00 / 5.00 | +0.95 [-0.85, +2.70] | 0.450 / 0.470 | 2/20 | 1.9 / 4.55 | 2.4x [2.3, 2.5] |
| 10 | lingam | 20 | 13.30 / 11.15 / 10.00 | +2.15 [-0.20, +4.60] | 0.415 / 0.415 | 0/20 | 7.76 / 23.6 | 3.0x [2.9, 3.2] |
| 20 | lingam | 10 | 32.40 / 24.80 / 20.00 | +7.60 [-0.90, +16.10] | 0.460 / 0.425 | 0/10 | 42.5 / 131 | 3.1x [2.8, 3.2] |

## Other changes in 0.13.0

CAM-UV and DYNOTEARS are experimental, and their docstrings, on the [API pages](../api.md), give
their measurements, not this card. The [changelog](../changelog.md)
measures the other changes: the location-scale noise generator, the rank that breaks ties without
the row order, the faster chi-square and G-square tests, the fixes to RCD, GIN, and VARMA-LiNGAM
under a constant offset, and the child and insurance networks in `load_dataset`.

## Limits

The linear-Gaussian settings cover 20 to 200 variables and 1,000 to 5,000 rows with Fisher-z tests;
the discrete settings cover five networks with up to 70 variables and the same two sample sizes.
Other data types, other tests, and larger discrete networks are not measured here. At 100 variables,
1,000 rows, and mean degree 2 on unstandardized data, the mean CPDAG of IAMB makes 79 errors against
100 for the empty graph, the closest of any setting (bnlearn: 80); on standardized data it is 20
variables, 1,000 rows, and mean degree 4, with 24 against 40.

Several CPDAG settings have fewer than 10 paired datasets and are not judged, because bnlearn
reached the cap: IAMB at 100 variables, 5,000 rows, and mean degree 4 on standardized data (9 pairs,
11 caps) and at 200 variables, 1,000 rows, and mean degree 4 (9 pairs, 1 cap). MMPC's CPDAG has 2
such settings on standardized data (100 variables and mean degree 4, where Andrey reached the cap in
6 and 19 fits and bnlearn in 18 and 20), 6 on unstandardized data, and 1 at 200 variables (5 pairs).
The MMPC skeleton at 200 variables and 5,000 rows is the one MMPC setting worse than bnlearn's on
two runs. Two discrete settings of RSMAX2 at insurance and 1,000 rows have 8 pairs and are not
judged. At 200 variables the skeleton and CPDAG of 5,000 rows and mean degree 4 are not measured,
because bnlearn did not finish all fits at 100 variables there.

CAM's settings cover 10 to 100 variables at 1,000 rows. Location-scale noise and linear-Gaussian
data lie outside its model and are reported, not judged: there CAM returns 77% to 81% (location-scale
noise) and 61% to 64% (linear Gaussian) of the true edges with the right direction, and linear-Gaussian
data identify only the equivalence class. CAM makes fewer errors than the empty graph in all 32
settings, and so does CAM 1.0. Larger graphs, more rows, hidden common causes, and effects that
interact are not measured here.

RESIT's settings cover 5 to 20 variables at 1,000 rows. Its accuracy claim holds up to 10 variables.
At 20 variables with Gaussian additive noise no implementation clearly beats the empty graph: Andrey
makes 19.4 mean errors, the authors' code 20.2, and lingam 44.8, against 20.0 for the empty graph
(19.8 for Andrey on columns in their own units, 10 datasets), so 20 variables is a range for speed.
With Gumbel noise at 20 variables Andrey (15.6) and the authors' code (14.5) are below the empty
graph and lingam (46.1) is not. Location-scale noise and linear-Gaussian data lie outside RESIT's
model and are reported, not judged: there RESIT returns 25% to 33% (location-scale noise) and 41% to
46% (linear Gaussian) of the true edges with the right direction, and its mean errors are above the
empty graph's in all 6 location-scale settings and in 4 of the 6 linear-Gaussian settings. Larger
graphs, more rows, hidden common causes, and effects that interact are not measured here.

## Reproduce

You can find environment setup, campaign drivers, and scores in the [benchmark harness
guide][harness]. To rerun the comparison, install Andrey (andrey-core), bnlearn 5.2.1, CAM 1.0, and
the authors' RESIT code with R 4.6.1 in the harness's R environment, and lingam 1.12.2. Every driver
shuffles the columns by default. Times will differ with the hardware and build; graphs should match
closely.

| Comparison | Lane | Sizes | Datasets x repeats | Wall cap (s) |
|---|---|---|---|---:|
| IAMB and MMPC against bnlearn | the harness's R lane | d 20, 50, 100, 200; n 1,000, 5,000; mean degree 2, 4 | 10 to 20 x 1 | 600 |
| IAMB, Inter-IAMB, MMPC, MMHC, and RSMAX2 against bnlearn on discrete networks | the harness's R lane | asia, child, insurance, alarm, hepar2; n 1,000, 5,000 | 10 to 20 x 1 | 600 |
| CAM against CAM 1.0 | the harness's R lane | d 10, 20, 50, 100; n 1,000 | 20 x 1 | 1,800 |
| RESIT against the authors' code and lingam | the harness's R lane | d 5, 10, 20; n 1,000 | 20 x 1 | 1,800 |

[harness]: https://github.com/Abel-ai-lab/andrey/blob/main/benchmarks/README.md