0.13.0 release card: functional causal models

Release card

Functional causal models

0.13.0

This card measures the four methods 0.13.0 makes supported: CAM and RESIT against their authors’ code, and IAMB and MMPC against bnlearn, on linear-Gaussian data and on five discrete networks. Each result holds for the measured conditions; it depends on the data, settings, and hardware.

IAMB, MMPC, CAM, and RESIT are supported methods. We compare IAMB and MMPC with bnlearn 5.2.1 on the graph each returns, scored against that graph’s own truth: the CPDAG for structure="cpdag", the skeleton for structure="skeleton", the default of MMPC, and the moral graph for structure="moral", which bnlearn’s learn.mb blankets joined by the AND rule give. Three grids of linear-Gaussian data measure them: standardized columns, columns left in their own units, and 200 variables. The same two methods, Inter-IAMB, MMHC, and RSMAX2 are then compared on five discrete networks (asia, child, insurance, alarm, and hepar2). There each package runs its own tests: Andrey’s calibrated chi-square and G-square tests (indep_test="chisq" and "gsq") against bnlearn’s x2-adf and mi-adf. The hybrids score with BDeu, equivalent sample size 1, in both packages. CAM is compared with CAM 1.0, the R package of its authors, on nonlinear additive-noise data, the model it assumes, and then on two kinds of data outside that model. RESIT is compared with the authors’ R code and with lingam’s implementation on the same kinds of data, at 5 to 20 variables.

Summary

Linear-Gaussian data:

Grid	Method	Graph	Settings	Better	Tie	Worse	Not judged	Worst setting: D, mean [95%]	Median time ratio
Standardized data, 20 to 100 variables	IAMB	CPDAG	12	0	10	1	1	+0.63 [-0.42, +1.74] (d 50, n 5,000, degree 4)	4.6
Standardized data, 20 to 100 variables	IAMB	skeleton	12	1	11	0	0	+0.10 [-0.25, +0.45] (d 50, n 1,000, degree 4)	3.9
Standardized data, 20 to 100 variables	IAMB	moral graph	12	1	11	0	0	+2.25 [-0.55, +5.50] (d 100, n 5,000, degree 4)	1.6
Standardized data, 20 to 100 variables	MMPC	skeleton	12	0	12	0	0	+0.60 [-0.40, +1.65] (d 100, n 1,000, degree 4)	7.6
Standardized data, 20 to 100 variables	MMPC	CPDAG	12	0	9	1	2	+0.70 [+0.10, +1.55] (d 100, n 1,000, degree 2)	18.8
Unstandardized data, 20 to 100 variables	IAMB	CPDAG	12	1	11	0	0	+0.90 [+0.40, +1.50] (d 20, n 5,000, degree 4)	2.7
Unstandardized data, 20 to 100 variables	IAMB	skeleton	12	1	11	0	0	+0.00 [+0.00, +0.00] (d 20, n 1,000, degree 2)	2.7
Unstandardized data, 20 to 100 variables	IAMB	moral graph	12	2	10	0	0	+0.00 [+0.00, +0.00] (d 20, n 1,000, degree 2)	1.7
Unstandardized data, 20 to 100 variables	MMPC	skeleton	12	1	11	0	0	+0.30 [-0.70, +1.20] (d 20, n 5,000, degree 4)	5.6
Unstandardized data, 20 to 100 variables	MMPC	CPDAG	12	0	6	0	6	+1.50 [-0.30, +3.50] (d 20, n 5,000, degree 4)	20.7
200 variables	IAMB	CPDAG	3	0	2	0	1	+0.00 [+0.00, +0.00] (d 200, n 1,000, degree 2)	2.2
200 variables	IAMB	skeleton	3	0	3	0	0	+0.70 [-0.40, +2.30] (d 200, n 1,000, degree 4)	2.0
200 variables	IAMB	moral graph	4	1	3	0	0	+0.40 [-2.00, +3.10] (d 200, n 1,000, degree 4)	2.9
200 variables	MMPC	skeleton	4	0	3	1	0	+1.60 [+0.10, +3.10] (d 200, n 5,000, degree 2)	16.1
200 variables	MMPC	CPDAG	1	0	0	0	1	-	29.1

Discrete networks, 10 settings per row (5 networks, 1,000 and 5,000 rows):

Method	Graph	Test	Settings	Better	Tie	Worse	Not judged	Worst setting: D, mean [95%]	Median time ratio	hepar2 time ratio, 1,000 / 5,000 rows
IAMB	CPDAG	chi-square	10	6	4	0	0	+0.05 [-0.25, +0.35] (asia, n 5,000)	0.4	0.28 / 0.35
IAMB	CPDAG	G-square	10	7	3	0	0	+1.00 [-1.35, +3.65] (alarm, n 5,000)	0.1	0.10 / 0.24
IAMB	skeleton	chi-square	10	7	3	0	0	+0.05 [-0.25, +0.35] (asia, n 5,000)	0.4	0.27 / 0.38
IAMB	skeleton	G-square	10	8	2	0	0	+0.20 [-0.10, +0.50] (asia, n 5,000)	0.1	0.10 / 0.24
IAMB	moral graph	chi-square	10	4	4	2	0	+5.81 [+1.81, +9.62] (hepar2, n 5,000)	0.4	0.41 / 0.41
IAMB	moral graph	G-square	10	8	2	0	0	+0.20 [-0.15, +0.55] (asia, n 5,000)	0.2	0.15 / 0.28
Inter-IAMB	CPDAG	chi-square	10	7	3	0	0	+0.05 [-0.15, +0.30] (asia, n 5,000)	0.4	0.29 / 0.38
Inter-IAMB	CPDAG	G-square	10	6	4	0	0	+2.50 [+1.10, +3.95] (alarm, n 5,000)	0.1	0.09 / 0.22
Inter-IAMB	skeleton	chi-square	10	7	3	0	0	+0.05 [-0.15, +0.30] (asia, n 5,000)	0.4	0.25 / 0.38
Inter-IAMB	skeleton	G-square	10	8	2	0	0	+0.15 [-0.10, +0.40] (asia, n 5,000)	0.1	0.09 / 0.23
Inter-IAMB	moral graph	chi-square	10	6	3	1	0	+4.67 [+1.47, +8.33] (hepar2, n 1,000)	0.4	0.38 / 0.46
Inter-IAMB	moral graph	G-square	10	7	3	0	0	+0.10 [-0.20, +0.40] (asia, n 5,000)	0.1	0.14 / 0.27
MMPC	skeleton	chi-square	10	8	2	0	0	+0.00 [+0.00, +0.00] (asia, n 1,000)	0.7	0.83 / 0.74
MMPC	skeleton	G-square	10	8	2	0	0	+0.00 [+0.00, +0.00] (asia, n 5,000)	0.2	0.29 / 0.29
MMPC	CPDAG	chi-square	10	7	3	0	0	+0.00 [+0.00, +0.00] (asia, n 1,000)	0.5	0.80 / 0.16
MMPC	CPDAG	G-square	10	7	3	0	0	+0.00 [+0.00, +0.00] (asia, n 5,000)	0.1	0.24 / 0.06
MMHC	CPDAG	chi-square	10	6	4	0	0	+0.25 [-0.65, +1.10] (asia, n 1,000)	0.7	0.83 / 0.77
MMHC	CPDAG	G-square	10	5	5	0	0	+0.40 [-1.40, +2.20] (asia, n 5,000)	0.2	0.31 / 0.28
RSMAX2	CPDAG	chi-square	10	5	4	0	1	+0.20 [-1.00, +1.40] (asia, n 1,000)	0.7	0.86 / 0.74
RSMAX2	CPDAG	G-square	10	5	5	0	0	+0.40 [-1.40, +2.20] (asia, n 5,000)	0.2	0.31 / 0.26
RSMAX2, IAMB restrict	CPDAG	chi-square	10	5	4	0	1	+0.30 [-0.60, +1.30] (asia, n 1,000)	0.4	0.27 / 0.43
RSMAX2, IAMB restrict	CPDAG	G-square	10	4	6	0	0	+0.30 [-0.40, +1.00] (asia, n 1,000)	0.2	0.12 / 0.26

A setting is one size, sample size, and mean degree, or one network and sample size. D is Andrey’s error minus bnlearn’s on the same dataset. A setting with at least 10 paired datasets is worse when the paired 95% bootstrap interval of the mean D lies wholly above 0, better when wholly below, and a tie otherwise; a setting with fewer is listed, not judged. A worse setting counts as a tie only if a rerun on fresh seeds with at least 10 paired datasets is not worse; one whose rerun has fewer pairs counts as worse. The time ratio is bnlearn’s fit time over Andrey’s, the median of the settings’ median ratios, so a ratio below 1 means Andrey is slower. The worst setting is the one with the largest D from the first run.

Nonlinear additive-noise data, CAM against CAM 1.0 (the DAG each returns, scored against the true DAG):

Grid	Method	Graph	Settings	Better	Tie	Worse	Not judged	Worst setting: D, mean [95%]	Median time ratio
Additive noise, Gaussian and Gumbel, 10 to 100 variables	CAM	DAG	16	2	14	0	0	+0.15 [+0.00, +0.40] (d 50, Gaussian noise, standardized)	9.7

A CAM setting is one size, noise, and column scale (standardized or in their own units) at 1,000 rows and 20 datasets, with the same verdict rule. The two better settings are Gumbel noise at 100 variables, on both scales. Only the settings of the model CAM assumes carry a verdict. The 16 settings of location-scale noise and of linear-Gaussian data, which the model does not cover, are reported in the tables and not judged: none has CAM 1.0 making fewer errors, and one, linear-Gaussian data in their own units at 100 variables, has Andrey making fewer. CAM returns the same graph as CAM 1.0 on 278 of the 320 in-model datasets. CAM 1.0 caps a variable’s parents at the smaller of the variables minus one and the rows over 20, 50 at 1,000 rows, which binds at 100 variables; Andrey sets no cap by default.

Nonlinear additive-noise data, RESIT against the authors’ code and against lingam (the DAG each returns, scored against the true DAG):

Grid	Method	Graph	Settings	Better	Tie	Worse	Not judged	Worst setting: D, mean [95%]	Median time ratio
Additive noise, Gaussian and Gumbel, 5 to 20 variables	RESIT, authors’ code	DAG	12	0	12	0	0	+1.50 [-3.30, +6.30] (d 20, Gumbel noise, own units)	3.7
Additive noise, Gaussian and Gumbel, 5 to 20 variables	RESIT, lingam	DAG	12	11	1	0	0	-2.65 [-4.15, -1.20] (d 5, Gaussian noise, standardized)	2.4

A RESIT setting is one size, noise, and column scale (standardized or in their own units) at 1,000 rows and 20 datasets (10 at 20 variables in their own units), with the same verdict rule. Against the authors’ code no setting is better and none is worse: 12 tie, one of them after a rerun, because Gaussian noise at 10 variables on standardized columns was worse on its first 20 datasets and is not on the next 20 (described below). Orientation is better in 2 settings against the authors’ code and in all 12 against lingam. Against lingam 11 settings are better and one ties. Only the settings of the model RESIT assumes carry a verdict. The 24 settings of location-scale noise and of linear-Gaussian data, which the model does not cover, are reported in the tables and not judged. In them the authors’ code makes fewer errors in 3 (location-scale noise: +0.65 [+0.20, +1.20] at 10 variables standardized, +0.35 [+0.05, +0.75] at 5 variables and +15.80 [+5.00, +27.70] at 20 variables in their own units, the last on 10 datasets), lingam in 2 (location-scale noise in their own units: +3.65 [+1.45, +5.85] at 10 variables and +21.00 [+4.90, +35.80] at 20 variables on 10 datasets), and RESIT makes fewer than lingam in 1 (location-scale noise, standardized, 20 variables: -10.90 [-16.50, -5.60]). RESIT returns the authors’ graph on 87 of the 220 in-model datasets and lingam’s on 19. Andrey’s RESIT returns the same graph on standardized columns and on columns in their own units on all 200 datasets of the four data types, the authors’ code on 55, and lingam on 43.

Where bnlearn makes fewer errors

10 of the 355 settings are worse in their first run:

Method	Graph	Grid	Setting	Andrey / bnlearn (mean errors)	D, first run [95%]	Rerun on fresh seeds: D [95%]	Counts as
IAMB	CPDAG	standardized	d 50, n 5,000, degree 2	9.15 / 8.65	+0.50 [+0.05, +1.05]	not rerun; cause traced below	worse
MMPC	CPDAG	standardized	d 100, n 1,000, degree 2	37.05 / 36.35	+0.70 [+0.10, +1.55]	+0.56, 9 datasets, seeds 20-29: not judged	worse, rerun not judged
IAMB	CPDAG	unstandardized	d 20, n 1,000, degree 4	22.10 / 21.50	+0.60 [+0.10, +1.10]	-0.10 [-0.30, +0.00] (10 datasets, seeds 20-29)	tie
IAMB	CPDAG	unstandardized	d 20, n 5,000, degree 4	17.10 / 16.20	+0.90 [+0.40, +1.50]	-0.10 [-0.30, +0.00] (10 datasets, seeds 20-29)	tie
MMPC	skeleton	200 variables	d 200, n 5,000, degree 2	75.50 / 73.90	+1.60 [+0.10, +3.10]	+2.70 [+1.20, +4.40] (10 datasets, seeds 20-29)	worse
IAMB	moral graph	networks, chi-square	alarm, n 1,000	35.85 / 34.55	+1.30 [+0.15, +2.55]	+2.10 [-0.30, +4.30] (10 datasets, seeds 20-29)	tie
IAMB	moral graph	networks, chi-square	hepar2, n 1,000	142.00 / 136.27	+5.73 [+3.27, +8.40]	+4.11, 9 datasets, seeds 20-29: not judged	worse, rerun not judged
IAMB	moral graph	networks, chi-square	hepar2, n 5,000	121.94 / 116.12	+5.81 [+1.81, +9.62]	+10.30 [+5.20, +15.20] (10 datasets, seeds 20-29)	worse
Inter-IAMB	CPDAG	networks, G-square	alarm, n 5,000	17.10 / 14.60	+2.50 [+1.10, +3.95]	-2.00 [-4.50, +0.30] (10 datasets, seeds 20-29)	tie
Inter-IAMB	moral graph	networks, chi-square	hepar2, n 1,000	141.47 / 136.80	+4.67 [+1.47, +8.33]	+2.00, 9 datasets, seeds 20-29: not judged	worse, rerun not judged

IAMB on linear-Gaussian data. The two unstandardized settings at 20 variables and mean degree 4 do not replicate: the rerun’s D is -0.10 on both, and 7 and 9 of its 10 graphs are identical. The standardized setting with 50 variables, 5,000 rows, and mean degree 2 is worse (16 of its 20 graphs are identical). Most of its gap, +7 pairs of the +10 over its 20 datasets, sits in 2 datasets whose skeletons differ, and the skeleton difference starts at one node’s first grow step, where the p-values of several candidates underflow to exactly 0 in bnlearn, which takes the first column among them. Andrey takes the candidate with the strongest test statistic, the association measure of Tsamardinos, Aliferis, and Statnikov (2003), so its result does not depend on the column order. The choice bnlearn makes, applied in Andrey’s search, reproduces bnlearn’s skeleton in all 4 traced datasets. The other +3 are orientation differences in 2 datasets with equal skeletons: Meek’s rules on a cyclic partially directed graph, and a collider that bnlearn skips for closing a directed cycle. The skeleton of the same cell is a tie, and so is its CPDAG on unstandardized data (-0.20 [-0.60, +0.00]).

MMPC, skeleton at 200 variables. The skeleton at 200 variables, 5,000 rows, and mean degree 2 is worse (D +1.60 [+0.10, +3.10] on seeds 0-9 and +2.70 [+1.20, +4.40] on seeds 20-29). At this size MMPC’s forward step often faces candidates whose dependence is too strong to rank by p-value, and Andrey admits the one with the strongest test statistic, the association measure of Tsamardinos, Brown, and Aliferis (2006), which is a true neighbor in 2,376 of 2,384 first picks (99.7%); the first column, bnlearn’s pick, is one in 49%. The extra errors are false edges between unrelated pairs whose statistic is just past the cutoff (z 2.0 to 3.1, the cutoff is 1.96); an arbitrary pick happens to admit a strongly correlated non-neighbor that removes some of them. Strongest-first keeps 44 such edges that the other order removes, about 2 per graph.

MMPC, CPDAG at 100 variables. The setting with 100 variables, 1,000 rows, and mean degree 2 is worse on seeds 0-19 (D +0.70 [+0.10, +1.55]). Its rerun on seeds 20-29 has 9 paired datasets, since bnlearn did not finish one in 600 s, so it is not judged: the mean D is +0.56 (an interval of -0.22 to +1.78), 6 of the 9 graphs are identical, and the whole difference sits in 3 datasets (-1, +1, +5). The setting counts as worse.

IAMB on hepar2. The moral graph of hepar2 at 5,000 rows is worse on both runs (D +5.81 [+1.81, +9.63] on seeds 0-19 and +10.30 [+5.20, +15.20] on seeds 20-29). The two packages run different discrete tests: Andrey’s chi-square test is calibrated by the exact permutation mean of the statistic as its degrees of freedom (Tsamardinos and Borboudakis 2010), and bnlearn’s x2-adf adjusts the degrees of freedom by a rule for sparse tables. Their graphs are not equal on any of the 3,183 datasets of child, insurance, alarm, and hepar2 (595 of 790 on asia), and the two tests agree on 97.7% of random triples.

Reruns that cannot judge. The moral graph of hepar2 at 1,000 rows is worse in its first run for IAMB (+5.73) and Inter-IAMB (+4.67) with chi-square tests. Each rerun on seeds 20-29 has 9 paired datasets, since the datasets on which both packages return an error (a constant column at 1,000 rows) are left out, so it is reported, not judged. Both reruns keep a positive mean D (+4.11 and +2.00). We have not traced these two settings separately; they involve the same pair of chi-square tests that differ at 5,000 rows. The two alarm settings that were worse in their first run (IAMB moral graph with chi-square at 1,000 rows, Inter-IAMB CPDAG with G-square at 5,000 rows) tie on their reruns.

Where the authors’ RESIT code makes fewer errors

One of the 12 settings of the model is worse in its first run: Gaussian additive noise at 10 variables on standardized columns, where Andrey makes 6.80 mean errors and the authors’ code 5.60 (D +1.20 [+0.05, +2.50], seeds 0-19). The rerun on 20 fresh datasets (seeds 20-39) has the interval [-0.60, +2.55] around D +0.95 (8.00 and 7.05 mean errors), which includes 0, so the setting counts as a tie and both runs are in the summary. The orientation ties in both runs. The 40 datasets together, which was not planned, give +1.08 [+0.13, +2.10].

RESIT, Gaussian noise, 10 variables. We traced the gap on the 40 datasets. It comes from the order in which RESIT tries to remove a variable’s parents, which the published algorithm does not fix: the authors’ code takes the columns in their order in the file, and Andrey takes a rank computed from the data that does not depend on the column order. Several predecessors can fit nearly as well, so two orders keep different parents. With the authors’ column order Andrey returns the authors’ graph on 38 of the 40 datasets and the gap falls to +0.20 [+0.00, +0.50]; 48 random parent orders moved the mean errors from 0.8 below to 0.8 above the column order’s. Whether Andrey’s order is systematically worse than a random one was then tested on 40 datasets not used before (seeds 40-79) with the rule written down in advance: its mean errors are +0.16 [-0.49, +0.85] from those of 16 random orders; 5 of the 16 random orders do as badly or worse (P = 0.35), so it is no worse than a random order. On those datasets Andrey is +0.68 [-0.53, +1.88] from the authors’ code, and a random order +0.51 [-0.28, +1.26]. What is left of the gap comes from near ties in the choice of the causal order, where Andrey’s smoothing parameters and the authors’ GCV minimum differ slightly, as in CAM. The order is arbitrary in the algorithm, so we do not choose it by its score on these data.

Method

Data. Linear-Gaussian data from Erdos-Renyi graphs with 20, 50, and 100 variables, 1,000 and 5,000 rows, and mean degree 2 and 4, as standardized columns and as columns left in their own units, and with 200 variables, 1,000 and 5,000 rows, and mean degree 2 and 4, on the outputs whose fits at 100 variables all finished. Each setting has up to 20 datasets (10 at 200 variables), one per seed, with the columns in a shuffled order. The CPU budget stopped some settings at 10 to 16 datasets. A setting keeps at least 10, and the ones that bnlearn could not finish keep fewer pairs. The discrete data are samples of 1,000 and 5,000 rows, up to 20 per setting, from the published networks asia (8 variables), child (20), insurance (27), alarm (37), and hepar2 (70), loaded with andrey.data.load_dataset. A dataset with a constant column is left out when both packages return an error on it.

Fits. One fit at a time on one core, a 600 s cap per fit, and one untimed warm-up fit except at 5,000 rows, mean degree 4, and 50 or more variables, where there is none. Linear-Gaussian data uses Fisher-z tests at alpha 0.05 (bnlearn test="zf"); discrete data uses chi-square and G-square tests at alpha 0.05 (bnlearn x2-adf and mi-adf). Andrey’s chi-square and G-square tests take as their degrees of freedom the exact permutation mean of the statistic (Tsamardinos and Borboudakis, 2010); bnlearn’s adjusted tests change the degrees of freedom by a rule for sparse tables. A fit that reaches the cap is a timeout. It is counted as coverage, left out of the pairs, and never scored as an empty graph. No discrete-network fit reached the cap. The discrete-network fit times were measured again with the faster discrete tests of this release, on the same datasets and seeds with the same protocol; the graphs and errors are those of the first campaign, and the 8,350 comparisons of the two sets of graphs found no difference.

Errors. The structural Hamming distance of the returned graph to its own truth. The tables give the median over the paired datasets, for Andrey, bnlearn, and the empty graph; D is the mean of the per-dataset differences, the statistic of the verdict. A DAG that bnlearn returns is compared through its CPDAG.

CAM data. Erdos-Renyi graphs of mean degree 2 with 10, 20, 50, and 100 variables and 1,000 rows, 20 datasets per setting, drawn by andrey.data.sample_scm in eight regimes: additive noise with Gaussian or Gumbel noise, which CAM assumes; location-scale noise, whose scale depends on the parents; and linear-Gaussian data, which identify only an equivalence class. Each is standardized (the default) and left in its own units. The columns are in a shuffled order.

CAM fits. One fit at a time on one core, a 1,800 s cap per fit, and a short warm-up fit in each process. CAM 1.0 runs with its preselection and pruning on (variableSel and pruning, tests at 0.001), which are Andrey’s settings. Two differences are named, not tuned away: CAM 1.0’s parent cap above, and smoothing parameters that come from different GCV optimizers, so two edges of nearly equal gain can be added in either order. Errors are the structural Hamming distance of the returned DAG, where a reversed edge counts 1, and the share of true edges returned with the right direction. No fit reached the cap. The reference fits are those of a campaign on the pre-release build; Andrey’s fits were run again on this release’s code, on the same datasets and seeds, and the 685 graphs of CAM from 10 to 100 variables are the campaign’s graphs.

RESIT data. Erdos-Renyi graphs of mean degree 2 with 5, 10, and 20 variables and 1,000 rows, 20 datasets per setting (10 at 20 variables in their own units, where the CPU budget guard skipped the rest), drawn by andrey.data.sample_scm in the eight regimes of the CAM data, standardized and in their own units, with the columns in a shuffled order.

RESIT fits. One fit at a time on one core, a 1,800 s cap per fit, and a short warm-up fit in each process. Andrey’s resit is compared with the code of the authors (Peters, Mooij, Janzing, and Scholkopf, 2014; R, additive models from mgcv::gam with GCV smoothing, an HSIC independence test, alpha 0.05) and with lingam 1.12.2 (lingam.RESIT with a random forest of depth 4, alpha 0.01). Each package runs its documented settings, and the differences are named, not tuned away. lingam regresses with a random forest, tests at alpha 0.01, and takes as the next sink the variable with the smallest HSIC statistic, where Andrey and the authors’ code take the largest p-value. The authors’ code removes a variable’s parents in the order of the columns, where Andrey removes them in a rank computed from the data, and it compares p-values themselves, so p-values below the smallest double tie at 0. The algorithm of the paper fixes neither order. Errors are the structural Hamming distance of the returned DAG, where a reversed edge counts 1, and the share of true edges returned with the right direction. No fit reached the cap. The reference fits are those of a campaign on a pre-release build; Andrey’s fits were run again on this release’s code, on the same datasets and seeds.

Rerun. A worse setting is rerun once on fresh seeds (20 to 29 for the other methods and 20 to 39 for RESIT, which has 20 datasets per setting), with the same build and arms. The verdict of the rerun is its own interval.

Speed

On linear-Gaussian data Andrey is faster in the median on every grid and for both methods: IAMB 3.8x on standardized data, 2.1x on unstandardized data, and 2.1x at 200 variables; MMPC 13.1x, 6.6x, and 23.4x, each the median over the grid’s settings and graphs. The ratio of a setting is the median of the per-dataset ratios of the two fit times on one core. 4 of the 135 settings are slower than bnlearn, none by more than 15%: IAMB moral graph at d 20, n 1,000, degree 2 (0.88, standardized); IAMB moral graph at d 50, n 1,000, degree 2 (0.98, standardized); IAMB CPDAG at d 100, n 1,000, degree 2 (0.93, unstandardized); IAMB moral graph at d 20, n 1,000, degree 4 (0.85, unstandardized).

On the discrete networks Andrey is slower than bnlearn. 186 of the 220 settings have a time ratio below 1, including all 44 settings on hepar2, the largest network (70 variables): the ratios there are 0.09 to 0.86 at 1,000 rows and 0.06 to 0.77 at 5,000 rows, and the G-square lanes are the slowest. The faster discrete tests of this release cut Andrey’s fit time on hepar2 by 1.0x to 6.0x between the two measurements (about, since they ran on different machines; the CPDAG of MMPC at 5,000 rows did not change, because most of its time goes to the collider tests, which still run one at a time). The ratios at hepar2 were 0.01 to 0.37 before and are 0.06 to 0.86 now; they stay below 1. The median ratio over the lanes of each network:

Network	Variables	Chi-square	G-square
asia	8	2.22	1.04
child	20	0.51	0.15
insurance	27	0.38	0.11
alarm	37	0.43	0.16
hepar2	70	0.40	0.24

Fit times at hepar2, in seconds for Andrey (the first campaign, then with the faster tests) and as the ratio of bnlearn’s time to Andrey’s, before -> after. The two measurements ran on different machines, so seconds do not compare across them; the ratio pairs the two packages in one process slot.

Method	Graph	Test	1,000 rows: Andrey seconds	1,000 rows: bnlearn time / Andrey time	5,000 rows: Andrey seconds	5,000 rows: bnlearn time / Andrey time
IAMB	CPDAG	chi-square	6.26 -> 1.29	0.07 -> 0.28	15.29 -> 4.77	0.06 -> 0.35
IAMB	CPDAG	G-square	19.81 -> 3.71	0.02 -> 0.10	35.09 -> 7.16	0.04 -> 0.24
IAMB	skeleton	chi-square	6.25 -> 1.21	0.06 -> 0.27	15.12 -> 4.50	0.07 -> 0.38
IAMB	skeleton	G-square	21.65 -> 3.59	0.01 -> 0.10	33.10 -> 7.13	0.03 -> 0.24
IAMB	moral graph	chi-square	6.51 -> 1.30	0.09 -> 0.41	15.32 -> 4.65	0.08 -> 0.41
IAMB	moral graph	G-square	19.53 -> 3.54	0.02 -> 0.15	33.65 -> 7.09	0.04 -> 0.28
Inter-IAMB	CPDAG	chi-square	6.52 -> 1.63	0.07 -> 0.29	15.05 -> 4.91	0.07 -> 0.38
Inter-IAMB	CPDAG	G-square	19.31 -> 4.40	0.02 -> 0.09	36.44 -> 9.20	0.03 -> 0.22
Inter-IAMB	skeleton	chi-square	6.37 -> 1.67	0.06 -> 0.25	14.68 -> 4.86	0.08 -> 0.38
Inter-IAMB	skeleton	G-square	20.21 -> 4.54	0.02 -> 0.09	33.96 -> 8.44	0.03 -> 0.23
Inter-IAMB	moral graph	chi-square	6.72 -> 1.63	0.09 -> 0.38	14.83 -> 4.63	0.09 -> 0.46
Inter-IAMB	moral graph	G-square	21.56 -> 4.64	0.02 -> 0.14	33.22 -> 8.21	0.04 -> 0.27
MMPC	skeleton	chi-square	0.59 -> 0.33	0.34 -> 0.83	1.95 -> 1.15	0.37 -> 0.74
MMPC	skeleton	G-square	2.18 -> 0.90	0.09 -> 0.29	5.21 -> 2.54	0.13 -> 0.29
MMPC	CPDAG	chi-square	0.68 -> 0.39	0.33 -> 0.80	6.20 -> 6.47	0.14 -> 0.16
MMPC	CPDAG	G-square	2.59 -> 1.15	0.09 -> 0.24	15.65 -> 14.00	0.05 -> 0.06
MMHC	CPDAG	chi-square	0.92 -> 0.33	0.28 -> 0.83	2.14 -> 0.97	0.33 -> 0.77
MMHC	CPDAG	G-square	2.79 -> 0.90	0.10 -> 0.31	6.12 -> 2.94	0.12 -> 0.28
RSMAX2	CPDAG	chi-square	0.93 -> 0.32	0.36 -> 0.86	1.76 -> 1.02	0.28 -> 0.74
RSMAX2	CPDAG	G-square	2.30 -> 0.95	0.10 -> 0.31	5.21 -> 3.13	0.10 -> 0.26
RSMAX2, IAMB restrict	CPDAG	chi-square	6.41 -> 1.34	0.05 -> 0.27	13.54 -> 3.91	0.07 -> 0.43
RSMAX2, IAMB restrict	CPDAG	G-square	16.45 -> 3.02	0.02 -> 0.12	30.63 -> 6.98	0.04 -> 0.26

CAM is faster than CAM 1.0 in every setting. In the 16 settings of its model, the time ratio of the fit times on one core is 6.5x to 14.7x, from 6.5x to 8.9x at 10 and 20 variables up to 14.0x to 14.7x at 100, with a median of 9.7x; outside the model it is 7.4x to 14.9x. Andrey’s CPU time packed with other jobs, against alone, differed by at most 4%.

RESIT is faster than both references in every setting. In the 12 settings of its model, the time ratio of the fit times on one core is 3.2x to 4.6x against the authors’ code, with a median of 3.7x, and 2.1x to 2.9x against lingam, with a median of 2.4x; outside the model it is 3.5x to 5.6x and 2.3x to 3.1x. A fit takes 1.9 s at 5 variables, 8 s at 10, and 53 s at 20. Packed with other jobs, against alone (10 datasets at 10 variables), the CPU time of Andrey’s fits was 1.00 times its time alone, lingam’s 1.00, and the authors’ code’s 1.08, so the ratios against the authors’ code may be up to 8% higher than on an idle machine.

Tables

IAMB

Standardized data, 20 to 100 variables

CPDAG, structure="cpdag", against bnlearn iamb():

Degree	d	n	Datasets	Andrey	bnlearn	Empty graph	D, mean [95%]	Verdict	bnlearn time / Andrey time
2	20	1,000	10	2.5	2.5	20.0	+0.00 [+0.00, +0.00]	tie	1.8
4	20	1,000	10	24.0	23.5	40.0	+0.20 [-0.20, +0.70]	tie	2.5
2	20	5,000	10	2.0	2.0	20.0	+0.00 [+0.00, +0.00]	tie	4.6
4	20	5,000	20	18.0	18.0	40.0	+0.20 [-0.40, +0.75]	tie	15.1
2	50	1,000	20	7.5	7.5	50.0	+0.00 [+0.00, +0.00]	tie	1.1
4	50	1,000	20	50.5	51.0	100.0	+0.05 [-0.55, +0.65]	tie	4.5
2	50	5,000	20	8.5	8.5	50.0	+0.50 [+0.05, +1.05]	worse	5.4
4	50	5,000	19	42.0	41.0	100.0	+0.63 [-0.42, +1.74]	tie	17.6
2	100	1,000	20	32.0	32.5	100.0	-0.20 [-0.75, +0.25]	tie	1.2
4	100	1,000	17	97.0	97.0	200.0	-0.12 [-0.82, +0.59]	tie	4.2
2	100	5,000	14	25.5	25.5	100.0	+0.36 [-0.14, +1.14]	tie	4.6
4	100	5,000	9	71.0	74.0	200.0	+0.00, under 10 datasets	not judged	35.7

Skeleton, structure="skeleton", against bnlearn iamb(undirected=TRUE):

Degree	d	n	Datasets	Andrey	bnlearn	Empty graph	D, mean [95%]	Verdict	bnlearn time / Andrey time
2	20	1,000	10	1.5	1.5	20.0	+0.00 [+0.00, +0.00]	tie	1.3
4	20	1,000	10	15.0	15.0	40.0	+0.00 [+0.00, +0.00]	tie	1.9
2	20	5,000	10	1.0	1.0	20.0	+0.00 [+0.00, +0.00]	tie	4.2
4	20	5,000	20	10.5	10.0	40.0	+0.05 [-0.25, +0.30]	tie	11.4
2	50	1,000	20	6.0	6.0	50.0	+0.00 [+0.00, +0.00]	tie	1.1
4	50	1,000	20	34.0	34.0	100.0	+0.10 [-0.25, +0.45]	tie	3.8
2	50	5,000	20	5.0	5.0	50.0	+0.10 [+0.00, +0.25]	tie	4.4
4	50	5,000	19	24.0	26.0	100.0	-0.26 [-0.79, +0.11]	tie	11.0
2	100	1,000	20	23.0	24.0	100.0	-0.20 [-0.55, +0.00]	tie	1.1
4	100	1,000	17	63.0	63.0	200.0	-0.18 [-0.65, +0.24]	tie	3.5
2	100	5,000	14	19.5	19.0	100.0	+0.07 [+0.00, +0.21]	tie	4.0
4	100	5,000	14	45.5	46.5	200.0	-0.64 [-1.21, -0.14]	better	29.5

Moral graph, structure="moral", against bnlearn learn.mb joined by the AND rule:

Degree	d	n	Datasets	Andrey	bnlearn	Empty graph	D, mean [95%]	Verdict	bnlearn time / Andrey time
2	20	1,000	10	1.5	1.5	31.5	+0.00 [+0.00, +0.00]	tie	0.88
4	20	1,000	10	10.5	10.5	74.0	+0.00 [+0.00, +0.00]	tie	1.1
2	20	5,000	10	1.0	1.0	31.5	-0.10 [-0.30, +0.00]	tie	2.9
4	20	5,000	20	1.5	2.0	71.0	-0.45 [-0.85, -0.05]	better	4.5
2	50	1,000	20	8.0	8.0	80.0	-0.15 [-0.40, +0.00]	tie	0.98
4	50	1,000	20	38.5	37.0	208.5	+0.40 [-0.60, +1.50]	tie	1.5
2	50	5,000	20	6.5	7.0	80.0	-0.20 [-0.70, +0.20]	tie	4.3
4	50	5,000	20	7.5	7.0	208.5	+0.35 [-1.00, +1.65]	tie	1.8
2	100	1,000	20	33.0	33.0	168.5	-0.20 [-0.55, +0.00]	tie	1.0
4	100	1,000	17	125.0	126.0	445.0	-0.88 [-1.71, +0.00]	tie	1.4
2	100	5,000	14	22.0	23.0	170.5	-0.36 [-0.93, +0.07]	tie	3.7
4	100	5,000	20	32.0	33.0	445.0	+2.25 [-0.55, +5.50]	tie	4.5

Unstandardized data, 20 to 100 variables

CPDAG, structure="cpdag", against bnlearn iamb():

Degree	d	n	Datasets	Andrey	bnlearn	Empty graph	D, mean [95%]	Verdict	bnlearn time / Andrey time
2	20	1,000	10	8.0	8.0	20.0	+0.10 [+0.00, +0.30]	tie	2.5
4	20	1,000	10	23.0	22.0	40.0	+0.60 [+0.10, +1.10]	worse, tie on rerun	1.5
2	20	5,000	10	5.0	5.0	20.0	+0.00 [+0.00, +0.00]	tie	2.9
4	20	5,000	10	17.5	17.0	40.0	+0.90 [+0.40, +1.50]	worse, tie on rerun	10.3
2	50	1,000	10	26.5	26.0	50.0	+0.30 [+0.00, +0.60]	tie	1.7
4	50	1,000	10	47.5	47.0	100.0	-0.30 [-1.20, +0.50]	tie	1.9
2	50	5,000	10	24.5	24.5	50.0	-0.20 [-0.60, +0.00]	tie	4.3
4	50	5,000	10	34.0	34.0	100.0	-0.40 [-1.50, +0.50]	tie	9.3
2	100	1,000	10	78.5	79.5	100.0	-0.60 [-1.10, -0.20]	better	0.93
4	100	1,000	10	95.5	95.0	200.0	+0.60 [-0.60, +1.90]	tie	1.8
2	100	5,000	10	68.0	67.5	100.0	-0.10 [-0.90, +0.50]	tie	4.6
4	100	5,000	16	65.5	67.5	200.0	-0.69 [-1.88, +0.31]	tie	21.5

Skeleton, structure="skeleton", against bnlearn iamb(undirected=TRUE):

Degree	d	n	Datasets	Andrey	bnlearn	Empty graph	D, mean [95%]	Verdict	bnlearn time / Andrey time
2	20	1,000	10	5.0	5.0	20.0	+0.00 [+0.00, +0.00]	tie	2.2
4	20	1,000	10	14.5	14.5	40.0	+0.00 [+0.00, +0.00]	tie	1.4
2	20	5,000	10	3.5	3.5	20.0	+0.00 [+0.00, +0.00]	tie	4.1
4	20	5,000	10	11.0	11.0	40.0	+0.00 [+0.00, +0.00]	tie	9.5
2	50	1,000	10	18.0	18.0	50.0	+0.00 [+0.00, +0.00]	tie	1.6
4	50	1,000	10	30.5	30.5	100.0	+0.00 [+0.00, +0.00]	tie	1.6
2	50	5,000	10	15.5	15.5	50.0	+0.00 [+0.00, +0.00]	tie	4.7
4	50	5,000	10	20.0	20.0	100.0	-0.20 [-0.50, +0.00]	tie	7.4
2	100	1,000	10	59.0	59.0	100.0	+0.00 [+0.00, +0.00]	tie	1.1
4	100	1,000	10	68.0	68.0	200.0	+0.00 [+0.00, +0.00]	tie	1.8
2	100	5,000	10	49.5	49.5	100.0	-0.10 [-0.30, +0.00]	tie	3.2
4	100	5,000	16	43.5	44.0	200.0	-0.44 [-0.75, -0.19]	better	13.4

Moral graph, structure="moral", against bnlearn learn.mb joined by the AND rule:

Degree	d	n	Datasets	Andrey	bnlearn	Empty graph	D, mean [95%]	Verdict	bnlearn time / Andrey time
2	20	1,000	10	8.0	8.0	31.5	+0.00 [+0.00, +0.00]	tie	1.0
4	20	1,000	10	31.0	31.0	74.0	+0.00 [+0.00, +0.00]	tie	0.85
2	20	5,000	10	6.0	6.0	31.5	+0.00 [+0.00, +0.00]	tie	2.1
4	20	5,000	10	15.5	15.5	74.0	-0.10 [-0.30, +0.00]	tie	3.9
2	50	1,000	10	35.0	35.0	81.0	+0.00 [+0.00, +0.00]	tie	1.4
4	50	1,000	10	96.0	95.5	215.0	-0.10 [-0.40, +0.20]	tie	1.1
2	50	5,000	10	30.0	30.0	81.0	-0.30 [-0.70, +0.00]	tie	3.3
4	50	5,000	10	58.0	59.0	215.0	-0.60 [-1.00, -0.20]	better	1.9
2	100	1,000	10	106.0	106.0	172.0	+0.00 [+0.00, +0.00]	tie	1.1
4	100	1,000	10	226.5	227.0	454.5	-0.10 [-0.30, +0.00]	tie	1.5
2	100	5,000	10	84.0	84.0	172.0	-0.40 [-0.90, +0.00]	tie	4.2
4	100	5,000	16	150.0	151.0	449.5	-1.00 [-1.88, -0.25]	better	3.4

200 variables

CPDAG, structure="cpdag", against bnlearn iamb():

Degree	d	n	Datasets	Andrey	bnlearn	Empty graph	D, mean [95%]	Verdict	bnlearn time / Andrey time
2	200	1,000	10	92.5	92.5	200.0	+0.00 [+0.00, +0.00]	tie	1.3
4	200	1,000	9	175.0	172.0	400.0	+1.67, under 10 datasets	not judged	2.2
2	200	5,000	10	87.5	87.5	200.0	-0.20 [-1.20, +0.60]	tie	4.2

Skeleton, structure="skeleton", against bnlearn iamb(undirected=TRUE):

Degree	d	n	Datasets	Andrey	bnlearn	Empty graph	D, mean [95%]	Verdict	bnlearn time / Andrey time
2	200	1,000	10	63.0	63.0	200.0	+0.00 [+0.00, +0.00]	tie	1.1
4	200	1,000	10	126.5	125.5	400.0	+0.70 [-0.40, +2.30]	tie	2.0
2	200	5,000	10	60.5	60.5	200.0	-0.20 [-0.50, +0.00]	tie	4.3

Moral graph, structure="moral", against bnlearn learn.mb joined by the AND rule:

Degree	d	n	Datasets	Andrey	bnlearn	Empty graph	D, mean [95%]	Verdict	bnlearn time / Andrey time
2	200	1,000	10	113.5	113.5	325.5	+0.00 [-0.30, +0.30]	tie	1.1
4	200	1,000	10	327.5	331.0	914.0	+0.40 [-2.00, +3.10]	tie	1.5
2	200	5,000	10	94.0	95.5	325.5	-0.90 [-1.60, -0.10]	better	4.2
4	200	5,000	10	116.5	122.5	914.0	-1.30 [-6.20, +3.40]	tie	5.0

Discrete networks

CPDAG, structure="cpdag", against bnlearn iamb(), with Andrey’s chisq and gsq against bnlearn’s x2-adf and mi-adf:

Network	d	n	Test	Datasets	Andrey	bnlearn	Empty graph	D, mean [95%]	Verdict	bnlearn time / Andrey time
asia	8	1,000	chi-square	20	6.0	6.0	8.0	+0.00 [+0.00, +0.00]	tie	2.5
asia	8	1,000	G-square	20	5.0	5.5	8.0	-0.10 [-0.30, +0.10]	tie	1.1
asia	8	5,000	chi-square	20	6.0	6.0	8.0	+0.05 [-0.25, +0.35]	tie	2.3
asia	8	5,000	G-square	20	6.0	6.0	8.0	+0.20 [-0.10, +0.50]	tie	1.1
child	20	1,000	chi-square	20	15.0	17.0	25.0	-2.10 [-3.35, -0.95]	better	0.38
child	20	1,000	G-square	20	14.5	16.5	25.0	-3.20 [-4.30, -2.15]	better	0.10
child	20	5,000	chi-square	20	7.0	13.0	25.0	-5.95 [-8.10, -3.85]	better	0.52
child	20	5,000	G-square	20	6.0	12.5	25.0	-5.55 [-8.10, -2.90]	better	0.15
insurance	27	1,000	chi-square	17	43.0	44.0	52.0	-2.06 [-4.06, -0.12]	better	0.29
insurance	27	1,000	G-square	17	40.0	46.0	52.0	-6.88 [-8.88, -4.82]	better	0.07
insurance	27	5,000	chi-square	20	32.0	40.5	52.0	-6.65 [-9.30, -3.85]	better	0.42
insurance	27	5,000	G-square	20	34.0	39.0	52.0	-6.70 [-9.75, -3.80]	better	0.12
alarm	37	1,000	chi-square	20	29.0	29.5	46.0	-0.85 [-3.20, +1.45]	tie	0.30
alarm	37	1,000	G-square	20	28.0	31.0	46.0	-2.85 [-4.55, -1.10]	better	0.10
alarm	37	5,000	chi-square	20	20.5	23.0	46.0	-2.75 [-5.70, -0.15]	better	0.43
alarm	37	5,000	G-square	20	18.5	18.0	46.0	+1.00 [-1.35, +3.65]	tie	0.15
hepar2	70	1,000	chi-square	15	117.0	117.0	123.0	-2.27 [-4.73, +0.20]	tie	0.28
hepar2	70	1,000	G-square	14	108.0	114.5	123.0	-4.36 [-7.43, -1.36]	better	0.10
hepar2	70	5,000	chi-square	16	94.0	97.5	123.0	-4.31 [-7.56, -1.12]	better	0.35
hepar2	70	5,000	G-square	17	89.0	94.0	123.0	-6.24 [-8.76, -3.41]	better	0.24

Skeleton, structure="skeleton", against bnlearn iamb(undirected=TRUE), with Andrey’s chisq and gsq against bnlearn’s x2-adf and mi-adf:

Network	d	n	Test	Datasets	Andrey	bnlearn	Empty graph	D, mean [95%]	Verdict	bnlearn time / Andrey time
asia	8	1,000	chi-square	20	4.0	4.0	8.0	+0.00 [+0.00, +0.00]	tie	2.5
asia	8	1,000	G-square	20	4.0	4.0	8.0	-0.15 [-0.35, +0.05]	tie	0.90
asia	8	5,000	chi-square	20	4.0	4.0	8.0	+0.05 [-0.25, +0.35]	tie	2.3
asia	8	5,000	G-square	20	4.0	4.0	8.0	+0.20 [-0.10, +0.50]	tie	1.2
child	20	1,000	chi-square	20	6.0	10.0	25.0	-4.00 [-4.60, -3.45]	better	0.40
child	20	1,000	G-square	20	5.5	10.0	25.0	-3.90 [-4.55, -3.20]	better	0.11
child	20	5,000	chi-square	20	2.0	7.0	25.0	-5.30 [-6.25, -4.40]	better	0.54
child	20	5,000	G-square	20	2.0	6.0	25.0	-4.55 [-5.65, -3.50]	better	0.14
insurance	27	1,000	chi-square	17	26.0	32.0	52.0	-5.94 [-7.00, -4.82]	better	0.28
insurance	27	1,000	G-square	17	27.0	34.0	52.0	-7.29 [-8.24, -6.47]	better	0.07
insurance	27	5,000	chi-square	20	18.0	26.0	52.0	-8.50 [-10.05, -6.95]	better	0.44
insurance	27	5,000	G-square	20	18.0	27.0	52.0	-8.95 [-10.15, -7.80]	better	0.12
alarm	37	1,000	chi-square	20	16.0	17.0	46.0	-1.25 [-2.10, -0.35]	better	0.30
alarm	37	1,000	G-square	20	14.0	17.0	46.0	-2.60 [-3.75, -1.50]	better	0.09
alarm	37	5,000	chi-square	20	12.0	14.0	46.0	-3.05 [-4.70, -1.40]	better	0.47
alarm	37	5,000	G-square	20	8.0	10.0	46.0	-1.95 [-2.80, -1.10]	better	0.16
hepar2	70	1,000	chi-square	15	96.0	98.0	123.0	-1.33 [-3.13, +0.47]	tie	0.27
hepar2	70	1,000	G-square	14	92.5	95.0	123.0	-3.07 [-4.50, -1.71]	better	0.10
hepar2	70	5,000	chi-square	16	74.5	77.0	123.0	-3.25 [-5.25, -1.44]	better	0.38
hepar2	70	5,000	G-square	17	67.0	73.0	123.0	-5.41 [-7.47, -3.35]	better	0.24

Moral graph, structure="moral", against bnlearn learn.mb joined by the AND rule, with Andrey’s chisq and gsq against bnlearn’s x2-adf and mi-adf:

Network	d	n	Test	Datasets	Andrey	bnlearn	Empty graph	D, mean [95%]	Verdict	bnlearn time / Andrey time
asia	8	1,000	chi-square	20	5.0	5.0	10.0	+0.00 [+0.00, +0.00]	tie	1.1
asia	8	1,000	G-square	20	5.0	5.0	10.0	-0.15 [-0.35, +0.05]	tie	0.43
asia	8	5,000	chi-square	20	5.0	5.0	10.0	+0.10 [-0.20, +0.40]	tie	1.7
asia	8	5,000	G-square	20	5.0	5.0	10.0	+0.20 [-0.15, +0.55]	tie	0.77
child	20	1,000	chi-square	20	11.0	15.5	30.0	-4.45 [-5.20, -3.65]	better	0.36
child	20	1,000	G-square	20	10.0	15.0	30.0	-5.05 [-5.70, -4.35]	better	0.10
child	20	5,000	chi-square	20	5.5	11.5	30.0	-5.90 [-7.15, -4.60]	better	0.57
child	20	5,000	G-square	20	5.0	11.0	30.0	-5.65 [-6.70, -4.60]	better	0.16
insurance	27	1,000	chi-square	17	38.0	47.0	70.0	-9.88 [-10.88, -8.94]	better	0.32
insurance	27	1,000	G-square	17	38.0	48.0	70.0	-10.29 [-11.18, -9.41]	better	0.09
insurance	27	5,000	chi-square	20	28.5	42.0	70.0	-13.30 [-14.65, -11.95]	better	0.47
insurance	27	5,000	G-square	20	29.0	42.5	70.0	-13.25 [-14.40, -12.05]	better	0.13
alarm	37	1,000	chi-square	20	36.5	33.5	65.0	+1.30 [+0.15, +2.55]	worse, tie on rerun	0.35
alarm	37	1,000	G-square	20	30.0	31.5	65.0	-2.25 [-3.65, -0.80]	better	0.11
alarm	37	5,000	chi-square	20	32.0	30.0	65.0	+2.15 [-0.30, +4.60]	tie	0.48
alarm	37	5,000	G-square	20	20.0	23.5	65.0	-2.45 [-4.05, -0.90]	better	0.18
hepar2	70	1,000	chi-square	15	142.0	135.0	158.0	+5.73 [+3.27, +8.40]	worse, rerun has 9 pairs	0.41
hepar2	70	1,000	G-square	14	129.5	130.5	158.0	-2.14 [-3.64, -0.71]	better	0.15
hepar2	70	5,000	chi-square	16	122.0	116.5	158.0	+5.81 [+1.81, +9.62]	worse, worse on rerun	0.41
hepar2	70	5,000	G-square	17	101.0	107.0	158.0	-8.35 [-11.82, -5.06]	better	0.28

MMPC

Standardized data, 20 to 100 variables

Skeleton, structure="skeleton", the default, against bnlearn mmpc(undirected=TRUE):

Degree	d	n	Datasets	Andrey	bnlearn	Empty graph	D, mean [95%]	Verdict	bnlearn time / Andrey time
2	20	1,000	20	2.5	2.5	20.0	+0.05 [-0.10, +0.20]	tie	4.8
4	20	1,000	10	20.5	21.5	40.0	-0.50 [-1.10, +0.00]	tie	4.1
2	20	5,000	20	1.0	1.0	20.0	+0.00 [-0.25, +0.25]	tie	13.1
4	20	5,000	20	20.5	20.0	40.0	-0.75 [-2.05, +0.60]	tie	22.2
2	50	1,000	20	8.5	8.0	50.0	+0.10 [-0.10, +0.35]	tie	6.7
4	50	1,000	20	50.5	51.0	100.0	+0.35 [-0.40, +1.15]	tie	5.3
2	50	5,000	20	6.5	7.0	50.0	-0.10 [-0.40, +0.25]	tie	14.2
4	50	5,000	20	47.0	47.5	100.0	-0.15 [-1.55, +1.30]	tie	5.8
2	100	1,000	20	27.0	27.0	100.0	+0.20 [-0.10, +0.50]	tie	6.4
4	100	1,000	20	91.5	92.5	200.0	+0.60 [-0.40, +1.65]	tie	8.5
2	100	5,000	16	23.0	23.5	100.0	+0.31 [-0.25, +0.81]	tie	22.6
4	100	5,000	20	82.5	84.5	200.0	-0.30 [-2.30, +1.55]	tie	29.4

CPDAG, structure="cpdag", against bnlearn mmpc(undirected=FALSE):

Degree	d	n	Datasets	Andrey	bnlearn	Empty graph	D, mean [95%]	Verdict	bnlearn time / Andrey time
2	20	1,000	20	5.5	6.5	20.0	-0.10 [-0.55, +0.20]	tie	4.2
4	20	1,000	10	31.5	30.5	40.0	+0.10 [-1.60, +1.60]	tie	4.9
2	20	5,000	20	2.5	2.5	20.0	+0.20 [-0.45, +0.85]	tie	12.3
4	20	5,000	20	29.0	29.5	40.0	-0.90 [-2.60, +0.70]	tie	22.0
2	50	1,000	20	13.5	13.0	50.0	+0.25 [-0.30, +1.00]	tie	7.2
4	50	1,000	19	66.0	67.0	100.0	-0.16 [-1.79, +1.68]	tie	18.8
2	50	5,000	20	7.0	7.5	50.0	+0.25 [-0.30, +0.95]	tie	25.0
4	50	5,000	10	66.0	68.0	100.0	-1.40 [-5.90, +3.10]	tie	26.3
2	100	1,000	20	37.0	36.0	100.0	+0.70 [+0.10, +1.55]	worse, rerun has 9 pairs	17.9
4	100	1,000	2	127.5	118.0	200.0	+9.50, under 10 datasets	not judged	27.1
2	100	5,000	14	27.0	28.5	100.0	+0.57 [-0.50, +1.79]	tie	98.7
4	100	5,000	0	-	-	-	no pairs	not judged	-

Unstandardized data, 20 to 100 variables

Skeleton, structure="skeleton", the default, against bnlearn mmpc(undirected=TRUE):

Degree	d	n	Datasets	Andrey	bnlearn	Empty graph	D, mean [95%]	Verdict	bnlearn time / Andrey time
2	20	1,000	10	4.5	4.5	20.0	+0.00 [+0.00, +0.00]	tie	3.8
4	20	1,000	10	16.5	16.5	40.0	+0.00 [+0.00, +0.00]	tie	3.2
2	20	5,000	10	3.0	3.0	20.0	-0.20 [-0.50, +0.00]	tie	5.8
4	20	5,000	10	12.0	13.0	40.0	+0.30 [-0.70, +1.20]	tie	10.4
2	50	1,000	10	16.0	16.0	50.0	+0.00 [+0.00, +0.00]	tie	5.9
4	50	1,000	10	31.0	31.0	100.0	+0.20 [+0.00, +0.60]	tie	4.9
2	50	5,000	10	15.0	15.0	50.0	+0.30 [+0.00, +0.80]	tie	6.6
4	50	5,000	14	22.5	23.5	100.0	-0.50 [-1.00, -0.14]	better	2.4
2	100	1,000	10	57.5	57.5	100.0	+0.00 [+0.00, +0.00]	tie	5.3
4	100	1,000	14	65.0	65.0	200.0	+0.00 [+0.00, +0.00]	tie	4.4
2	100	5,000	14	49.0	50.5	100.0	-0.07 [-0.57, +0.29]	tie	12.0
4	100	5,000	14	43.0	42.5	200.0	+0.29 [-0.14, +0.71]	tie	6.2

CPDAG, structure="cpdag", against bnlearn mmpc(undirected=FALSE):

Degree	d	n	Datasets	Andrey	bnlearn	Empty graph	D, mean [95%]	Verdict	bnlearn time / Andrey time
2	20	1,000	10	6.5	6.0	20.0	+0.10 [-0.30, +0.60]	tie	4.7
4	20	1,000	10	23.0	23.5	40.0	+0.00 [-0.30, +0.30]	tie	7.4
2	20	5,000	10	6.0	6.5	20.0	-0.20 [-0.50, +0.00]	tie	6.8
4	20	5,000	10	21.0	18.5	40.0	+1.50 [-0.30, +3.50]	tie	57.5
2	50	1,000	10	25.0	25.0	50.0	+0.00 [+0.00, +0.00]	tie	6.6
4	50	1,000	9	53.0	50.0	100.0	+0.44, under 10 datasets	not judged	26.9
2	50	5,000	10	26.0	25.5	50.0	+0.70 [-0.20, +2.20]	tie	27.2
4	50	5,000	0	-	-	-	no pairs	not judged	-
2	100	1,000	8	73.5	73.5	100.0	+0.12, under 10 datasets	not judged	20.7
4	100	1,000	0	-	-	-	no pairs	not judged	-
2	100	5,000	3	66.0	70.0	100.0	-2.00, under 10 datasets	not judged	111.0
4	100	5,000	0	-	-	-	no pairs	not judged	-

200 variables

Skeleton, structure="skeleton", the default, against bnlearn mmpc(undirected=TRUE):

Degree	d	n	Datasets	Andrey	bnlearn	Empty graph	D, mean [95%]	Verdict	bnlearn time / Andrey time
2	200	1,000	10	68.5	68.5	200.0	+0.40 [+0.00, +0.80]	tie	8.8
4	200	1,000	10	162.5	164.0	400.0	-1.70 [-3.90, +0.20]	tie	6.4
2	200	5,000	10	70.5	67.0	200.0	+1.60 [+0.10, +3.10]	worse, worse on rerun	23.4
4	200	5,000	10	136.5	140.0	400.0	-2.70 [-7.00, +1.50]	tie	52.9

CPDAG, structure="cpdag", against bnlearn mmpc(undirected=FALSE):

Degree	d	n	Datasets	Andrey	bnlearn	Empty graph	D, mean [95%]	Verdict	bnlearn time / Andrey time
2	200	1,000	5	89.0	88.0	200.0	+0.00, under 10 datasets	not judged	29.1

Discrete networks

Skeleton, structure="skeleton", the default, against bnlearn mmpc(undirected=TRUE), with Andrey’s chisq and gsq against bnlearn’s x2-adf and mi-adf:

Network	d	n	Test	Datasets	Andrey	bnlearn	Empty graph	D, mean [95%]	Verdict	bnlearn time / Andrey time
asia	8	1,000	chi-square	20	3.0	3.0	8.0	+0.00 [+0.00, +0.00]	tie	3.0
asia	8	1,000	G-square	20	3.0	3.0	8.0	-0.20 [-0.50, +0.05]	tie	1.6
asia	8	5,000	chi-square	20	2.0	2.0	8.0	+0.00 [+0.00, +0.00]	tie	3.4
asia	8	5,000	G-square	20	2.5	2.5	8.0	+0.00 [+0.00, +0.00]	tie	2.0
child	20	1,000	chi-square	10	5.0	9.5	25.0	-4.80 [-5.20, -4.40]	better	0.66
child	20	1,000	G-square	20	5.0	10.0	25.0	-4.90 [-5.30, -4.45]	better	0.15
child	20	5,000	chi-square	20	1.0	5.0	25.0	-4.45 [-5.25, -3.70]	better	0.68
child	20	5,000	G-square	20	1.0	5.0	25.0	-3.70 [-4.45, -3.05]	better	0.19
insurance	27	1,000	chi-square	17	24.0	30.0	52.0	-6.29 [-7.06, -5.53]	better	0.58
insurance	27	1,000	G-square	17	23.0	29.0	52.0	-7.18 [-8.18, -6.18]	better	0.14
insurance	27	5,000	chi-square	20	15.0	24.0	52.0	-9.10 [-9.90, -8.30]	better	0.55
insurance	27	5,000	G-square	20	15.0	22.5	52.0	-7.85 [-8.70, -6.95]	better	0.16
alarm	37	1,000	chi-square	20	11.0	16.0	46.0	-5.05 [-5.55, -4.60]	better	0.68
alarm	37	1,000	G-square	20	10.0	15.0	46.0	-5.05 [-5.50, -4.60]	better	0.20
alarm	37	5,000	chi-square	20	5.0	11.0	46.0	-6.25 [-7.20, -5.35]	better	0.89
alarm	37	5,000	G-square	20	5.0	8.0	46.0	-3.40 [-4.00, -2.80]	better	0.29
hepar2	70	1,000	chi-square	15	96.0	99.0	123.0	-1.47 [-2.20, -0.80]	better	0.83
hepar2	70	1,000	G-square	15	93.0	97.0	123.0	-1.47 [-2.60, -0.40]	better	0.29
hepar2	70	5,000	chi-square	20	70.5	73.5	123.0	-3.70 [-4.35, -3.05]	better	0.74
hepar2	70	5,000	G-square	20	68.0	72.0	123.0	-3.60 [-4.55, -2.65]	better	0.29

CPDAG, structure="cpdag", against bnlearn mmpc(undirected=FALSE), with Andrey’s chisq and gsq against bnlearn’s x2-adf and mi-adf:

Network	d	n	Test	Datasets	Andrey	bnlearn	Empty graph	D, mean [95%]	Verdict	bnlearn time / Andrey time
asia	8	1,000	chi-square	20	4.0	4.0	8.0	+0.00 [+0.00, +0.00]	tie	3.0
asia	8	1,000	G-square	20	4.0	4.0	8.0	-0.20 [-0.50, +0.05]	tie	1.4
asia	8	5,000	chi-square	20	3.0	3.0	8.0	+0.00 [+0.00, +0.00]	tie	3.1
asia	8	5,000	G-square	20	3.5	3.5	8.0	+0.00 [+0.00, +0.00]	tie	1.9
child	20	1,000	chi-square	10	13.5	19.0	25.0	-5.40 [-6.60, -4.00]	better	0.47
child	20	1,000	G-square	20	14.0	19.0	25.0	-5.45 [-6.25, -4.60]	better	0.11
child	20	5,000	chi-square	20	9.5	17.0	25.0	-7.80 [-8.70, -6.95]	better	0.33
child	20	5,000	G-square	20	10.5	16.0	25.0	-6.85 [-7.70, -6.10]	better	0.10
insurance	27	1,000	chi-square	17	34.0	47.0	52.0	-11.82 [-13.47, -10.18]	better	0.26
insurance	27	1,000	G-square	17	34.0	46.0	52.0	-12.88 [-14.24, -11.53]	better	0.06
insurance	27	5,000	chi-square	20	27.0	44.0	52.0	-17.35 [-19.70, -15.00]	better	0.08
insurance	27	5,000	G-square	20	23.5	43.0	52.0	-18.35 [-20.30, -16.30]	better	0.02
alarm	37	1,000	chi-square	20	28.0	38.0	46.0	-11.55 [-14.00, -9.30]	better	0.49
alarm	37	1,000	G-square	20	26.5	38.0	46.0	-11.95 [-13.40, -10.45]	better	0.16
alarm	37	5,000	chi-square	20	11.5	30.0	46.0	-18.10 [-20.35, -16.05]	better	0.48
alarm	37	5,000	G-square	20	11.5	26.0	46.0	-15.40 [-18.10, -12.75]	better	0.18
hepar2	70	1,000	chi-square	15	126.0	128.0	123.0	-2.80 [-6.87, +1.00]	tie	0.80
hepar2	70	1,000	G-square	15	119.0	128.0	123.0	-3.60 [-11.07, +3.80]	tie	0.24
hepar2	70	5,000	chi-square	20	94.5	98.0	123.0	-9.45 [-14.75, -4.80]	better	0.16
hepar2	70	5,000	G-square	20	92.0	95.0	123.0	-3.60 [-5.10, -2.20]	better	0.06

Inter-IAMB

Discrete networks

CPDAG, structure="cpdag", against bnlearn inter.iamb(), with Andrey’s chisq and gsq against bnlearn’s x2-adf and mi-adf:

Network	d	n	Test	Datasets	Andrey	bnlearn	Empty graph	D, mean [95%]	Verdict	bnlearn time / Andrey time
asia	8	1,000	chi-square	20	6.0	6.0	8.0	+0.00 [+0.00, +0.00]	tie	2.4
asia	8	1,000	G-square	20	5.0	5.5	8.0	-0.10 [-0.30, +0.10]	tie	0.83
asia	8	5,000	chi-square	20	6.0	6.0	8.0	+0.05 [-0.15, +0.30]	tie	1.9
asia	8	5,000	G-square	20	6.0	6.0	8.0	+0.15 [-0.10, +0.40]	tie	0.88
child	20	1,000	chi-square	20	15.5	17.0	25.0	-2.65 [-4.30, -1.15]	better	0.36
child	20	1,000	G-square	20	14.0	16.5	25.0	-3.60 [-4.75, -2.50]	better	0.09
child	20	5,000	chi-square	20	5.0	13.0	25.0	-6.90 [-9.55, -4.20]	better	0.43
child	20	5,000	G-square	20	6.0	12.5	25.0	-5.60 [-7.50, -3.50]	better	0.15
insurance	27	1,000	chi-square	17	40.0	44.0	52.0	-3.65 [-5.47, -2.00]	better	0.27
insurance	27	1,000	G-square	17	39.0	46.0	52.0	-6.94 [-8.59, -5.24]	better	0.07
insurance	27	5,000	chi-square	20	31.5	39.0	52.0	-7.35 [-9.45, -5.25]	better	0.36
insurance	27	5,000	G-square	20	30.0	38.0	52.0	-8.05 [-10.90, -5.45]	better	0.10
alarm	37	1,000	chi-square	20	28.5	29.5	46.0	-2.15 [-4.90, +0.45]	tie	0.27
alarm	37	1,000	G-square	20	28.0	31.0	46.0	-4.15 [-6.15, -2.15]	better	0.09
alarm	37	5,000	chi-square	20	18.0	21.0	46.0	-2.70 [-4.75, -0.80]	better	0.38
alarm	37	5,000	G-square	20	17.0	14.0	46.0	+2.50 [+1.10, +3.95]	worse, tie on rerun	0.15
hepar2	70	1,000	chi-square	15	116.0	117.0	123.0	-2.60 [-5.40, -0.07]	better	0.29
hepar2	70	1,000	G-square	12	111.5	113.5	123.0	-1.92 [-5.67, +1.75]	tie	0.09
hepar2	70	5,000	chi-square	15	89.0	97.0	123.0	-7.73 [-10.73, -4.47]	better	0.38
hepar2	70	5,000	G-square	15	89.0	95.0	123.0	-6.20 [-8.93, -3.73]	better	0.22

Skeleton, structure="skeleton", against bnlearn inter.iamb(undirected=TRUE), with Andrey’s chisq and gsq against bnlearn’s x2-adf and mi-adf:

Network	d	n	Test	Datasets	Andrey	bnlearn	Empty graph	D, mean [95%]	Verdict	bnlearn time / Andrey time
asia	8	1,000	chi-square	20	4.0	4.0	8.0	+0.00 [+0.00, +0.00]	tie	2.1
asia	8	1,000	G-square	20	4.0	4.0	8.0	-0.15 [-0.35, +0.05]	tie	0.88
asia	8	5,000	chi-square	20	4.0	4.0	8.0	+0.05 [-0.15, +0.30]	tie	1.9
asia	8	5,000	G-square	20	4.0	4.0	8.0	+0.15 [-0.10, +0.40]	tie	0.96
child	20	1,000	chi-square	20	6.0	10.0	25.0	-4.70 [-5.30, -4.15]	better	0.30
child	20	1,000	G-square	20	5.0	10.0	25.0	-4.40 [-4.95, -3.80]	better	0.09
child	20	5,000	chi-square	20	1.5	6.0	25.0	-5.80 [-6.65, -5.00]	better	0.51
child	20	5,000	G-square	20	1.0	6.0	25.0	-5.25 [-6.10, -4.45]	better	0.14
insurance	27	1,000	chi-square	17	25.0	32.0	52.0	-6.88 [-7.82, -5.88]	better	0.28
insurance	27	1,000	G-square	17	26.0	34.0	52.0	-8.47 [-9.12, -7.76]	better	0.07
insurance	27	5,000	chi-square	20	16.0	24.0	52.0	-8.65 [-9.75, -7.55]	better	0.35
insurance	27	5,000	G-square	20	14.5	24.0	52.0	-9.30 [-10.45, -8.05]	better	0.10
alarm	37	1,000	chi-square	20	14.0	17.0	46.0	-2.75 [-3.80, -1.75]	better	0.25
alarm	37	1,000	G-square	20	13.5	17.0	46.0	-3.30 [-4.65, -2.00]	better	0.10
alarm	37	5,000	chi-square	20	9.5	13.0	46.0	-3.35 [-4.80, -1.95]	better	0.39
alarm	37	5,000	G-square	20	7.0	8.0	46.0	-1.05 [-1.60, -0.50]	better	0.15
hepar2	70	1,000	chi-square	15	97.0	98.0	123.0	-1.27 [-3.27, +1.00]	tie	0.25
hepar2	70	1,000	G-square	12	92.5	95.0	123.0	-2.83 [-4.75, -1.00]	better	0.09
hepar2	70	5,000	chi-square	15	72.0	77.0	123.0	-6.07 [-7.60, -4.40]	better	0.38
hepar2	70	5,000	G-square	15	67.0	74.0	123.0	-5.33 [-7.00, -3.67]	better	0.23

Moral graph, structure="moral", against bnlearn learn.mb with inter.iamb, joined by the AND rule, with Andrey’s chisq and gsq against bnlearn’s x2-adf and mi-adf:

Network	d	n	Test	Datasets	Andrey	bnlearn	Empty graph	D, mean [95%]	Verdict	bnlearn time / Andrey time
asia	8	1,000	chi-square	20	5.0	5.0	10.0	+0.00 [+0.00, +0.00]	tie	1.0
asia	8	1,000	G-square	20	5.0	5.0	10.0	-0.10 [-0.30, +0.10]	tie	0.37
asia	8	5,000	chi-square	20	5.0	5.0	10.0	+0.05 [-0.15, +0.30]	tie	1.3
asia	8	5,000	G-square	20	5.0	5.0	10.0	+0.10 [-0.20, +0.40]	tie	0.56
child	20	1,000	chi-square	20	10.0	15.5	30.0	-5.60 [-6.35, -4.90]	better	0.38
child	20	1,000	G-square	20	9.0	15.0	30.0	-5.65 [-6.45, -4.90]	better	0.10
child	20	5,000	chi-square	20	5.0	11.0	30.0	-6.05 [-7.05, -5.10]	better	0.54
child	20	5,000	G-square	20	4.5	10.5	30.0	-6.20 [-7.10, -5.30]	better	0.15
insurance	27	1,000	chi-square	17	36.0	47.0	70.0	-11.12 [-11.94, -10.24]	better	0.30
insurance	27	1,000	G-square	17	37.0	49.0	70.0	-11.94 [-12.47, -11.35]	better	0.07
insurance	27	5,000	chi-square	20	28.0	40.0	70.0	-12.80 [-13.80, -11.80]	better	0.40
insurance	27	5,000	G-square	20	25.0	39.5	70.0	-14.00 [-15.30, -12.55]	better	0.11
alarm	37	1,000	chi-square	20	33.0	33.5	65.0	-2.10 [-3.60, -0.60]	better	0.31
alarm	37	1,000	G-square	20	28.0	31.5	65.0	-3.70 [-5.20, -2.20]	better	0.11
alarm	37	5,000	chi-square	20	25.0	27.0	65.0	-1.90 [-3.75, -0.20]	better	0.43
alarm	37	5,000	G-square	20	18.5	21.0	65.0	-3.10 [-3.85, -2.35]	better	0.17
hepar2	70	1,000	chi-square	15	140.0	137.0	158.0	+4.67 [+1.47, +8.33]	worse, rerun has 9 pairs	0.38
hepar2	70	1,000	G-square	12	129.0	130.5	158.0	-1.67 [-3.33, +0.08]	tie	0.14
hepar2	70	5,000	chi-square	15	116.0	117.0	158.0	-0.53 [-3.80, +2.87]	tie	0.46
hepar2	70	5,000	G-square	15	100.0	109.0	158.0	-9.40 [-11.67, -7.20]	better	0.27

MMHC

Discrete networks

CPDAG with the defaults, against bnlearn mmhc(), with Andrey’s chisq and gsq against bnlearn’s x2-adf and mi-adf:

Network	d	n	Test	Datasets	Andrey	bnlearn	Empty graph	D, mean [95%]	Verdict	bnlearn time / Andrey time
asia	8	1,000	chi-square	20	4.5	4.0	8.0	+0.25 [-0.65, +1.10]	tie	2.2
asia	8	1,000	G-square	10	5.5	5.5	8.0	+0.20 [-1.00, +1.40]	tie	1.2
asia	8	5,000	chi-square	10	4.0	4.0	8.0	+0.20 [-1.40, +1.80]	tie	2.4
asia	8	5,000	G-square	10	4.0	4.0	8.0	+0.40 [-1.40, +2.20]	tie	1.7
child	20	1,000	chi-square	20	17.5	19.0	25.0	-2.30 [-4.00, -0.80]	better	0.63
child	20	1,000	G-square	20	17.0	19.0	25.0	-2.45 [-3.95, -1.10]	better	0.17
child	20	5,000	chi-square	10	12.0	11.0	25.0	-2.30 [-6.60, +1.70]	tie	0.70
child	20	5,000	G-square	10	12.5	10.0	25.0	-0.20 [-4.40, +3.50]	tie	0.20
insurance	27	1,000	chi-square	17	37.0	40.0	52.0	-4.35 [-7.00, -1.82]	better	0.52
insurance	27	1,000	G-square	17	36.0	41.0	52.0	-5.29 [-7.29, -3.29]	better	0.16
insurance	27	5,000	chi-square	20	30.5	34.0	52.0	-3.75 [-6.65, -0.65]	better	0.60
insurance	27	5,000	G-square	10	32.0	35.5	52.0	-4.40 [-6.70, -2.40]	better	0.17
alarm	37	1,000	chi-square	10	23.5	29.0	46.0	-5.00 [-6.80, -3.10]	better	0.61
alarm	37	1,000	G-square	20	23.5	26.0	46.0	-2.60 [-4.20, -1.05]	better	0.20
alarm	37	5,000	chi-square	20	19.5	22.0	46.0	-3.00 [-5.55, -0.60]	better	0.78
alarm	37	5,000	G-square	20	20.0	22.0	46.0	-0.80 [-3.00, +1.35]	tie	0.31
hepar2	70	1,000	chi-square	15	110.0	112.0	123.0	-1.20 [-3.60, +1.07]	tie	0.83
hepar2	70	1,000	G-square	15	112.0	111.0	123.0	+0.13 [-2.33, +2.40]	tie	0.31
hepar2	70	5,000	chi-square	20	90.0	92.5	123.0	-3.60 [-6.70, -1.00]	better	0.77
hepar2	70	5,000	G-square	20	90.0	92.0	123.0	-3.60 [-6.30, -1.50]	better	0.28

RSMAX2

Discrete networks

CPDAG with restrict="mmpc" and maximize="hc", the defaults, against bnlearn rsmax2() with the same phases, with Andrey’s chisq and gsq against bnlearn’s x2-adf and mi-adf:

Network	d	n	Test	Datasets	Andrey	bnlearn	Empty graph	D, mean [95%]	Verdict	bnlearn time / Andrey time
asia	8	1,000	chi-square	10	5.0	5.0	8.0	+0.20 [-1.00, +1.40]	tie	2.1
asia	8	1,000	G-square	20	5.0	4.0	8.0	+0.30 [-0.60, +1.15]	tie	1.3
asia	8	5,000	chi-square	10	4.0	4.0	8.0	+0.20 [-1.40, +1.80]	tie	2.4
asia	8	5,000	G-square	10	4.0	4.0	8.0	+0.40 [-1.40, +2.20]	tie	1.3
child	20	1,000	chi-square	20	17.5	19.0	25.0	-2.30 [-4.05, -0.75]	better	0.63
child	20	1,000	G-square	10	17.0	19.0	25.0	-2.30 [-4.40, -0.60]	better	0.19
child	20	5,000	chi-square	20	12.0	13.5	25.0	-2.10 [-4.55, +0.10]	tie	0.69
child	20	5,000	G-square	20	12.0	10.5	25.0	-0.40 [-2.85, +1.85]	tie	0.20
insurance	27	1,000	chi-square	8	37.0	40.5	52.0	-5.75, under 10 datasets	not judged	0.55
insurance	27	1,000	G-square	17	36.0	41.0	52.0	-5.29 [-7.35, -3.29]	better	0.18
insurance	27	5,000	chi-square	20	30.5	34.0	52.0	-3.75 [-6.65, -0.65]	better	0.58
insurance	27	5,000	G-square	20	31.5	34.5	52.0	-3.40 [-6.05, -0.60]	better	0.17
alarm	37	1,000	chi-square	20	23.5	29.0	46.0	-4.75 [-6.45, -2.85]	better	0.74
alarm	37	1,000	G-square	20	23.5	26.0	46.0	-2.60 [-4.15, -1.05]	better	0.20
alarm	37	5,000	chi-square	20	19.5	22.0	46.0	-3.00 [-5.50, -0.60]	better	0.82
alarm	37	5,000	G-square	20	20.0	22.0	46.0	-0.80 [-3.00, +1.30]	tie	0.30
hepar2	70	1,000	chi-square	15	110.0	112.0	123.0	-1.20 [-3.47, +1.13]	tie	0.86
hepar2	70	1,000	G-square	15	112.0	111.0	123.0	+0.13 [-2.33, +2.47]	tie	0.31
hepar2	70	5,000	chi-square	20	90.0	92.5	123.0	-3.60 [-6.70, -1.00]	better	0.74
hepar2	70	5,000	G-square	20	90.0	92.0	123.0	-3.60 [-6.25, -1.50]	better	0.26

CPDAG with restrict="iamb" and maximize="hc", against bnlearn rsmax2(restrict="iamb"), with Andrey’s chisq and gsq against bnlearn’s x2-adf and mi-adf:

Network	d	n	Test	Datasets	Andrey	bnlearn	Empty graph	D, mean [95%]	Verdict	bnlearn time / Andrey time
asia	8	1,000	chi-square	10	6.0	5.0	8.0	+0.30 [-0.60, +1.30]	tie	1.9
asia	8	1,000	G-square	20	5.0	5.0	8.0	+0.30 [-0.40, +1.00]	tie	0.87
asia	8	5,000	chi-square	10	5.5	5.0	8.0	-0.10 [-1.30, +1.00]	tie	1.8
asia	8	5,000	G-square	10	6.0	5.0	8.0	+0.20 [-1.20, +1.50]	tie	1.0
child	20	1,000	chi-square	20	17.5	19.0	25.0	-2.05 [-3.50, -0.85]	better	0.40
child	20	1,000	G-square	10	17.0	19.0	25.0	-2.50 [-4.50, -0.90]	better	0.12
child	20	5,000	chi-square	20	13.0	15.0	25.0	-2.40 [-4.85, -0.25]	better	0.51
child	20	5,000	G-square	20	14.0	12.5	25.0	-0.60 [-3.35, +1.90]	tie	0.18
insurance	27	1,000	chi-square	8	41.5	46.5	52.0	-5.75, under 10 datasets	not judged	0.35
insurance	27	1,000	G-square	17	43.0	48.0	52.0	-5.82 [-8.18, -3.35]	better	0.09
insurance	27	5,000	chi-square	20	35.5	40.0	52.0	-6.20 [-9.65, -2.60]	better	0.39
insurance	27	5,000	G-square	20	35.0	41.5	52.0	-6.40 [-10.15, -2.80]	better	0.13
alarm	37	1,000	chi-square	20	29.0	30.5	46.0	-1.10 [-3.20, +1.05]	tie	0.30
alarm	37	1,000	G-square	20	27.5	30.0	46.0	-1.45 [-3.80, +0.70]	tie	0.10
alarm	37	5,000	chi-square	20	21.5	25.5	46.0	-3.40 [-5.90, -0.85]	better	0.42
alarm	37	5,000	G-square	20	20.5	21.0	46.0	-0.20 [-2.15, +1.85]	tie	0.18
hepar2	70	1,000	chi-square	15	112.0	112.0	123.0	-1.33 [-3.87, +1.20]	tie	0.27
hepar2	70	1,000	G-square	15	112.0	112.0	123.0	-2.07 [-4.53, +0.47]	tie	0.12
hepar2	70	5,000	chi-square	20	93.0	95.0	123.0	-2.80 [-5.30, -0.55]	better	0.43
hepar2	70	5,000	G-square	20	90.0	94.5	123.0	-4.95 [-7.90, -2.50]	better	0.26

CAM

Andrey’s CAM against CAM 1.0 at 1,000 rows and 20 datasets per setting. D is Andrey’s SHD minus CAM 1.0’s on the same dataset. Orientation is the share of true edges returned with the right direction. The settings outside the model carry no verdict.

Standardized columns, in the claim

Additive noise, Gaussian:

d	Pairs	SHD: Andrey / CAM 1.0 / empty	D, mean [95%]	Verdict	Orientation: Andrey / CAM 1.0	Same graph	CPU s: Andrey / CAM 1.0	Time ratio [95%]
10	20	1.35 / 1.35 / 10.00	+0.00 [+0.00, +0.00]	tie	0.915 / 0.915	20/20	0.677 / 5.2	7.2x [6.6, 7.6]
20	20	4.25 / 4.25 / 20.00	+0.00 [+0.00, +0.00]	tie	0.897 / 0.897	19/20	3.11 / 24.1	8.0x [7.4, 8.9]
50	20	10.85 / 10.70 / 50.00	+0.15 [+0.00, +0.40]	tie	0.910 / 0.911	18/20	8.32 / 90	10.5x [9.4, 11.3]
100	20	20.55 / 20.80 / 100.00	-0.25 [-0.65, +0.05]	tie	0.917 / 0.916	16/20	19.8 / 281	14.1x [13.4, 14.5]

Additive noise, Gumbel:

d	Pairs	SHD: Andrey / CAM 1.0 / empty	D, mean [95%]	Verdict	Orientation: Andrey / CAM 1.0	Same graph	CPU s: Andrey / CAM 1.0	Time ratio [95%]
10	20	4.65 / 4.70 / 10.00	-0.05 [-0.15, +0.00]	tie	0.745 / 0.745	19/20	0.697 / 5.29	7.6x [6.8, 7.8]
20	20	8.65 / 8.60 / 20.00	+0.05 [+0.00, +0.15]	tie	0.762 / 0.762	19/20	3.29 / 24	8.0x [7.4, 9.0]
50	20	18.15 / 18.30 / 50.00	-0.15 [-0.30, +0.00]	tie	0.817 / 0.816	17/20	7.56 / 92.9	11.8x [11.3, 12.7]
100	20	31.65 / 32.35 / 100.00	-0.70 [-1.40, -0.20]	better	0.839 / 0.837	10/20	18.3 / 280	14.7x [14.0, 15.7]

Columns in their own units, in the claim

Additive noise, Gaussian:

d	Pairs	SHD: Andrey / CAM 1.0 / empty	D, mean [95%]	Verdict	Orientation: Andrey / CAM 1.0	Same graph	CPU s: Andrey / CAM 1.0	Time ratio [95%]
10	20	1.35 / 1.35 / 10.00	+0.00 [+0.00, +0.00]	tie	0.915 / 0.915	20/20	0.755 / 5.31	6.5x [5.8, 7.7]
20	20	4.25 / 4.25 / 20.00	+0.00 [+0.00, +0.00]	tie	0.897 / 0.897	19/20	3 / 26.6	8.9x [8.5, 10.2]
50	20	10.85 / 10.80 / 50.00	+0.05 [+0.00, +0.15]	tie	0.910 / 0.910	19/20	8.65 / 98	12.3x [11.0, 12.7]
100	20	20.55 / 21.45 / 100.00	-0.90 [-2.60, +0.05]	tie	0.917 / 0.914	16/20	19.6 / 272	14.0x [13.4, 14.5]

Additive noise, Gumbel:

d	Pairs	SHD: Andrey / CAM 1.0 / empty	D, mean [95%]	Verdict	Orientation: Andrey / CAM 1.0	Same graph	CPU s: Andrey / CAM 1.0	Time ratio [95%]
10	20	4.65 / 4.70 / 10.00	-0.05 [-0.15, +0.00]	tie	0.745 / 0.745	19/20	0.728 / 5.05	6.8x [6.5, 7.3]
20	20	8.65 / 8.60 / 20.00	+0.05 [+0.00, +0.15]	tie	0.762 / 0.762	19/20	2.68 / 23.6	8.6x [8.0, 10.1]
50	20	18.15 / 18.30 / 50.00	-0.15 [-0.30, +0.00]	tie	0.817 / 0.816	17/20	7.43 / 87.8	11.6x [10.8, 12.1]
100	20	31.65 / 32.50 / 100.00	-0.85 [-1.65, -0.20]	better	0.839 / 0.836	11/20	18.7 / 274	14.4x [13.9, 15.0]

Outside the claim, standardized columns

Location-scale noise:

d	Pairs	SHD: Andrey / CAM 1.0 / empty	D, mean [95%]	Orientation: Andrey / CAM 1.0	Same graph	CPU s: Andrey / CAM 1.0	Time ratio [95%]
10	20	2.70 / 2.70 / 10.00	+0.00 [+0.00, +0.00]	0.800 / 0.800	20/20	0.682 / 4.63	7.4x [6.2, 8.3]
20	20	8.85 / 8.85 / 20.00	+0.00 [+0.00, +0.00]	0.765 / 0.765	20/20	2.5 / 22.8	9.0x [8.1, 9.5]
50	20	18.30 / 18.40 / 50.00	-0.10 [-0.30, +0.00]	0.798 / 0.798	18/20	8.22 / 89	10.8x [9.8, 11.6]
100	20	34.80 / 35.00 / 100.00	-0.20 [-0.55, +0.00]	0.810 / 0.810	18/20	18.1 / 255	14.9x [13.5, 15.2]

Linear Gaussian:

d	Pairs	SHD: Andrey / CAM 1.0 / empty	D, mean [95%]	Orientation: Andrey / CAM 1.0	Same graph	CPU s: Andrey / CAM 1.0	Time ratio [95%]
10	20	5.50 / 5.70 / 10.00	-0.20 [-0.90, +0.20]	0.640 / 0.630	16/20	0.904 / 7.45	8.8x [8.4, 9.2]
20	20	12.95 / 12.05 / 20.00	+0.90 [-0.05, +2.25]	0.608 / 0.620	15/20	3.31 / 27.7	8.4x [7.9, 9.5]
50	20	33.85 / 33.70 / 50.00	+0.15 [-1.40, +1.65]	0.617 / 0.618	7/20	9.22 / 110	11.4x [11.1, 12.6]
100	20	67.50 / 69.55 / 100.00	-2.05 [-4.55, +0.30]	0.627 / 0.620	5/20	20.9 / 266	12.6x [11.7, 13.7]

Outside the claim, columns in their own units

Location-scale noise:

d	Pairs	SHD: Andrey / CAM 1.0 / empty	D, mean [95%]	Orientation: Andrey / CAM 1.0	Same graph	CPU s: Andrey / CAM 1.0	Time ratio [95%]
10	20	2.70 / 2.70 / 10.00	+0.00 [+0.00, +0.00]	0.800 / 0.800	20/20	0.67 / 5.12	8.0x [6.7, 8.5]
20	20	8.85 / 8.85 / 20.00	+0.00 [+0.00, +0.00]	0.765 / 0.765	20/20	2.42 / 23	8.8x [8.4, 10.2]
50	20	18.30 / 18.40 / 50.00	-0.10 [-0.30, +0.00]	0.798 / 0.798	18/20	8.07 / 92.3	10.8x [10.0, 11.9]
100	20	34.80 / 34.85 / 100.00	-0.05 [-0.15, +0.00]	0.810 / 0.810	18/20	18.8 / 262	14.1x [13.2, 15.1]

Linear Gaussian:

d	Pairs	SHD: Andrey / CAM 1.0 / empty	D, mean [95%]	Orientation: Andrey / CAM 1.0	Same graph	CPU s: Andrey / CAM 1.0	Time ratio [95%]
10	20	5.50 / 6.00 / 10.00	-0.50 [-1.25, +0.10]	0.640 / 0.610	12/20	0.912 / 7.08	8.5x [8.1, 8.7]
20	20	12.95 / 13.00 / 20.00	-0.05 [-1.50, +1.55]	0.608 / 0.588	13/20	3.4 / 27.4	8.1x [7.4, 8.7]
50	20	33.85 / 34.60 / 50.00	-0.75 [-1.75, +0.30]	0.617 / 0.604	7/20	8.84 / 104	11.5x [11.2, 12.2]
100	20	67.50 / 72.75 / 100.00	-5.25 [-7.70, -2.90]	0.627 / 0.595	1/20	20.9 / 286	14.1x [12.7, 14.9]

RESIT

Andrey’s RESIT against the authors’ code and lingam at 1,000 rows and 20 datasets per setting (10 at 20 variables in their own units). D is Andrey’s SHD minus the reference’s on the same dataset. Orientation is the share of true edges returned with the right direction. The settings outside the model carry no verdict. The setting marked “tie after rerun” is described above.

Standardized columns, in the claim

Additive noise, Gaussian:

d	Reference	Pairs	SHD: Andrey / reference / empty	D, mean [95%]	Verdict	Orientation: Andrey / reference	Same graph	CPU s: Andrey / reference	Time ratio [95%]
5	authors’ code	20	0.85 / 0.85 / 5.00	+0.00 [+0.00, +0.00]	tie	0.910 / 0.910	20/20	1.86 / 6.76	3.6x [3.5, 3.7]
10	authors’ code	20	6.80 / 5.60 / 10.00	+1.20 [+0.05, +2.50]	tie after rerun	0.815 / 0.810	9/20	8.18 / 29.3	3.6x [3.4, 3.7]
20	authors’ code	20	19.40 / 20.20 / 20.00	-0.80 [-3.20, +1.60]	tie	0.685 / 0.688	0/20	53.2 / 232	4.4x [3.8, 4.6]
5	authors’ code	20	0.85 / 3.50 / 5.00	-2.65 [-4.15, -1.20]	better	0.910 / 0.680	5/20	1.86 / 4.33	2.3x [2.2, 2.4]
10	authors’ code	20	6.80 / 12.25 / 10.00	-5.45 [-8.45, -2.40]	better	0.815 / 0.610	0/20	8.18 / 22.2	2.7x [2.6, 2.8]
20	authors’ code	20	19.40 / 44.80 / 20.00	-25.40 [-31.90, -19.15]	better	0.685 / 0.458	0/20	53.2 / 126	2.4x [2.1, 2.5]
5	lingam	20	0.85 / 3.50 / 5.00	-2.65 [-4.15, -1.20]	better	0.910 / 0.680	5/20	1.86 / 4.33	2.3x [2.2, 2.4]
10	lingam	20	6.80 / 12.25 / 10.00	-5.45 [-8.45, -2.40]	better	0.815 / 0.610	0/20	8.18 / 22.2	2.7x [2.6, 2.8]
20	lingam	20	19.40 / 44.80 / 20.00	-25.40 [-31.90, -19.15]	better	0.685 / 0.458	0/20	53.2 / 126	2.4x [2.1, 2.5]

Additive noise, Gumbel:

d	Reference	Pairs	SHD: Andrey / reference / empty	D, mean [95%]	Verdict	Orientation: Andrey / reference	Same graph	CPU s: Andrey / reference	Time ratio [95%]
5	authors’ code	20	0.15 / 0.25 / 5.00	-0.10 [-0.60, +0.30]	tie	0.980 / 0.980	18/20	1.94 / 6.79	3.5x [3.4, 3.6]
10	authors’ code	20	3.60 / 3.50 / 10.00	+0.10 [-0.90, +1.05]	tie	0.905 / 0.905	9/20	7.9 / 29.1	3.7x [3.5, 3.9]
20	authors’ code	20	15.55 / 14.45 / 20.00	+1.10 [-1.70, +3.80]	tie	0.765 / 0.773	0/20	52.7 / 235	4.6x [4.4, 4.8]
5	authors’ code	20	0.15 / 2.85 / 5.00	-2.70 [-3.95, -1.60]	better	0.980 / 0.800	6/20	1.94 / 4.45	2.3x [2.3, 2.4]
10	authors’ code	20	3.60 / 12.75 / 10.00	-9.15 [-12.50, -5.70]	better	0.905 / 0.700	0/20	7.9 / 22.5	2.9x [2.8, 2.9]
20	authors’ code	20	15.55 / 46.10 / 20.00	-30.55 [-36.80, -24.15]	better	0.765 / 0.607	0/20	52.7 / 132	2.5x [2.4, 2.6]
5	lingam	20	0.15 / 2.85 / 5.00	-2.70 [-3.95, -1.60]	better	0.980 / 0.800	6/20	1.94 / 4.45	2.3x [2.3, 2.4]
10	lingam	20	3.60 / 12.75 / 10.00	-9.15 [-12.50, -5.70]	better	0.905 / 0.700	0/20	7.9 / 22.5	2.9x [2.8, 2.9]
20	lingam	20	15.55 / 46.10 / 20.00	-30.55 [-36.80, -24.15]	better	0.765 / 0.607	0/20	52.7 / 132	2.5x [2.4, 2.6]

Columns in their own units, in the claim

Additive noise, Gaussian:

d	Reference	Pairs	SHD: Andrey / reference / empty	D, mean [95%]	Verdict	Orientation: Andrey / reference	Same graph	CPU s: Andrey / reference	Time ratio [95%]
5	authors’ code	20	0.85 / 1.15 / 5.00	-0.30 [-0.70, +0.10]	tie	0.910 / 0.870	13/20	1.94 / 6.73	3.6x [3.3, 3.7]
10	authors’ code	20	6.80 / 7.30 / 10.00	-0.50 [-3.00, +2.15]	tie	0.815 / 0.710	1/20	8.48 / 30	3.4x [3.3, 3.6]
20	authors’ code	10	19.80 / 18.90 / 20.00	+0.90 [-4.10, +5.80]	tie	0.685 / 0.610	0/10	53.5 / 224	4.4x [3.7, 4.5]
5	authors’ code	20	0.85 / 3.55 / 5.00	-2.70 [-4.05, -1.40]	better	0.910 / 0.620	3/20	1.94 / 4.42	2.4x [2.1, 2.5]
10	authors’ code	20	6.80 / 9.80 / 10.00	-3.00 [-5.65, -0.20]	better	0.815 / 0.565	0/20	8.48 / 23.2	2.7x [2.6, 2.8]
20	authors’ code	10	19.80 / 22.70 / 20.00	-2.90 [-7.40, +1.70]	tie	0.685 / 0.525	0/10	53.5 / 121	2.3x [2.0, 2.4]
5	lingam	20	0.85 / 3.55 / 5.00	-2.70 [-4.05, -1.40]	better	0.910 / 0.620	3/20	1.94 / 4.42	2.4x [2.1, 2.5]
10	lingam	20	6.80 / 9.80 / 10.00	-3.00 [-5.65, -0.20]	better	0.815 / 0.565	0/20	8.48 / 23.2	2.7x [2.6, 2.8]
20	lingam	10	19.80 / 22.70 / 20.00	-2.90 [-7.40, +1.70]	tie	0.685 / 0.525	0/10	53.5 / 121	2.3x [2.0, 2.4]

Additive noise, Gumbel:

d	Reference	Pairs	SHD: Andrey / reference / empty	D, mean [95%]	Verdict	Orientation: Andrey / reference	Same graph	CPU s: Andrey / reference	Time ratio [95%]
5	authors’ code	20	0.15 / 0.40 / 5.00	-0.25 [-0.60, +0.00]	tie	0.980 / 0.940	15/20	2.26 / 7.16	3.2x [3.1, 3.5]
10	authors’ code	20	3.60 / 4.15 / 10.00	-0.55 [-2.35, +1.25]	tie	0.905 / 0.865	2/20	7.91 / 30.5	3.8x [3.7, 3.9]
20	authors’ code	10	16.00 / 14.50 / 20.00	+1.50 [-3.30, +6.30]	tie	0.780 / 0.725	0/10	54.6 / 233	4.3x [4.2, 4.7]
5	authors’ code	20	0.15 / 3.10 / 5.00	-2.95 [-4.10, -1.85]	better	0.980 / 0.770	5/20	2.26 / 4.59	2.1x [2.0, 2.2]
10	authors’ code	20	3.60 / 11.10 / 10.00	-7.50 [-10.90, -4.20]	better	0.905 / 0.675	0/20	7.91 / 23.2	2.9x [2.8, 3.0]
20	authors’ code	10	16.00 / 31.20 / 20.00	-15.20 [-23.50, -6.40]	better	0.780 / 0.540	0/10	54.6 / 124	2.3x [2.2, 2.5]
5	lingam	20	0.15 / 3.10 / 5.00	-2.95 [-4.10, -1.85]	better	0.980 / 0.770	5/20	2.26 / 4.59	2.1x [2.0, 2.2]
10	lingam	20	3.60 / 11.10 / 10.00	-7.50 [-10.90, -4.20]	better	0.905 / 0.675	0/20	7.91 / 23.2	2.9x [2.8, 3.0]
20	lingam	10	16.00 / 31.20 / 20.00	-15.20 [-23.50, -6.40]	better	0.780 / 0.540	0/10	54.6 / 124	2.3x [2.2, 2.5]

Outside the claim, standardized columns

Location-scale noise:

d	Reference	Pairs	SHD: Andrey / reference / empty	D, mean [95%]	Orientation: Andrey / reference	Same graph	CPU s: Andrey / reference	Time ratio [95%]
5	authors’ code	20	7.75 / 7.75 / 5.00	+0.00 [+0.00, +0.00]	0.330 / 0.330	20/20	1.84 / 7.85	4.2x [3.8, 4.8]
10	authors’ code	20	28.95 / 28.30 / 10.00	+0.65 [+0.20, +1.20]	0.255 / 0.260	13/20	8.17 / 29.7	3.6x [3.5, 3.7]
20	authors’ code	20	87.70 / 86.45 / 20.00	+1.25 [-0.35, +2.95]	0.300 / 0.297	0/20	54.4 / 240	4.4x [4.1, 4.6]
5	authors’ code	20	7.75 / 7.60 / 5.00	+0.15 [-0.30, +0.60]	0.330 / 0.360	6/20	1.84 / 4.69	2.5x [2.4, 2.6]
10	authors’ code	20	28.95 / 29.50 / 10.00	-0.55 [-2.90, +1.55]	0.255 / 0.280	1/20	8.17 / 23.3	2.8x [2.7, 2.9]
20	authors’ code	20	87.70 / 98.60 / 20.00	-10.90 [-16.50, -5.60]	0.300 / 0.245	0/20	54.4 / 139	2.5x [2.4, 2.8]
5	lingam	20	7.75 / 7.60 / 5.00	+0.15 [-0.30, +0.60]	0.330 / 0.360	6/20	1.84 / 4.69	2.5x [2.4, 2.6]
10	lingam	20	28.95 / 29.50 / 10.00	-0.55 [-2.90, +1.55]	0.255 / 0.280	1/20	8.17 / 23.3	2.8x [2.7, 2.9]
20	lingam	20	87.70 / 98.60 / 20.00	-10.90 [-16.50, -5.60]	0.300 / 0.245	0/20	54.4 / 139	2.5x [2.4, 2.8]

Linear Gaussian:

d	Reference	Pairs	SHD: Andrey / reference / empty	D, mean [95%]	Orientation: Andrey / reference	Same graph	CPU s: Andrey / reference	Time ratio [95%]
5	authors’ code	20	4.95 / 5.30 / 5.00	-0.35 [-0.95, +0.00]	0.450 / 0.420	18/20	1.87 / 6.8	3.6x [3.5, 3.7]
10	authors’ code	20	13.30 / 13.35 / 10.00	-0.05 [-0.95, +0.85]	0.415 / 0.420	4/20	8.03 / 28.5	3.5x [3.3, 3.7]
20	authors’ code	20	32.20 / 31.80 / 20.00	+0.40 [-3.25, +4.05]	0.420 / 0.427	0/20	42.7 / 230	5.6x [5.3, 6.2]
5	authors’ code	20	4.95 / 3.90 / 5.00	+1.05 [-0.80, +2.75]	0.450 / 0.480	1/20	1.87 / 4.45	2.4x [2.3, 2.4]
10	authors’ code	20	13.30 / 11.40 / 10.00	+1.90 [-0.70, +4.40]	0.415 / 0.470	0/20	8.03 / 22.1	2.8x [2.6, 2.9]
20	authors’ code	20	32.20 / 34.10 / 20.00	-1.90 [-6.85, +2.90]	0.420 / 0.450	0/20	42.7 / 135	3.1x [2.9, 3.3]
5	lingam	20	4.95 / 3.90 / 5.00	+1.05 [-0.80, +2.75]	0.450 / 0.480	1/20	1.87 / 4.45	2.4x [2.3, 2.4]
10	lingam	20	13.30 / 11.40 / 10.00	+1.90 [-0.70, +4.40]	0.415 / 0.470	0/20	8.03 / 22.1	2.8x [2.6, 2.9]
20	lingam	20	32.20 / 34.10 / 20.00	-1.90 [-6.85, +2.90]	0.420 / 0.450	0/20	42.7 / 135	3.1x [2.9, 3.3]

Outside the claim, columns in their own units

Location-scale noise:

d	Reference	Pairs	SHD: Andrey / reference / empty	D, mean [95%]	Orientation: Andrey / reference	Same graph	CPU s: Andrey / reference	Time ratio [95%]
5	authors’ code	20	7.75 / 7.40 / 5.00	+0.35 [+0.05, +0.75]	0.330 / 0.360	10/20	2.05 / 7.41	3.6x [3.4, 3.9]
10	authors’ code	20	28.95 / 27.55 / 10.00	+1.40 [-0.10, +3.05]	0.255 / 0.250	2/20	8.05 / 32	3.9x [3.9, 4.0]
20	authors’ code	10	95.80 / 80.00 / 20.00	+15.80 [+5.00, +27.70]	0.265 / 0.310	0/10	53.4 / 246	4.6x [3.8, 4.9]
5	authors’ code	20	7.75 / 7.80 / 5.00	-0.05 [-0.60, +0.40]	0.330 / 0.330	8/20	2.05 / 4.68	2.3x [2.2, 2.4]
10	authors’ code	20	28.95 / 25.30 / 10.00	+3.65 [+1.45, +5.85]	0.255 / 0.315	1/20	8.05 / 24.2	3.0x [2.9, 3.1]
20	authors’ code	10	95.80 / 74.80 / 20.00	+21.00 [+4.90, +35.80]	0.265 / 0.345	0/10	53.4 / 132	2.5x [2.1, 2.6]
5	lingam	20	7.75 / 7.80 / 5.00	-0.05 [-0.60, +0.40]	0.330 / 0.330	8/20	2.05 / 4.68	2.3x [2.2, 2.4]
10	lingam	20	28.95 / 25.30 / 10.00	+3.65 [+1.45, +5.85]	0.255 / 0.315	1/20	8.05 / 24.2	3.0x [2.9, 3.1]
20	lingam	10	95.80 / 74.80 / 20.00	+21.00 [+4.90, +35.80]	0.265 / 0.345	0/10	53.4 / 132	2.5x [2.1, 2.6]

Linear Gaussian:

d	Reference	Pairs	SHD: Andrey / reference / empty	D, mean [95%]	Orientation: Andrey / reference	Same graph	CPU s: Andrey / reference	Time ratio [95%]
5	authors’ code	20	4.95 / 5.55 / 5.00	-0.60 [-1.75, +0.45]	0.450 / 0.410	6/20	1.9 / 7.2	3.8x [3.7, 3.9]
10	authors’ code	20	13.30 / 12.85 / 10.00	+0.45 [-1.70, +2.45]	0.415 / 0.420	1/20	7.76 / 36.6	4.7x [4.4, 4.9]
20	authors’ code	10	32.40 / 33.80 / 20.00	-1.40 [-8.90, +6.20]	0.460 / 0.340	0/10	42.5 / 237	5.5x [5.1, 5.8]
5	authors’ code	20	4.95 / 4.00 / 5.00	+0.95 [-0.85, +2.70]	0.450 / 0.470	2/20	1.9 / 4.55	2.4x [2.3, 2.5]
10	authors’ code	20	13.30 / 11.15 / 10.00	+2.15 [-0.20, +4.60]	0.415 / 0.415	0/20	7.76 / 23.6	3.0x [2.9, 3.2]
20	authors’ code	10	32.40 / 24.80 / 20.00	+7.60 [-0.90, +16.10]	0.460 / 0.425	0/10	42.5 / 131	3.1x [2.8, 3.2]
5	lingam	20	4.95 / 4.00 / 5.00	+0.95 [-0.85, +2.70]	0.450 / 0.470	2/20	1.9 / 4.55	2.4x [2.3, 2.5]
10	lingam	20	13.30 / 11.15 / 10.00	+2.15 [-0.20, +4.60]	0.415 / 0.415	0/20	7.76 / 23.6	3.0x [2.9, 3.2]
20	lingam	10	32.40 / 24.80 / 20.00	+7.60 [-0.90, +16.10]	0.460 / 0.425	0/10	42.5 / 131	3.1x [2.8, 3.2]

Other changes in 0.13.0

CAM-UV and DYNOTEARS are experimental, and their docstrings, on the API pages, give their measurements, not this card. The changelog measures the other changes: the location-scale noise generator, the rank that breaks ties without the row order, the faster chi-square and G-square tests, the fixes to RCD, GIN, and VARMA-LiNGAM under a constant offset, and the child and insurance networks in load_dataset.

Limits

The linear-Gaussian settings cover 20 to 200 variables and 1,000 to 5,000 rows with Fisher-z tests; the discrete settings cover five networks with up to 70 variables and the same two sample sizes. Other data types, other tests, and larger discrete networks are not measured here. At 100 variables, 1,000 rows, and mean degree 2 on unstandardized data, the mean CPDAG of IAMB makes 79 errors against 100 for the empty graph, the closest of any setting (bnlearn: 80); on standardized data it is 20 variables, 1,000 rows, and mean degree 4, with 24 against 40.

Several CPDAG settings have fewer than 10 paired datasets and are not judged, because bnlearn reached the cap: IAMB at 100 variables, 5,000 rows, and mean degree 4 on standardized data (9 pairs, 11 caps) and at 200 variables, 1,000 rows, and mean degree 4 (9 pairs, 1 cap). MMPC’s CPDAG has 2 such settings on standardized data (100 variables and mean degree 4, where Andrey reached the cap in 6 and 19 fits and bnlearn in 18 and 20), 6 on unstandardized data, and 1 at 200 variables (5 pairs). The MMPC skeleton at 200 variables and 5,000 rows is the one MMPC setting worse than bnlearn’s on two runs. Two discrete settings of RSMAX2 at insurance and 1,000 rows have 8 pairs and are not judged. At 200 variables the skeleton and CPDAG of 5,000 rows and mean degree 4 are not measured, because bnlearn did not finish all fits at 100 variables there.

CAM’s settings cover 10 to 100 variables at 1,000 rows. Location-scale noise and linear-Gaussian data lie outside its model and are reported, not judged: there CAM returns 77% to 81% (location-scale noise) and 61% to 64% (linear Gaussian) of the true edges with the right direction, and linear-Gaussian data identify only the equivalence class. CAM makes fewer errors than the empty graph in all 32 settings, and so does CAM 1.0. Larger graphs, more rows, hidden common causes, and effects that interact are not measured here.

RESIT’s settings cover 5 to 20 variables at 1,000 rows. Its accuracy claim holds up to 10 variables. At 20 variables with Gaussian additive noise no implementation clearly beats the empty graph: Andrey makes 19.4 mean errors, the authors’ code 20.2, and lingam 44.8, against 20.0 for the empty graph (19.8 for Andrey on columns in their own units, 10 datasets), so 20 variables is a range for speed. With Gumbel noise at 20 variables Andrey (15.6) and the authors’ code (14.5) are below the empty graph and lingam (46.1) is not. Location-scale noise and linear-Gaussian data lie outside RESIT’s model and are reported, not judged: there RESIT returns 25% to 33% (location-scale noise) and 41% to 46% (linear Gaussian) of the true edges with the right direction, and its mean errors are above the empty graph’s in all 6 location-scale settings and in 4 of the 6 linear-Gaussian settings. Larger graphs, more rows, hidden common causes, and effects that interact are not measured here.

Reproduce

You can find environment setup, campaign drivers, and scores in the benchmark harness guide. To rerun the comparison, install Andrey (andrey-core), bnlearn 5.2.1, CAM 1.0, and the authors’ RESIT code with R 4.6.1 in the harness’s R environment, and lingam 1.12.2. Every driver shuffles the columns by default. Times will differ with the hardware and build; graphs should match closely.

Comparison	Lane	Sizes	Datasets x repeats	Wall cap (s)
IAMB and MMPC against bnlearn	the harness’s R lane	d 20, 50, 100, 200; n 1,000, 5,000; mean degree 2, 4	10 to 20 x 1	600
IAMB, Inter-IAMB, MMPC, MMHC, and RSMAX2 against bnlearn on discrete networks	the harness’s R lane	asia, child, insurance, alarm, hepar2; n 1,000, 5,000	10 to 20 x 1	600
CAM against CAM 1.0	the harness’s R lane	d 10, 20, 50, 100; n 1,000	20 x 1	1,800
RESIT against the authors’ code and lingam	the harness’s R lane	d 5, 10, 20; n 1,000	20 x 1	1,800