0.12.0 release card: calibration#

Release card

Calibration

0.12.0

This card measures the three methods 0.12.0 makes supported, Inter-IAMB, MMHC, and RSMAX2, against bnlearn on the graph each returns. Each result holds for the measured conditions; it depends on the data, settings, and hardware.

Inter-IAMB, MMHC, and RSMAX2 are supported methods. We compare each with bnlearn 5.2.1 on the graph it returns, scored against that graph’s own truth: the CPDAG for MMHC and RSMAX2 and for Inter-IAMB’s default structure="cpdag", the skeleton for structure="skeleton", and the moral graph for structure="moral", which bnlearn’s learn.mb blankets joined by the AND rule give.

Summary#

Method	Graph	Settings	Tie	Better	Largest difference D, mean [95%]	Median time ratio
Inter-IAMB	CPDAG	12	12	0	+0.26 [-0.58, +1.16] (d 50, n 5,000, degree 4)	3.8
Inter-IAMB	skeleton	12	12	0	+0.07 [+0.00, +0.20] (d 100, n 5,000, degree 2)	3.4
Inter-IAMB	moral graph	12	12	0	+0.50 [-0.30, +1.55] (d 100, n 5,000, degree 4)	1.9
MMHC	CPDAG	12	11	1	+1.80 [-0.10, +3.50] (d 50, n 1,000, degree 2)	6.6
RSMAX2	CPDAG	12	11	1	+1.80 [-0.10, +3.50] (d 50, n 1,000, degree 2)	7.8
RSMAX2, IAMB restrict	CPDAG, restrict iamb	12	11	1	+1.60 [-0.30, +3.20] (d 50, n 1,000, degree 2)	3.5

A setting is one size, sample size, and mean degree. D is Andrey’s error minus bnlearn’s on the same dataset. A setting with at least 10 paired datasets is worse when the paired 95% bootstrap interval of the mean D lies wholly above 0, better when wholly below, and a tie otherwise. The time ratio is bnlearn’s fit time over Andrey’s, the median of the settings’ median ratios. In no setting does bnlearn make significantly fewer errors.

Method#

Data. Linear-Gaussian data from Erdos-Renyi graphs, standardized, with 20, 50, and 100 variables, 1,000 and 5,000 rows, and mean degree 2 and 4. Each setting has up to 20 datasets, one per seed, with the columns in a shuffled order; every setting keeps at least 10 paired datasets.

Fits. One fit at a time on one core, Fisher-z tests at alpha 0.05 (bnlearn test="zf") and the Gaussian BIC for the score phase (bnlearn score="bic-g"), a 600 s cap per fit, and one untimed warm-up fit except at 5,000 rows, mean degree 4, and 50 or more variables, where there is none. A fit that reaches the cap is a timeout. It is counted as coverage, left out of the pairs, and never scored as an empty graph. The Andrey fits all finished; the timeouts are bnlearn’s.

Errors. The structural Hamming distance of the returned graph to its own truth. The tables give the median over the paired datasets, for Andrey, bnlearn, and the empty graph; D is the mean of the per-dataset differences, the statistic of the verdict. A DAG that bnlearn returns is compared through its CPDAG.

Inter-IAMB, MMHC, and RSMAX2#

Inter-IAMB#

CPDAG, structure="cpdag", against bnlearn inter.iamb():

Degree	d	n	Datasets	Andrey	bnlearn	Empty graph	D, mean [95%]	Verdict	bnlearn time / Andrey time
2	20	1,000	20	4.0	4.0	20.0	+0.00 [+0.00, +0.00]	tie	1.9
4	20	1,000	20	24.0	23.5	40.0	+0.25 [+0.00, +0.55]	tie	2.5
2	20	5,000	20	1.0	1.0	20.0	+0.00 [+0.00, +0.00]	tie	6.6
4	20	5,000	20	18.0	18.0	40.0	+0.00 [-0.45, +0.40]	tie	14.9
2	50	1,000	20	7.5	7.5	50.0	+0.00 [+0.00, +0.00]	tie	1.7
4	50	1,000	20	54.5	51.5	100.0	+0.25 [-0.50, +1.15]	tie	2.7
2	50	5,000	20	8.5	8.5	50.0	+0.15 [+0.00, +0.40]	tie	3.5
4	50	5,000	19	42.0	41.0	100.0	+0.26 [-0.58, +1.16]	tie	20.8
2	100	1,000	20	32.0	32.0	100.0	+0.15 [+0.00, +0.30]	tie	1.4
4	100	1,000	14	104.5	101.5	200.0	-0.14 [-1.86, +1.57]	tie	4.2
2	100	5,000	15	24.0	24.0	100.0	+0.00 [-0.20, +0.20]	tie	4.4
4	100	5,000	12	71.5	74.0	200.0	-0.58 [-1.67, +0.58]	tie	30.3

Skeleton, structure="skeleton", against bnlearn inter.iamb(undirected=TRUE):

Degree	d	n	Datasets	Andrey	bnlearn	Empty graph	D, mean [95%]	Verdict	bnlearn time / Andrey time
2	20	1,000	20	1.0	1.0	20.0	+0.00 [+0.00, +0.00]	tie	1.7
4	20	1,000	20	14.5	14.5	40.0	+0.00 [+0.00, +0.00]	tie	2.5
2	20	5,000	20	1.0	1.0	20.0	+0.00 [+0.00, +0.00]	tie	4.7
4	20	5,000	20	10.5	10.5	40.0	+0.00 [+0.00, +0.00]	tie	10.7
2	50	1,000	20	6.0	6.0	50.0	+0.00 [+0.00, +0.00]	tie	1.7
4	50	1,000	20	32.5	32.5	100.0	+0.05 [+0.00, +0.15]	tie	2.4
2	50	5,000	20	5.0	5.0	50.0	+0.00 [+0.00, +0.00]	tie	3.4
4	50	5,000	19	24.0	24.0	100.0	+0.00 [+0.00, +0.00]	tie	11.6
2	100	1,000	20	23.0	23.0	100.0	+0.00 [+0.00, +0.00]	tie	1.3
4	100	1,000	14	64.0	64.0	200.0	+0.00 [+0.00, +0.00]	tie	3.4
2	100	5,000	15	19.0	19.0	100.0	+0.07 [+0.00, +0.20]	tie	4.2
4	100	5,000	15	44.0	44.0	200.0	+0.07 [+0.00, +0.20]	tie	25.0

Moral graph, structure="moral", against bnlearn learn.mb joined by the AND rule:

Degree	d	n	Datasets	Andrey	bnlearn	Empty graph	D, mean [95%]	Verdict	bnlearn time / Andrey time
2	20	1,000	20	1.5	1.5	30.5	+0.00 [+0.00, +0.00]	tie	1.2
4	20	1,000	20	8.0	8.0	71.0	+0.00 [+0.00, +0.00]	tie	1.0
2	20	5,000	20	1.0	1.0	30.5	-0.05 [-0.15, +0.00]	tie	4.1
4	20	5,000	20	1.5	1.5	71.0	+0.00 [+0.00, +0.00]	tie	3.7
2	50	1,000	20	8.0	8.0	80.0	+0.00 [+0.00, +0.00]	tie	1.5
4	50	1,000	20	40.0	38.0	208.5	+0.30 [+0.00, +0.80]	tie	1.2
2	50	5,000	20	6.5	6.5	80.0	-0.15 [-0.45, +0.00]	tie	3.0
4	50	5,000	20	7.0	7.0	208.5	+0.05 [+0.00, +0.15]	tie	2.0
2	100	1,000	20	33.0	33.0	168.5	+0.00 [+0.00, +0.00]	tie	1.3
4	100	1,000	14	125.5	125.5	448.0	+0.07 [+0.00, +0.21]	tie	1.8
2	100	5,000	15	23.0	23.0	171.0	+0.07 [+0.00, +0.20]	tie	4.2
4	100	5,000	20	35.0	35.0	445.0	+0.50 [-0.30, +1.55]	tie	3.4

MMHC#

CPDAG against bnlearn mmhc():

Degree	d	n	Datasets	Andrey	bnlearn	Empty graph	D, mean [95%]	Verdict	bnlearn time / Andrey time
2	20	1,000	10	5.0	5.5	20.0	-0.10 [-2.00, +1.90]	tie	3.7
4	20	1,000	20	32.0	31.5	40.0	-0.40 [-2.35, +1.85]	tie	3.8
2	20	5,000	20	5.5	5.0	20.0	+0.05 [-1.25, +1.45]	tie	10.8
4	20	5,000	20	28.5	30.5	40.0	-1.45 [-3.55, +0.55]	tie	24.3
2	50	1,000	10	16.5	15.0	50.0	+1.80 [-0.10, +3.50]	tie	4.1
4	50	1,000	20	65.5	64.5	100.0	-0.50 [-3.10, +1.90]	tie	6.2
2	50	5,000	20	11.5	12.5	50.0	-0.05 [-1.50, +1.30]	tie	11.4
4	50	5,000	14	62.0	62.0	100.0	+0.00 [-3.07, +3.07]	tie	6.2
2	100	1,000	10	24.0	33.5	100.0	-5.80 [-10.20, -1.20]	better	4.5
4	100	1,000	20	121.0	127.5	200.0	-2.05 [-4.95, +0.70]	tie	7.0
2	100	5,000	20	22.5	26.5	100.0	-2.70 [-6.70, +1.35]	tie	14.0
4	100	5,000	16	112.5	119.0	200.0	-1.12 [-5.31, +3.50]	tie	19.4

RSMAX2#

CPDAG with restrict="mmpc" and maximize="hc", the defaults, against bnlearn rsmax2() with the same phases:

Degree	d	n	Datasets	Andrey	bnlearn	Empty graph	D, mean [95%]	Verdict	bnlearn time / Andrey time
2	20	1,000	20	6.5	6.0	20.0	-0.35 [-2.30, +1.45]	tie	4.4
4	20	1,000	20	32.0	31.5	40.0	-0.40 [-2.35, +1.75]	tie	4.2
2	20	5,000	20	5.5	5.0	20.0	+0.05 [-1.30, +1.45]	tie	7.5
4	20	5,000	20	28.5	30.5	40.0	-1.45 [-3.55, +0.50]	tie	19.2
2	50	1,000	10	16.5	15.0	50.0	+1.80 [-0.10, +3.50]	tie	3.9
4	50	1,000	20	65.5	64.5	100.0	-0.50 [-3.10, +1.90]	tie	4.8
2	50	5,000	20	11.5	12.5	50.0	-0.05 [-1.50, +1.30]	tie	11.2
4	50	5,000	15	62.0	62.0	100.0	-0.80 [-3.40, +2.00]	tie	8.0
2	100	1,000	20	28.5	34.0	100.0	-5.80 [-9.40, -2.20]	better	7.2
4	100	1,000	20	121.0	127.5	200.0	-2.05 [-4.95, +0.65]	tie	8.9
2	100	5,000	20	22.5	26.5	100.0	-2.70 [-6.85, +1.35]	tie	12.0
4	100	5,000	20	112.5	119.0	200.0	-1.50 [-5.05, +2.25]	tie	14.6

CPDAG with restrict="iamb", whose restrict phase keeps the skeleton of IAMB, as bnlearn’s does:

Degree	d	n	Datasets	Andrey	bnlearn	Empty graph	D, mean [95%]	Verdict	bnlearn time / Andrey time
2	20	1,000	20	4.0	5.5	20.0	-0.60 [-2.50, +1.10]	tie	1.4
4	20	1,000	20	24.5	25.5	40.0	-0.30 [-1.95, +1.50]	tie	3.1
2	20	5,000	20	4.0	4.5	20.0	-0.10 [-1.45, +1.45]	tie	4.0
4	20	5,000	20	20.5	20.5	40.0	+0.20 [-2.60, +2.75]	tie	14.3
2	50	1,000	10	14.0	11.5	50.0	+1.60 [-0.30, +3.20]	tie	1.1
4	50	1,000	20	49.0	51.5	100.0	-0.95 [-3.85, +1.80]	tie	2.9
2	50	5,000	20	10.5	11.0	50.0	-0.25 [-2.30, +1.65]	tie	3.8
4	50	5,000	14	36.5	39.0	100.0	-2.00 [-4.71, +0.50]	tie	13.9
2	100	1,000	20	26.5	31.0	100.0	-5.20 [-8.50, -1.85]	better	1.7
4	100	1,000	20	94.5	92.0	200.0	+0.65 [-3.10, +4.50]	tie	3.2
2	100	5,000	20	23.5	24.0	100.0	-1.85 [-6.15, +2.45]	tie	3.7
4	100	5,000	14	74.0	74.0	200.0	+1.57 [-2.64, +5.93]	tie	26.0

Other changes in 0.12.0#

The calibration changes of 0.12.0 are measured in the docstrings of the code they change, on the API pages, and in the changelog, not on this card: the normal_scores test, the calibrated G-square and chi-square tests, the false discovery rate control of PCMCI and Granger, the Markov check (andrey.markov_check), and the bootstrap’s subsampling and averaged graph. The upgrade guide shows the renamed and removed arguments.

Limits#

The settings cover 20 to 100 variables and 1,000 to 5,000 rows on linear-Gaussian data with Fisher-z tests. Larger graphs, other data types, and other tests are not measured. bnlearn reached the 600 s cap in some settings at 5,000 rows and mean degree 4, so those settings have fewer paired datasets; each table lists the datasets it pairs. At 20 variables and mean degree 4, MMHC and RSMAX2 make 28.5 to 32 errors against 40 for the empty graph, the closest of any setting.

Reproduce#

You can find environment setup, campaign drivers, and scores in the benchmark harness guide. To rerun the comparison, install Andrey 0.12.0 (andrey-core) and bnlearn 5.2.1 with R 4.6.1 in the harness’s R environment. Every driver shuffles the columns by default. Times will differ with the hardware and build; graphs should match closely.

Comparison	Lane	Sizes	Datasets x repeats	Wall cap (s)
Inter-IAMB, MMHC, and RSMAX2 against bnlearn	the harness’s R lane	d 20, 50, 100; n 1,000, 5,000; mean degree 2, 4	10 to 20 x 1	600

Builds and versions#

Every fit ran on build 31f418d on AMD EPYC 7713 and 9534 and Intel Xeon Gold 6542Y processors. Later commits change src/ only in method status, docstrings, the text of the argument concepts, and the bootstrap’s averaged-graph fallback; none changes the output of a method fit, checked bit for bit on every method. The bootstrap’s averaged graph changes only where its fallback would have returned an invalid graph: 6 of the 120 bootstraps checked, now valid.