---
name: benchmarks-0.12.0
description: The Andrey 0.12.0 benchmarks - Inter-IAMB, MMHC, and RSMAX2 against bnlearn, on the CPDAG, the skeleton, and the moral graph each returns.
meta:
  type: guide
  tags: [benchmarks, performance, accuracy, constraint-based, hybrid]
---

# 0.12.0 release card: calibration

:::::{div} release-opening

:::{card}
:class-card: release-cover release-art release-art-0-12-0
:shadow: none

Release card

Calibration

0.12.0
:::

::::{div} release-opening-text

This card measures the three methods 0.12.0 makes supported, Inter-IAMB, MMHC, and RSMAX2, against
bnlearn on the graph each returns. Each result holds for the measured conditions; it depends on the
data, settings, and hardware.

::::
:::::

Inter-IAMB, MMHC, and RSMAX2 are supported methods. We compare each with bnlearn 5.2.1 on the
graph it returns, scored against that graph's own truth: the CPDAG for MMHC and RSMAX2 and for
Inter-IAMB's default `structure="cpdag"`, the skeleton for `structure="skeleton"`, and the moral
graph for `structure="moral"`, which bnlearn's `learn.mb` blankets joined by the AND rule give.

## Summary

| Method | Graph | Settings | Worse | Tie | Better | Largest difference D, mean [95%] | Median time ratio |
|---|---|---:|---:|---:|---:|---|---:|
| Inter-IAMB | CPDAG | 12 | 0 | 12 | 0 | +0.26 [-0.58, +1.16] (d 50, n 5,000, degree 4) | 3.8 |
| Inter-IAMB | skeleton | 12 | 0 | 12 | 0 | +0.07 [+0.00, +0.20] (d 100, n 5,000, degree 2) | 3.4 |
| Inter-IAMB | moral graph | 12 | 0 | 12 | 0 | +0.50 [-0.30, +1.55] (d 100, n 5,000, degree 4) | 1.9 |
| MMHC | CPDAG | 12 | 0 | 11 | 1 | +1.80 [-0.10, +3.50] (d 50, n 1,000, degree 2) | 6.6 |
| RSMAX2 | CPDAG | 12 | 0 | 11 | 1 | +1.80 [-0.10, +3.50] (d 50, n 1,000, degree 2) | 7.8 |
| RSMAX2, IAMB restrict | CPDAG, restrict iamb | 12 | 0 | 11 | 1 | +1.60 [-0.30, +3.20] (d 50, n 1,000, degree 2) | 3.5 |

A setting is one size, sample size, and mean degree. D is Andrey's error minus bnlearn's on the
same dataset. A setting with at least 10 paired datasets is worse when the paired 95% bootstrap
interval of the mean D lies wholly above 0, better when wholly below, and a tie otherwise. The
time ratio is bnlearn's fit time over Andrey's, the median of the settings' median ratios. In no
setting does bnlearn make significantly fewer errors.

## Method

**Data.** Linear-Gaussian data from Erdos-Renyi graphs, standardized, with 20, 50, and 100
variables, 1,000 and 5,000 rows, and mean degree 2 and 4. Each setting has up to 20 datasets, one
per seed, with the columns in a shuffled order; every setting keeps at least 10 paired datasets.

**Fits.** One fit at a time on one core, Fisher-z tests at alpha 0.05 (bnlearn `test="zf"`) and the
Gaussian BIC for the score phase (bnlearn `score="bic-g"`), a 600 s cap per fit, and one untimed
warm-up fit except at 5,000 rows, mean degree 4, and 50 or more variables, where there is none. A
fit that reaches the cap is a timeout. It is counted as coverage, left out of the pairs, and never
scored as an empty graph. The Andrey fits all finished; the timeouts are bnlearn's.

**Errors.** The structural Hamming distance of the returned graph to its own truth. The tables give
the median over the paired datasets, for Andrey, bnlearn, and the empty graph; D is the mean of the
per-dataset differences, the statistic of the verdict. A DAG that bnlearn returns is compared
through its CPDAG.

## Inter-IAMB, MMHC, and RSMAX2

### Inter-IAMB

CPDAG, `structure="cpdag"`, against bnlearn `inter.iamb()`:

| Degree | d | n | Datasets | Andrey | bnlearn | Empty graph | D, mean [95%] | Verdict | bnlearn time / Andrey time |
|---:|---:|---:|---:|---:|---:|---:|---|---|---:|
| 2 | 20 | 1,000 | 20 | 4.0 | 4.0 | 20.0 | +0.00 [+0.00, +0.00] | tie | 1.9 |
| 4 | 20 | 1,000 | 20 | 24.0 | 23.5 | 40.0 | +0.25 [+0.00, +0.55] | tie | 2.5 |
| 2 | 20 | 5,000 | 20 | 1.0 | 1.0 | 20.0 | +0.00 [+0.00, +0.00] | tie | 6.6 |
| 4 | 20 | 5,000 | 20 | 18.0 | 18.0 | 40.0 | +0.00 [-0.45, +0.40] | tie | 14.9 |
| 2 | 50 | 1,000 | 20 | 7.5 | 7.5 | 50.0 | +0.00 [+0.00, +0.00] | tie | 1.7 |
| 4 | 50 | 1,000 | 20 | 54.5 | 51.5 | 100.0 | +0.25 [-0.50, +1.15] | tie | 2.7 |
| 2 | 50 | 5,000 | 20 | 8.5 | 8.5 | 50.0 | +0.15 [+0.00, +0.40] | tie | 3.5 |
| 4 | 50 | 5,000 | 19 | 42.0 | 41.0 | 100.0 | +0.26 [-0.58, +1.16] | tie | 20.8 |
| 2 | 100 | 1,000 | 20 | 32.0 | 32.0 | 100.0 | +0.15 [+0.00, +0.30] | tie | 1.4 |
| 4 | 100 | 1,000 | 14 | 104.5 | 101.5 | 200.0 | -0.14 [-1.86, +1.57] | tie | 4.2 |
| 2 | 100 | 5,000 | 15 | 24.0 | 24.0 | 100.0 | +0.00 [-0.20, +0.20] | tie | 4.4 |
| 4 | 100 | 5,000 | 12 | 71.5 | 74.0 | 200.0 | -0.58 [-1.67, +0.58] | tie | 30.3 |

Skeleton, `structure="skeleton"`, against bnlearn `inter.iamb(undirected=TRUE)`:

| Degree | d | n | Datasets | Andrey | bnlearn | Empty graph | D, mean [95%] | Verdict | bnlearn time / Andrey time |
|---:|---:|---:|---:|---:|---:|---:|---|---|---:|
| 2 | 20 | 1,000 | 20 | 1.0 | 1.0 | 20.0 | +0.00 [+0.00, +0.00] | tie | 1.7 |
| 4 | 20 | 1,000 | 20 | 14.5 | 14.5 | 40.0 | +0.00 [+0.00, +0.00] | tie | 2.5 |
| 2 | 20 | 5,000 | 20 | 1.0 | 1.0 | 20.0 | +0.00 [+0.00, +0.00] | tie | 4.7 |
| 4 | 20 | 5,000 | 20 | 10.5 | 10.5 | 40.0 | +0.00 [+0.00, +0.00] | tie | 10.7 |
| 2 | 50 | 1,000 | 20 | 6.0 | 6.0 | 50.0 | +0.00 [+0.00, +0.00] | tie | 1.7 |
| 4 | 50 | 1,000 | 20 | 32.5 | 32.5 | 100.0 | +0.05 [+0.00, +0.15] | tie | 2.4 |
| 2 | 50 | 5,000 | 20 | 5.0 | 5.0 | 50.0 | +0.00 [+0.00, +0.00] | tie | 3.4 |
| 4 | 50 | 5,000 | 19 | 24.0 | 24.0 | 100.0 | +0.00 [+0.00, +0.00] | tie | 11.6 |
| 2 | 100 | 1,000 | 20 | 23.0 | 23.0 | 100.0 | +0.00 [+0.00, +0.00] | tie | 1.3 |
| 4 | 100 | 1,000 | 14 | 64.0 | 64.0 | 200.0 | +0.00 [+0.00, +0.00] | tie | 3.4 |
| 2 | 100 | 5,000 | 15 | 19.0 | 19.0 | 100.0 | +0.07 [+0.00, +0.20] | tie | 4.2 |
| 4 | 100 | 5,000 | 15 | 44.0 | 44.0 | 200.0 | +0.07 [+0.00, +0.20] | tie | 25.0 |

Moral graph, `structure="moral"`, against bnlearn `learn.mb` joined by the AND rule:

| Degree | d | n | Datasets | Andrey | bnlearn | Empty graph | D, mean [95%] | Verdict | bnlearn time / Andrey time |
|---:|---:|---:|---:|---:|---:|---:|---|---|---:|
| 2 | 20 | 1,000 | 20 | 1.5 | 1.5 | 30.5 | +0.00 [+0.00, +0.00] | tie | 1.2 |
| 4 | 20 | 1,000 | 20 | 8.0 | 8.0 | 71.0 | +0.00 [+0.00, +0.00] | tie | 1.0 |
| 2 | 20 | 5,000 | 20 | 1.0 | 1.0 | 30.5 | -0.05 [-0.15, +0.00] | tie | 4.1 |
| 4 | 20 | 5,000 | 20 | 1.5 | 1.5 | 71.0 | +0.00 [+0.00, +0.00] | tie | 3.7 |
| 2 | 50 | 1,000 | 20 | 8.0 | 8.0 | 80.0 | +0.00 [+0.00, +0.00] | tie | 1.5 |
| 4 | 50 | 1,000 | 20 | 40.0 | 38.0 | 208.5 | +0.30 [+0.00, +0.80] | tie | 1.2 |
| 2 | 50 | 5,000 | 20 | 6.5 | 6.5 | 80.0 | -0.15 [-0.45, +0.00] | tie | 3.0 |
| 4 | 50 | 5,000 | 20 | 7.0 | 7.0 | 208.5 | +0.05 [+0.00, +0.15] | tie | 2.0 |
| 2 | 100 | 1,000 | 20 | 33.0 | 33.0 | 168.5 | +0.00 [+0.00, +0.00] | tie | 1.3 |
| 4 | 100 | 1,000 | 14 | 125.5 | 125.5 | 448.0 | +0.07 [+0.00, +0.21] | tie | 1.8 |
| 2 | 100 | 5,000 | 15 | 23.0 | 23.0 | 171.0 | +0.07 [+0.00, +0.20] | tie | 4.2 |
| 4 | 100 | 5,000 | 20 | 35.0 | 35.0 | 445.0 | +0.50 [-0.30, +1.55] | tie | 3.4 |

### MMHC

CPDAG against bnlearn `mmhc()`:

| Degree | d | n | Datasets | Andrey | bnlearn | Empty graph | D, mean [95%] | Verdict | bnlearn time / Andrey time |
|---:|---:|---:|---:|---:|---:|---:|---|---|---:|
| 2 | 20 | 1,000 | 10 | 5.0 | 5.5 | 20.0 | -0.10 [-2.00, +1.90] | tie | 3.7 |
| 4 | 20 | 1,000 | 20 | 32.0 | 31.5 | 40.0 | -0.40 [-2.35, +1.85] | tie | 3.8 |
| 2 | 20 | 5,000 | 20 | 5.5 | 5.0 | 20.0 | +0.05 [-1.25, +1.45] | tie | 10.8 |
| 4 | 20 | 5,000 | 20 | 28.5 | 30.5 | 40.0 | -1.45 [-3.55, +0.55] | tie | 24.3 |
| 2 | 50 | 1,000 | 10 | 16.5 | 15.0 | 50.0 | +1.80 [-0.10, +3.50] | tie | 4.1 |
| 4 | 50 | 1,000 | 20 | 65.5 | 64.5 | 100.0 | -0.50 [-3.10, +1.90] | tie | 6.2 |
| 2 | 50 | 5,000 | 20 | 11.5 | 12.5 | 50.0 | -0.05 [-1.50, +1.30] | tie | 11.4 |
| 4 | 50 | 5,000 | 14 | 62.0 | 62.0 | 100.0 | +0.00 [-3.07, +3.07] | tie | 6.2 |
| 2 | 100 | 1,000 | 10 | 24.0 | 33.5 | 100.0 | -5.80 [-10.20, -1.20] | better | 4.5 |
| 4 | 100 | 1,000 | 20 | 121.0 | 127.5 | 200.0 | -2.05 [-4.95, +0.70] | tie | 7.0 |
| 2 | 100 | 5,000 | 20 | 22.5 | 26.5 | 100.0 | -2.70 [-6.70, +1.35] | tie | 14.0 |
| 4 | 100 | 5,000 | 16 | 112.5 | 119.0 | 200.0 | -1.12 [-5.31, +3.50] | tie | 19.4 |

### RSMAX2

CPDAG with `restrict="mmpc"` and `maximize="hc"`, the defaults, against bnlearn `rsmax2()` with the
same phases:

| Degree | d | n | Datasets | Andrey | bnlearn | Empty graph | D, mean [95%] | Verdict | bnlearn time / Andrey time |
|---:|---:|---:|---:|---:|---:|---:|---|---|---:|
| 2 | 20 | 1,000 | 20 | 6.5 | 6.0 | 20.0 | -0.35 [-2.30, +1.45] | tie | 4.4 |
| 4 | 20 | 1,000 | 20 | 32.0 | 31.5 | 40.0 | -0.40 [-2.35, +1.75] | tie | 4.2 |
| 2 | 20 | 5,000 | 20 | 5.5 | 5.0 | 20.0 | +0.05 [-1.30, +1.45] | tie | 7.5 |
| 4 | 20 | 5,000 | 20 | 28.5 | 30.5 | 40.0 | -1.45 [-3.55, +0.50] | tie | 19.2 |
| 2 | 50 | 1,000 | 10 | 16.5 | 15.0 | 50.0 | +1.80 [-0.10, +3.50] | tie | 3.9 |
| 4 | 50 | 1,000 | 20 | 65.5 | 64.5 | 100.0 | -0.50 [-3.10, +1.90] | tie | 4.8 |
| 2 | 50 | 5,000 | 20 | 11.5 | 12.5 | 50.0 | -0.05 [-1.50, +1.30] | tie | 11.2 |
| 4 | 50 | 5,000 | 15 | 62.0 | 62.0 | 100.0 | -0.80 [-3.40, +2.00] | tie | 8.0 |
| 2 | 100 | 1,000 | 20 | 28.5 | 34.0 | 100.0 | -5.80 [-9.40, -2.20] | better | 7.2 |
| 4 | 100 | 1,000 | 20 | 121.0 | 127.5 | 200.0 | -2.05 [-4.95, +0.65] | tie | 8.9 |
| 2 | 100 | 5,000 | 20 | 22.5 | 26.5 | 100.0 | -2.70 [-6.85, +1.35] | tie | 12.0 |
| 4 | 100 | 5,000 | 20 | 112.5 | 119.0 | 200.0 | -1.50 [-5.05, +2.25] | tie | 14.6 |

CPDAG with `restrict="iamb"`, whose restrict phase keeps the skeleton of IAMB, as bnlearn's does:

| Degree | d | n | Datasets | Andrey | bnlearn | Empty graph | D, mean [95%] | Verdict | bnlearn time / Andrey time |
|---:|---:|---:|---:|---:|---:|---:|---|---|---:|
| 2 | 20 | 1,000 | 20 | 4.0 | 5.5 | 20.0 | -0.60 [-2.50, +1.10] | tie | 1.4 |
| 4 | 20 | 1,000 | 20 | 24.5 | 25.5 | 40.0 | -0.30 [-1.95, +1.50] | tie | 3.1 |
| 2 | 20 | 5,000 | 20 | 4.0 | 4.5 | 20.0 | -0.10 [-1.45, +1.45] | tie | 4.0 |
| 4 | 20 | 5,000 | 20 | 20.5 | 20.5 | 40.0 | +0.20 [-2.60, +2.75] | tie | 14.3 |
| 2 | 50 | 1,000 | 10 | 14.0 | 11.5 | 50.0 | +1.60 [-0.30, +3.20] | tie | 1.1 |
| 4 | 50 | 1,000 | 20 | 49.0 | 51.5 | 100.0 | -0.95 [-3.85, +1.80] | tie | 2.9 |
| 2 | 50 | 5,000 | 20 | 10.5 | 11.0 | 50.0 | -0.25 [-2.30, +1.65] | tie | 3.8 |
| 4 | 50 | 5,000 | 14 | 36.5 | 39.0 | 100.0 | -2.00 [-4.71, +0.50] | tie | 13.9 |
| 2 | 100 | 1,000 | 20 | 26.5 | 31.0 | 100.0 | -5.20 [-8.50, -1.85] | better | 1.7 |
| 4 | 100 | 1,000 | 20 | 94.5 | 92.0 | 200.0 | +0.65 [-3.10, +4.50] | tie | 3.2 |
| 2 | 100 | 5,000 | 20 | 23.5 | 24.0 | 100.0 | -1.85 [-6.15, +2.45] | tie | 3.7 |
| 4 | 100 | 5,000 | 14 | 74.0 | 74.0 | 200.0 | +1.57 [-2.64, +5.93] | tie | 26.0 |

## Other changes in 0.12.0

The calibration changes of 0.12.0 are measured in the docstrings of the code they change, on the
[API pages](../api.md), and in the [changelog](../changelog.md), not on this card: the
`normal_scores` test, the calibrated G-square and chi-square tests, the false discovery rate
control of PCMCI and Granger, the Markov check (`andrey.markov_check`), and the bootstrap's
subsampling and averaged graph. The [upgrade guide](../guides/upgrading.md) shows the renamed and
removed arguments.

## Limits

The settings cover 20 to 100 variables and 1,000 to 5,000 rows on linear-Gaussian data with
Fisher-z tests. Larger graphs, other data types, and other tests are not measured. bnlearn reached
the 600 s cap in some settings at 5,000 rows and mean degree 4, so those settings have fewer paired
datasets; each table lists the datasets it pairs. At 20 variables and mean degree 4, MMHC and
RSMAX2 make 28.5 to 32 errors against 40 for the empty graph, the closest of any setting.

## Reproduce

You can find environment setup, campaign drivers, and scores in the [benchmark harness
guide][harness]. To rerun the comparison, install Andrey 0.12.0 (andrey-core) and bnlearn 5.2.1 with
R 4.6.1 in the harness's R environment. Every driver shuffles the columns by default. Times will
differ with the hardware and build; graphs should match closely.

| Comparison | Lane | Sizes | Datasets x repeats | Wall cap (s) |
|---|---|---|---|---:|
| Inter-IAMB, MMHC, and RSMAX2 against bnlearn | the harness's R lane | d 20, 50, 100; n 1,000, 5,000; mean degree 2, 4 | 10 to 20 x 1 | 600 |

[harness]: https://github.com/Abel-ai-lab/andrey/blob/main/benchmarks/README.md

## Builds and versions

Every fit ran on build `31f418d` on AMD EPYC 7713 and 9534 and Intel Xeon Gold 6542Y processors.
Later commits change `src/` only in method status, docstrings, the text of the argument concepts,
and the bootstrap's averaged-graph fallback; none changes the output of a method fit, checked bit
for bit on every method. The bootstrap's averaged graph changes only where its fallback would have
returned an invalid graph: 6 of the 120 bootstraps checked, now valid.