---
name: benchmarks-0.4.0
description: The Andrey 0.4.0 release card - ANM and PNL compared with causal-learn on simulated cause-effect pairs.
meta:
  type: guide
  tags: [benchmarks, performance, accuracy, pairwise]
---

# 0.4.0 release card: pairwise direction

% The release's cover, as on the Release cards page, beside the opening (_static/benchmarks.css).
:::::{div} release-opening

:::{card}
:class-card: release-cover release-art release-art-0-4-0
:shadow: none

Release card

Pairwise direction

0.4.0
:::

::::{div} release-opening-text

We compare Andrey's two pairwise direction tests, ANM and PNL, with causal-learn, which implements
both. ANM names the same direction as causal-learn on every pair and runs about 2x faster. PNL runs
up to 1,470x faster and names the true direction on more pairs than causal-learn. We support both
from 0.4.0.

A direction test names a direction on every pair, right or wrong. At 200 rows, both methods name the
reversed direction on a quarter of the pairs (see [Known limits](#known-limits)). You can find
speedups and accuracy in the [summary](#summary), and measurement and scoring definitions in [how to
read the results](#how-to-read-the-results). These results apply to the measured data and hardware.
::::
:::::

## Summary

causal-learn is the baseline. We report every paired speed ratio, regardless of accuracy.

**Speed.**

| Method | Compared with | Rows (`n`) | Cores per fit | Andrey speedup |
|---|---|---|---:|---|
| ANM | causal-learn | 200-8,000 | 4 | 1.88-2.47x |
| PNL | causal-learn | 200-5,000 | 4 | 275-1,470x |

```{raw} html
:file: 0.4.0/speedup.html
```

**Accuracy.**

| Method | Compared with | Same graph | True direction, Andrey | True direction, causal-learn |
|---|---|---|---|---|
| ANM | causal-learn | 95 of 95 | 84 of 95 | 84 of 95 |
| PNL | causal-learn | 47 of 100 | 94 of 100 | 45 of 100 |

## How to read the results

**Data.** We evaluate ANM on `anm_pair` and PNL on `pnl_pair`: cause-effect pairs of two
standardized variables with one edge, in a direction each seed draws. In `anm_pair`, the effect is
`w * (x + sin x)` plus Gaussian noise; in `pnl_pair`, it is `g2(w * x + e)` with
`g2(z) = z + 0.5 tanh(z)` and Laplace noise `e`. In both, `|w|` lies between 0.5 and 2.

**Accuracy.** Each package returns a two-node DAG with at most one edge. Both name the direction
with the larger p-value; on equal p-values, Andrey takes the direction with the smaller HSIC
statistic relative to its mean under independence, and causal-learn draws no edge. Each answer is
the true direction, the reversed edge, or no edge; every fit here drew an edge. "Same graph" means
equal `structure_hash` values: both packages return the same edge. The empty graph names no
direction, so it names the true direction on no pair.

**Timing.** Times are seconds per fit, the median over a size's pairs; each pair is fitted once. For
each pair, we divide causal-learn's time by Andrey's, then report the median of these paired ratios.
This need not equal the ratio of the displayed median times. No fit reached its time limit.

**Hardware.** Every fit ran on an Intel Core i7-7820X with 4 cores per fit.

## Recorded campaign

| Comparison | n | Andrey build | Measured package versions |
|---|---|---|---|
| ANM | 200-8,000 | Pre-release 0.4.0, October 12, 2026 | causal-learn 0.1.4.8, torch 2.14.1+cpu |
| PNL | 200-5,000 | Pre-release 0.4.0, October 12, 2026 | causal-learn 0.1.4.8, torch 2.14.1+cpu |

We measured a pre-release Andrey 0.4.0 build of October 12, 2026. The
{download}`summary <../../../benchmarks/published/0.4.0/summary.json>` contains one row
per method and size `n`. Each row gives median seconds, the paired ratio, the pairs both packages
finished (`paired`) and those with the same graph (`same_graph`), and for each package the pairs it
answered with the true direction, the reversed edge, or no edge, with the share of the first.
`fits`, `andrey_ok`, and `comparator_ok` count the fits and those that finished.

## Pairwise methods

ANM and PNL take two columns, `x` and `y`, and test each direction. Each fits a model of the effect
given the proposed cause, then tests whether the estimated noise is independent of that cause, which
holds only in the causal direction.

### ANM

ANM fits `y = f(x) + e` in each direction by Gaussian-process regression and tests the residual with
HSIC. Both packages fit the same Gaussian process and the same HSIC test. Andrey standardizes each
column first, so its answer does not depend on the units; these pairs are standardized already.

| n | Andrey (s) | causal-learn (s) | Speedup |
|---:|---:|---:|---:|
| 200 | 0.0907 | 0.178 | 2.08x |
| 500 | 0.624 | 1.11 | 1.88x |
| 1,000 | 2.00 | 3.94 | 2.05x |
| 2,000 | 7.65 | 18.0 | 2.38x |
| 4,000 | 35.9 | 87.5 | 2.47x |
| 8,000 | 196 | 472 | 2.39x |

| n | Pairs | Same graph | True direction, Andrey | True direction, causal-learn |
|---:|---:|---:|---:|---:|
| 200 | 20 | 20 of 20 | 15 of 20 | 15 of 20 |
| 500 | 20 | 20 of 20 | 16 of 20 | 16 of 20 |
| 1,000 | 20 | 20 of 20 | 19 of 20 | 19 of 20 |
| 2,000 | 20 | 20 of 20 | 19 of 20 | 19 of 20 |
| 4,000 | 10 | 10 of 10 | 10 of 10 | 10 of 10 |
| 8,000 | 5 | 5 of 5 | 5 of 5 | 5 of 5 |

ANM returns the same graph as causal-learn on every pair, so their direction columns are equal.

### PNL

PNL fits `y = f2(f1(x) + e)` in each direction. Andrey fits each transform as a sum of smooth basis
functions by Newton's method and tests the estimated noise's independence of the proposed cause with
HSIC. causal-learn trains two small neural networks per direction and compares the mean of the
proposed cause with the mean of the estimated noise by a two-sample t-test, with p-values rounded to
three decimals.

| n | Andrey (s) | causal-learn (s) | Speedup |
|---:|---:|---:|---:|
| 200 | 0.00892 | 13.2 | 1,470x |
| 500 | 0.0283 | 37.3 | 1,310x |
| 1,000 | 0.0861 | 85.5 | 993x |
| 2,000 | 0.270 | 187 | 696x |
| 5,000 | 1.72 | 475 | 275x |

| n | Pairs | Same graph | True direction, Andrey | True direction, causal-learn |
|---:|---:|---:|---:|---:|
| 200 | 20 | 11 of 20 | 15 of 20 | 10 of 20 |
| 500 | 20 | 11 of 20 | 19 of 20 | 10 of 20 |
| 1,000 | 20 | 10 of 20 | 20 of 20 | 10 of 20 |
| 2,000 | 20 | 10 of 20 | 20 of 20 | 10 of 20 |
| 5,000 | 20 | 5 of 20 | 20 of 20 | 5 of 20 |

Andrey names the true direction on more pairs than causal-learn at every size, and on every pair
from 1,000 rows.

## Known limits

- **Data coverage.** We measure one simulated regime per method, standardized. These results do not
  establish performance on measured pairs or on other mechanisms and noise.
- **Small samples.** At 200 rows, both methods name the reversed direction on a quarter of the
  pairs.
- **A direction on every pair.** With the default `alpha=None`, both methods name a direction
  whenever the p-values differ, even for unrelated variables or a pair whose direction cannot be
  identified, such as a linear pair with Gaussian noise. A significance level draws an edge only
  when exactly one direction's p-value exceeds it; these campaigns measure the default only.
- **ANM fit times.** ANM's regression time grows with the cube of the rows: its median fit takes
  over three minutes at 8,000 rows.

## Reproduce

The [benchmark harness guide][harness] documents environment setup, the campaign drivers, and the
scores. To rerun the comparisons, install Andrey 0.4.0 (andrey-core), causal-learn 0.1.4.8, and
torch 2.14.1+cpu, then run the drivers with the settings below. Times will differ with the hardware
and build; directions should match closely.

| Comparison | Driver | n | Pairs x repeats | Wall cap (s) | Cores |
|---|---|---|---|---:|---:|
| ANM | `anm_pairs.py` | 200-2,000 | 20 x 1 | 1,800 | 4 |
| ANM | `anm_pairs.py` | 4,000 | 10 x 1 | 1,800 | 4 |
| ANM | `anm_pairs.py` | 8,000 | 5 x 1 | 1,800 | 4 |
| PNL | `pnl_pairs.py` | 200-2,000 | 20 x 1 | 1,200 | 4 |
| PNL | `pnl_pairs.py` | 5,000 | 20 x 1 | 2,400 | 4 |

```shell
$ANDREY_BENCH_PYTHON benchmarks/anm_pairs.py --out outputs/anm-pairs \
    --sizes 200 500 1000 2000 --seeds 20 --repeats 1 --cap-wall-s 1800 --cores 4
$ANDREY_BENCH_PYTHON benchmarks/anm_pairs.py --out outputs/anm-pairs-4000 \
    --sizes 4000 --seeds 10 --repeats 1 --cap-wall-s 1800 --cores 4
$ANDREY_BENCH_PYTHON benchmarks/anm_pairs.py --out outputs/anm-pairs-8000 \
    --sizes 8000 --seeds 5 --repeats 1 --cap-wall-s 1800 --cores 4
$ANDREY_BENCH_PYTHON benchmarks/pnl_pairs.py --out outputs/pnl-pairs \
    --sizes 200 500 1000 2000 --seeds 20 --repeats 1 --cap-wall-s 1200 --cores 4
$ANDREY_BENCH_PYTHON benchmarks/pnl_pairs.py --out outputs/pnl-pairs-5000 \
    --sizes 5000 --seeds 20 --repeats 1 --cap-wall-s 2400 --cores 4
```

[harness]: https://github.com/Abel-ai-lab/andrey/blob/main/benchmarks/README.md