0.4.0 release card: pairwise direction#
Release card
Pairwise direction
0.4.0
We compare Andrey’s two pairwise direction tests, ANM and PNL, with causal-learn, which implements both. ANM names the same direction as causal-learn on every pair and runs about 2x faster. PNL runs up to 1,470x faster and names the true direction on more pairs than causal-learn. We support both from 0.4.0.
A direction test names a direction on every pair, right or wrong. At 200 rows, both methods name the reversed direction on a quarter of the pairs (see Known limits). You can find speedups and accuracy in the summary, and measurement and scoring definitions in how to read the results. These results apply to the measured data and hardware.
Summary#
causal-learn is the baseline. We report every paired speed ratio, regardless of accuracy.
Speed.
Method |
Compared with |
Rows ( |
Cores per fit |
Andrey speedup |
|---|---|---|---|---|
ANM |
causal-learn |
200-8,000 |
4 |
1.88-2.47x |
PNL |
causal-learn |
200-5,000 |
4 |
275-1,470x |
Each bar is the median, over the pairs of that size, of causal-learn's time divided by Andrey's on the same pair. Each method's bars share its own scale.
Accuracy.
Method |
Compared with |
Same graph |
True direction, Andrey |
True direction, causal-learn |
|---|---|---|---|---|
ANM |
causal-learn |
95 of 95 |
84 of 95 |
84 of 95 |
PNL |
causal-learn |
47 of 100 |
94 of 100 |
45 of 100 |
How to read the results#
Data. We evaluate ANM on anm_pair and PNL on pnl_pair: cause-effect pairs of two
standardized variables with one edge, in a direction each seed draws. In anm_pair, the effect is
w * (x + sin x) plus Gaussian noise; in pnl_pair, it is g2(w * x + e) with
g2(z) = z + 0.5 tanh(z) and Laplace noise e. In both, |w| lies between 0.5 and 2.
Accuracy. Each package returns a two-node DAG with at most one edge. Both name the direction
with the larger p-value; on equal p-values, Andrey takes the direction with the smaller HSIC
statistic relative to its mean under independence, and causal-learn draws no edge. Each answer is
the true direction, the reversed edge, or no edge; every fit here drew an edge. “Same graph” means
equal structure_hash values: both packages return the same edge. The empty graph names no
direction, so it names the true direction on no pair.
Timing. Times are seconds per fit, the median over a size’s pairs; each pair is fitted once. For each pair, we divide causal-learn’s time by Andrey’s, then report the median of these paired ratios. This need not equal the ratio of the displayed median times. No fit reached its time limit.
Hardware. Every fit ran on an Intel Core i7-7820X with 4 cores per fit.
Recorded campaign#
Comparison |
n |
Andrey build |
Measured package versions |
|---|---|---|---|
ANM |
200-8,000 |
Pre-release 0.4.0, October 12, 2026 |
causal-learn 0.1.4.8, torch 2.14.1+cpu |
PNL |
200-5,000 |
Pre-release 0.4.0, October 12, 2026 |
causal-learn 0.1.4.8, torch 2.14.1+cpu |
We measured a pre-release Andrey 0.4.0 build of October 12, 2026. The
summary contains one row
per method and size n. Each row gives median seconds, the paired ratio, the pairs both packages
finished (paired) and those with the same graph (same_graph), and for each package the pairs it
answered with the true direction, the reversed edge, or no edge, with the share of the first.
fits, andrey_ok, and comparator_ok count the fits and those that finished.
Pairwise methods#
ANM and PNL take two columns, x and y, and test each direction. Each fits a model of the effect
given the proposed cause, then tests whether the estimated noise is independent of that cause, which
holds only in the causal direction.
ANM#
ANM fits y = f(x) + e in each direction by Gaussian-process regression and tests the residual with
HSIC. Both packages fit the same Gaussian process and the same HSIC test. Andrey standardizes each
column first, so its answer does not depend on the units; these pairs are standardized already.
n |
Andrey (s) |
causal-learn (s) |
Speedup |
|---|---|---|---|
200 |
0.0907 |
0.178 |
2.08x |
500 |
0.624 |
1.11 |
1.88x |
1,000 |
2.00 |
3.94 |
2.05x |
2,000 |
7.65 |
18.0 |
2.38x |
4,000 |
35.9 |
87.5 |
2.47x |
8,000 |
196 |
472 |
2.39x |
n |
Pairs |
Same graph |
True direction, Andrey |
True direction, causal-learn |
|---|---|---|---|---|
200 |
20 |
20 of 20 |
15 of 20 |
15 of 20 |
500 |
20 |
20 of 20 |
16 of 20 |
16 of 20 |
1,000 |
20 |
20 of 20 |
19 of 20 |
19 of 20 |
2,000 |
20 |
20 of 20 |
19 of 20 |
19 of 20 |
4,000 |
10 |
10 of 10 |
10 of 10 |
10 of 10 |
8,000 |
5 |
5 of 5 |
5 of 5 |
5 of 5 |
ANM returns the same graph as causal-learn on every pair, so their direction columns are equal.
PNL#
PNL fits y = f2(f1(x) + e) in each direction. Andrey fits each transform as a sum of smooth basis
functions by Newton’s method and tests the estimated noise’s independence of the proposed cause with
HSIC. causal-learn trains two small neural networks per direction and compares the mean of the
proposed cause with the mean of the estimated noise by a two-sample t-test, with p-values rounded to
three decimals.
n |
Andrey (s) |
causal-learn (s) |
Speedup |
|---|---|---|---|
200 |
0.00892 |
13.2 |
1,470x |
500 |
0.0283 |
37.3 |
1,310x |
1,000 |
0.0861 |
85.5 |
993x |
2,000 |
0.270 |
187 |
696x |
5,000 |
1.72 |
475 |
275x |
n |
Pairs |
Same graph |
True direction, Andrey |
True direction, causal-learn |
|---|---|---|---|---|
200 |
20 |
11 of 20 |
15 of 20 |
10 of 20 |
500 |
20 |
11 of 20 |
19 of 20 |
10 of 20 |
1,000 |
20 |
10 of 20 |
20 of 20 |
10 of 20 |
2,000 |
20 |
10 of 20 |
20 of 20 |
10 of 20 |
5,000 |
20 |
5 of 20 |
20 of 20 |
5 of 20 |
Andrey names the true direction on more pairs than causal-learn at every size, and on every pair from 1,000 rows.
Known limits#
Data coverage. We measure one simulated regime per method, standardized. These results do not establish performance on measured pairs or on other mechanisms and noise.
Small samples. At 200 rows, both methods name the reversed direction on a quarter of the pairs.
A direction on every pair. With the default
alpha=None, both methods name a direction whenever the p-values differ, even for unrelated variables or a pair whose direction cannot be identified, such as a linear pair with Gaussian noise. A significance level draws an edge only when exactly one direction’s p-value exceeds it; these campaigns measure the default only.ANM fit times. ANM’s regression time grows with the cube of the rows: its median fit takes over three minutes at 8,000 rows.
Reproduce#
The benchmark harness guide documents environment setup, the campaign drivers, and the scores. To rerun the comparisons, install Andrey 0.4.0 (andrey-core), causal-learn 0.1.4.8, and torch 2.14.1+cpu, then run the drivers with the settings below. Times will differ with the hardware and build; directions should match closely.
Comparison |
Driver |
n |
Pairs x repeats |
Wall cap (s) |
Cores |
|---|---|---|---|---|---|
ANM |
|
200-2,000 |
20 x 1 |
1,800 |
4 |
ANM |
|
4,000 |
10 x 1 |
1,800 |
4 |
ANM |
|
8,000 |
5 x 1 |
1,800 |
4 |
PNL |
|
200-2,000 |
20 x 1 |
1,200 |
4 |
PNL |
|
5,000 |
20 x 1 |
2,400 |
4 |
$ANDREY_BENCH_PYTHON benchmarks/anm_pairs.py --out outputs/anm-pairs \
--sizes 200 500 1000 2000 --seeds 20 --repeats 1 --cap-wall-s 1800 --cores 4
$ANDREY_BENCH_PYTHON benchmarks/anm_pairs.py --out outputs/anm-pairs-4000 \
--sizes 4000 --seeds 10 --repeats 1 --cap-wall-s 1800 --cores 4
$ANDREY_BENCH_PYTHON benchmarks/anm_pairs.py --out outputs/anm-pairs-8000 \
--sizes 8000 --seeds 5 --repeats 1 --cap-wall-s 1800 --cores 4
$ANDREY_BENCH_PYTHON benchmarks/pnl_pairs.py --out outputs/pnl-pairs \
--sizes 200 500 1000 2000 --seeds 20 --repeats 1 --cap-wall-s 1200 --cores 4
$ANDREY_BENCH_PYTHON benchmarks/pnl_pairs.py --out outputs/pnl-pairs-5000 \
--sizes 5000 --seeds 20 --repeats 1 --cap-wall-s 2400 --cores 4