---
name: benchmarks-0.5.0
description: The Andrey 0.5.0 release card - temporal causal discovery methods compared with tigramite, causal-learn, and lingam on simulated series.
meta:
  type: guide
  tags: [benchmarks, performance, accuracy, time-series]
---

# 0.5.0 release card: temporal causal discovery

% The release's cover, as on the Release cards page, beside the opening (_static/benchmarks.css).
:::::{div} release-opening

:::{card}
:class-card: release-cover release-art release-art-0-5-0
:shadow: none

Release card

Temporal causal discovery

0.5.0
:::

::::{div} release-opening-text

We compare Andrey's time-series methods with the packages that implement them: PCMCI, PCMCI+, and
LPCMCI with tigramite, lasso Granger causality with causal-learn, and VARMA-LiNGAM and
LongitudinalLiNGAM with lingam. PCMCI+ runs more than 50x faster than tigramite at every size, and
more than 100x at 80 variables and 2,000 time steps. PCMCI runs more than 20x faster at 10
variables, where it has fewer errors than the empty graph. Both return the same graph as tigramite
on every dataset. LPCMCI runs more than 8x faster at every size, and LongitudinalLiNGAM more than
1.3x faster than lingam. We support these methods from 0.5.0.

Speed is not accuracy. On the larger graphs, both packages' PCMCI results have more errors than the
empty graph (see [Known limits](#known-limits)). You can find speedups and accuracy in the
[summary](#summary), and measurement and scoring definitions in [how to read the
results](#how-to-read-the-results). These results apply to the measured workloads. Speed and
accuracy depend on the data, settings, and hardware.
::::
:::::

## Summary

tigramite, causal-learn, and lingam are the baselines, each for the methods it implements. We report
every paired speed ratio, regardless of accuracy.

**Speed.**

| Method | Compared with | Variables (`d`) | Length (`n`) | Cores per fit | Andrey speedup |
|---|---|---|---|---:|---|
| PCMCI | tigramite | 10-160 | 500-2,000 | 4 | 23.2-166x |
| PCMCI+ | tigramite | 10-80 | 500-2,000 | 4 | 52.0-116x |
| LPCMCI | tigramite | 5-20 | 500-2,000 | 4 | 8.56-18.7x |
| Granger | causal-learn | 10-160 | 1,000 | 4 | 1.59-3.34x |
| VARMA-LiNGAM | lingam | 3-20 | 100-1,000 | 4 | 0.962-1.13x |
| LongitudinalLiNGAM | lingam | 2-5 | 20-50 | 4 | 1.33-2.24x |

```{raw} html
:file: 0.5.0/speedup.html
```

**Accuracy.**

| Method | Compared with | Datasets | Same graph | Andrey ahead | Other ahead |
|---|---|---:|---|---:|---:|
| PCMCI | tigramite | 125 | 125 of 125 | 0 | 0 |
| PCMCI+ | tigramite | 110 | 110 of 110 | 0 | 0 |
| LPCMCI | tigramite | 90 | 73 of 90 | 12 | 5 |
| Granger | causal-learn | 50 | 50 of 50 | 0 | 0 |
| VARMA-LiNGAM | lingam | 105 | 51 of 105 | 18 | 18 |
| LongitudinalLiNGAM | lingam | 240 | 231 of 240 | 8 | 1 |

"Ahead" counts the datasets on which that package's graph has fewer errors than the other's, and
"Other" is the compared package. On the rest, the two have the same number of errors.

## How to read the results

**Data.** We run each method on its paper's simulation, with one dataset per seed. PCMCI runs on
`pcmci_paper` (Runge et al., 2019): linear-Gaussian series in which every variable depends on its
own previous step, with one cross-link per variable at lag 1 or 2 and no same-step links. PCMCI+
runs on `pcmciplus_paper` (Runge, 2020), with strong autocorrelation and 1.5 cross-links per
variable, 30% of them within a time step and the rest at lags 1 to 5. LPCMCI runs on `lpcmci_paper`
(Gerhardus and Runge, 2020), like PCMCI+'s simulation, with lags up to 3 and 30% of the series
unobserved. Granger runs on `var2_lagged`: linear-Gaussian VAR(2) series with no same-step links and
each variable z-scored. VARMA-LiNGAM runs on `varma_lingam_paper` (Hyvarinen et al., 2010): a linear
series with non-Gaussian noise, a same-step DAG, a lag-1 matrix, and a moving-average matrix.
LongitudinalLiNGAM runs on `longitudinal_lingam_paper` (Kadowaki et al., 2013): `M` units observed
at `n` time points, with a same-step DAG and a lag-1 matrix drawn anew at every time point and a
hidden common cause at the first. `d` is the number of variables (series), and `n` is the number of
time steps. There are 10 datasets per rung, except 5 for PCMCI at 160 variables and VARMA-LiNGAM at
20 variables.

**Accuracy.** We count each method's errors in its own unit, defined in its section below. Lower is
better. The empty graph's errors are the true links it misses. F1 measures the window causal graph's
cross-links: links between two different variables, at a lag or within a time step. It leaves out
each variable's links from its own past (Assaad et al., 2022, Sect. 9.1). LPCMCI's adjacency F1 is
defined in its section. Errors are medians over a rung's datasets, and F1 is reported per rung.
"Same graph" means equal `structure_hash` values: the same links at every lag, with the same marks
or weights.

**Timing.** Times are seconds per fit, the median over a rung's datasets; each dataset is fitted
once. For each dataset, we divide the other package's time by Andrey's, then report the median of
these paired ratios. This need not equal the ratio of the displayed median times. No fit reached its
time limit. Times and ratios have three significant figures, as in the summary.

**Hardware.** Every fit ran on an Intel Core i7-7820X with 4 cores per fit.

## Benchmark results

| Comparison | d | Andrey build | Measured package versions |
|---|---|---|---|
| PCMCI | 10-160 | Pre-release 0.5.0, October 13, 2026 | tigramite 5.2.10.1 |
| PCMCI+ | 10-80 | Pre-release 0.5.0, October 13, 2026 | tigramite 5.2.10.1 |
| LPCMCI | 5-20 | Pre-release 0.5.0, October 13, 2026 | tigramite 5.2.10.1 |
| Granger | 10-160 | Pre-release 0.5.0, October 13, 2026 | causal-learn 0.1.4.8 |
| VARMA-LiNGAM | 3-20 | Pre-release 0.5.0, October 15, 2026 | lingam 1.13.0 |
| LongitudinalLiNGAM | 2-5 | Pre-release 0.5.0, October 15, 2026 | lingam 1.13.0 |

We measured pre-release Andrey 0.5.0 builds with the dates shown in the table. VARMA-LiNGAM ran in
lingam's own environment (Python 3.12.3, NumPy 2.2.6, SciPy 1.13.1, statsmodels 0.15.0), so both
packages fit the model with the same numerical libraries. The
{download}`summary <../../../benchmarks/published/0.5.0/summary.json>` contains one
row per method and rung. Each row gives median seconds, the paired ratio, the datasets both packages
finished (`paired`), and those with the same graph (`same_graph`). It also gives each package's
errors beside the empty graph's, the datasets on which each has fewer errors, and F1. `fits`,
`andrey_ok`, and `comparator_ok` count the fits and those that finished.

## Time-series methods

These methods learn causal links in data ordered in time, from a variable's past to another's
present. PCMCI+, LPCMCI, and the two LiNGAM methods also learn links within a time step.

### PCMCI

PCMCI finds the lagged links of a multivariate series in two steps. Its condition-selection step
keeps, for each variable, the past values its own tests cannot remove. It then runs a momentary
conditional independence test for each lagged pair, conditioned on the parents kept at both ends.
Both packages test with Student-t partial correlation at lags 1 to 2, with `pc_alpha` 0.2 and
`alpha_level` 0.05. The output is one directed graph per lag. Errors are the lag-resolved SHD: we
count the links that differ from the true links at each lag, with a reversed link counting 2, then
sum over the lags.

| d | n | Andrey (s) | tigramite (s) | Speedup |
|---:|---:|---:|---:|---:|
| 10 | 500 | 0.0316 | 0.688 | 23.2x |
| 10 | 1,000 | 0.0308 | 0.816 | 24.0x |
| 10 | 2,000 | 0.0354 | 0.988 | 27.4x |
| 20 | 500 | 0.0481 | 2.56 | 51.4x |
| 20 | 1,000 | 0.0573 | 3.19 | 58.8x |
| 20 | 2,000 | 0.0518 | 3.95 | 69.5x |
| 40 | 500 | 0.127 | 11.2 | 86.6x |
| 40 | 1,000 | 0.131 | 13.9 | 104x |
| 40 | 2,000 | 0.140 | 17.6 | 125x |
| 80 | 500 | 0.626 | 57.7 | 88.6x |
| 80 | 1,000 | 0.549 | 69.9 | 132x |
| 80 | 2,000 | 0.608 | 103 | 166x |
| 160 | 1,000 | 5.35 | 477 | 91.6x |

| d | n | Datasets | Same graph | Errors, Andrey | Errors, tigramite | Errors, empty graph | Cross-link F1, Andrey | Cross-link F1, tigramite |
|---:|---:|---:|---:|---:|---:|---:|---:|---:|
| 10 | 500 | 10 | 10 of 10 | 8 | 8 | 20 | 0.717 | 0.717 |
| 10 | 1,000 | 10 | 10 of 10 | 9.5 | 9.5 | 20 | 0.704 | 0.704 |
| 10 | 2,000 | 10 | 10 of 10 | 8 | 8 | 20 | 0.728 | 0.728 |
| 20 | 500 | 10 | 10 of 10 | 35 | 35 | 39 | 0.526 | 0.526 |
| 20 | 1,000 | 10 | 10 of 10 | 38.5 | 38.5 | 39 | 0.519 | 0.519 |
| 20 | 2,000 | 10 | 10 of 10 | 37.5 | 37.5 | 39 | 0.523 | 0.523 |
| 40 | 500 | 10 | 10 of 10 | 160 | 160 | 78 | 0.332 | 0.332 |
| 40 | 1,000 | 10 | 10 of 10 | 161.5 | 161.5 | 78 | 0.333 | 0.333 |
| 40 | 2,000 | 10 | 10 of 10 | 157.5 | 157.5 | 78 | 0.340 | 0.340 |
| 80 | 500 | 10 | 10 of 10 | 644.5 | 644.5 | 155.5 | 0.195 | 0.195 |
| 80 | 1,000 | 10 | 10 of 10 | 634 | 634 | 155.5 | 0.203 | 0.203 |
| 80 | 2,000 | 10 | 10 of 10 | 634 | 634 | 155.5 | 0.203 | 0.203 |
| 160 | 1,000 | 5 | 5 of 5 | 2,605 | 2,605 | 300 | 0.110 | 0.110 |

PCMCI returns the same graph as tigramite on every dataset, so their error and F1 columns are equal.

### PCMCI+

PCMCI+ adds links within a time step. It runs PCMCI's tests over lags 0 to 5, then orients the
same-step links as PC does, with the majority collider rule. Both packages test with Student-t
partial correlation at `pc_alpha` 0.01. The output is a CPDAG of the same-step links and a directed
graph for each lag. Errors are the endpoint SHD over lags 0 to 5: two for a missing or extra link
and one per wrong end. An unoriented same-step link counts as an error at its open end.

| d | n | Andrey (s) | tigramite (s) | Speedup |
|---:|---:|---:|---:|---:|
| 10 | 500 | 0.0243 | 1.21 | 52.0x |
| 10 | 1,000 | 0.0269 | 1.50 | 54.8x |
| 10 | 2,000 | 0.0282 | 1.95 | 67.8x |
| 20 | 500 | 0.0575 | 4.22 | 72.9x |
| 20 | 1,000 | 0.0655 | 5.59 | 82.9x |
| 20 | 2,000 | 0.0735 | 7.01 | 96.6x |
| 40 | 500 | 0.241 | 15.3 | 64.7x |
| 40 | 1,000 | 0.267 | 18.8 | 69.4x |
| 40 | 2,000 | 0.270 | 25.1 | 84.3x |
| 80 | 1,000 | 0.772 | 75.9 | 96.4x |
| 80 | 2,000 | 0.852 | 98.7 | 116x |

| d | n | Datasets | Same graph | Errors, Andrey | Errors, tigramite | Errors, empty graph | Cross-link F1, Andrey | Cross-link F1, tigramite |
|---:|---:|---:|---:|---:|---:|---:|---:|---:|
| 10 | 500 | 10 | 10 of 10 | 12 | 12 | 50 | 0.786 | 0.786 |
| 10 | 1,000 | 10 | 10 of 10 | 8 | 8 | 50 | 0.873 | 0.873 |
| 10 | 2,000 | 10 | 10 of 10 | 5 | 5 | 50 | 0.916 | 0.916 |
| 20 | 500 | 10 | 10 of 10 | 34 | 34 | 100 | 0.697 | 0.697 |
| 20 | 1,000 | 10 | 10 of 10 | 26 | 26 | 100 | 0.807 | 0.807 |
| 20 | 2,000 | 10 | 10 of 10 | 19.5 | 19.5 | 100 | 0.842 | 0.842 |
| 40 | 500 | 10 | 10 of 10 | 71.5 | 71.5 | 200 | 0.695 | 0.695 |
| 40 | 1,000 | 10 | 10 of 10 | 56 | 56 | 200 | 0.773 | 0.773 |
| 40 | 2,000 | 10 | 10 of 10 | 60.5 | 60.5 | 200 | 0.777 | 0.777 |
| 80 | 1,000 | 10 | 10 of 10 | 179.5 | 179.5 | 400 | 0.668 | 0.668 |
| 80 | 2,000 | 10 | 10 of 10 | 167 | 167 | 400 | 0.715 | 0.715 |

PCMCI+ returns the same graph as tigramite on every dataset, so their error and F1 columns are
equal.

### LPCMCI

LPCMCI allows for unobserved series. It returns a time-series PAG over lags 0 to 3, with a tail, an
arrowhead, or a circle at each end of each link. Both packages test with Student-t partial
correlation at `pc_alpha` 0.05 with one preliminary round. In this regime, 30% of the series are
unobserved, so we score each fit against the true PAG over the observed series (Gerhardus and Runge,
2020, Sect. 2.3). Errors are the endpoint SHD against that PAG; a circle in both counts as a match.
F1 counts links regardless of their marks, including autodependencies.

| d | Observed | n | Andrey (s) | tigramite (s) | Speedup |
|---:|---:|---:|---:|---:|---:|
| 5 | 4 | 500 | 0.0746 | 0.647 | 8.56x |
| 5 | 4 | 1,000 | 0.0841 | 0.928 | 12.0x |
| 5 | 4 | 2,000 | 0.0920 | 1.33 | 15.9x |
| 10 | 7 | 500 | 0.195 | 2.32 | 10.4x |
| 10 | 7 | 1,000 | 0.276 | 3.56 | 12.6x |
| 10 | 7 | 2,000 | 0.255 | 4.59 | 18.7x |
| 20 | 14 | 500 | 1.04 | 10.6 | 10.8x |
| 20 | 14 | 1,000 | 0.940 | 11.2 | 12.2x |
| 20 | 14 | 2,000 | 1.06 | 16.2 | 16.5x |

| d | Observed | n | Datasets | Same graph | Errors, Andrey | Errors, tigramite | Errors, empty graph | Adjacency F1, Andrey | Adjacency F1, tigramite |
|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
| 5 | 4 | 500 | 10 | 9 of 10 | 4.5 | 4.5 | 16 | 0.883 | 0.883 |
| 5 | 4 | 1,000 | 10 | 9 of 10 | 14.5 | 15.5 | 28 | 0.791 | 0.775 |
| 5 | 4 | 2,000 | 10 | 8 of 10 | 16 | 16 | 29 | 0.827 | 0.827 |
| 10 | 7 | 500 | 10 | 6 of 10 | 55 | 56 | 78 | 0.554 | 0.577 |
| 10 | 7 | 1,000 | 10 | 9 of 10 | 41.5 | 41.5 | 62 | 0.644 | 0.644 |
| 10 | 7 | 2,000 | 10 | 8 of 10 | 45 | 45 | 74 | 0.653 | 0.653 |
| 20 | 14 | 500 | 10 | 9 of 10 | 181.5 | 181 | 221 | 0.422 | 0.422 |
| 20 | 14 | 1,000 | 10 | 9 of 10 | 146.5 | 146.5 | 182 | 0.486 | 0.486 |
| 20 | 14 | 2,000 | 10 | 6 of 10 | 131 | 131 | 170 | 0.555 | 0.555 |

The two return different graphs on 17 of 90 datasets: Andrey's has fewer errors on 12 of them and
tigramite's on 5.

### Granger

Lasso Granger causality fits each variable on every variable's past at lags 1 to 2, using a lasso
with 5-fold cross-validation. It keeps lagged edges with nonzero coefficients. Both packages fit the
same lasso on the same z-scored series. Errors are the lagged edges missed plus those added. We also
ran Andrey's per-variable fits on threads across its 4 cores (`andrey.granger.numpy.parallel`),
shown in the parallel columns. This returns the same coefficients.

| d | n | Andrey (s) | causal-learn (s) | Speedup | Andrey, parallel (s) | Speedup, parallel |
|---:|---:|---:|---:|---:|---:|---:|
| 10 | 1,000 | 0.149 | 0.480 | 3.34x | 0.434 | 1.11x |
| 20 | 1,000 | 0.327 | 0.995 | 3.09x | 0.984 | 1.04x |
| 40 | 1,000 | 0.836 | 2.38 | 2.83x | 1.88 | 1.27x |
| 80 | 1,000 | 2.98 | 6.52 | 2.15x | 2.50 | 2.64x |
| 160 | 1,000 | 21.6 | 33.8 | 1.59x | 7.00 | 4.85x |

| d | n | Datasets | Same graph | Errors, Andrey | Errors, causal-learn | Errors, empty graph | Cross-link F1, Andrey | Cross-link F1, causal-learn |
|---:|---:|---:|---:|---:|---:|---:|---:|---:|
| 10 | 1,000 | 10 | 10 of 10 | 59 | 59 | 56.5 | 0.559 | 0.559 |
| 20 | 1,000 | 10 | 10 of 10 | 166 | 166 | 124 | 0.448 | 0.448 |
| 40 | 1,000 | 10 | 10 of 10 | 447 | 447 | 246.5 | 0.358 | 0.358 |
| 80 | 1,000 | 10 | 10 of 10 | 1,027.5 | 1,027.5 | 494 | 0.315 | 0.315 |
| 160 | 1,000 | 10 | 10 of 10 | 2,444 | 2,444 | 973 | 0.284 | 0.284 |

### VARMA-LiNGAM

VARMA-LiNGAM fits a vector autoregressive moving-average model of order (1, 1), with no order
search. It then estimates the same-step DAG from the residuals using LiNGAM and prunes the lag
blocks using adaptive lasso. It returns weighted same-step, lag-1, and moving-average matrices.
Errors are the cross-links that differ from the true ones: missing plus extra. We count a lagged
link with its direction and a same-step link by adjacency.

| d | n | Andrey (s) | lingam (s) | Speedup |
|---:|---:|---:|---:|---:|
| 3 | 100 | 2.14 | 2.27 | 1.05x |
| 3 | 300 | 3.23 | 3.16 | 1.00x |
| 3 | 1,000 | 5.04 | 5.14 | 1.02x |
| 5 | 100 | 6.64 | 6.90 | 1.06x |
| 5 | 300 | 12.4 | 11.7 | 0.962x |
| 5 | 1,000 | 29.5 | 30.4 | 1.03x |
| 10 | 100 | 20.2 | 20.6 | 1.03x |
| 10 | 300 | 36.1 | 36.1 | 1.01x |
| 10 | 1,000 | 96.6 | 99.4 | 1.04x |
| 20 | 100 | 43.0 | 47.4 | 1.11x |
| 20 | 300 | 85.7 | 97.2 | 1.13x |
| 20 | 1,000 | 241 | 270 | 1.12x |

| d | n | Datasets | Same graph | Errors, Andrey | Errors, lingam | Errors, empty graph | Cross-link F1, Andrey | Cross-link F1, lingam |
|---:|---:|---:|---:|---:|---:|---:|---:|---:|
| 3 | 100 | 10 | 9 of 10 | 3 | 3 | 7 | 0.748 | 0.748 |
| 3 | 300 | 10 | 9 of 10 | 2 | 2 | 7 | 0.833 | 0.833 |
| 3 | 1,000 | 10 | 9 of 10 | 1 | 1 | 7 | 0.916 | 0.916 |
| 5 | 100 | 10 | 4 of 10 | 9 | 8.5 | 12 | 0.653 | 0.667 |
| 5 | 300 | 10 | 6 of 10 | 5 | 5 | 12 | 0.781 | 0.781 |
| 5 | 1,000 | 10 | 7 of 10 | 4 | 3.5 | 12 | 0.840 | 0.863 |
| 10 | 100 | 10 | 0 of 10 | 29 | 27.5 | 26 | 0.556 | 0.565 |
| 10 | 300 | 10 | 0 of 10 | 16 | 17 | 26 | 0.684 | 0.686 |
| 10 | 1,000 | 10 | 5 of 10 | 9 | 9 | 26 | 0.834 | 0.834 |
| 20 | 100 | 5 | 0 of 5 | 78 | 77 | 55 | 0.466 | 0.469 |
| 20 | 300 | 5 | 0 of 5 | 39 | 37 | 55 | 0.650 | 0.646 |
| 20 | 1,000 | 5 | 2 of 5 | 33 | 34 | 55 | 0.741 | 0.741 |

The two return different graphs on 54 of 105 datasets: Andrey's has fewer errors on 18 of them and
lingam's on 18.

### LongitudinalLiNGAM

LongitudinalLiNGAM takes `M` units observed at `n` time points. It estimates a same-step DAG and a
lag-1 matrix at every time point but the first, using one lag and the pairwise likelihood-ratio
measure. As for VARMA-LiNGAM, errors count the cross-links that differ from the true ones, pooled
over the time points.

| d | n | M | Andrey (s) | lingam (s) | Speedup |
|---:|---:|---:|---:|---:|---:|
| 2 | 20 | 1,000 | 0.0716 | 0.101 | 1.37x |
| 2 | 50 | 100 | 0.151 | 0.263 | 1.60x |
| 2 | 50 | 200 | 0.153 | 0.217 | 1.42x |
| 2 | 50 | 500 | 0.166 | 0.225 | 1.34x |
| 2 | 50 | 1,000 | 0.174 | 0.225 | 1.33x |
| 3 | 20 | 100 | 0.116 | 0.184 | 1.55x |
| 3 | 20 | 200 | 0.120 | 0.179 | 1.47x |
| 3 | 20 | 500 | 0.125 | 0.193 | 1.52x |
| 3 | 20 | 1,000 | 0.131 | 0.218 | 1.66x |
| 3 | 50 | 100 | 0.288 | 0.442 | 1.52x |
| 3 | 50 | 200 | 0.304 | 0.455 | 1.54x |
| 3 | 50 | 500 | 0.314 | 0.508 | 1.59x |
| 3 | 50 | 1,000 | 0.334 | 0.516 | 1.55x |
| 5 | 20 | 100 | 0.238 | 0.460 | 1.90x |
| 5 | 20 | 200 | 0.249 | 0.503 | 2.02x |
| 5 | 20 | 500 | 0.277 | 0.619 | 2.24x |
| 5 | 20 | 1,000 | 0.300 | 0.539 | 1.77x |
| 5 | 50 | 100 | 0.622 | 1.17 | 1.87x |
| 5 | 50 | 200 | 0.628 | 1.23 | 1.95x |
| 5 | 50 | 500 | 0.682 | 1.33 | 1.96x |
| 5 | 50 | 1,000 | 0.766 | 1.41 | 1.84x |

| d | n | M | Datasets | Same graph | Errors, Andrey | Errors, lingam | Errors, empty graph | Cross-link F1, Andrey | Cross-link F1, lingam |
|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
| 2 | 20 | 100 | 10 | 10 of 10 | 0 | 0 | 57 | 1.00 | 1.00 |
| 2 | 20 | 200 | 10 | 10 of 10 | 0 | 0 | 57 | 1.00 | 1.00 |
| 2 | 20 | 500 | 10 | 10 of 10 | 0 | 0 | 57 | 1.00 | 1.00 |
| 2 | 20 | 1,000 | 10 | 10 of 10 | 0 | 0 | 57 | 1.00 | 1.00 |
| 2 | 50 | 100 | 10 | 10 of 10 | 0 | 0 | 147 | 1.00 | 1.00 |
| 2 | 50 | 200 | 10 | 10 of 10 | 0 | 0 | 147 | 1.00 | 1.00 |
| 2 | 50 | 500 | 10 | 10 of 10 | 0 | 0 | 147 | 1.00 | 1.00 |
| 2 | 50 | 1,000 | 10 | 10 of 10 | 0 | 0 | 147 | 1.00 | 1.00 |
| 3 | 20 | 100 | 10 | 10 of 10 | 2 | 2 | 171 | 0.994 | 0.994 |
| 3 | 20 | 200 | 10 | 10 of 10 | 0 | 0 | 171 | 1.00 | 1.00 |
| 3 | 20 | 500 | 10 | 10 of 10 | 0 | 0 | 171 | 1.00 | 1.00 |
| 3 | 20 | 1,000 | 10 | 10 of 10 | 0 | 0 | 171 | 1.00 | 1.00 |
| 3 | 50 | 100 | 10 | 10 of 10 | 4.5 | 4.5 | 441 | 0.995 | 0.995 |
| 3 | 50 | 200 | 10 | 10 of 10 | 1 | 1 | 441 | 0.999 | 0.999 |
| 3 | 50 | 500 | 10 | 10 of 10 | 1 | 1 | 441 | 0.999 | 0.999 |
| 3 | 50 | 1,000 | 10 | 10 of 10 | 0.5 | 0.5 | 441 | 0.999 | 0.999 |
| 5 | 20 | 100 | 10 | 10 of 10 | 15 | 15 | 570 | 0.987 | 0.987 |
| 5 | 20 | 200 | 10 | 10 of 10 | 5.5 | 5.5 | 570 | 0.995 | 0.995 |
| 5 | 20 | 500 | 10 | 10 of 10 | 3 | 3 | 570 | 0.997 | 0.997 |
| 5 | 20 | 1,000 | 10 | 10 of 10 | 1.5 | 1.5 | 570 | 0.999 | 0.999 |
| 5 | 50 | 100 | 10 | 7 of 10 | 31.5 | 31.5 | 1,470 | 0.989 | 0.989 |
| 5 | 50 | 200 | 10 | 7 of 10 | 12.5 | 15 | 1,470 | 0.996 | 0.995 |
| 5 | 50 | 500 | 10 | 9 of 10 | 3 | 3 | 1,470 | 0.999 | 0.999 |
| 5 | 50 | 1,000 | 10 | 8 of 10 | 2 | 2 | 1,470 | 0.999 | 0.999 |

The two return different graphs on 9 of 240 datasets: Andrey's has fewer errors on 8 of them and
lingam's on 1.

The panels of 2 variables at 20 time points with 100, 200, and 500 units report accuracy only. We
omit speed because both fits take under 0.1 s.

## Known limits

- **Data coverage.** We measure one simulated regime per method. These results do not establish
  performance on measured series or on other mechanisms, noise, or lags.
- **PCMCI accuracy.** At 40 and 80 variables, at every length, and at 160, both packages return more
  errors than the empty graph on every dataset. At 20 variables, Andrey does so on 4 or 5 of 10
  datasets, as tigramite does.
- **Granger accuracy.** At 1,000 time steps, both packages return more errors than the empty graph
  on every dataset from 20 variables. At 10 variables, Andrey does so on 5 of 10, as causal-learn
  does.
- **VARMA-LiNGAM.** Andrey's VARMA-LiNGAM gains little speed over lingam: its ratios range from
  0.962x, slower than lingam, to 1.13x. At 100 time steps from 10 variables, both packages return
  more errors than the empty graph on most datasets. lingam has fewer errors on more datasets at two
  rungs: 10 variables at 100 steps (5 against 4) and 20 at 300 (4 against 1). This comes from a
  defect in lingam: its pruning builds the lagged columns with `np.roll`, so its first regression
  row pairs the first step with values from the end of the series. Andrey starts the regressions at
  the first step with a full lag.
- **LPCMCI agreement.** On series with unobserved variables, the two LPCMCI implementations return
  the same PAG on 6 to 9 of 10 datasets per rung. At 20 variables with 14 observed and 500 steps,
  tigramite has fewer errors on 1 dataset and Andrey on 0. This comes from the order of candidate
  separating sets: tigramite follows the variable indices, and Andrey's order does not depend on the
  column order.
- **LongitudinalLiNGAM agreement.** The two implementations return different graphs on 9 of 240
  datasets, from a defect in lingam: its lag regression uses scikit-learn's LinearRegression, whose
  default tolerance zeroes singular values below 1e-6 of the largest, so on the fast-growing panels
  it drops to rank 1 of 5. With an exact lag regression, lingam returns the same graph as Andrey on
  all 240 datasets.
- **LongitudinalLiNGAM timing.** The smallest panels, 2 variables at 20 time points with 100, 200,
  and 500 units, are not timed: both fits take under 0.1 s.
- **Hardware.** Ratios are specific to the measured hardware, core budget, and settings.

## Reproduce

The [benchmark harness guide][harness] documents environment setup, the campaign drivers, and the
scores. tigramite and lingam run in their own environments (`benchmarks/start_tigramite.sh`,
`benchmarks/start_lingam_ts.sh`), listed in the guide. To rerun the comparisons, install Andrey
0.5.0 (andrey-core), tigramite 5.2.10.1, causal-learn 0.1.4.8, and lingam 1.13.0, then run the
drivers with the settings below. Times will differ with the hardware and build; graphs should match
closely.

| Comparison | Driver | d | n | Datasets x repeats | Wall cap (s) | Cores |
|---|---|---|---|---|---:|---:|
| PCMCI | `pcmci_cpu.py` | 10, 20, 40, and 80 | 500, 1,000, and 2,000 | 10 x 1 | 3,600 | 4 |
| PCMCI | `pcmci_cpu.py` | 160 | 1,000 | 5 x 1 | 3,600 | 4 |
| PCMCI+ | `pcmci_cpu.py` | 10, 20, and 40 | 500, 1,000, and 2,000 | 10 x 1 | 3,600 | 4 |
| PCMCI+ | `pcmci_cpu.py` | 80 | 1,000 and 2,000 | 10 x 1 | 3,600 | 4 |
| LPCMCI | `pcmci_cpu.py` | 5, 10, and 20 | 500, 1,000, and 2,000 | 10 x 1 | 3,600 | 4 |
| Granger | `granger_var.py` | 10, 20, 40, 80, and 160 | 1,000 | 10 x 1 | 3,600 | 4 |
| VARMA-LiNGAM | `lingam_ts.py` | 3, 5, and 10 | 100, 300, and 1,000 | 10 x 1 | 3,600 | 4 |
| VARMA-LiNGAM | `lingam_ts.py` | 20 | 100, 300, and 1,000 | 5 x 1 | 3,600 | 4 |
| LongitudinalLiNGAM | `lingam_ts.py` | 2, 3, and 5 | 20 and 50 | 10 x 1 | 3,600 | 4 |

```shell
$ANDREY_BENCH_PYTHON benchmarks/pcmci_cpu.py --method pcmci --out outputs/pcmci \
    --ladder 10 20 40 80 --lengths 500 1000 2000 --tau-max 2 --seeds 10 --repeats 1 \
    --cap-wall-s 3600 --cores 4
$ANDREY_BENCH_PYTHON benchmarks/pcmci_cpu.py --method pcmci --out outputs/pcmci-160 --ladder 160 \
    --lengths 1000 --tau-max 2 --seeds 5 --repeats 1 --cap-wall-s 3600 --cores 4
$ANDREY_BENCH_PYTHON benchmarks/pcmci_cpu.py --method pcmciplus --out outputs/pcmciplus \
    --ladder 10 20 40 --lengths 500 1000 2000 --tau-max 5 --seeds 10 --repeats 1 --cap-wall-s 3600 \
    --cores 4
$ANDREY_BENCH_PYTHON benchmarks/pcmci_cpu.py --method pcmciplus --out outputs/pcmciplus-80 \
    --ladder 80 --lengths 1000 2000 --tau-max 5 --seeds 10 --repeats 1 --cap-wall-s 3600 --cores 4
$ANDREY_BENCH_PYTHON benchmarks/pcmci_cpu.py --method lpcmci --out outputs/lpcmci --ladder 5 10 20 \
    --lengths 500 1000 2000 --tau-max 3 --seeds 10 --repeats 1 --cap-wall-s 3600 --cores 4
$ANDREY_BENCH_PYTHON benchmarks/granger_var.py --out outputs/granger --ladder 10 20 40 80 160 \
    --n 1000 --maxlag 2 --cv 5 --seeds 10 --repeats 1 --cap-wall-s 3600 --cores 4
$ANDREY_BENCH_PYTHON benchmarks/lingam_ts.py --method varma --out outputs/varma --ladder 3 5 10 \
    --lengths 100 300 1000 --seeds 10 --repeats 1 --cap-wall-s 3600 --cores 4
$ANDREY_BENCH_PYTHON benchmarks/lingam_ts.py --method varma --out outputs/varma-20 --ladder 20 \
    --lengths 100 300 1000 --seeds 5 --repeats 1 --cap-wall-s 3600 --cores 4
$ANDREY_BENCH_PYTHON benchmarks/lingam_ts.py --method longitudinal --out outputs/longitudinal \
    --ladder 2 3 5 --lengths 20 50 --samples 100 200 500 1000 --seeds 10 --repeats 1 \
    --cap-wall-s 3600 --cores 4
```

[harness]: https://github.com/Abel-ai-lab/andrey/blob/main/benchmarks/README.md