---
name: benchmarks-0.2.0
description: The Andrey 0.2.0 release card - RCD and BottomUpParceLiNGAM compared with lingam on data with hidden common causes.
meta:
  type: guide
  tags: [benchmarks, performance, accuracy, latent]
---

# 0.2.0 release card: hidden common causes

% The release's cover, as on the Release cards page, beside the opening (_static/benchmarks.css).
:::::{div} release-opening

:::{card}
:class-card: release-cover release-art
:shadow: none

Release card

Hidden common causes

0.2.0
:::

::::{div} release-opening-text

We compare RCD and BottomUpParceLiNGAM with lingam, which implements both methods for data with
hidden common causes. Each returns the same graph as lingam on 25 of 25 datasets and runs
1.7-10.8x faster. We therefore support both from 0.2.0.

Graph agreement does not ensure accuracy. Here `d` counts all nodes, including hidden nodes.
From `d = 10`, BottomUpParceLiNGAM's directed edges have more errors than the empty graph's in
both packages (see [Known limits](#known-limits)). You can find speedups and accuracy in the
[summary](#summary), and measurement and scoring definitions in
[how to read the results](#how-to-read-the-results). [Faster small fits](#faster-small-fits)
times PC, FCI, GES, and the experimental hill climbing with and without a 0.2.0 change that makes
them 2.5-12.6x faster at `d = 10`, with identical output. These results apply to the measured
workloads; speed and accuracy depend on the data, settings, and hardware.
::::
:::::

## Summary

lingam is the baseline. We report every paired speed ratio, regardless of accuracy.

**Speed.**

| Family | Method | Compared with | Variables (`d`) | Andrey speedup |
|---|---|---|---|---|
| Hidden common causes | RCD | lingam | 5-30 | 1.7-2.9x |
| Hidden common causes | BottomUpParceLiNGAM | lingam | 5-30 | 2.5-10.8x |

```{raw} html
:file: 0.2.0/speedup.html
```

**Accuracy.**

| Method | Compared with | Accuracy |
|---|---|---|
| RCD | lingam | Same graph on 25 of 25 datasets |
| BottomUpParceLiNGAM | lingam | Same graph on 25 of 25 datasets |

```{raw} html
:file: 0.2.0/errors.html
```

## How to read the results

These definitions cover the comparison with lingam; [Faster small fits](#faster-small-fits) gives
its own.

**Data.** We evaluate both methods on `latent_lingam_er`: Erdos-Renyi directed acyclic graphs
(DAGs) with mean degree 2, linear mechanisms, uniform noise, standardized data, and 2,000 samples.
One node in five is hidden, with at least one hidden node. Hidden nodes have no data columns and
act as hidden common causes of their children. The sizes `d = 5, 10, 15, 20, 30` give 4, 8, 12,
16, and 24 observed variables. There are five seeds per size.

**Accuracy.** "Same graph" means equal `structure_hash` values: the packages return the same
edges with the same marks. We also score each estimate against the generated ground truth for
the two types of relationships these methods estimate:

- **Directed edges**, compared with the latent projection: an observed edge `i -> j` exists when
  a directed path runs from `i` to `j` through hidden nodes only, including a direct edge, but
  excluding pairs that share a hidden cause. The error count is the number of edges in one set
  but not the other. A reversed edge counts twice; lower is better. The empty graph's count is
  the number of true edges.
- **Confounded pairs**, compared with the observed pairs that a single hidden node reaches
  through hidden nodes only. RCD marks such a pair `<->`. BottomUpParceLiNGAM leaves a block of
  variables unordered, joined by `o-o`. This block includes the confounded variables, their
  observed ancestors, and any other confounded pairs, so some pairs have no shared cause.
  We score RCD by F1 and BottomUpParceLiNGAM by recall, the share of true pairs it leaves
  unordered. Higher is better.

The observed partial ancestral graph (PAG), the target of FCI in the
[0.1.0 release card](0.1.0.md), is unsuitable for ranking these methods. Their functional
assumptions let them mark `<->` where the PAG has `o-o`, so PAG scores would count a correct
`<->` as an error. Compare each method across the two packages, not RCD with BottomUpParceLiNGAM.

**Timing.** Times are seconds per fit. Each fit runs in its own process and is timed after
imports, with no warm-up fit and one fit per dataset. Wall caps include setup and the timed fit;
memory caps cover the process tree. For each dataset, we divide lingam's time by Andrey's, then
report the median of these paired ratios. This need not equal the ratio of the displayed median
times.

**Hardware.** Every fit used 2 cores of an AMD EPYC 7713 on a shared node, so times may include
contention. Ratios apply within this campaign and do not transfer between machines.

## Recorded campaign

| Comparison | d | Date | Measured package versions |
|---|---|---|---|
| RCD, BottomUpParceLiNGAM | 5-30 | 11 October 2026 | lingam 1.12.2 |
| [Small fits](#faster-small-fits): PC, FCI, GES, hill climbing (experimental) | 10-400 | 11 October 2026 | Andrey only |

We measured a pre-release Andrey 0.2.0 build installed from source. The
{download}`summary <../../../benchmarks/published/0.2.0/summary.json>` contains
one row per comparison and size. Each row gives median seconds, the paired ratio, the number of
datasets with the same graph, each package's median directed-edge errors alongside the empty
graph's, and each package's mean pair score. `pair_score` names the pair metric; `seeds` counts
the paired datasets.

For the small fits, we measured two pre-release builds installed from source: one without the
change and one with it. The
{download}`small-fits summary <../../../benchmarks/published/0.2.0-small-fits/summary.json>`
contains one row per method and size. Each row gives median seconds without and with the change,
the paired ratio, the number of paired fits, and how many of them returned identical output.
`paired_fits` and `same_output` count every paired fit in both jobs.

## Hidden-confounder methods

We use each method's defaults in both packages. RCD regresses out at most 2 variables at a time
and uses a significance level of 0.01 for its correlation, independence, and normality tests.
BottomUpParceLiNGAM accepts a sink at a Bonferroni-corrected significance level of 0.1.

### RCD

RCD regresses variables on candidate ancestors and tests the residuals for independence. It finds
each variable's ancestors, then its parents, then the pairs that share a hidden common cause.

| d | Observed variables | Andrey | lingam | Speedup |
|---:|---:|---:|---:|---:|
| 5 | 4 | 1.99 | 3.31 | 1.66x |
| 10 | 8 | 3.75 | 8.81 | 2.35x |
| 15 | 12 | 6.83 | 16.7 | 2.17x |
| 20 | 16 | 11.9 | 34.7 | 2.88x |
| 30 | 24 | 17.9 | 52.5 | 2.55x |

| d | Same graph | Directed errors, Andrey | Directed errors, lingam | Empty-graph directed errors | Pair F1, Andrey | Pair F1, lingam |
|---:|---:|---:|---:|---:|---:|---:|
| 5 | 5 of 5 | 0 | 0 | 3 | 0.657 | 0.657 |
| 10 | 5 of 5 | 4 | 4 | 4 | 0.344 | 0.344 |
| 15 | 5 of 5 | 6 | 6 | 8 | 0.454 | 0.454 |
| 20 | 5 of 5 | 8 | 8 | 11 | 0.543 | 0.543 |
| 30 | 5 of 5 | 11 | 11 | 19 | 0.467 | 0.467 |

RCD's directed errors are below the empty graph's at every size except `d = 10`, where they
are equal.

### BottomUpParceLiNGAM

BottomUpParceLiNGAM finds sinks one at a time, working backward from the last variable in the
causal order. Variables it cannot order because of a hidden common cause remain in one unordered
block.

| d | Observed variables | Andrey | lingam | Speedup |
|---:|---:|---:|---:|---:|
| 5 | 4 | 1.84 | 4.59 | 2.48x |
| 10 | 8 | 8.66 | 46.9 | 5.08x |
| 15 | 12 | 21.7 | 126 | 5.68x |
| 20 | 16 | 39.9 | 279 | 6.61x |
| 30 | 24 | 95.9 | 1,030 | 10.8x |

| d | Same graph | Directed errors, Andrey | Directed errors, lingam | Empty-graph directed errors | Pair recall, Andrey | Pair recall, lingam |
|---:|---:|---:|---:|---:|---:|---:|
| 5 | 5 of 5 | 1 | 1 | 3 | 0.233 | 0.233 |
| 10 | 5 of 5 | 11 | 11 | 4 | 0.145 | 0.145 |
| 15 | 5 of 5 | 17 | 17 | 8 | 0.05 | 0.05 |
| 20 | 5 of 5 | 22 | 22 | 11 | 0 | 0 |
| 30 | 5 of 5 | 40 | 40 | 19 | 0 | 0 |

From `d = 10`, BottomUpParceLiNGAM's directed errors exceed the empty graph's. From `d = 20`,
it leaves none of the confounded pairs unordered.

### CAM-UV is not reported

The campaign driver also runs CAM-UV, compared with lingam on `latent_anm_er`: the same graphs
and hidden nodes, with nonlinear mechanisms and Gaussian additive noise. Andrey uses spline
features and least squares for regression; lingam fits a generalized additive model. Their
independence tests also use different kernel bandwidths and null approximations. These
differences produce different graphs under the same settings. CAM-UV remains experimental,
and we exclude its results from this card.

## Faster small fits

In 0.2.0, PC, FCI, GES, and the experimental hill climbing find the Basic Linear Algebra
Subprograms (BLAS) libraries once per process. Each method limits these libraries to one thread
while it fits. Before 0.2.0, it found them again each time: once per fit in GES and hill climbing,
and once per pass of the skeleton search in PC and FCI. Finding them takes the same time whatever
the data, so it weighed most on small fits.

We timed each method with and without the change on 4 cores of an AMD EPYC 7713. Both builds ran in
one job on one node, alternating which ran first. Each fit ran in its own process after one warm-up
fit. The data come from sparse linear-Gaussian Erdos-Renyi graphs (mean degree 2), with `n = 10d`
samples; FCI's graphs hide one variable in ten, and `d` counts them. GES and hill climbing use NumPy
with one worker, their default. A second job re-measured PC at `d = 200` and 400 and FCI at
`d = 200` with 40 paired fits each, and the table reports these sizes from it. The times are median
milliseconds per fit; the speedup is the median over paired fits of the time without the change
divided by the time with it.

| Method | d | Paired fits | Without the change (ms) | With the change (ms) | Speedup |
|---|---:|---:|---:|---:|---:|
| PC | 10 | 10 | 30.4 | 2.61 | 12.2x |
| | 20 | 10 | 59.8 | 6.52 | 8.72x |
| | 50 | 10 | 299 | 94.3 | 3.22x |
| | 100 | 10 | 379 | 216 | 1.56x |
| | 200 | 40 | 345 | 214 | 1.59x |
| | 400 | 40 | 1,020 | 801 | 1.30x |
| FCI | 10 | 10 | 29.1 | 2.29 | 12.6x |
| | 20 | 10 | 56.9 | 5.72 | 7.27x |
| | 50 | 10 | 264 | 92.0 | 3.14x |
| | 100 | 10 | 486 | 271 | 1.73x |
| | 200 | 40 | 1,340 | 1,140 | 1.18x |
| GES | 10 | 10 | 49.5 | 19.9 | 2.49x |
| | 20 | 10 | 177 | 151 | 1.18x |
| | 50 | 10 | 1,960 | 1,900 | 1.03x |
| | 100 | 5 | 10,700 | 10,600 | 1.01x |
| Hill climbing (experimental) | 10 | 10 | 32.5 | 4.27 | 7.74x |
| | 20 | 10 | 45.1 | 17.3 | 2.62x |
| | 50 | 10 | 315 | 268 | 1.23x |
| | 100 | 10 | 1,290 | 1,260 | 1.05x |
| | 400 | 5 | 96,000 | 98,600 | 1.00x |

At `d = 10`, PC and FCI take about a twelfth of the time they took without the change, hill
climbing an eighth, and GES two fifths. The gain shrinks as fits grow: from `d = 100`, the
speedup is 1.00-1.73x, and no size is slower. All 410 paired fits in the two jobs returned
identical output, so the change leaves every graph as it was. The count includes GES's parallel
and Numba builds and hill climbing's parallel build, which the jobs also timed.

The time saved grows much more slowly than the fit, so a fit that takes seconds gains little.
These speedups come from one CPU model with 4 cores.

## Known limits

- **Data coverage.** We measure RCD and BottomUpParceLiNGAM in one regime: sparse linear
  Erdos-Renyi graphs with uniform noise, one node in five hidden, and 2,000 samples, up to
  `d = 30`. These results do not establish performance across all data regimes.
- **Scale.** Fit times for both methods grow quickly with the number of variables. From 4 to 24
  observed variables, Andrey's median time grows from 1.99 to 17.9 s for RCD and from 1.84 to
  95.9 s for BottomUpParceLiNGAM. In lingam, it grows from 3.31 to 52.5 s and from 4.59 to
  1,030 s.
- **Errors against the empty graph.** BottomUpParceLiNGAM's median directed-edge errors are higher
  than the empty graph's at `d = 10`, 15, 20, and 30 (11, 17, 22, and 40 compared with 4, 8, 11,
  and 19). RCD's equal the empty graph's at `d = 10` (4 compared with 4). Andrey and lingam return
  the same graphs, so these results reflect the methods on this data, not differences between
  implementations.
- **BottomUpParceLiNGAM's block.** Edges inside the unordered block remain unoriented, limiting
  directed-edge recall.
- **Small fits.** Finding the BLAS libraries takes a fixed time, so the gain from finding them
  once shrinks as fits grow: 1.01x for GES at `d = 100` and 1.00x for the experimental hill
  climbing at `d = 400`, fits of about 10 and 100 s.
- **Hardware.** Ratios are specific to the measured hardware, core budget, and settings.

## Reproduce

The [benchmark harness guide][harness] documents environment setup, the campaign driver, and the
scores. To rerun the comparison with lingam, install Andrey 0.2.0 (andrey-core) and lingam
1.12.2, then run the driver with the settings below. Times will differ with the hardware and
build; graphs and scores should match closely.

| Comparison | Driver | d | Seeds x repeats | Warm-up fits | Wall cap (s) | Cores |
|---|---|---|---|---:|---:|---:|
| RCD, BottomUpParceLiNGAM | `confounded_cpu.py` | 5-30 | 5 x 1 | 0 | 1,800 | 2 |
| Small fits: PC, FCI, GES, hill climbing (experimental) | `benchmark.py`, `fci_latent.py`, `ges_cpu.py`, `r_packages.py` | 10-400 | 5 x 1-2; re-check 5 x 8 | 1 | 900 | 4 |

For the comparison with lingam, memory was capped at 20,000 MB per fit. The driver runs each
size's datasets for every method it includes, CAM-UV included:

```shell
$ANDREY_BENCH_PYTHON benchmarks/confounded_cpu.py --out outputs/confounded-cpu \
    --ladder 5 10 15 20 30 --seeds 5 --n 2000 --cap-wall-s 1800 --cap-mem-mb 20000 --cores 2
```

To rerun the small fits, install two builds of Andrey from source: the commit with this change,
linked from the 0.2.0 [changelog](../changelog.md), and the commit before it. On each build, run the
standard driver for each method with Andrey's solutions and the empty baseline in its solution list:
`benchmark.py` for PC, `fci_latent.py` for FCI, `ges_cpu.py` for GES, and `r_packages.py --lane hc`
for hill climbing. Run both builds in one job, alternating which runs first, with memory capped at
12,000 MB per fit. Sizes up to `d = 100` (GES: 50) take 2 repeats, larger ones 1. For PC on one
build:

```shell
$ANDREY_BENCH_PYTHON benchmarks/benchmark.py --out outputs/small-fits/pc \
    --ladder 10 20 50 100 --seeds 5 --repeats 2 --cap-wall-s 900 --cap-mem-mb 12000 --cores 4
```

The re-check ran PC at `d = 200` and 400 and FCI at `d = 200` with 4 repeats, twice per build, in
the order without, with, with, without: 40 paired fits per size.

[harness]: https://github.com/Abel-ai-lab/andrey/blob/main/benchmarks/README.md