---
name: benchmarks-0.10.0
description: The Andrey 0.10.0 release card - for each method, the problems it finishes in time and
  solves better than the empty graph, by size and density.
meta:
  type: guide
  tags: [benchmarks, performance, feasibility]
---

# 0.10.0 release card: the boundary

:::::{div} release-opening

:::{card}
:class-card: release-cover release-art release-art-0-10-0
:shadow: none

Release card

The boundary

0.10.0
:::

::::{div} release-opening-text

This card maps, for each supported method, the problems it can handle: those it finishes within
the cap and solves with fewer errors than the empty graph. It measures Andrey alone, on the 0.9.0
code, with no other package. Each boundary is a reference for the measured conditions, not a hard
limit; it depends on the data, settings, and hardware.

::::
:::::

## Summary

**How far each method goes.** At 2,000 samples and a mean degree of 2, every size up to these
finished and beat the empty graph: BOSS 1,600 variables; GRaSP 800; PC, CALM, and DirectLiNGAM 400;
GES, GIES, and XGES 200 (400 with 10,000 samples); CD-NOD, tabu search, and hill climbing 100;
ICA-LiNGAM 50; exact search 22. The [post's table](../../blog/20261027-andrey-0-10-0.md) gives 500
and 10,000 samples, and the maps below every size.

**What stops them.** Most score-based searches stop on accuracy, and reach further with more
samples. The permutation-based searches, exact search, and CALM stop on time. PC stops on accuracy.

## The boundary

**Inside the boundary.** A size, a number of variables with a number of samples and a mean degree,
is inside a method's boundary when it is through, fewer than 3 of its fits cut by the time cap, the
memory cap, or an error, and the median errors of its finished fits are fewer than the empty
graph's. A fit that did not finish never counts as an empty graph. A ladder's boundary is the
largest size up to which every size is inside.

**How a ladder ends.** A ladder steps the number of variables on one setting until it stops: on
time, memory, or an error (3 fits of a size cut), past the accuracy boundary (two sizes in a row
that do not beat the empty graph), or at its last planned size. Larger sizes past the accuracy
boundary cost compute and tell you nothing more. Seven ladders ran past it before the rule, and
keep those sizes, marked.

**What a boundary is not.** Each boundary is a reference measured under the conditions below: our
graph and data generators, the listed sizes and densities, each method's settings as listed under
[Reproduce](#reproduce), and the listed machines. Other graphs, data, settings, or hardware move
it. It is not a guarantee that a method fails past it, or succeeds before it.

## Method

**Data.** Each method runs on the data its release card used: Erdos-Renyi graphs with linear
relations and Gaussian noise for the Gaussian searches, with non-Gaussian noise for the LiNGAM
methods; for RCD and BottomUpParceLiNGAM, one node in five is hidden (at least one), the number of
variables counts the hidden ones, and the errors are against the true PAG of the observed nodes; for
GIES, 40% of the variables get single-variable interventions, with the samples split evenly between
observation and each intervention, so the samples per setting shrink as the variables grow. Each
ladder's label names its setting.

**Two maps.** The size map: variables from 20, doubling, by 500, 2,000, and 10,000 samples, with a
mean degree of 2. The density map: the same variables by a mean degree of 1, 2, 4, and 8, with
2,000 samples. Exact search runs from 10 variables.

**Datasets.** 5 datasets per size, one per seed, 10 where the median errors were within 10% of the
empty graph's, and 20 per size for the pairwise methods. A size stops after 3 fits that did not
finish (on time, memory, or an error), so a cut size lists 3 or 4 datasets. For RCD and
BottomUpParceLiNGAM, a seed whose graph has too few nodes with two or more children to hide is
skipped, so a size can have fewer datasets.

**Errors.** The structural Hamming distance against the true graph in the form each method
returns: the CPDAG for the constraint-based, score-based, permutation-based, and LiNGAM searches,
with a DAG estimate (from CALM, exact search, hill climbing, and tabu search) scored against its
equivalence class; the PAG, marks included, for FCI and GFCI; the true PAG of the observed nodes,
endpoint by endpoint, for RCD and BottomUpParceLiNGAM; and the time-series graph for PCMCI, PCMCI+,
and LPCMCI. F1 for VARMA-LiNGAM and LongitudinalLiNGAM, the share of pairs with the right direction
for ANM and PNL, and the adjusted Rand index of the indicator clusters for GIN. The empty graph's
errors are the number of true edges.

**Limits.** A one-hour wall cap and a 28,000 MB memory cap per fit, on 4 CPUs: almost all on an AMD
EPYC 7713, some on an AMD EPYC 9534 or an Intel Xeon Gold 6542Y. Nine sizes ran on two CPU models;
their errors are the median over every finished fit, and they quote no time.

## Builds and versions

On the 1,252 fits of our release checks, 0.10.0 returns the same results as these builds.

| Measurement | Build |
|---|---|
| Exact search | `b94e647` |
| LongitudinalLiNGAM | `2fd784a` |
| The other methods | `52b814f`, `8a134cd`, and `1fce718` |

## Constraint-based methods

```{raw} html
:file: ../../blog/0.10.0/boundary-constraint.html
```

## Score-based methods

```{raw} html
:file: ../../blog/0.10.0/boundary-score.html
```

## Permutation-based methods

```{raw} html
:file: ../../blog/0.10.0/boundary-permutation.html
```

## Linear non-Gaussian methods

```{raw} html
:file: ../../blog/0.10.0/boundary-lingam.html
```

## Methods on their own axes

FCI and GFCI run with the samples growing with the variables (10 or 50 samples per variable). The
latent-variable, time-series, and pairwise methods run on their own axes. Granger is not measured in
this release: its data grow denser as the variables grow, so its sizes are not comparable. PCMCI+'s
ladders end at 320 variables, where the campaign's generator found no stationary series; 0.10.0's
generator draws one.

```{raw} html
:file: 0.10.0/ladders.html
```

## Comparisons with Tetrad

Each table answers one question: on the same datasets, how fast and how accurate is Andrey's PC,
GES, BOSS, GRaSP, and FCI against Tetrad 7.6.10's? Both run with the same score or test, penalty,
and depth, on the same node, with one worker; Tetrad's time is its search inside the Java virtual
machine, without its start or the data loading. Each time is the median over 5 datasets. The errors
are the mean SHD over every dataset of the matched rerun (Datasets), and for FCI the median over its
5.

```{raw} html
:file: 0.10.0/tetrad.html
```

Where the errors differ, the cause, from a matched rerun on more datasets:

- **PC.** The two find the same skeleton on every dataset and differ only in how they orient
  colliders: Tetrad keeps the first separating set it finds, and Andrey the union of the sets of
  that size. Andrey makes fewer errors at every size.
- **GES.** At the BIC penalty both packages use by default, both searches return graphs denser than
  the truth on these data, and Andrey's single pass and FGES's three passes stop in different local
  optima. With a penalty of 2 on both sides, the errors are equal within the run-to-run spread.
- **BOSS.** Each package runs one start of a local search: Tetrad from the data's column order,
  Andrey from a seeded order, and each reaches the true graph on some datasets and not others. Up
  to 100 variables the errors are equal within the run-to-run spread. At 200 variables Tetrad's
  start from the column order has fewer errors (3.0 against 8.8); started from random orders, as
  Andrey's search is, Tetrad's mean is 7.5, within the spread of Andrey's.
- **GRaSP.** A difference in search quality, not in the depth or the score. At 100 variables,
  Andrey's search ends at a worse BIC than the true graph's in 58% to 75% of fits, and Tetrad's in
  45% to 55%, so Andrey makes more errors; the gap sits in the fits that go furthest wrong.

## Other changes in 0.10.0

- Every method's docstring gives its measured boundary in its **Limitations** paragraph: how far
  it beat the empty graph and how far it finished, on each setting measured here.
- With several workers, the worker pool starts when `TMPDIR` is on a file system that rejects the
  forkserver's socket, such as a cluster's parallel file system, where `ges`, `gfci`, and the
  skeleton search raised `OSError`. Results do not change.
- `categorical=[]` is accepted by every test and score, and every score search takes both names of
  the linear-Gaussian BIC, `"local_score_BIC"` and `"local_score_BIC_from_cov"`.
- `andrey.data.storage` and the `data` extra are removed.

The [changelog](../changelog.md) lists every change.

## Limits

- **A reference, not a hard limit.** The boundaries depend on the graph and data generators, the
  densities, each method's settings, and the machine; see [What a boundary is not](#the-boundary).
- **Settings.** Most methods run at their defaults, but not all: BOSS and GRaSP run with a BIC
  penalty of 4, GES with a limit of 4 parents, hill climbing and tabu search with up to 100,000
  iterations, and the time-series methods with their own lag window; [Reproduce](#reproduce) lists
  every setting. A smaller `alpha`, a `max_depth`, or another score can move a boundary either way.
- **The cap.** A longer cap moves a time boundary to the right; it does not move an accuracy
  boundary.

## Reproduce

Every size can be rerun from the protocol on this card. Each fit is one call of the method on one
dataset, with a 3,600 s wall cap, a 28,000 MB memory cap, and 4 cores. The datasets per size and the
stop rules are under [Method](#method) and [The boundary](#the-boundary). Each method runs with the
settings below, passed as keyword arguments, and every setting not listed is at its default; each
dataset is drawn by the call below, from Andrey or the repository's `benchmarks/` folder, with its
variables, samples, density, and seed.

```{raw} html
:file: 0.10.0/settings.html
```