---
name: benchmarks-0.15.0
description: The Andrey 0.15.0 benchmarks - FCI-JCI, BOSS-FCI, GRaSP-FCI, and J-PCMCI+ against
  their authors' implementations in pcalg, Tetrad, and tigramite.
meta:
  type: guide
  tags: [benchmarks, performance, accuracy, multi-domain, latent, time-series]
---

# 0.15.0 release card: multi-domain data

:::::{div} release-opening

:::{card}
:class-card: release-cover release-art release-art-0-15-0
:shadow: none

Release card

Multi-domain data

0.15.0
:::

::::{div} release-opening-text

We compare FCI-JCI, BOSS-FCI, GRaSP-FCI, and J-PCMCI+, which become supported in 0.15.0, with their
authors' implementations. The first three run on simulations that allow hidden common causes, and
FCI-JCI's data come from an observational environment and an interventional one. J-PCMCI+'s data
are time series from several datasets that differ through context variables, some of them hidden.
Each result holds for the measured conditions; it depends on the data, settings, and hardware.

::::
:::::

We use the authors' implementation of FCI-JCI in pcalg, those of BOSS-FCI and GRaSP-FCI in Tetrad,
and that of J-PCMCI+ in tigramite on data simulated as in their papers. FCI-JCI learns from data
collected in several environments, with the environment index as one more variable, and allows
hidden common causes. BOSS-FCI and GRaSP-FCI also allow hidden common causes. They start from the
graph of a BOSS or GRaSP search and remove with conditional-independence tests the edges whose
endpoints are independent given some set of other variables. J-PCMCI+ runs PCMCI+ on time series
from several datasets, with the contexts it is given and dummies of the dataset and the time step
standing in for the hidden ones.

## Summary

| Method | Reference | Comparisons | Better | Tie | Worse | Speed against the reference |
|---|---|---:|---:|---:|---:|---|
| FCI-JCI | [pcalg 2.7-12](https://cran.r-project.org/package=pcalg) (R) | 3 settings x 2 assumption sets x 2 levels | 0 | 12 | 0 | 4.0x to 4.6x faster at 4 variables, 72x to 79x at 10 |
| BOSS-FCI | [Tetrad 7.6.10](https://github.com/cmu-phil/tetrad) (Java) | 36 settings x 2 levels, and 100 variables | 1 | 72 | 0 | 1.05x to 5.8x faster at 20 variables, 1.85x at 100 |
| GRaSP-FCI | [Tetrad 7.6.10](https://github.com/cmu-phil/tetrad) (Java) | 36 settings x 2 levels, and 100 variables | 1 | 72 | 0 | 1.16x to 5.8x faster at 20 variables, 7.6x at 100 |
| J-PCMCI+ | [tigramite 5.2.10.1](https://github.com/jakobrunge/tigramite) (Python) | 21 settings, and the context links in 7 | 0 | 28 | 0 | 14x to 168x faster |

Andrey ties the reference in all 12 FCI-JCI comparisons, in all 28 J-PCMCI+ ones, and in 144 of the
146 BOSS-FCI and GRaSP-FCI ones, and is better in the other two. A tie means we found no
reproducible difference in errors under the criteria below; it does not mean the returned graphs are
identical.

BOSS-FCI and GRaSP-FCI also take `edge_removal="triangles"`, an option that tests only the edges in
a triangle of the completed partially directed acyclic graph (CPDAG) for removal. With it, Andrey
follows the same published rule as Tetrad 7.6.3, the GFCI paper's (Ogarrio, Spirtes, and Ramsey
2016, Sec. 5), so we compared the option with Tetrad 7.6.3 on the same datasets and settings:

| Method | Option | Reference | Comparisons | Better | Tie | Worse |
|---|---|---|---|---:|---:|---:|
| BOSS-FCI | `edge_removal="triangles"` | [Tetrad 7.6.3](https://github.com/cmu-phil/tetrad) (Java) | 36 settings x 2 levels, and 100 variables | 1 | 72 | 0 |
| GRaSP-FCI | `edge_removal="triangles"` | [Tetrad 7.6.3](https://github.com/cmu-phil/tetrad) (Java) | 36 settings x 2 levels, and 100 variables | 2 | 71 | 0 |

With the option, Andrey ties Tetrad 7.6.3 in 143 of the 146 comparisons and is better in the other
three. Here is how to read the tables:

- The speed ratio is the median, over the datasets, of the reference's fit time divided by Andrey's
  on the same dataset, written as how many times faster or slower Andrey is.
- A comparison is better or worse when the 95% interval of the mean paired difference in errors lies
  entirely below or above 0. One with fewer than 10 paired datasets is reported and not judged.
- A better or worse result has to hold twice. It counts only where a rerun on fresh seeds agreed,
  and one on the first seeds alone counts as a tie. No FCI-JCI or J-PCMCI+ comparison came out
  better or worse, so none was rerun; 12 BOSS-FCI and GRaSP-FCI comparisons did, 6 each way, and 2
  of them held on the rerun. With the option against Tetrad 7.6.3, 19 did, 17 better and 2 worse,
  and 3 better ones held.
- Andrey must also make fewer errors than the empty graph wherever the reference does: this is the
  floor. Where the reference does not beat the empty graph either, the floor is reported and not
  judged. The floor is judged on the first seeds; a rerun only confirms a failure there.

## How we measured

Each dataset was simulated as the method's paper simulates its data and given to both packages, so
each comparison pairs Andrey's error with the reference's on the same data. D in the tables is the
mean of those paired differences, Andrey's error minus the reference's, with its 95% bootstrap
interval.

- FCI-JCI ran one fit at a time on one core of an AMD EPYC 7713 with one BLAS thread, and BOSS-FCI
  and GRaSP-FCI the same way on an AMD EPYC 7713 or 9534, and out of the claim at mean degree 10 on
  an Intel Xeon Gold 6542Y. J-PCMCI+ ran the same way on an AMD EPYC 7713, and out of the claim
  with GPDC at 60 datasets on an AMD EPYC 7713 or an Intel Xeon Gold 6542Y.
- Each fit had a time limit of 1,800 s, including imports and warm-up; BOSS-FCI and GRaSP-FCI had
  3,600 s at 100 variables, and J-PCMCI+ also a memory limit of 16,000 MB. We record a fit that
  exceeds its limit as a timeout, not an empty graph.
- Both packages run at the comparison's settings, given in each method's section below.

## FCI-JCI

FCI-JCI (Mooij, Magliacane, and Claassen 2020) runs FCI on data from several environments, with the
environment index as one more variable that no data variable causes. We compared it with pcalg's
`fci` with `jci`, the implementation by the method's authors, on their simulation. All 12
comparisons tie, and Andrey is 4 to 79 times faster.

- The data follow the paper's simulation (Sec. 5.2): linear Gaussian equations with hidden common
  causes, and one binary environment index that changes one variable, in an observational
  environment and an interventional one.
- The claim holds the paper's small setting, 4 variables with mechanism changes and with perfect
  interventions, and its larger one, 10 variables with mechanism changes. Each has 500 rows per
  environment, with 20 datasets per setting.
- Both packages run the Fisher z test at alpha 0.01, the paper's level, and at 0.05, Andrey's
  default, under two of the paper's assumption sets: `confounded_index=True` with pcalg's
  `jci = "1"`, and `False` with `"123"`.
- We count the endpoint SHD over the data variables and the environment index against FCI-JCI's PAG
  at the true distribution, built from the true graph and checked against pcalg's on every dataset.

| Setting | JCI | alpha | Andrey | pcalg | Empty graph | D [95%] |
|---|---|---|---:|---:|---:|---|
| 4 variables, mechanism change | 1 | 0.01 | 5.95 | 5.90 | 12.50 | +0.05 [-0.30, +0.45] |
| 4 variables, mechanism change | 123 | 0.01 | 5.90 | 5.80 | 12.50 | +0.10 [+0.00, +0.30] |
| 4 variables, mechanism change | 1 | 0.05 | 5.05 | 5.05 | 12.50 | +0.00 [-0.15, +0.15] |
| 4 variables, mechanism change | 123 | 0.05 | 5.00 | 5.05 | 12.50 | -0.05 [-0.15, +0.00] |
| 4 variables, perfect intervention | 1 | 0.01 | 6.05 | 6.15 | 12.50 | -0.10 [-0.25, +0.00] |
| 4 variables, perfect intervention | 123 | 0.01 | 6.05 | 6.15 | 12.50 | -0.10 [-0.25, +0.00] |
| 4 variables, perfect intervention | 1 | 0.05 | 5.45 | 5.40 | 12.50 | +0.05 [-0.15, +0.30] |
| 4 variables, perfect intervention | 123 | 0.05 | 5.45 | 5.40 | 12.50 | +0.05 [-0.15, +0.30] |
| 10 variables, mechanism change | 1 | 0.01 | 37.85 | 37.80 | 54.80 | +0.05 [-0.90, +0.95] |
| 10 variables, mechanism change | 123 | 0.01 | 37.85 | 37.80 | 54.80 | +0.05 [-0.90, +0.95] |
| 10 variables, mechanism change | 1 | 0.05 | 36.25 | 36.55 | 54.80 | -0.30 [-1.20, +0.65] |
| 10 variables, mechanism change | 123 | 0.05 | 36.05 | 36.45 | 54.80 | -0.40 [-1.30, +0.60] |

Both packages make fewer errors than the empty graph in every comparison, so the floor applies in
all 12, and Andrey passes each. The two returned the same PAG on 18 or 19 of the 20 datasets at 4
variables and on 4 to 6 at 10. There they differ by design in how they pick separating sets and
possible-d-separation sets: Andrey's result does not depend on the column order, and pcalg's stable
skeleton (Colombo and Maathuis 2014) keeps the first separating set it finds in column order.

The paper's large setting, 100 variables with 100 rows per environment, is in the claim where pcalg
finishes, and pcalg did not: on the two datasets of a dry run, every pcalg fit ran past 1,800 s, so
we ran Andrey alone there. Each of Andrey's fits took under half a second, and it made fewer errors
than the empty graph on every dataset, 328 against 412 on average at alpha 0.01.

Out of the claim, with the paper's several context variables merged into one environment index, and
with 100 or 2,000 rows per environment at 10 variables, no interval for the mean paired error
difference lies entirely above or below 0. At 20 variables pcalg finished 4 and 6 of the 10 datasets
within 1,800 s in the respective settings, too few to judge; at 50 variables it finished neither
dataset of the dry run. Andrey finished every dataset of the campaign.

At 10 variables and alpha 0.01, FCI on the pooled rows or on the observational rows alone made
more errors over the data variables than FCI-JCI, and so did CD-NOD, which assumes no hidden common
cause.

At 4 variables Andrey's median fit took 0.004 to 0.005 s against pcalg's 0.017 to 0.021 s, and at
10 variables 0.011 to 0.013 s against 0.83 to 1.31 s, counting pcalg's time from the transfer into
R. Over the paired datasets the median ratio ran from 4.0 to 4.6 at 4 variables and from 72 to 79
at 10.

## BOSS-FCI

BOSS-FCI (Ramsey, Andrews, and Spirtes 2025) runs BOSS's search for a CPDAG, removes with
conditional-independence tests the adjacencies whose endpoints are independent given some set of
other variables, and orients the rest with FCI's rules, as GFCI does with GES's graph. We compared
it with Tetrad 7.6.10's `BossFci`, the implementation by the method's authors, on their simulation.
It ties in 72 of 73 comparisons and is better in one, at 100 variables, and it is 1.05 to 5.8 times
faster at 20 variables and 1.85 times at 100.

- The data follow the paper's simulation (Sec. 8.1.1): linear Gaussian equations on random forward
  DAGs, with up to 0, 4, or 8 of the nodes with two or more children hidden.
- The claim holds the paper's settings of 20 measured variables at mean degree 2, 4, or 6 with 200
  to 5,000 rows, and 100 measured variables with 10 hidden, mean degree 6, 1,000 rows, and
  conditioning sets of at most 7, with 20 datasets each.
- Both packages run the BIC score and Fisher's z test at the paper's penalty discount 2 and alpha
  0.01, and at Andrey's defaults, 1 and 0.05. Tetrad tests the edges in an unseeded order, so it
  runs five times per dataset and its error there is the mean of the five.
- We count the endpoint SHD against the PAG of the measured variables, built from the true graph and
  checked against pcalg's on every graph of up to 30 measured variables.

The error columns below are means over each group of comparisons, averaged over the hidden-variable
counts and the four sample sizes at 20 variables. The [comparison page][bgf] lists every comparison.

| Settings | Variables | Mean degree | Comparisons | Andrey | 7.6.10 | Empty graph | Better | Tie | Worse |
|---|---|---|---:|---:|---:|---:|---:|---:|---:|
| paper's | 20 + 0, 4, or 8 hidden | 2 | 12 | 18.68 | 18.55 | 39.2 | 0 | 12 | 0 |
| paper's | 20 + 0, 4, or 8 hidden | 4 | 12 | 56.67 | 56.21 | 89.7 | 0 | 12 | 0 |
| paper's | 20 + 0, 4, or 8 hidden | 6 | 12 | 122.94 | 123.56 | 158.77 | 0 | 12 | 0 |
| paper's | 100 + 10 hidden | 6 | 1 | 636.65 | 640.95 | 950.3 | 1 | 0 | 0 |
| defaults | 20 + 0, 4, or 8 hidden | 2 | 12 | 20.99 | 20.96 | 39.2 | 0 | 12 | 0 |
| defaults | 20 + 0, 4, or 8 hidden | 4 | 12 | 55.26 | 55.12 | 89.7 | 0 | 12 | 0 |
| defaults | 20 + 0, 4, or 8 hidden | 6 | 12 | 120.72 | 121.05 | 158.77 | 0 | 12 | 0 |

Both packages make fewer errors than the empty graph in every comparison, so the floor applies in
all 73, and Andrey passes each. Three comparisons came out worse and three better on seeds 0 to 19,
each by at most 4.3 errors; rerun on seeds 100 to 119, the worse ones and the better ones at 20
variables tie, and the one at 100 variables held: D was -4.3 [-8.32, -0.57] on the first seeds and
-5.01 [-7.68, -2.36] on the rerun. Andrey returned the same PAG as one of Tetrad's runs on 475 of
the 1,460 datasets. They differ by design in how they remove edges: Andrey runs FCI's adjacency
search level by level, so its result does not depend on the order of the edges, and Tetrad tests
each edge once in a random order.

Tetrad 7.6.3, the authors' release in causal-cmd 1.12.0, tests only the edges that lie in a
triangle of the CPDAG, the rule of the GFCI paper (Ogarrio, Spirtes, and Ramsey 2016, Sec. 5),
rather than Algorithm 5's every edge. It keeps more edges. At the paper's settings it made fewer
errors at degree 4 and 6, 53.6 and 119.3 on average against Andrey's 56.67 and 122.94, and at 100
variables, 583.8 against 636.65; at Andrey's defaults at degree 2 it made more, 24.02 against 20.99.

With `edge_removal="triangles"`, Andrey follows the same rule as Tetrad 7.6.3, and against it on the
same datasets it ties in 72 of 73 comparisons and is better in one. At the paper's settings it made
53.3 and 118.52 errors on average at degree 4 and 6, against Tetrad 7.6.3's 53.6 and 119.3, and
579.75 against 583.8 at 100 variables; at the defaults at degree 2 it made 23.61 against 24.02.
Eight comparisons came out better and two worse on seeds 0 to 19; rerun on seeds 100 to 119, the
worse ones tie, and one better one held: at the defaults, degree 2 with 8 hidden and 1,000 rows, D
was -1.55 [-2.7, -0.45] and then -1.4 [-3.15, -0.15]. The floor applies in 72 comparisons, and
Andrey passes each; at the defaults at degree 2 with 8 hidden and 200 rows neither Tetrad 7.6.3
(38.45 errors) nor the option (38.55) beats the empty graph (37.8). Andrey returned the same PAG as
Tetrad 7.6.3 on 343 of the 1,460 datasets: the rule is the same, and they still differ in how they
search for separating sets and where BOSS starts. With the option, Andrey's median fit took 0.012 to
0.41 s at 20 variables and 2.5 s at 100, against Tetrad 7.6.3's 0.041 to 3.0 s and 24.2 s, timed in
separate runs on AMD EPYC 7713 or 9534 processors.

At 20 variables Andrey's median fit took 0.014 to 0.49 s against Tetrad 7.6.10's 0.058 to 0.5 s,
counting Tetrad's search inside the JVM, and at 100 variables 1.42 s against 2.6 s. Over the paired
datasets the median ratio ran from 1.05 to 5.8 at 20 variables and was 1.85 at 100.

## GRaSP-FCI

GRaSP-FCI runs the same steps with GRaSP's search in place of BOSS's. Against Tetrad 7.6.10's
`GraspFci` on the same datasets and settings it ties in 72 of 73 comparisons and is better in one,
and it is 1.16 to 5.8 times faster at 20 variables and 7.6 times at 100.

| Settings | Variables | Mean degree | Comparisons | Andrey | 7.6.10 | Empty graph | Better | Tie | Worse |
|---|---|---|---:|---:|---:|---:|---:|---:|---:|
| paper's | 20 + 0, 4, or 8 hidden | 2 | 12 | 18.73 | 18.7 | 39.2 | 0 | 12 | 0 |
| paper's | 20 + 0, 4, or 8 hidden | 4 | 12 | 56.94 | 56.76 | 89.7 | 0 | 12 | 0 |
| paper's | 20 + 0, 4, or 8 hidden | 6 | 12 | 123.52 | 123.81 | 158.77 | 1 | 11 | 0 |
| paper's | 100 + 10 hidden | 6 | 1 | 638.85 | 643.11 | 950.3 | 0 | 1 | 0 |
| defaults | 20 + 0, 4, or 8 hidden | 2 | 12 | 21.11 | 20.9 | 39.2 | 0 | 12 | 0 |
| defaults | 20 + 0, 4, or 8 hidden | 4 | 12 | 55.23 | 55.33 | 89.7 | 0 | 12 | 0 |
| defaults | 20 + 0, 4, or 8 hidden | 6 | 12 | 120.84 | 121.65 | 158.77 | 0 | 12 | 0 |

The floor applies in all 73 comparisons, and Andrey passes each. Three comparisons came out worse
and three better on seeds 0 to 19; rerun on seeds 100 to 119, the worse ones tie, and one better one
held: at the paper's settings, degree 6 with no hidden variable and 1,000 rows, D was -2.49 [-4.87,
-0.27] and again -2.32 [-4.61, -0.13]. Tetrad sets GRaSP's tuck depth from `depth`, unlimited by
default and 7 at 100 variables, and Andrey's GRaSP-FCI runs a tuck depth of 3; at the paper's
settings a tuck depth of 7 changed Andrey's mean errors by less than 0.3 at 20 variables and by 1.7
at 100. As for BOSS-FCI, Tetrad 7.6.3 made fewer errors at the paper's settings at degree 4 and 6
(54.36 and 119.99 against 56.94 and 123.52) and more at the defaults at degree 2 (23.47 against
21.11).

With `edge_removal="triangles"`, against Tetrad 7.6.3, Andrey's GRaSP-FCI ties in 71 of 73
comparisons and is better in two: at the paper's settings, degree 6 with no hidden variable and
1,000 rows, D was -2.25 [-4.65, -0.15] and then -3.75 [-6.8, -0.4], and at the defaults, degree 4
with no hidden variable and 5,000 rows, -3.25 [-5.4, -1.25] and then -2.65 [-5.35, -0.3]. The other
seven better on seeds 0 to 19 tie on the rerun. At the paper's settings it made 53.9 and 119.49
errors on average at degree 4 and 6 against 54.36 and 119.99, and 585.5 against 589.45 at 100
variables. The floor applies in 72 comparisons, all but the same one as for BOSS-FCI, and Andrey
passes each. Andrey's median fit took 0.013 to 0.42 s at 20 variables and 2.2 s at 100, against
Tetrad 7.6.3's 0.048 to 3.7 s and 28.2 s.

At 20 variables Andrey's median fit took 0.014 to 0.5 s against Tetrad 7.6.10's 0.071 to 0.57 s, and
at 100 variables 1.3 s against 11.9 s. At 200 measured variables with 20 hidden, out of the claim,
one dataset took Andrey 10 s and Tetrad 502 s with GRaSP-FCI, and 9.3 s against 16.9 s with
BOSS-FCI.

Out of the claim, at 100 measured variables with mean degree 10, both hybrids made about 21 fewer
errors than Tetrad 7.6.10 on 10 datasets and about 80 more than Tetrad 7.6.3. On the setting of
Andrey's recovery test, 20 measured variables with 4 hidden, every comparison ties except BOSS-FCI's
at the defaults, where Andrey's mean error is lower by 3.9.

## J-PCMCI+

J-PCMCI+ (Gunther, Ninad, and Runge 2023) runs PCMCI+ on time series from several datasets of the
same system variables that differ through context variables: time contexts, the same series in
every dataset, and space contexts, fixed within a dataset. One-hot dummies of the dataset and of the
time step stand in for the contexts it is not given. We compared it with tigramite's `JPCMCIplus`,
the implementation in the package of the method's authors, on the model their experiment script
draws. All 28 comparisons tie, and Andrey is 14 to 168 times faster. The [comparison page][jp]
lists every comparison.

- The data follow the authors' experiment script: 5 system variables with lagged links, 2 time and 2
  space contexts, links from the contexts into the system variables, and Gaussian noise. The script
  draws every link between system variables at lag 1 or 2, though it asks for lag 0, so the claimed
  settings have no same-step link between them, as in the paper's figures. Unlike the paper's
  text, it also draws links between the contexts, which is how a hidden context can break the
  paper's Assumption 3.
- The claim holds the settings of the paper's Figures 2 and 3: every context observed with 10
  datasets of 10 to 200 steps, and one context of each kind observed with 10 datasets of 10 to 200
  steps or with 5 to 200 datasets of 10 steps. Each setting has 20 draws of its datasets, or 40 with
  hidden contexts, as about half of those break the paper's Assumption 3: a hidden context
  confounds an observed one and a system variable. We report those draws apart and do not judge
  them.
- Both packages run the partial-correlation test with both dummies and `tau_max=2`, at alpha 0.05,
  the paper's level, where the claim is judged, and at 0.01, both packages' default.
- We count the endpoint SHD over the links between system variables at lags 0 to 2 and, where every
  context is observed, the links from the contexts into the system variables that a fit misses or
  adds.
- tigramite 5.2.10.1 raises on a same-step link between two observed contexts, which most draws with
  every context observed have, so we ran it with a fix that leaves that link unoriented. Every fit
  that finished without the fix gave the same graph with it.

| Contexts observed | Datasets x steps | Andrey | tigramite | Empty graph | D [95%] |
|---|---|---:|---:|---:|---|
| all | 10 x 10 | 7.70 | 7.60 | 20.00 | +0.10 [+0.00, +0.30] |
| all | 10 x 20 | 4.90 | 5.00 | 20.00 | -0.10 [-0.30, +0.00] |
| all | 10 x 30 | 4.00 | 3.90 | 20.00 | +0.10 [+0.00, +0.30] |
| all | 10 x 50 | 3.70 | 3.60 | 20.00 | +0.10 [-0.20, +0.50] |
| all | 10 x 100 | 3.10 | 3.10 | 20.00 | +0.00 [+0.00, +0.00] |
| all | 10 x 150 | 3.60 | 3.60 | 20.00 | +0.00 [+0.00, +0.00] |
| all | 10 x 200 | 3.10 | 3.10 | 20.00 | +0.00 [+0.00, +0.00] |
| one of each | 10 x 10 | 7.24 | 7.33 | 20.00 | -0.10 [-0.29, +0.00] |
| one of each | 10 x 20 | 4.67 | 5.05 | 20.00 | -0.38 [-0.86, +0.00] |
| one of each | 10 x 30 | 5.52 | 5.62 | 20.00 | -0.10 [-0.29, +0.00] |
| one of each | 10 x 50 | 4.67 | 4.86 | 20.00 | -0.19 [-0.95, +0.38] |
| one of each | 10 x 100 | 3.71 | 4.10 | 20.00 | -0.38 [-0.86, +0.00] |
| one of each | 10 x 150 | 3.62 | 3.81 | 20.00 | -0.19 [-0.67, +0.29] |
| one of each | 10 x 200 | 3.71 | 3.43 | 20.00 | +0.29 [+0.00, +0.57] |
| one of each | 5 x 10 | 12.95 | 12.76 | 20.00 | +0.19 [-0.48, +0.86] |
| one of each | 20 x 10 | 5.90 | 5.81 | 20.00 | +0.10 [-0.19, +0.38] |
| one of each | 30 x 10 | 6.19 | 5.71 | 20.00 | +0.48 [-0.10, +1.24] |
| one of each | 50 x 10 | 5.33 | 5.33 | 20.00 | +0.00 [-0.29, +0.29] |
| one of each | 100 x 10 | 5.05 | 5.05 | 20.00 | +0.00 [+0.00, +0.00] |
| one of each | 150 x 10 | 5.33 | 5.14 | 20.00 | +0.19 [+0.00, +0.48] |
| one of each | 200 x 10 | 4.76 | 4.76 | 20.00 | +0.00 [-0.57, +0.48] |

The context links missed plus added, where every context is observed:

| Datasets x steps | Andrey | tigramite | Empty graph | D [95%] |
|---|---:|---:|---:|---|
| 10 x 10 | 7.75 | 7.75 | 8.90 | +0.00 [+0.00, +0.00] |
| 10 x 20 | 4.30 | 4.30 | 8.90 | +0.00 [+0.00, +0.00] |
| 10 x 30 | 3.85 | 3.85 | 8.90 | +0.00 [+0.00, +0.00] |
| 10 x 50 | 2.85 | 2.85 | 8.90 | +0.00 [+0.00, +0.00] |
| 10 x 100 | 1.65 | 1.65 | 8.90 | +0.00 [+0.00, +0.00] |
| 10 x 150 | 1.45 | 1.45 | 8.90 | +0.00 [+0.00, +0.00] |
| 10 x 200 | 1.05 | 1.05 | 8.90 | +0.00 [+0.00, +0.00] |

Both packages make fewer errors than the empty graph in every comparison, so the floor applies in
all 28, and Andrey passes each. The two returned the same graph on 15 to 19 of the 20 draws of a
setting with every context observed, and on 11 to 21 of the 21 with one of each kind.

The comparisons outside the claim ran once, on the first seeds. At alpha 0.01, Andrey made fewer
errors in two settings with one context of each kind observed, 10 datasets of 10 and of 150 steps,
and tied in the rest. On the draws that break Assumption 3, it made fewer errors at 10 datasets of
200 steps at both levels and tied elsewhere. Every comparison ties on
`andrey.data.sample_context_series`, whose links are half at lag 0, and with 10 and 15 system
variables. With GPDC on the authors' nonlinear model, Andrey made fewer errors in three of the 24
comparisons, at 50 and 60 datasets, and more in two at alpha 0.01: at 20 datasets, and in the
context links at 50. On the datasets pooled without an index, PCMCI+, the paper's baseline, made
15.6 to 23.0 errors on average with every context observed, against J-PCMCI+'s 3.1 to 7.7 and the
empty graph's 20.

Andrey's median fit took 0.017 to 0.17 s in the claim against tigramite's 0.24 to 11.5 s. Over the
paired draws the median ratio was 14 at 5 datasets, 18 to 52 at 10, and up to 168 at 150. With
15 system variables, out of the claim, Andrey's median fit took 11.5 s and tigramite's 664 s.

## Limits

Each result holds for the settings above. These are the places where the method, or both packages,
fall short:

- FCI-JCI takes one environment index. Out of the claim, we merged the paper's several context
  variables into one label. Andrey ties pcalg there too. Reading that label as a category
  (`indep_test="cond_gauss"`) reduced errors over the data variables and the environment index at
  alpha 0.01, from 7.75 to 6.80 at 4 variables and from 49.10 to 44.50 at 10. The reduction came
  from the environment index's edges.
- At 100 variables with 100 rows per environment, pcalg finished no fit within 1,800 s, so there is
  no reference to compare with. Andrey's errors there are measured only against the empty graph's.
- BOSS-FCI and GRaSP-FCI recover only part of the PAG on their paper's simulation, and so does
  Tetrad. Hidden nodes leave many pairs joined only through them. At degree 6 their errors were
  about three quarters of the empty graph's, and at degree 2 with 8 hidden and 200 rows at the
  defaults 34.35 and 33.3 against the empty graph's 37.8.
- At the paper's settings, the GFCI paper's rule of testing only the edges in a triangle of the
  CPDAG, which Tetrad 7.6.3 runs, made fewer errors than Algorithm 5 at degree 4 and 6 and at 100
  variables; at Andrey's defaults on sparse graphs it made more. Andrey's default and Tetrad 7.6.10
  follow Algorithm 5, and `edge_removal="triangles"` follows the GFCI paper's rule.
- Tetrad 7.6.10's rules R5 and R9 read a covered path as uncovered, so they can put a tail where no
  Markov-equivalent maximal ancestral graph (MAG) has one, a defect of the reference. Its tail
  precision here was close to Andrey's.
- J-PCMCI+'s claimed settings, from its authors' script, have no same-step link between the system
  variables, so they measure only its false same-step links. On `andrey.data.sample_context_series`,
  out of the claim, half the links are at lag 0, and Andrey ties tigramite there.
- J-PCMCI+'s Assumption 3 breaks in about half the models with hidden contexts: in 19 of the 40
  here, the same in every setting. We judge neither package on their draws; Andrey made fewer errors
  in 2 of their 28 comparisons and tied in the rest.
- With GPDC on the authors' nonlinear model, out of the claim, J-PCMCI+ is slower than tigramite
  from 20 datasets of 20 steps: at 50 datasets its median fit took 385 s against 203 s. tigramite's
  GPDC reads one column of each dummy, so the two do not run the same tests there.

## Reproduce

You can find environment setup, campaign drivers, and scores in the [benchmark harness
guide][harness]. Times will differ with the hardware and build; graphs should match closely.

| Comparison | Lane | Sizes | Datasets x repeats | Wall cap (s) |
|---|---|---|---|---:|
| FCI-JCI against pcalg 2.7-12 | `fci_jci_cpu.py` | 4, 10, and 100 variables; 100 to 500 rows per environment | 20 x 1 | 1,800 |
| BOSS-FCI and GRaSP-FCI against Tetrad 7.6.10, and with the option against 7.6.3 | `boss_grasp_fci_cpu.py` | 20, 100, and 200 measured variables; 200 to 5,000 rows | 20 x 5; out of the claim 10 x 5, and 1 x 1 at 200 variables | 1,800 to 3,600 |
| J-PCMCI+ against tigramite 5.2.10.1 | `jpcmciplus_cpu.py` | 5 system variables, and 2, 10, and 15 out of the claim; 5 to 200 datasets of 10 to 200 steps | 20 x 1; 40 x 1 with hidden contexts | 1,800 |

[harness]: https://github.com/Abel-ai-lab/andrey/blob/main/benchmarks/README.md
[bgf]: https://github.com/Abel-ai-lab/andrey/blob/main/benchmarks/comparisons/boss-fci-grasp-fci.md
[jp]: https://github.com/Abel-ai-lab/andrey/blob/main/benchmarks/comparisons/jpcmciplus.md