Release cards

0.11.0 release card: ready for real data#

Release card

Ready for real data

0.11.0

This card measures what 0.11.0 changes for real data: the extended BIC as the default penalty of ten score searches, PC and FCI on data with missing values, and the bootstrap’s edge frequencies. Each result holds for the measured conditions; it depends on the data, settings, and hardware.

Summary#

Extended BIC. On the same 299 simulated datasets, the extended BIC (ebic_gamma=1.0) lowers the errors of each of the ten methods it is the default for. Per true edge, the mean change is -0.74 to -0.83 for GES, XGES, GIES, hill climbing, Tabu search, BOSS, and GRaSP, -0.24 for exact search, and -0.10 for MMHC and RSMAX2, each with its 95% interval below 0, and no size is significantly worse for any of them. GFCI keeps the standard BIC: 3 of its 65 sizes are significantly worse with the extended BIC.

Missing values. With 20 variables and 1,000 samples, dropping the incomplete samples (listwise deletion) keeps a median of 124.5 samples with a tenth of the values missing, 10.5 with a fifth, and 0.5 with 30%, where no fit runs. Test-wise deletion fits every size: PC’s median errors are 6.5, 7.0, and 7.5, against 4.0 on the complete data and 20 for the empty graph. Over every size, test-wise deletion is never significantly worse than listwise deletion.

Bootstrap. The edge frequencies rank edges: of the adjacencies in more than nine fits in ten, 98% to 100% are true, and of those in fewer than one in ten, at most 1.1%. The frequencies in between are not probabilities. The averaged graph is not consistently more accurate than a single fit: fewer errors for PC, more for GES, the same for DirectLiNGAM.

Extended BIC, the new default#

Each row compares one method with ebic_gamma at 0 and at 1 on the same datasets, with every other setting at its default. Sachs, 853 real samples of 11 proteins, is left out of the datasets the table counts. On it and 50 resamples of it, every method has fewer errors with the extended BIC too: about 2 fewer for most, and under 1 for MMHC, RSMAX2, and GFCI.

Errors with the standard and the extended BIC

MethodStandard BICExtended BICChange per true edgeBetter / equal / worse
GES87.025.0-0.78[-0.88, -0.69]291 / 6 / 2
XGES73.011.0-0.75[-0.84, -0.67]285 / 4 / 3
GIES83.024.0-0.76[-0.86, -0.67]283 / 6 / 3
Hill climbing109.062.0-0.74[-0.84, -0.65]290 / 6 / 5
Tabu search112.056.0-0.80[-0.90, -0.71]293 / 3 / 5
Exact search8.01.5-0.24[-0.29, -0.19]44 / 4 / 2
BOSS69.015.0-0.83[-0.97, -0.71]282 / 8 / 9
GRaSP72.018.0-0.82[-0.95, -0.70]281 / 7 / 11
MMHC61.051.0-0.10[-0.11, -0.08]232 / 55 / 11
RSMAX258.047.0-0.10[-0.11, -0.09]236 / 56 / 7
GFCIkeeps the standard BIC93.088.0-0.06[-0.07, -0.05]201 / 41 / 56
Median errors (SHD) over every dataset with each penalty; the change is the mean of the extended BIC's errors minus the standard BIC's, per true edge, with its 95% interval; better, equal, and worse count the datasets.

Missing values#

Each table is one method with one test. Listwise deletion runs out of samples as the variables and the missing share grow; test-wise deletion stays near the complete data. With 10 variables, where listwise deletion keeps more samples, the two are close.

FCI with Fisher's z, missing values

Test-wiseListwiseComplete dataEmpty graphTest-wise minus listwise
10 variables, 1,000 rows0%13.013.013.020-
10%11.012.0347 rows13.020+0.2[-1.8, +2.4]
20%12.014.5104 rows13.020-1.5[-4.3, +1.0]
30%13.020.028 rows13.020-5.8[-8.0, -3.5]
20%at random12.517.076 rows13.020-4.8[-6.7, -2.9]
10 variables, 5,000 rows0%12.012.012.020-
10%12.013.51,732 rows12.020-0.7[-1.9, +0.4]
20%12.013.5530 rows12.020-0.5[-1.7, +0.5]
30%11.015.0139 rows12.020-2.1[-3.9, -0.3]
20%at random13.013.0376 rows12.020+1.0[-1.1, +3.7]
20 variables, 1,000 rows0%28.028.028.040-
10%28.532.5124 rows28.040-5.9[-8.8, -3.3]
20%30.542.010 rows28.040-12.1[-16.0, -8.5]
30%29.5-0 rows28.040-
20%at random29.043.05 rows28.040-13.6[-18.1, -8.8]
20 variables, 5,000 rows0%27.527.527.540-
10%28.529.5611 rows27.540-1.8[-3.1, -0.5]
20%28.036.556 rows27.540-9.9[-13.3, -6.7]
30%27.542.04 rows27.540-16.2[-20.2, -12.4]
20%at random27.039.532 rows27.540-12.9[-15.9, -10.2]
50 variables, 1,000 rows0%68.568.568.5100-
10%65.5104.06 rows68.5100-40.4[-46.9, -33.9]
20%68.0-0 rows68.5100-
30%70.0-0 rows68.5100-
20%at random69.5-0 rows68.5100-
50 variables, 5,000 rows0%64.064.064.0100-
10%61.0102.032 rows64.0100-38.9[-45.3, -32.1]
20%58.5-0 rows64.0100-
30%65.0-0 rows64.0100-
20%at random63.0-0 rows64.0100-
Median errors over the seeds. The missing share is of every value, completely at random unless marked; at random, the odd columns miss twice as often. Listwise deletion fits the complete rows only, whose median count is under its errors; a dash is a size where no fit finished, such as no complete rows left, never scored as an empty graph. The difference is the mean over the seeds, with its 95% interval.

FCI with Student's t, missing values

Test-wiseListwiseComplete dataEmpty graphTest-wise minus listwise
10 variables, 1,000 rows0%13.013.013.020-
10%11.012.0347 rows13.020+0.2[-1.8, +2.4]
20%12.014.5104 rows13.020-1.5[-4.3, +1.1]
30%13.020.028 rows13.020-5.8[-8.0, -3.5]
20%at random12.517.076 rows13.020-4.8[-6.7, -2.9]
10 variables, 5,000 rows0%12.012.012.020-
10%12.013.51,732 rows12.020-0.7[-2.0, +0.4]
20%12.013.5530 rows12.020-0.5[-1.7, +0.5]
30%11.015.0139 rows12.020-2.1[-3.8, -0.3]
20%at random13.013.0376 rows12.020+1.0[-1.1, +3.7]
20 variables, 1,000 rows0%28.028.028.040-
10%28.532.5124 rows28.040-5.9[-8.8, -3.3]
20%30.542.010 rows28.040-12.0[-15.9, -8.5]
30%29.5-0 rows28.040-
20%at random29.044.05 rows28.040-15.0[-18.5, -10.8]
20 variables, 5,000 rows0%27.527.527.540-
10%28.529.5611 rows27.540-1.8[-3.1, -0.4]
20%28.036.556 rows27.540-9.9[-13.4, -6.8]
30%27.542.04 rows27.540-16.2[-20.4, -12.4]
20%at random27.039.532 rows27.540-12.7[-15.8, -9.8]
50 variables, 1,000 rows0%68.568.568.5100-
10%65.5106.06 rows68.5100-41.1[-47.1, -35.0]
20%68.0-0 rows68.5100-
30%70.0-0 rows68.5100-
20%at random69.5-0 rows68.5100-
50 variables, 5,000 rows0%64.064.064.0100-
10%61.0102.032 rows64.0100-38.9[-45.2, -32.2]
20%58.5-0 rows64.0100-
30%65.0-0 rows64.0100-
20%at random63.0-0 rows64.0100-
Median errors over the seeds. The missing share is of every value, completely at random unless marked; at random, the odd columns miss twice as often. Listwise deletion fits the complete rows only, whose median count is under its errors; a dash is a size where no fit finished, such as no complete rows left, never scored as an empty graph. The difference is the mean over the seeds, with its 95% interval.

PC with Fisher's z, missing values

Test-wiseListwiseComplete dataEmpty graphTest-wise minus listwise
10 variables, 1,000 rows0%2.02.02.010-
10%1.53.0347 rows2.010-1.0[-2.1, +0.0]
20%2.05.5104 rows2.010-2.2[-4.1, -0.4]
30%2.58.528 rows2.010-5.3[-6.6, -4.0]
20%at random3.57.076 rows2.010-2.9[-4.6, -1.1]
10 variables, 5,000 rows0%1.01.01.010-
10%1.01.51,732 rows1.010-1.0[-2.4, +0.2]
20%1.03.0530 rows1.010-1.2[-2.2, -0.2]
30%1.06.0139 rows1.010-3.5[-5.7, -1.3]
20%at random2.04.5376 rows1.010-2.3[-4.1, -0.7]
20 variables, 1,000 rows0%4.04.04.020-
10%6.58.5124 rows4.020-3.3[-4.8, -1.8]
20%7.020.010 rows4.020-12.8[-15.2, -10.5]
30%7.5-0 rows4.020-
20%at random7.520.55 rows4.020-13.8[-16.5, -11.1]
20 variables, 5,000 rows0%4.04.04.020-
10%3.07.5611 rows4.020-3.5[-5.1, -1.9]
20%4.011.556 rows4.020-8.1[-9.2, -6.9]
30%4.021.04 rows4.020-15.9[-17.8, -14.0]
20%at random4.516.532 rows4.020-12.5[-14.1, -10.7]
50 variables, 1,000 rows0%12.512.512.550-
10%10.551.06 rows12.550-39.0[-43.1, -34.4]
20%14.0-0 rows12.550-
30%13.5-0 rows12.550-
20%at random13.0-0 rows12.550-
50 variables, 5,000 rows0%9.59.59.550-
10%10.040.032 rows9.550-30.9[-34.3, -27.9]
20%7.0-0 rows9.550-
30%7.5-0 rows9.550-
20%at random7.0-0 rows9.550-
Median errors over the seeds. The missing share is of every value, completely at random unless marked; at random, the odd columns miss twice as often. Listwise deletion fits the complete rows only, whose median count is under its errors; a dash is a size where no fit finished, such as no complete rows left, never scored as an empty graph. The difference is the mean over the seeds, with its 95% interval.

PC with Student's t, missing values

Test-wiseListwiseComplete dataEmpty graphTest-wise minus listwise
10 variables, 1,000 rows0%2.02.02.010-
10%1.53.0347 rows2.010-1.0[-2.1, +0.0]
20%2.05.5104 rows2.010-2.2[-4.0, -0.3]
30%2.58.528 rows2.010-5.3[-6.6, -4.0]
20%at random3.57.076 rows2.010-2.9[-4.6, -1.1]
10 variables, 5,000 rows0%1.01.01.010-
10%1.01.51,732 rows1.010-1.0[-2.4, +0.2]
20%1.03.0530 rows1.010-1.2[-2.2, -0.2]
30%1.06.0139 rows1.010-3.5[-5.7, -1.3]
20%at random2.04.5376 rows1.010-2.3[-4.1, -0.7]
20 variables, 1,000 rows0%4.04.04.020-
10%6.58.5124 rows4.020-3.3[-4.8, -1.8]
20%7.020.010 rows4.020-12.6[-15.1, -10.4]
30%7.5-0 rows4.020-
20%at random7.521.55 rows4.020-14.2[-16.8, -11.7]
20 variables, 5,000 rows0%4.04.04.020-
10%3.07.5611 rows4.020-3.5[-5.0, -1.9]
20%4.011.556 rows4.020-8.1[-9.2, -6.9]
30%4.021.04 rows4.020-15.9[-17.8, -14.2]
20%at random4.516.532 rows4.020-12.5[-14.1, -10.8]
50 variables, 1,000 rows0%12.512.512.550-
10%10.553.06 rows12.550-39.3[-43.3, -34.9]
20%14.0-0 rows12.550-
30%13.5-0 rows12.550-
20%at random13.0-0 rows12.550-
50 variables, 5,000 rows0%9.59.59.550-
10%10.040.032 rows9.550-30.9[-34.3, -28.0]
20%7.0-0 rows9.550-
30%7.5-0 rows9.550-
20%at random7.0-0 rows9.550-
Median errors over the seeds. The missing share is of every value, completely at random unless marked; at random, the odd columns miss twice as often. Listwise deletion fits the complete rows only, whose median count is under its errors; a dash is a size where no fit finished, such as no complete rows left, never scored as an empty graph. The difference is the mean over the seeds, with its 95% interval.

Bootstrap#

Each table is one method. A row is a tenth of edge frequency: how many edges appeared in that share of the 100 bootstrap fits, and how many of them are in the true graph, for adjacencies and for orientations.

PC: how often an edge appears, and how often it is true

AdjacenciesTrueOrientationsTrue
1,000 rows0.9 to 1.017398%54100%
0.8 to 0.9888%48100%
0.7 to 0.8450%3385%
0.6 to 0.7475%1889%
0.5 to 0.6850%2264%
0.4 to 0.560%2941%
0.3 to 0.4922%2528%
0.2 to 0.31612%4312%
0.1 to 0.2385%985%
0.0 to 0.15231%1,1121%
5,000 rows0.9 to 1.0185100%66100%
0.8 to 0.90-48100%
0.7 to 0.81100%32100%
0.6 to 0.710%1894%
0.5 to 0.620%13100%
0.4 to 0.51020%1735%
0.3 to 0.4922%2421%
0.2 to 0.31718%388%
0.1 to 0.2518%1224%
0.0 to 0.15391%1,1400%
Each tenth of frequency counts the edges that appeared in that share of the bootstrap fits, over 10 datasets of 20 variables, and the share of them in the true graph. 1,000 rows: the averaged graph's median errors 3.5, a single fit's 4.0; 5,000 rows: the averaged graph's median errors 1.5, a single fit's 4.0.

GES: how often an edge appears, and how often it is true

AdjacenciesTrueOrientationsTrue
1,000 rows0.9 to 1.018998%11396%
0.8 to 0.91050%1283%
0.7 to 0.8633%3063%
0.6 to 0.7520%1644%
0.5 to 0.6425%1947%
0.4 to 0.5450%3047%
0.3 to 0.4157%3219%
0.2 to 0.3390%6825%
0.1 to 0.2811%1225%
0.0 to 0.18810%1,3320%
5,000 rows0.9 to 1.019799%11698%
0.8 to 0.9367%24100%
0.7 to 0.830%2688%
0.6 to 0.750%2250%
0.5 to 0.6110%1839%
0.4 to 0.5812%2635%
0.3 to 0.460%2129%
0.2 to 0.3150%329%
0.1 to 0.2580%1001%
0.0 to 0.17230%1,0140%
Each tenth of frequency counts the edges that appeared in that share of the bootstrap fits, over 10 datasets of 20 variables, and the share of them in the true graph. 1,000 rows: the averaged graph's median errors 4.5, a single fit's 0.5; 5,000 rows: the averaged graph's median errors 3.0, a single fit's 0.0.

DirectLiNGAM: how often an edge appears, and how often it is true

AdjacenciesTrueOrientationsTrue
1,000 rows0.9 to 1.0200100%200100%
0.8 to 0.90-0-
0.7 to 0.80-0-
0.6 to 0.70-0-
0.5 to 0.60-0-
0.4 to 0.530%20%
0.3 to 0.480%30%
0.2 to 0.3370%240%
0.1 to 0.21120%960%
0.0 to 0.11,1710%1,6820%
5,000 rows0.9 to 1.0200100%200100%
0.8 to 0.90-0-
0.7 to 0.80-0-
0.6 to 0.710%0-
0.5 to 0.60-0-
0.4 to 0.520%20%
0.3 to 0.450%30%
0.2 to 0.370%60%
0.1 to 0.2450%400%
0.0 to 0.18640%1,1040%
Each tenth of frequency counts the edges that appeared in that share of the bootstrap fits, over 10 datasets of 20 variables, and the share of them in the true graph. 1,000 rows: the averaged graph's median errors 0.0, a single fit's 0.0; 5,000 rows: the averaged graph's median errors 0.0, a single fit's 0.0.

The boundary#

The boundary is carried over from the 0.10.0 release card: its ladders were measured on the 0.10.0 builds and not run again on 0.11.0. Two changes can move it. The extended BIC, now the default of ten score searches, changes their graphs and so their accuracy boundary, and the simulated data are standardized by default.

Other changes in 0.11.0#

Background knowledge reaches 27 of the 34 methods. With no test or score named, a method picks one from the column types you declare. Arguments share one name across methods, the former names working in this release with a warning. The changelog lists every change, and the upgrade guide shows the common calls before and after.

Limits#

  • Missing values. Values missing completely at random, 10% to 30% of them, and one pattern missing at random, 20%, on linear-Gaussian data, with PC and FCI and the Fisher’s z and Student’s t tests. Values missing not at random, other methods, and the mixed-data tests are not measured.

  • Bootstrap. 20 variables with a mean degree of 2, 1,000 and 5,000 samples, and 100 resamples. A frequency measures how stable an edge is across resamples, not the probability that it is true.

  • Extended BIC. Linear-Gaussian data, up to 400 variables. The extended BIC applies to the linear-Gaussian BIC only; the categorical and mixed-data scores ignore ebic_gamma.

Reproduce#

Extended BIC. Each method at ebic_gamma 0.0 and 1.0 on every dataset, at its other defaults; BOSS and GRaSP with seed=0; RSMAX2 as restrict="pc", maximize="tabu". The datasets: Erdos-Renyi graphs with a mean degree of 2 and 4 and scale-free graphs with a mean degree of 2, by 20 to 400 variables and 200 to 4,000 samples, 5 seeds each; the same Erdos-Renyi graphs with unitless weights, 20 to 100 variables; four bnlearn Gaussian networks (arth150, ecoli70, magic-irri, magic-niab) at 200, 1,000, and 5,000 samples, 3 seeds each; and Sachs. All are linear Gaussian and standardized. Exact search runs up to 20 variables. The errors are the SHD of the CPDAG against the true CPDAG, and for GFCI of the PAG; the change per true edge averages each data source’s mean, with a bootstrap interval. A 900-second and 10 GB cap per fit; 2 of 352 datasets reached the time cap at both settings.

Missing values. andrey.data.sample_scm at its defaults, Erdos-Renyi graphs with a mean degree of 2, at 10, 20, and 50 variables by 1,000 and 5,000 samples, 10 seeds each. Completely at random, each value is missing with the stated probability; at random, the even columns are always observed and an odd column goes missing more often when the column before it is high. andrey.pc and andrey.fci with indep_test="fisherz" or "student_t", other settings at their defaults. PC is scored by CPDAG SHD, FCI by endpoint SHD. A 900-second and 6 GB cap per fit.

Bootstrap. andrey.data.sample_scm at its defaults, 20 variables, a mean degree of 2, and 1,000 and 5,000 samples, 10 seeds each, Gaussian noise for PC and GES and uniform noise for DirectLiNGAM. Each method at its defaults once on the full data, then andrey.bootstrap(method, X, n_boot=100, seed=seed) with one worker. Edges are judged against the true CPDAG for PC and GES and the true DAG for DirectLiNGAM; the averaged graph keeps the pairs adjacent in at least half the fits.

Builds and versions#

Every lane ran on build e15e826 on AMD EPYC 7713 processors, with some extended-BIC fits on an AMD EPYC 9534 or an Intel Xeon Gold 6542Y. The code of 0.11.0 is that build’s: no file under src/ changed after it.