--- name: benchmarks-0.11.0 description: The Andrey 0.11.0 release card - the extended BIC before and after, missing values by rate, and how often the bootstrap's frequent edges are true. meta: type: guide tags: [benchmarks, missing values, bootstrap] --- # 0.11.0 release card: ready for real data :::::{div} release-opening :::{card} :class-card: release-cover release-art release-art-0-11-0 :shadow: none Release card Ready for real data 0.11.0 ::: ::::{div} release-opening-text This card measures what 0.11.0 changes for real data: the extended BIC as the default penalty of ten score searches, PC and FCI on data with missing values, and the bootstrap's edge frequencies. Each result holds for the measured conditions; it depends on the data, settings, and hardware. :::: ::::: ## Summary **Extended BIC.** On the same 299 simulated datasets, the extended BIC (`ebic_gamma=1.0`) lowers the errors of each of the ten methods it is the default for. Per true edge, the mean change is -0.74 to -0.83 for GES, XGES, GIES, hill climbing, Tabu search, BOSS, and GRaSP, -0.24 for exact search, and -0.10 for MMHC and RSMAX2, each with its 95% interval below 0, and no size is significantly worse for any of them. GFCI keeps the standard BIC: 3 of its 65 sizes are significantly worse with the extended BIC. **Missing values.** With 20 variables and 1,000 samples, dropping the incomplete samples (listwise deletion) keeps a median of 124.5 samples with a tenth of the values missing, 10.5 with a fifth, and 0.5 with 30%, where no fit runs. Test-wise deletion fits every size: PC's median errors are 6.5, 7.0, and 7.5, against 4.0 on the complete data and 20 for the empty graph. Over every size, test-wise deletion is never significantly worse than listwise deletion. **Bootstrap.** The edge frequencies rank edges: of the adjacencies in more than nine fits in ten, 98% to 100% are true, and of those in fewer than one in ten, at most 1.1%. The frequencies in between are not probabilities. The averaged graph is not consistently more accurate than a single fit: fewer errors for PC, more for GES, the same for DirectLiNGAM. ## Extended BIC, the new default Each row compares one method with `ebic_gamma` at 0 and at 1 on the same datasets, with every other setting at its default. Sachs, 853 real samples of 11 proteins, is left out of the datasets the table counts. On it and 50 resamples of it, every method has fewer errors with the extended BIC too: about 2 fewer for most, and under 1 for MMHC, RSMAX2, and GFCI. ```{raw} html :file: 0.11.0/ebic.html ``` ## Missing values Each table is one method with one test. Listwise deletion runs out of samples as the variables and the missing share grow; test-wise deletion stays near the complete data. With 10 variables, where listwise deletion keeps more samples, the two are close. ```{raw} html :file: 0.11.0/missing.html ``` ## Bootstrap Each table is one method. A row is a tenth of edge frequency: how many edges appeared in that share of the 100 bootstrap fits, and how many of them are in the true graph, for adjacencies and for orientations. ```{raw} html :file: 0.11.0/bootstrap.html ``` ## The boundary The boundary is carried over from the [0.10.0 release card](0.10.0.md): its ladders were measured on the 0.10.0 builds and not run again on 0.11.0. Two changes can move it. The extended BIC, now the default of ten score searches, changes their graphs and so their accuracy boundary, and the simulated data are standardized by default. ## Other changes in 0.11.0 Background knowledge reaches 27 of the 34 methods. With no test or score named, a method picks one from the column types you declare. Arguments share one name across methods, the former names working in this release with a warning. The [changelog](../changelog.md) lists every change, and the [upgrade guide](../guides/upgrading.md) shows the common calls before and after. ## Limits - **Missing values.** Values missing completely at random, 10% to 30% of them, and one pattern missing at random, 20%, on linear-Gaussian data, with PC and FCI and the Fisher's z and Student's t tests. Values missing not at random, other methods, and the mixed-data tests are not measured. - **Bootstrap.** 20 variables with a mean degree of 2, 1,000 and 5,000 samples, and 100 resamples. A frequency measures how stable an edge is across resamples, not the probability that it is true. - **Extended BIC.** Linear-Gaussian data, up to 400 variables. The extended BIC applies to the linear-Gaussian BIC only; the categorical and mixed-data scores ignore `ebic_gamma`. ## Reproduce **Extended BIC.** Each method at `ebic_gamma` 0.0 and 1.0 on every dataset, at its other defaults; BOSS and GRaSP with `seed=0`; RSMAX2 as `restrict="pc"`, `maximize="tabu"`. The datasets: Erdos-Renyi graphs with a mean degree of 2 and 4 and scale-free graphs with a mean degree of 2, by 20 to 400 variables and 200 to 4,000 samples, 5 seeds each; the same Erdos-Renyi graphs with unitless weights, 20 to 100 variables; four bnlearn Gaussian networks (arth150, ecoli70, magic-irri, magic-niab) at 200, 1,000, and 5,000 samples, 3 seeds each; and Sachs. All are linear Gaussian and standardized. Exact search runs up to 20 variables. The errors are the SHD of the CPDAG against the true CPDAG, and for GFCI of the PAG; the change per true edge averages each data source's mean, with a bootstrap interval. A 900-second and 10 GB cap per fit; 2 of 352 datasets reached the time cap at both settings. **Missing values.** `andrey.data.sample_scm` at its defaults, Erdos-Renyi graphs with a mean degree of 2, at 10, 20, and 50 variables by 1,000 and 5,000 samples, 10 seeds each. Completely at random, each value is missing with the stated probability; at random, the even columns are always observed and an odd column goes missing more often when the column before it is high. `andrey.pc` and `andrey.fci` with `indep_test="fisherz"` or `"student_t"`, other settings at their defaults. PC is scored by CPDAG SHD, FCI by endpoint SHD. A 900-second and 6 GB cap per fit. **Bootstrap.** `andrey.data.sample_scm` at its defaults, 20 variables, a mean degree of 2, and 1,000 and 5,000 samples, 10 seeds each, Gaussian noise for PC and GES and uniform noise for DirectLiNGAM. Each method at its defaults once on the full data, then `andrey.bootstrap(method, X, n_boot=100, seed=seed)` with one worker. Edges are judged against the true CPDAG for PC and GES and the true DAG for DirectLiNGAM; the averaged graph keeps the pairs adjacent in at least half the fits. ## Builds and versions Every lane ran on build `e15e826` on AMD EPYC 7713 processors, with some extended-BIC fits on an AMD EPYC 9534 or an Intel Xeon Gold 6542Y. The code of 0.11.0 is that build's: no file under `src/` changed after it.