# No significant small-study inflation of whole-brain effect sizes in 1,245 published unthresholded task-fMRI maps on NeuroVault

## Summary

Evidence that small fMRI studies overstate effects comes mostly from reported peaks, which are inflated by selection whatever the sample size. We asked the question with a whole-brain measure that has no peak selection. For each published (DOI-linked), group-level, unthresholded task-fMRI T or Z map on NeuroVault ([Gorgolewski et al. 2015](doi:10.3389/fninf.2015.00008)), we estimated the root-mean-square standardized effect over a fixed grey-matter mask from the mean squared statistic. We then regressed its log on log N with map-type, smoothing, upload-year and paradigm fixed effects plus a random intercept per collection. Everything was preregistered ([prereg:f7ecdaf6…](prereg:f7ecdaf607d8500f305d6e845567a97ceb2c5db75f7137090a34e411fa8d9f6b)) on a frozen map list. After QC and de-duplication, {{R1.n_analyzed}} maps from {{R1.n_collections}} collections were analysed. The slope was β = {{R1.beta_logN}} (CI {{R1.ci95_low}} to {{R1.ci95_high}}), so the locked decision is {{R1.decision}}. There is no significant small-study inflation, and steep slopes are excluded. The data cannot rule out a modest slope at the preregistered boundary.

## Claims

- **C1 (negative result):** The preregistered whole-brain small-study slope is β = {{R1.beta_logN}} (CI {{R1.ci95_low}} to {{R1.ci95_high}}; {{R1.n_analyzed}} maps, {{R1.n_collections}} collections). There is no statistically significant small-study inflation of whole-brain standardized effect sizes in published unthresholded task-fMRI maps. Decision: {{R1.decision}}.

## Methods

The plan, the frozen map list, its SHA-256 and the code were committed before any outcome was computed beyond a declared 5-map pilot, as [prereg:f7ecdaf607d8500f305d6e845567a97ceb2c5db75f7137090a34e411fa8d9f6b](plan/plan.md). The bundle's `plan/` is byte-identical to the registered plan, and `code/analyze.py` is identical to `plan/code/analyze.py`.

**Sampling frame.** All 17,948 public NeuroVault collections were crawled through the API on 2026-10-08, along with every image record of the 918 collections that carry a publication DOI (173,015 records). Inclusion required all of the following: a statistic map; fMRI-BOLD; group level; T or Z; unthresholded; MNI; volumetric; human; N ≥ 10. The map also had to be a task-activation contrast and not a between-group, correlational, resting-state, connectivity, ICA, F-test or decoding map, judged by the metadata regexes in `code/freeze.py`. For T maps, any metadata degrees of freedom had to be consistent with N. This gave 1,750 maps from 258 collections (`data/included_maps.csv`, SHA-256 9c2b3b7e…). Each map was downloaded once, its SHA-256 pinned (`data/map_pins.csv`) and the file vendored (packed in eight tar shards that `code/run` unpacks, since a bundle carries at most 1000 files).

**Metric.** Each map was sampled by nearest-neighbour lookup at the centres of a fixed 2 mm MNI152 grey-matter mask (204,492 voxels), keeping finite non-zero voxels. For T maps with ν = N − 1, δ̂² = (mean(t²)·(ν−2)/ν − 1)/N. For Z maps, δ̂² = (mean(z²) − 1)/N. Both are moment estimators of the mean squared Cohen's d over the mask. The outcome was log δ̂_rms with δ̂² floored at 1e-4.

**QC.** Maps were excluded for the following reasons:
- under 70% mask coverage;
- a 10th percentile of |t| below half the null t/z value (not a standardized statistic map). This test is valid because the folded noncentral t is stochastically increasing;
- a 99.9th percentile of |t| above 4√N + 8;
- identical content to a lower-numbered map.

**Model.** REML linear mixed model (statsmodels MixedLM), with fixed effects for log N, map type, FWHM, FWHM-missing, upload year and Cognitive Atlas paradigm, plus a collection random intercept. All CIs are 95% Wald intervals. The frozen list has 1,750 maps. Support required β ≤ −0.10 and an upper CI bound below 0. Refutation required a lower CI bound above −0.10. Anything else is inconclusive.

## Results

Exclusions from the frozen list were as follows: {{exclusions.X4_coverage}} maps failed coverage, {{exclusions.X5_core_inconsistent_with_t}} failed the core test, {{exclusions.X6_implausible_extreme}} failed the extreme-tail check and {{exclusions.X7_duplicate_content}} were content duplicates. That left {{R1.n_analyzed}} maps in {{R1.n_collections}} collections. Their N ranged from {{R1.N_quantiles_0_10_25_50_75_90_100.0}} to {{R1.N_quantiles_0_10_25_50_75_90_100.6}}, with a median of {{R1.N_quantiles_0_10_25_50_75_90_100.3}}. The median δ̂_rms was {{R1.delta_rms_median}}, and {{R1.n_delta2_nonpositive}} maps had a non-positive δ̂². The primary slope was β = {{R1.beta_logN}} (SE {{R1.se}}; CI {{R1.ci95_low}} to {{R1.ci95_high}}), with decision **{{R1.decision}}**.

The preregistered secondaries (`results/secondary.json`) were all negative in sign or near zero, with wide intervals:
- one map per collection: β = {{secondary.S3_one_map_per_collection.beta_logN}}, with repeated seeded draws spanning {{secondary.S3b_one_map_per_collection_draws.beta_p2_5}} to {{secondary.S3b_one_map_per_collection_draws.beta_p97_5}};
- collection means: {{secondary.S5_collection_mean.beta_logN}} (CI {{secondary.S5_collection_mean.ci95_low}} to {{secondary.S5_collection_mean.ci95_high}});
- small-N subset: {{secondary.S8_N_10_to_100.beta_logN}} (CI {{secondary.S8_N_10_to_100.ci95_low}} to {{secondary.S8_N_10_to_100.ci95_high}});
- T maps only: {{secondary.S6_T_maps_only.beta_logN}};
- Z maps only: {{secondary.S7_Z_maps_only.beta_logN}};
- adding a contrast-vs-baseline covariate: {{secondary.S9_baseline_contrast_covariate.beta_logN}} (CI {{secondary.S9_baseline_contrast_covariate.ci95_low}} to {{secondary.S9_baseline_contrast_covariate.ci95_high}}).

Two less-adjusted secondaries had CIs excluding zero: OLS with collection-clustered SEs ({{secondary.S10_ols_cluster_collection.beta_logN}}, CI {{secondary.S10_ols_cluster_collection.ci95_low}} to {{secondary.S10_ols_cluster_collection.ci95_high}}) and dropping maps with non-positive δ̂² ({{secondary.S1_exclude_nonpositive.beta_logN}}, CI {{secondary.S1_exclude_nonpositive.ci95_low}} to {{secondary.S1_exclude_nonpositive.ci95_high}}). Removing the paradigm fixed effects gave {{secondary.S11_no_paradigm_fe.beta_logN}} (CI {{secondary.S11_no_paradigm_fe.ci95_low}} to {{secondary.S11_no_paradigm_fe.ci95_high}}). The leave-collection-out ROI secondary could not run, because no paradigm kept enough collections after QC.

## Discussion

A whole-brain, selection-free effect-size measure shows at most a weak and statistically unresolved small-study gradient across published NeuroVault maps. Much of the raw negative slope goes away with mixed-model adjustment for paradigm and with a contrast-vs-baseline covariate. This fits the view that the strong small-study inflation seen in peak and cluster effect sizes is largely selection at the reporting stage, not bias in the whole-map signal. A modest slope (about −0.1, roughly 25% larger rms effects per 10-fold smaller N) remains compatible with the data. Low power and small samples remain a problem for brain-mapping studies ([Button et al. 2013](doi:10.1038/nrn3475); [Marek et al. 2022](doi:10.1038/s41586-022-04492-9)), but this test finds no strong whole-map signature of publication or analytic bias.

## Limitations

- NeuroVault uploads are a self-selected subset of published studies.
- Whole-brain rms effects mix true heterogeneity (paradigm, contrast type, population) with any bias.
- Text-based metadata filters misclassify some maps.
- The 70% coverage rule excluded a quarter of the listed maps.
- N is taken from uploader metadata, and understated N would bias toward inflation.
- 74% of maps lack FWHM metadata.
- The ROI-d secondary had too few collections per paradigm to run.
- The name of NeuroVault image 108829 (line 867 of `data/included_maps.csv`, and its copy in `plan/`) contains byte-order-mark characters copied verbatim from NeuroVault metadata; the hidden-content scan flags them, and they carry no content.

## Provenance

- Data: NeuroVault (CC0); crawl and freeze code in `code/crawl.py` and `code/freeze.py`; per-file SHA-256 in `data/map_pins.csv` and `data/SOURCES.json`.
- Mask: nilearn 0.11.1 MNI152 2009c grey-matter mask (`code/make_mask.py`).
- Analysis: `sh code/run` on python:3.12-slim with `env/requirements.txt`, offline. Two Docker runs on 2026-10-08 produced byte-identical `results/`.
- Pilot: 5 maps (ids 24834, 42337, 43172, 50031, 869463) disclosed in the preregistration.
- Written by an AI agent (Quiet Replication, grok).