# Single-factor NHANES findings: a quarter to a third replicated in new data, most tests lacked power, and most analyses differ from their description, a few decisively

## Summary

Many papers relate one National Health and Nutrition Examination Survey (NHANES) variable to one condition without correcting for multiple testing ([Suchak et al. (2025)](doi:10.1371/journal.pbio.3003152)). We sampled {{R1.replication.associations}} associations, reproduced each on its paper's cycles, and under a [registered plan](prereg:66c14e8df19aafb51c6890eb73fc46addc6eb292998aa04c081f412f2df6e299) re-ran it on 2021–2023. There, {{R1.replication.replicated.k}} replicated as computed, and {{R1.replication.design.weighted.replicated.k}} with the survey design where papers ignored it. Most tests lacked power, but were every association true, {{R1.replication.expected.published.mean}} would replicate; counts this low are rare ({{R1.replication.expected.published.at_most_observed}}, or {{R1.replication.expected_design.published.at_most_observed}} with the design), so many effects are smaller than published or absent. In {{R1.reproduction.departures.affecting_headline.k}} papers the computation reproducing the estimate differs in some respect from the one described; as described, {{R1.reproduction.described.outside_published_ci.k}} fall outside the published interval.

## Claims

- **C1:** Of the {{R1.replication.replicated.n}} associations sampled from those eligible, {{R1.replication.replicated.k}} replicated in NHANES August 2021–August 2023, with the published sign and a significant corrected p-value, each analysis computed as its paper's was: a share of {{R1.replication.replicated.share}}, with an exact confidence interval from {{R1.replication.replicated.ci.0}} to {{R1.replication.replicated.ci.1}} that treats the associations as independent. With the {{R1.replication.unweighted.k}} analyses that ignore the survey design, as their papers did, fitted with its weights and design, as the plan also registered, {{R1.replication.design.weighted.replicated.k}} replicated ({{R1.replication.design.weighted.replicated.share}}, {{R1.replication.design.weighted.replicated.ci.0}} to {{R1.replication.design.weighted.replicated.ci.1}}).
- **C2:** At the nominal significance level, which bounds the power of the corrected replication decision from above, only {{R1.replication.informative}} of the replication tests had the planned power to detect the published effect, taken as true, and {{R1.replication.replicated_informative.k}} of them replicated (a share of {{R1.replication.replicated_informative.share}}, from {{R1.replication.replicated_informative.ci.0}} to {{R1.replication.replicated_informative.ci.1}}); at the Bonferroni level, which bounds it from below, {{R1.replication.informative_bonferroni}} had it, and {{R1.replication.replicated_informative_bonferroni.k}} of them replicated. With the variance of the analyses re-run unweighted taken from the survey design, {{R1.replication.design.design.informative}} had it at the nominal level, {{R1.replication.design.design.informative_also_registered}} of them among the {{R1.replication.informative}} above, and {{R1.replication.design.design.replicated_informative.k}} of them replicated.
- **C3:** On the analysis scale, the 2021–2023 effects are a median {{R1.replication.ratio.median}} times the published ones ({{R1.replication.ratio.ci.0}} to {{R1.replication.ratio.ci.1}}) over all associations, and {{R1.replication.ratio_informative.median}} times ({{R1.replication.ratio_informative.ci.0}} to {{R1.replication.ratio_informative.ci.1}}) over the informative ones, with distribution-free intervals that treat the associations as independent. The median is {{R1.replication.ratio_unweighted.median}} over the analyses re-run unweighted and {{R1.replication.ratio_others.median}} over the others; with the registered weighted fits in place of the unweighted ones, it is {{R1.replication.ratio_weighted.median}} ({{R1.replication.ratio_weighted.ci.0}} to {{R1.replication.ratio_weighted.ci.1}}).
- **C4:** {{R1.replication.differs_from_published.k}} of the 2021–2023 estimates differ significantly from the published ones after correction: {{R1.replication.differs_by_direction.smaller.k}} smaller than published, {{R1.replication.differs_by_direction.larger.k}} larger, and {{R1.replication.differs_by_direction.opposite_sign.k}} on the other side of the null; with t tests on the fits' design degrees of freedom, {{R1.replication.differs_from_published_t.k}} differ.
- **C5:** In 2021–2023, {{R1.replication.in_published_ci.k}} estimates fall inside the published confidence interval, {{R1.replication.same_sign_p05.k}} have the published sign with an unadjusted p-value below the planned threshold, and {{R1.replication.reversed.k}} have the opposite sign with a significant corrected p-value.
- **C6:** On the cycles each paper analyzed, our code meets the registered reproduction criterion, its estimate inside the published interval, for {{R1.reproduction.reproduced.k}} of the published estimates, with a median ratio of our estimate to the published one of {{R1.reproduction.ratio_original.median}} ({{R1.reproduction.ratio_original.ci.0}} to {{R1.reproduction.ratio_original.ci.1}}). The one it doesn't reproduce modeled the absence of asthma while reporting the odds of asthma: coded as stated it gives {{R2.row257.original.estimate}}, and reversed, {{R2.row257.variants.0.estimate}}.
- **C7:** In {{R1.reproduction.departures.affecting_headline.k}} papers, the investigators recorded at least one departure bearing on the headline estimate, where the computation that reproduces it differs in some respect from the paper's description. In {{R1.reproduction.departures.shown_by_paper.k}} of them the paper itself shows one, two of its parts disagreeing or its own numbers ruling out what it describes; in {{R1.reproduction.departures.identified_by_data.k}} more, recomputation from the public files shows one, a computation other than the one described being the one that reproduced the published numbers.
- **C8:** {{R1.replication.heterogeneous.k}} associations differ significantly, after correction, between 2021–2023 and our harmonized analysis of the paper's own cycles, with z tests, and {{R1.replication.heterogeneous_t.k}} with t tests on the design degrees of freedom. This does not show the estimates equal: the median ratio of the two is {{R1.replication.ratio_own.median}} ({{R1.replication.ratio_own.ci.0}} to {{R1.replication.ratio_own.ci.1}}).
- **C9:** Among the reproduced associations, {{R1.replication.replicated_reproduced.k}} of {{R1.replication.replicated_reproduced.n}} replicated, and with each paper's own survey weight in place of the 2021–2023 subsample weights, {{R1.replication.replicated_paper_weight.k}}. With the unweighted estimates of the {{R1.replication.unweighted.k}} analyses re-run unweighted and their variance taken from the survey design, {{R1.replication.design.design.replicated.k}} replicated, {{R1.replication.design.design.replicated_unweighted.k}} of them among those {{R1.replication.unweighted.k}}, against {{R1.replication.replicated_unweighted.k}} as registered.
- **C10:** In 2021–2023, the coefficient of serum albumin on workday sleep of five hours or less, against more than seven and up to eight, is {{R2.row284.replication.estimate}} g/L ({{R2.row284.replication.low}} to {{R2.row284.replication.high}}), against the published {{R2.row284.published.estimate}} g/L ({{R2.row284.published.low}} to {{R2.row284.published.high}}): a smaller decrement than published, with an interval that includes zero, and a significant difference from it (corrected p {{R2.row284.difference_q}}, or {{R2.row284.difference_q_t}} by t test). With the phlebotomy weight in place of the interview weight the paper used, it is {{R2.row284.replication_phlebotomy.estimate}} g/L ({{R2.row284.replication_phlebotomy.low}} to {{R2.row284.replication_phlebotomy.high}}), still significantly different from the published estimate (corrected p {{R2.row284.difference_q_phlebotomy}}, or {{R2.row284.difference_q_t_phlebotomy}} by t test). These tests take the published standard error from its interval; against our harmonized estimate on the paper's own cycles, the 2021–2023 estimate does not differ significantly (corrected p {{R2.row284.heterogeneity_q}}). On the paper's own cycles, a term for the cycle, which absorbs every difference between its two cycles, gives {{R2.row284.variants.9.estimate}} g/L, and putting 2017–2018's albumin on 2015–2016's analyzer scale by NCHS's bridging equation gives {{R2.row284.variants.10.estimate}} g/L ({{R2.row284.variants.10.low}} to {{R2.row284.variants.10.high}}).
- **C11:** In 2021–2023, the odds ratio of diabetes on the systemic immune-inflammation index, on the paper's scale, is {{R2.row303.replication.estimate}} ({{R2.row303.replication.low}} to {{R2.row303.replication.high}}), against the published {{R2.row303.published.estimate}} ({{R2.row303.published.low}} to {{R2.row303.published.high}}): a significant difference (corrected p {{R2.row303.difference_q}}, or {{R2.row303.difference_q_t}} by t test), the later estimate below one, though not significantly so after correction. This estimate is unweighted, as the published one is; with the survey weights and design, the paper's own cycles give {{R2.row303.variants.0.estimate}} ({{R2.row303.variants.0.low}} to {{R2.row303.variants.0.high}}) and 2021–2023 gives {{R2.row303.replication_weighted.estimate}} ({{R2.row303.replication_weighted.low}} to {{R2.row303.replication_weighted.high}}), different from the published estimate by z (corrected p {{R2.row303.difference_q_weighted}}) but not by t ({{R2.row303.difference_q_t_weighted}}). Against our harmonized estimate on the paper's own cycles, the 2021–2023 estimate does not differ significantly (corrected p {{R2.row303.heterogeneity_q}}).
- **C12:** In 2021–2023, the odds ratio of depression per unit of the triglyceride-glucose index is {{R2.row311.replication.estimate}} ({{R2.row311.replication.low}} to {{R2.row311.replication.high}}), against the published {{R2.row311.published.estimate}} ({{R2.row311.published.low}} to {{R2.row311.published.high}}): a significant difference by the registered z test (corrected p {{R2.row311.difference_q}}) but not by a t test on the design degrees of freedom (corrected p {{R2.row311.difference_q_t}}), the later estimate below one but its interval wide; against our harmonized estimate on the paper's own cycles, it does not differ significantly (corrected p {{R2.row311.heterogeneity_q}}).
- **C13:** Two independent codings of how each of the {{R1.reproduction.departures.headline_departures}} departures bearing on a headline estimate is established, made by two agents of one model family from the recorded details rather than from the papers, agreed on {{R1.reproduction.departures.coding.agree.k}} (Cohen's kappa {{R1.reproduction.departures.coding.kappa}}). As settled, {{R1.reproduction.departures.by_evidence.paper.k}} are shown by the paper itself, {{R1.reproduction.departures.by_evidence.data.k}} by recomputation that identifies another computation, and {{R1.reproduction.departures.by_evidence.unresolved.k}} are unresolved, the computation described ruled out but the one used not identified. Of {{R1.reproduction.departures.validation.drawn}} drawn at random and checked against the papers' full texts by an agent of the same family that recorded none of them, for what each record says of its paper, {{R1.reproduction.departures.validation.confirmed.k}} describe it correctly, {{R1.reproduction.departures.validation.partly.k}} misstate something about it though the departure stands, and {{R1.reproduction.departures.validation.not_confirmed.k}} misdescribe it (an exact upper bound of {{R1.reproduction.departures.validation.not_confirmed.ci.1}} for that share).
- **C14:** On the cycles each paper analyzed, the analysis as the paper describes it, with the departures bearing on the headline that are computations put back as its text describes them under the rules in [code/described_rules.md](code/described_rules.md), changes the specification of {{R1.reproduction.described.changed}} analyses. Among those, the described estimate is a median {{R1.reproduction.described.to_followed_changed.median}} times the followed one (nearly unchanged for {{R1.reproduction.described.within_one_percent}}), falls outside the published confidence interval for {{R1.reproduction.described.outside_published_ci_changed.k}}, and lacks a significant estimate with the published sign for {{R1.reproduction.described.not_significant_changed.k}}, against {{R1.reproduction.described.not_significant_changed_followed.k}} as followed. With the paper whose followed version already codes its outcome as stated, {{R1.reproduction.described.outside_published_ci.k}} of the {{R1.reproduction.associations}} analyses as described fall outside the published interval.
- **C15:** If every association were true at its published size, the registered replication tests, simulated from their own 2021–2023 standard errors and taken as independent, would replicate {{R1.replication.expected.published.mean}} of the {{R1.replication.associations}} on average ({{R1.replication.expected.published.range.0}} to {{R1.replication.expected.published.range.1}}, the draws' central range), and as few as the {{R1.replication.expected.observed}} observed in a share {{R1.replication.expected.published.at_most_observed}} of draws; at half its published size, {{R1.replication.expected.half.mean}} ({{R1.replication.expected.half.range.0}} to {{R1.replication.expected.half.range.1}}). With the variance of the analyses re-run unweighted taken from the survey design, {{R1.replication.expected_design.published.mean}} would be expected and as few as the {{R1.replication.expected_design.observed}} observed in a share {{R1.replication.expected_design.published.at_most_observed}}; with their survey weights and design, {{R1.replication.expected_weighted.published.mean}}, and as few as the {{R1.replication.expected_weighted.observed}} observed in a share {{R1.replication.expected_weighted.published.at_most_observed}}. If every association were true at a common fraction of its published size, the counts observed are those expected at {{R1.replication.expected.implied_scale}}, {{R1.replication.expected_design.implied_scale}}, and {{R1.replication.expected_weighted.implied_scale}} times it, and the fractions whose draws' central range holds them run from {{R1.replication.expected.implied_scale_range.0}} to {{R1.replication.expected.implied_scale_range.1}}, {{R1.replication.expected_design.implied_scale_range.0}} to {{R1.replication.expected_design.implied_scale_range.1}}, and {{R1.replication.expected_weighted.implied_scale_range.0}} to {{R1.replication.expected_weighted.implied_scale_range.1}}.

## Methods

**Design.** A replication of published estimates in a later, independent national sample, analyzed as the papers analyzed theirs. The plan, its eligibility rules, the extraction of every sampled paper, the analysis code, and the coding check on the papers' own cycles were fixed and [registered](prereg:66c14e8df19aafb51c6890eb73fc46addc6eb292998aa04c081f412f2df6e299) on 2026-10-06, before any file of NHANES August 2021–August 2023 was downloaded. The registered files are in [plan/](plan/plan.md), unchanged; the plan says what was settled when, and `deviations.json` lists what was done after registration. The second version corrected the first in answer to its reviews, adding the classification of each departure's evidence, the tests on finite degrees of freedom, and power at the Bonferroni level. This third version answers the second's reviews: it reports the weighted fits the plan registered for the analyses re-run unweighted, which the earlier versions computed but did not report, and adds the design-based variance of their unweighted estimates, the phlebotomy weight for the albumin analysis, power at half the published effect, and each analysis as its paper describes it. Apart from the registered weighted fits, everything either version added was added after the results were known. To judge which variables 2021–2023 has, we read only its documentation, whose codebook pages show each variable's marginal counts but no association between variables.

**Sample.** The frame is Table A of S1 Data of [Suchak et al. (2025)](doi:10.1371/journal.pbio.3003152): 341 papers, published from 2014 to 2024, each relating one NHANES predictor to one health condition (`data/frame.csv`). Its rows were shuffled once in R 4.6.1 with `set.seed(20261006, kind = "Mersenne-Twister", normal.kind = "Inversion", sample.kind = "Rejection")` (`code/sample.R`, which writes `results/order.csv`), and papers were judged in that order until 40 were eligible. A paper was eligible if it was not retracted (E0), its full text was in Europe PMC (E1), it analyzed continuous NHANES (E2), its headline association was cross-sectional (E3), the headline was a significant whole-population estimate from a generalized linear model with a 95% confidence interval (E4), and its exposure, outcome, and defining population could be built from 2021–2023's public files with the same measurement or question (E5). The headline is the first significant estimate the abstract reports for the paper's predictor and condition, its most-adjusted model, and for ordered categories its highest against its lowest; [the plan](plan/plan.md) gives the rules in full. Judging ran through rank 206: 2 papers were retracted, 53 had no full text in Europe PMC, 3 did not analyze continuous NHANES, 4 had no cross-sectional headline, 8 had no qualifying headline, and 96 could not be built from 2021–2023 (85 judged from Table A's labels and abstract, 11 from the full text). Each decision is in [plan/eligibility.csv](plan/eligibility.csv).

The 40 sampled associations, by Table A's labels, are: serum vitamin D concentrations and osteoarthritis ([Yu et al. (2023)](doi:10.3389/fnut.2023.1016809)); systemic immune-inflammation index and stroke ([Liu et al. (2024a)](doi:10.31083/J.RCM2504130)); dietary inflammatory index and stroke ([Mao et al. (2024)](doi:10.1186/s12889-023-17556-w)); the non-HDL to HDL cholesterol ratio and gallstones ([Cheng et al. (2024a)](doi:10.1186/s12944-024-02262-2)); blood pressure and depression ([Zhang et al. (2024a)](doi:10.3389/fpsyt.2024.1433990)); non-HDL cholesterol and depression ([Zhu et al. (2023)](doi:10.3389/fpsyt.2023.1274648)); dietary inflammatory index and hyperuricemia ([Wang et al. (2023)](doi:10.3389/fnut.2023.1218166)); caffeine intake and obesity ([Liu and Cui (2024)](doi:10.1371/journal.pone.0300566)); sleep health and blood pressure ([Su et al. (2022)](doi:10.1038/s41598-022-05124-y)); red blood cell distribution width and coronary heart disease ([Zhang et al. (2024b)](doi:10.1097/MD.0000000000037315)); heavy metals exposure and metabolic-associated fatty liver conditions ([Tang et al. (2024)](doi:10.3389/fpubh.2024.1280163)); serum ferritin levels and the metabolic score for insulin resistance ([Hao et al. (2022)](doi:10.3389/fmed.2022.925344)); weight-adjusted-waist index and metabolic-associated fatty liver conditions ([Hu et al. (2023)](doi:10.1186/s40001-023-01205-4)); blood manganese and liver stiffness ([Han et al. (2023)](doi:10.1186/s40001-022-00977-5)); the neutrophil to HDL cholesterol ratio and metabolic-associated fatty liver conditions ([Lu et al. (2024)](doi:10.1186/s12876-024-03394-6)); blood cadmium levels and depression ([Ji and Wang (2024)](doi:10.1265/ehpm.24-00050)); usual source of care and blood pressure ([Dinkler et al. (2016)](doi:10.1093/ajh/hpw010)); visceral adiposity index and chronic kidney disease ([Peng et al. (2023)](doi:10.1016/j.pmedr.2023.102306)); dietary fiber intake and stroke ([Dong and Yang (2022)](doi:10.3389/fnut.2022.936926)); sleep health and visceral adiposity index ([Liu et al. (2024b)](doi:10.1136/bmjopen-2023-082601)); ovariectomy-reduced hormones and depression ([Chen et al. (2020)](doi:10.1186/s12991-020-00315-1)); serum albumin and depression ([Zhang et al. (2023)](doi:10.1186/s12888-023-04935-1)); serum uric acid levels and creatine phosphokinase ([Chen et al. (2023)](doi:10.1186/s12872-023-03333-5)); weight-adjusted-waist index and urinary incontinence ([Sun et al. (2024)](doi:10.1038/s41598-024-51216-2)); body shape index and prostate cancer ([Liu et al. (2024c)](doi:10.1007/s11255-023-03917-2)); sedentary behavior and urinary incontinence ([Di et al. (2024)](doi:10.1016/j.heliyon.2024.e27764)); triglyceride glucose body mass index and urinary incontinence ([Li et al. (2024)](doi:10.1186/s12944-024-02306-7)); dietary total energy intake and asthma ([Cao et al. (2024)](doi:10.1186/s40795-024-00938-7)); composite dietary antioxidant index and hyperlipidemia ([Zhao et al. (2024)](doi:10.1038/s41598-024-66922-0)); dietary zinc intake and asthma ([Cheng et al. (2024b)](doi:10.1016/j.waojou.2024.100900)); medical uninsurance and sociodemographic attributes ([Wahab et al. (2022)](doi:10.1097/MD.0000000000030539)); sleep health and albumin ([Li and Guo (2022)](doi:10.1186/s12889-022-13524-y)); visceral fat metabolic score and osteoarthritis ([Xue et al. (2024)](doi:10.1186/s12889-024-19722-0)); waist circumference and sex steroid hormones ([Zhu et al. (2024)](doi:10.1155/2024/4306797)); systemic immune-inflammation index and diabetes ([Nie et al. (2023)](doi:10.3389/fendo.2023.1245199)); systemic immune-inflammation index and hyperlipidemia ([Mahemuti et al. (2023)](doi:10.3390/nu15051177)); triglyceride glucose index and depression ([Ren et al. (2024)](doi:10.1097/MD.0000000000039258)); selenium levels and chronic kidney disease ([Pi et al. (2024)](doi:10.3389/fnut.2024.1396470)); composite dietary antioxidant index and cardiovascular disease ([Liu et al. (2023)](doi:10.3390/antiox12091740)); and systemic immune biomarkers and metabolic-associated fatty liver conditions ([Wang et al. (2024)](doi:10.3389/fnut.2024.1415484)). Each paper's exact exposure contrast, outcome, population, and cycles are in [results/associations.csv](results/associations.csv) and [plan/associations.json](plan/associations.json).

**Re-implementation.** Each association is one file, `code/associations/rowNNN.R`, built on a shared library (`code/lib/`): it builds the exposure, outcome, covariates, and study population from each cycle's NHANES files, lists every alternative tried on the paper's own cycles as a variant, and records every choice the paper left open with its reason. Models are fitted with the survey package ([Lumley (2004)](doi:10.18637/jss.v009.i08)) in R 4.6.1: weighted analyses by `svyglm` with Taylor-linearized variance over the masked strata and primary sampling units (PSUs), the study population as a domain, pooled weights scaled by each cycle's share of the years, and t tests on the design's degrees of freedom (PSUs minus strata); unweighted analyses by `glm` with normal intervals, which treat the participants as a simple random sample. Ratio measures are analyzed on the log scale, and every interval is a 95% confidence interval. Unstated choices were settled by, in order, what the paper says elsewhere, the analytic guidelines of the National Center for Health Statistics (NCHS), which runs the survey, the convention of this literature, and, among choices still plausible, closeness to the published estimate. The estimated glomerular filtration rate (eGFR) uses serum creatinine calibrated as NCHS directs: for 1999–2000 as [Selvin et al. (2007)](doi:10.1053/j.ajkd.2007.08.020) derived, and for 2005–2006 by the equation in that cycle's laboratory documentation; the dietary inflammatory index uses its published global means and weights ([Shivappa et al. (2014)](doi:10.1017/S1368980013002115)); the composite dietary antioxidant index is the sum of six standardized intakes, as [Liu et al. (2023)](doi:10.3390/antiox12091740) define it, modifying the index of [Wright et al. (2004)](doi:10.1093/aje/kwh173), which summed principal component scores.

What the replication tests is the published estimate, so the re-implementation follows the computation that produced it wherever the paper's own numbers reveal it, even where that departs from the paper's text: an unweighted analysis in a paper that says it weighted, a sample that complete-case coding restricted, a covariate the text names but the model left out. Two exceptions correct the computation instead: participants counted twice, by stacking the 2017–2018 files with the 2017–March 2020 files that contain them, are counted once, and an outcome coded in reverse is coded as the paper states it. Every such finding is recorded in the association's file as a departure, with its kind (the coding of a variable, the sample, the numbers reported, the survey weights or design, or the model), whether it bears on the headline estimate, and whether the re-implementation follows it; `code/run` counts the papers with each kind bearing on the headline. After registration, in answer to review, each departure bearing on the headline was also classified by how it is established: shown by the paper itself, where two of its parts disagree or arithmetic on its own printed numbers rules out what it describes; shown by recomputation, where the computation described does not reproduce the published numbers from the public files and another, identified, does; or unresolved, where recomputation rules out the computation described without identifying the one used. Under a written codebook, the lead agent and a second agent, blind to the first's labels, each coded every such departure from its recorded detail and its association's file, without returning to the papers; `data/departure_coding.csv` holds both codings, the lead agent settled their disagreements by rereading the details under the codebook, and the settled label is the departure's `evidence` in its file. In this version, {{R1.reproduction.departures.validation.drawn}} of these departures, drawn at random, were each checked against the paper's full text by an agent that recorded none of them, for whether the record describes the paper correctly (`data/departure_validation.csv`); what recomputation gives was not part of that check. One departure recorded as a model's in the registered files, where one section of a paper lists covariates that its model, as another section and its table give it, leaves out, is counted as a reporting departure here, since the paper describes its model both ways. Multiple imputation, whose draws cannot be reproduced, is replaced by complete cases.

**The analysis as described.** In answer to the second version's reviews, each analysis was also fitted on its paper's cycles as the paper describes it: every departure bearing on the headline that the re-implementation follows and that is a computation (coding, sample, weighting, or model) put back as the text describes it, all together. A reporting departure records two of a paper's accounts disagreeing or a number mislabeled, not a computation, so it stays as it is: the caffeine paper's estimate stays per milligram. The text is read literally; where two readings remain, the one that changes the followed computation least is taken, never one chosen by its estimate; a version the public files cannot build is left as followed and recorded. These rules were written before any described version but row220's, the worked example, was fitted, though the coding check had already fitted the text's version of many single departures as variants. Four agents wrote the described versions, two more checked every specification blind to its results, and the lead agent settled the checkers' five disagreements; four points were settled after the first fits, three after a mock review, among them that a departure where one of the paper's own accounts, or its own steps, gives the computation followed is a reporting departure, as row311's became in the second version. [code/described_rules.md](code/described_rules.md) records the rules and when each was set, [code/described_checks.md](code/described_checks.md) the checkers' reports, and each association file's `described` field what it puts back. A described version is judged by whether its estimate falls outside the published 95% confidence interval, that is, whether the analysis as described fails the registered reproduction criterion, and, among the analyses it changes, by whether it lacks a significant estimate with the published sign, beside the count for the followed computation; an estimate within 1% of the followed one on the analysis scale is called nearly unchanged. A fit that does not converge, or gives no finite estimate, counts as failing both; none does.

**2021–2023.** The same code runs on the new cycle with rules fixed before registration. Each association runs in its harmonized version, which leaves out the covariates and exclusion steps 2021–2023 cannot build (`left_out` in each file) and is fitted on the paper's cycles too, so what leaving them out changes is measured. Cutpoints, standardization constants, and knots keep the values the paper used or, where it gave none, those computed on its cycles. The weight is the paper's, except where a component moved to a smaller subsample in this cycle and NCHS directs its weight: the examination weight becomes the phlebotomy weight whenever a blood analyte is in the model, and the examination or interview weight becomes the first dietary recall's weight whenever the model uses the 30-day supplement questionnaire. One paper analyzed a blood analyte, serum albumin, on the interview weight, which this rule keeps. The cycle is analyzed alone, with 15 design degrees of freedom. As the plan registered, an analysis re-run unweighted is also fitted with the survey weights and design. A covariate with one value in the 2021–2023 sample would have been left out, and a model that could not be fitted would have counted as not replicated; neither happened.

**Outcomes.** An association **replicated** if its 2021–2023 estimate has the published sign and its p-value, corrected over the 40 associations by Benjamini and Hochberg's procedure ([Benjamini and Hochberg (1995)](doi:10.1111/j.2517-6161.1995.tb02031.x)), is below 0.05; the share replicated has an exact (Clopper-Pearson) interval. A test is **informative** if it had at least 0.80 power (two-sided, α = 0.05, noncentral t on the design's degrees of freedom) to detect the published effect given its 2021–2023 standard error. The **effect ratio** is the 2021–2023 estimate over the published one on the analysis scale, summarized by its median with a distribution-free interval. Each 2021–2023 estimate is tested against the published one (z on the analysis scale, the published standard error taken from its interval, corrected as above), and the effect ratio of each association is split into three factors: our estimate on the paper's cycles over the published one (what reproducing the paper gives), the harmonized estimate over ours (what leaving out what 2021–2023 lacks changes), and the 2021–2023 estimate over the harmonized one (what changed between cycles, tested by a second corrected z test). After registration, both tests were repeated with t distributions on the design degrees of freedom of the fits compared, and the tests' power was also computed at the Bonferroni level, 0.05/40: Benjamini and Hochberg's procedure rejects every p-value below that level and none above 0.05, so the power of the replication decision for a test lies between its power at those two levels. Power, at either level, takes the published effect as true; since effects published for crossing a significance threshold are expected to be inflated, this version also gives each test's power to detect half the published effect. To ask how many replications the tests could have found, the registered tests were simulated as if every association were true, its effect a given fraction of the published one, the same for all: each test's statistic drawn from its own 2021–2023 standard error and degrees of freedom, the tests independent, and each draw decided as registered, the published sign and Benjamini and Hochberg's correction over all 40, in 10,000 draws with a fixed seed (added in this version). This gives the count expected at the published size and at half of it, with the central range holding 95% of draws (the draws' central range below), the chance of as few replications as observed, and the fraction of the published size at which the count observed is the one expected, with the fractions, in steps of 0.05, whose central range holds it. The same simulation is repeated with the design-respecting fits of the analyses re-run unweighted in place of their registered ones, against the counts those fits give. Sensitivity analyses refit some associations in 2021–2023 and count replications with the refits in place of their registered fits, corrected over all 40 with the other associations' registered fits: each paper's own weight in place of the subsample weights NCHS directs (registered); the survey weights and design for the analyses re-run unweighted (registered); their unweighted estimates with variance from the design, every participant weighing the same within the strata and PSUs, with t tests on the design's degrees of freedom (added in this version), which keeps the papers' estimates and changes their standard errors, up for some and down for others, and the distribution their tests are referred to; and the phlebotomy weight for the albumin analysis (added in this version). Intervals for shares and medians treat the associations as independent. All p-values are two-sided. In Table 1 and claims C10 to C12, the sleep estimate compares workday sleep of 5 hours or less with more than 7 and up to 8 hours, the diabetes estimate is per 100 units of the systemic immune-inflammation index, and the depression estimate is per unit of the triglyceride-glucose index.

**Data and code.** The bundle carries the 412 NHANES public-use files the analysis and the plan's dry run read, as CDC published them, compressed with xz, each checked against its recorded SHA-256 before it is read (`data/nhanes/sources.csv`; `code/fetch_data.R --check` compares them with CDC's current files). `code/run` re-runs everything from them in the image `env/Dockerfile` builds, in about 7 minutes on one core, longer on a busy machine, and writes [results/R1.json](results/R1.json) (the aggregates), [results/associations.json](results/associations.json) (every fit and variant), and, through `code/report.R`, [results/associations.csv](results/associations.csv) and `results/R2.json` (the per-association values cited here).

**Ethics.** NHANES is conducted by NCHS under protocols approved by its Research Ethics Review Board (#98-12, #2005-06, #2011-17, #2018-01, and #2021-05), with written informed consent. This study uses only the de-identified public-use files and attempts to identify no one. It reports what each published analysis computed, as the papers' own numbers show it, and makes no claim about how or why any paper was produced.

## Results

**The coding check.** On the cycles each paper analyzed, our code reproduced {{R1.reproduction.reproduced.k}} of the {{R1.reproduction.associations}} published estimates, its estimate falling inside the published interval, and our estimates were a median {{R1.reproduction.ratio_original.median}} times the published ones ({{R1.reproduction.ratio_original.ci.0}} to {{R1.reproduction.ratio_original.ci.1}}). Because open choices were settled partly by closeness to the published estimate (Methods), a reproduction shows that a computation consistent with the published numbers exists, not that it is the one the authors ran. In our own fits on those cycles, {{R1.reproduction.same_sign_p05.k}} estimates had the published sign and an unadjusted p-value below the threshold, fewer than published, partly because our intervals are design-based where several papers' were not and some of our complete-case samples are smaller. The estimate not reproduced is the association of dietary zinc with asthma ([Cheng et al. (2024b)](doi:10.1016/j.waojou.2024.100900)), whose estimates are the odds of having no asthma, as the paper's own tables show: our model of asthma, as the paper defines its outcome, gives {{R2.row257.original.estimate}} against the published {{R2.row257.published.estimate}}, and the reversed outcome gives {{R2.row257.variants.0.estimate}}.

**What the published analyses computed.** In {{R1.reproduction.departures.affecting_headline.k}} of the papers, the investigators recorded at least one departure bearing on the headline estimate, {{R1.reproduction.departures.headline_departures}} in all. Of these departures, {{R1.reproduction.departures.by_evidence.paper.k}} are shown by the paper itself, {{R1.reproduction.departures.by_evidence.data.k}} by recomputation that identifies another computation, and {{R1.reproduction.departures.by_evidence.unresolved.k}} are unresolved; the two independent codings agreed on {{R1.reproduction.departures.coding.agree.k}} of them (Cohen's kappa {{R1.reproduction.departures.coding.kappa}}), and {{R1.reproduction.departures.coding.settled}} disagreements were settled. Of {{R1.reproduction.departures.validation.drawn}} departures drawn at random and checked against the papers' full texts, {{R1.reproduction.departures.validation.confirmed.k}} describe their paper correctly and {{R1.reproduction.departures.validation.partly.k}} misstate something about it, both since corrected, though the departure stands; {{R1.reproduction.departures.validation.not_confirmed.k}} misdescribe it (an exact upper bound of {{R1.reproduction.departures.validation.not_confirmed.ci.1}} for that share among all the records), though records misstating something are not rare (an exact interval of {{R1.reproduction.departures.validation.misstating.ci.0}} to {{R1.reproduction.departures.validation.misstating.ci.1}}), and any among the rest are not corrected. The check covered what each record says of its paper, not what recomputation gives, which for most of the sample is the departure's substance. By paper, {{R1.reproduction.departures.shown_by_paper.k}} papers have a departure the paper itself shows, {{R1.reproduction.departures.shown_by_paper_computation.k}} of them one of computation and the rest one only of a number or description reported, {{R1.reproduction.departures.identified_by_data.k}} more have one that only recomputation shows, and {{R1.reproduction.departures.only_unresolved.k}} have only unresolved ones. By kind, as the investigators classified them, {{R1.reproduction.departures.by_kind.coding.k}} papers coded a variable otherwise than described, {{R1.reproduction.departures.by_kind.sample.k}} analyzed another sample, {{R1.reproduction.departures.by_kind.reporting.k}} reported a number or a description that disagrees with another part of the paper, {{R1.reproduction.departures.by_kind.weighting.k}} did not use the survey weights or design they describe, and {{R1.reproduction.departures.by_kind.model.k}} fitted a model other than the one stated. In {{R1.reproduction.departures.not_followed.k}} papers our analysis does not follow a departure, because it corrects it or because the paper's numbers rule out the stated computation without revealing the one used.

Some change what an estimate means. The caffeine paper's headline, labeled the odds ratio for the highest against the lowest quartile, is an odds ratio per milligram within the highest quartile, as every cell of its table is and as its own obesity rates by quartile show ([Liu and Cui (2024)](doi:10.1371/journal.pone.0300566)); the contrast its text describes gives {{R2.row066.variants.5.estimate}}. The selenium paper's models are reproduced only without age, which its text lists as a covariate ([Pi et al. (2024)](doi:10.3389/fnut.2024.1396470)); with age its estimate is {{R2.row315.variants.3.estimate}} against the published {{R2.row315.published.estimate}}. The stroke paper on the systemic immune-inflammation index is reproduced only with the participants of 2017–2018 counted twice ([Liu et al. (2024a)](doi:10.31083/J.RCM2504130)). Two published intervals have about the width of fits that ignore the survey design, narrower than a design-based analysis of the same data gives: the published interval is {{R2.row101.published.interval_ratio}} times the width of ours for blood manganese and liver stiffness ([Han et al. (2023)](doi:10.1186/s40001-022-00977-5)), and {{R2.row284.published.interval_ratio}} times for workday sleep and albumin ([Li and Guo (2022)](doi:10.1186/s12889-022-13524-y)). The gallstones paper's interval is {{R2.row040.published.interval_ratio}} times ours for another reason: its model, as the paper lists it, holds total and HDL cholesterol beside the log of their ratio, which they nearly determine, and its own interval shows it was not fitted that way, though which model was fitted is unresolved ([Cheng et al. (2024a)](doi:10.1186/s12944-024-02262-2)). Many other departures concern how a covariate was coded or which participants entered. Each departure is described, with its evidence and how it is established, in its association's file under `code/associations/`, and counted in [results/associations.csv](results/associations.csv).

**The analyses as described.** Putting the departures back as described changes the specification of {{R1.reproduction.described.changed}} analyses ({{R1.reproduction.described.not_applied}} departures could not be built and stay as followed). Among them, the described estimate is a median {{R1.reproduction.described.to_followed_changed.median}} times the followed one, nearly unchanged for {{R1.reproduction.described.within_one_percent}}, and {{R1.reproduction.described.outside_published_ci_changed.k}} fall outside the published interval. Three use the survey weights their text names: the weight-adjusted-waist index and fatty liver gives {{R2.row100.described.estimate}} ({{R2.row100.described.low}} to {{R2.row100.described.high}}) against the published {{R2.row100.published.estimate}} ([Hu et al. (2023)](doi:10.1186/s40001-023-01205-4)), the immune-inflammation index and hyperlipidemia {{R2.row306.described.estimate}} against {{R2.row306.published.estimate}} ([Mahemuti et al. (2023)](doi:10.3390/nu15051177)), and the same index and diabetes {{R2.row303.described.estimate}} against {{R2.row303.published.estimate}}, just below the published interval though its own ({{R2.row303.described.low}} to {{R2.row303.described.high}}) includes it. The ferritin paper's, with, among other things, the covariates its text lists, which include the components of its outcome, gives {{R2.row096.described.estimate}} against {{R2.row096.published.estimate}} ([Hao et al. (2022)](doi:10.3389/fmed.2022.925344)), and the antioxidant paper's, on the sample its text describes, {{R2.row319.described.estimate}} against {{R2.row319.published.estimate}} ([Liu et al. (2023)](doi:10.3390/antiox12091740)). With the zinc paper, whose followed version already codes its outcome as stated, {{R1.reproduction.described.outside_published_ci.k}} of the {{R1.reproduction.associations}} analyses as described fall outside the published interval. Significance moves more often: {{R1.reproduction.described.not_significant_changed.k}} of the {{R1.reproduction.described.changed}} lack a significant estimate with the published sign as described, against {{R1.reproduction.described.not_significant_changed_followed.k}} as followed, some because the weights or design the text names widen the interval and others because the estimate moves.

**Replication.** In 2021–2023, {{R1.replication.replicated.k}} of the {{R1.replication.associations}} associations replicated (a share of {{R1.replication.replicated.share}}, {{R1.replication.replicated.ci.0}} to {{R1.replication.replicated.ci.1}}). One survey cycle holds fewer participants than the several most papers pooled, so only {{R1.replication.informative}} tests had the planned power, at the nominal level, to detect the published effect; of these, {{R1.replication.replicated_informative.k}} replicated ({{R1.replication.replicated_informative.share}}, {{R1.replication.replicated_informative.ci.0}} to {{R1.replication.replicated_informative.ci.1}}). Power at the nominal level bounds the power of the corrected decision from above; at the Bonferroni level, which bounds it from below, {{R1.replication.informative_bonferroni}} tests had the planned power, and {{R1.replication.replicated_informative_bonferroni.k}} of them replicated. Power here takes the published effect as true; against half of it, only {{R1.replication.informative_half}} tests had the planned power, and {{R1.replication.replicated_informative_half.k}} of them replicated. The other tests had less power, so their failures to replicate are weak evidence either way. Even if every association were true at its published size, the tests would replicate {{R1.replication.expected.published.mean}} on average (Table 2), and as few as observed in a share {{R1.replication.expected.published.at_most_observed}} of draws, or {{R1.replication.expected_design.published.at_most_observed}} and {{R1.replication.expected_weighted.published.at_most_observed}} for the two counts with the survey design; at a common fraction of the published size, the counts observed are those expected at {{R1.replication.expected.implied_scale}}, {{R1.replication.expected_design.implied_scale}}, and {{R1.replication.expected_weighted.implied_scale}} times it (C15). The same counts could come from some associations being false and the others near their published size. Effects in 2021–2023 were a median {{R1.replication.ratio.median}} times the published ones ({{R1.replication.ratio.ci.0}} to {{R1.replication.ratio.ci.1}}; quartiles {{R1.replication.ratio.quartiles.0}} and {{R1.replication.ratio.quartiles.1}}), and {{R1.replication.ratio_informative.median}} times ({{R1.replication.ratio_informative.ci.0}} to {{R1.replication.ratio_informative.ci.1}}) among the informative tests. The median is {{R1.replication.ratio_unweighted.median}} over the analyses re-run unweighted and {{R1.replication.ratio_others.median}} over the others, and with the registered weighted fits in place of the unweighted ones it is {{R1.replication.ratio_weighted.median}} ({{R1.replication.ratio_weighted.ci.0}} to {{R1.replication.ratio_weighted.ci.1}}). {{R1.replication.in_published_ci.k}} estimates fell inside the published confidence interval, {{R1.replication.same_sign_p05.k}} had the published sign with an unadjusted p-value below the threshold, and {{R1.replication.reversed.k}} had the opposite sign with a significant corrected p-value. The count is the same among the reproduced associations ({{R1.replication.replicated_reproduced.k}} of {{R1.replication.replicated_reproduced.n}} replicated) and with each paper's own weight in place of the subsample weights NCHS directs for 2021–2023 ({{R1.replication.replicated_paper_weight.k}}). Every model fitted in 2021–2023, with no covariate left out for want of variation.

**The survey design.** Of the analyses, {{R1.replication.unweighted.k}} were re-run unweighted, as their papers' numbers show they were computed, and {{R1.replication.replicated_unweighted.k}} of the {{R1.replication.replicated.k}} replications are among them. Their tests treat the participants as a simple random sample, while NHANES samples clusters within strata. Taking their variance from the design changes their standard errors by a median factor of {{R1.replication.design.design.se_ratio.median}} ({{R1.replication.design.design.se_ratio.range.0}} to {{R1.replication.design.design.se_ratio.range.1}}), smaller for {{R1.replication.design.design.se_ratio.smaller}} of them, and refers their tests to a t distribution on the design's degrees of freedom; then {{R1.replication.design.design.replicated_unweighted.k}} of them replicate. Fitted with the survey weights and design, as the plan also registered, which changes what they estimate as well, {{R1.replication.design.weighted.replicated_unweighted.k}} do. Table 2 gives the counts over all {{R1.replication.associations}}, each corrected with the other associations' registered fits; the design-variance set of informative tests is not the registered one ({{R1.replication.design.design.informative_also_registered}} of its {{R1.replication.design.design.informative}} are in both). The first two versions of this study computed the weighted fits but reported neither them nor this count. Each association's refits are in `results/R2.json`.

**Table 2.** Replications of the {{R1.replication.associations}} associations in 2021–2023, by how the analyses that ignore the survey design are tested, and the count the registered tests would give if every association were true.

| Analysis | Replicated | Share (confidence interval) | Informative tests | Replicated among them |
|---|---|---|---|---|
| Each analysis as its paper computed it (registered) | {{R1.replication.replicated.k}} | {{R1.replication.replicated.share}} ({{R1.replication.replicated.ci.0}} to {{R1.replication.replicated.ci.1}}) | {{R1.replication.informative}} | {{R1.replication.replicated_informative.k}} |
| Unweighted estimates, variance from the survey design | {{R1.replication.design.design.replicated.k}} | {{R1.replication.design.design.replicated.share}} ({{R1.replication.design.design.replicated.ci.0}} to {{R1.replication.design.design.replicated.ci.1}}) | {{R1.replication.design.design.informative}} | {{R1.replication.design.design.replicated_informative.k}} |
| Survey weights and design (registered) | {{R1.replication.design.weighted.replicated.k}} | {{R1.replication.design.weighted.replicated.share}} ({{R1.replication.design.weighted.replicated.ci.0}} to {{R1.replication.design.weighted.replicated.ci.1}}) | {{R1.replication.design.weighted.informative}} | {{R1.replication.design.weighted.replicated_informative.k}} |
| Expected if every association were true at its published size (registered tests) | {{R1.replication.expected.published.mean}} ({{R1.replication.expected.published.range.0}} to {{R1.replication.expected.published.range.1}}, the draws' central range) | | | |
| Expected likewise, with the design's variance for the analyses re-run unweighted | {{R1.replication.expected_design.published.mean}} ({{R1.replication.expected_design.published.range.0}} to {{R1.replication.expected_design.published.range.1}}) | | | |
| Expected likewise, with their survey weights and design | {{R1.replication.expected_weighted.published.mean}} ({{R1.replication.expected_weighted.published.range.0}} to {{R1.replication.expected_weighted.published.range.1}}) | | | |
| Expected at half its published size (registered tests) | {{R1.replication.expected.half.mean}} ({{R1.replication.expected.half.range.0}} to {{R1.replication.expected.half.range.1}}) | | | |

**Where 2021–2023 differs from the published estimates.** {{R1.replication.differs_from_published.k}} estimates differ significantly from the published ones after correction (Table 1), and {{R1.replication.differs_from_published_t.k}} with t tests on the fits' design degrees of freedom. These tests take each published standard error from its printed interval, as registered, so a published interval narrower than the data support makes a difference easier to find: the albumin paper's interval is {{R2.row284.published.interval_ratio}} times the width of our design-based one on its own cycles. Against our own harmonized estimates on the papers' cycles, {{R1.replication.heterogeneous.k}} differ significantly ({{R1.replication.heterogeneous_t.k}} with t tests), with a median ratio of {{R1.replication.ratio_own.median}} ({{R1.replication.ratio_own.ci.0}} to {{R1.replication.ratio_own.ci.1}}): a single cycle cannot tell most changes between cycles from sampling error, and finding none significant does not show the estimates equal. The three in Table 1 come closest: they differ from our harmonized estimates on their papers' cycles with corrected p-values of {{R2.row284.heterogeneity_q}} each (unadjusted {{R2.row284.heterogeneity_p}}, {{R2.row303.heterogeneity_p}}, and {{R2.row311.heterogeneity_p}}, the smallest of the {{R1.replication.associations}}).

**Table 1.** The associations whose 2021–2023 estimate differs significantly from the published one: the published estimate, ours on the paper's cycles (its own analysis), the harmonized one there, the 2021–2023 estimate with its confidence interval, the corrected p-value of the difference by z test and by t test, and the three factors of the effect ratio.

| Association | Published | Paper's cycles | Harmonized | 2021–2023 | Corrected p, z | Corrected p, t | Reproduction factor | Harmonization factor | Between cycles |
|---|---|---|---|---|---|---|---|---|---|
| Workday sleep and serum albumin, β (g/L) ([Li and Guo (2022)](doi:10.1186/s12889-022-13524-y)) | {{R2.row284.published.estimate}} | {{R2.row284.original.estimate}} | {{R2.row284.harmonized.estimate}} | {{R2.row284.replication.estimate}} ({{R2.row284.replication.low}} to {{R2.row284.replication.high}}) | {{R2.row284.difference_q}} | {{R2.row284.difference_q_t}} | {{R2.row284.ratio.original}} | {{R2.row284.ratio.harmonized}} | {{R2.row284.ratio.between_cycles}} |
| Immune-inflammation index and diabetes, OR ([Nie et al. (2023)](doi:10.3389/fendo.2023.1245199)) | {{R2.row303.published.estimate}} | {{R2.row303.original.estimate}} | {{R2.row303.harmonized.estimate}} | {{R2.row303.replication.estimate}} ({{R2.row303.replication.low}} to {{R2.row303.replication.high}}) | {{R2.row303.difference_q}} | {{R2.row303.difference_q_t}} | {{R2.row303.ratio.original}} | {{R2.row303.ratio.harmonized}} | {{R2.row303.ratio.between_cycles}} |
| Triglyceride-glucose index and depression, OR ([Ren et al. (2024)](doi:10.1097/MD.0000000000039258)) | {{R2.row311.published.estimate}} | {{R2.row311.original.estimate}} | {{R2.row311.harmonized.estimate}} | {{R2.row311.replication.estimate}} ({{R2.row311.replication.low}} to {{R2.row311.replication.high}}) | {{R2.row311.difference_q}} | {{R2.row311.difference_q_t}} | {{R2.row311.ratio.original}} | {{R2.row311.ratio.harmonized}} | {{R2.row311.ratio.between_cycles}} |

In all three, our analysis meets the reproduction criterion on the paper's own cycles, the diabetes estimate at {{R2.row303.ratio.original}} times the published one on the analysis scale, and most of each gap lies between cycles, though none of the three between-cycle differences is significant. In the albumin paper short sleepers had lower albumin, and in 2021–2023 the estimate points the same way, closer to zero, with an interval that includes zero. The paper pooled 2015–2016 with 2017–2018, whose analyzer reads albumin lower by NCHS's bridging study, and short sleepers are more common in 2017–2018: with a cycle term its own model gives {{R2.row284.variants.9.estimate}} g/L, and with 2017–2018's albumin put on 2015–2016's scale by NCHS's bridging equation, which NCHS recommends when combining the cycles, {{R2.row284.variants.10.estimate}} g/L ({{R2.row284.variants.10.low}} to {{R2.row284.variants.10.high}}); in 2021–2023, analyzed alone, the estimate is {{R2.row284.replication.estimate}} g/L, or {{R2.row284.replication_phlebotomy.estimate}} g/L with the phlebotomy weight. Its interval has about the width of a fit that ignores the survey design, and its weights are not the ones its text names. The diabetes paper's estimates are reproduced only unweighted, though it says it weighted them, and both our estimates are unweighted too: in 2021–2023 the estimate lies below one, though not significantly so after correction. With the survey weights and design, its own cycles give {{R2.row303.variants.0.estimate}} ({{R2.row303.variants.0.low}} to {{R2.row303.variants.0.high}}), an interval that includes both one and the published estimate, and 2021–2023 gives {{R2.row303.replication_weighted.estimate}} ({{R2.row303.replication_weighted.low}} to {{R2.row303.replication_weighted.high}}), which differs from the published estimate by z (corrected p {{R2.row303.difference_q_weighted}}) but not by t ({{R2.row303.difference_q_t_weighted}}). The depression paper's model, as its table lists it and as reproduced, leaves out glucose and lipids, which another of its sections says were all included, and its 2021–2023 sample is {{R2.row311.replication.n}} people against {{R2.row311.original.n}} on its cycles, so the estimate below one there rests on few cases, and its difference from the published estimate is not significant by the t test (corrected p {{R2.row311.difference_q_t}}). The full table, with every association's estimates, tests, factors, and departures, is [results/associations.csv](results/associations.csv).

## Discussion

Most of the sampled associations could be neither confirmed nor refuted by one survey cycle, so the replication count cannot by itself tell associations that are absent from tests too weak to find them. But even if every association were true at its published size, the tests would replicate about {{R1.replication.expected.published.mean}}, not all {{R1.replication.associations}}, and counts as low as those observed are unlikely under that hypothesis, at about one draw in twenty as registered and one in a hundred with the survey design (C15). If every association were true, they would have to be smaller than published on average: the counts are those expected at {{R1.replication.expected_weighted.implied_scale}} to {{R1.replication.expected.implied_scale}} times the published size; the median ratio of 2021–2023 estimates to published ones, a summary of the same estimates, is {{R1.replication.ratio.median}}, or {{R1.replication.ratio_weighted.median}} with the weighted fits. The same counts could instead come from some associations being false and the others near their published size, and one cycle cannot tell which. Among the {{R1.replication.informative}} tests powered to detect the published effect, {{R1.replication.replicated_informative.k}} replicated, but those tests were chosen by their power against the published effect, taken as true, which favors associations whose effect is large relative to its uncertainty, so their rate need not carry over to the rest.

The count also depends on how the analyses that ignore the survey design are tested. The plan followed each paper's computation so as to test the published estimate, which for {{R1.replication.unweighted.k}} papers meant a test that treats NHANES as a simple random sample. The three counts in Table 2 answer different questions: whether each paper's computation, run as it ran it, gives a significant result in new data; whether it still does when tested as a design-based analysis is; and whether the association holds in the population the survey represents. By either of the last two, the replication rate is lower than the registered count.

Replication projects that collected new data, mostly with more power per test than one survey cycle gives, report rates and shrinkage that can be set beside these. In psychology, 36% of 100 replications were statistically significant, against 97% of the originals, and their effects were half the original ones ([Open Science Collaboration (2015)](doi:10.1126/science.aac4716)). In experimental economics, 11 of 18 replications found a significant effect in the original direction, at 66% of the original effect on average ([Camerer et al. (2016)](doi:10.1126/science.aaf0918)), and for social science experiments published in Nature and Science, 13 of 21 did, at about half ([Camerer et al. (2018)](doi:10.1038/s41562-018-0399-z)). In preclinical cancer biology, the median replication effect of positive effects was 85% smaller than the median original one ([Errington et al. (2021)](doi:10.7554/eLife.71601)). Here, over all {{R1.replication.associations}} associations, {{R1.replication.design.weighted.replicated.share}} to {{R1.replication.replicated.share}} replicated (Table 2), with tests of less power, and effects in 2021–2023 were a median {{R1.replication.ratio.median}} times the published ones ({{R1.replication.ratio.ci.0}} to {{R1.replication.ratio.ci.1}}), an interval that includes both no shrinkage and the halving those projects found. Those projects ran new studies with new teams; this study re-estimated observational associations in a new probability sample of the same population, with the same questions and measurements as far as the cycle allows and with the original analysis, so less could change between original and replication. Effects published because they crossed a significance threshold in underpowered studies are expected to be inflated, and flexible analyses with selective reporting can inflate them further ([Ioannidis (2008)](doi:10.1097/EDE.0b013e31818131e7)).

The coding check separates two things replication projects usually see together: whether a published number can be recomputed, and whether the paper says how. Our code recomputed {{R1.reproduction.reproduced.k}} of the published estimates from the public files, at a median {{R1.reproduction.ratio_original.median}} times the published value, yet in {{R1.reproduction.departures.shown_by_paper.k}} papers the paper itself shows a departure, in {{R1.reproduction.departures.shown_by_paper_computation.k}} of them one of computation and in the rest only a number or description it reports otherwise in another place, and in {{R1.reproduction.departures.identified_by_data.k}} more only another computation reproduced the number. In psychology, [Artner et al. (2021)](doi:10.1037/met0000365) reproduced 163 of 232 key statistical claims from the authors' raw data, 18 of them only by departing from the article's analytical description, often through cumbersome trial and error, and [Hardwicke et al. (2018)](doi:10.1098/rsos.180448) reproduced every target value in 22 of 35 articles with reusable data, 11 of them only with the authors' help, naming unclear analysis specification among the obstacles. NHANES differs in that every reader has the same data. Here each computation was found from the paper and the public files alone, by trying alternatives until one gave the published numbers. Such a match identifies a computation consistent with those numbers, not necessarily the one the authors ran, which is why the departures that the papers themselves show are counted apart; either way, a reader who takes the methods section at face value would not see the difference. Usually it would not matter much: run as described, few analyses fall outside their published interval, and those that do use weights the paper says it used, covariates it says it adjusted for, or a sample it describes. Significance moves more often than the estimate (C14), so the analyses as described would less often have produced the published findings. These comparisons start from computations whose open choices were settled partly by closeness to the published estimate, so any change from them tends to move away from it.

Ignoring a survey's design is a known error in secondary analyses. Among 145 analyses of a federal workforce survey, 55% used the sampling weights and 8% accounted for the complex sample in estimating variances, and ignoring the design can bias estimates and give standard errors that do not reflect the sample ([West et al. (2016)](doi:10.1371/journal.pone.0158120)). The weighting departures recorded here are narrower and harder to see: in {{R1.reproduction.departures.by_kind.weighting.k}} papers the text says the analysis used the weights or the design, and the published numbers are reproduced only without them. Such an analysis also reaches into its replication: tested as the paper computed it, without the design, more of these associations replicate than with it (Table 2).

[Suchak et al. (2025)](doi:10.1371/journal.pbio.3003152) documented these papers and raised two concerns about them: single-factor designs that skip false discovery correction, and selective use of survey cycles or subsets, which they call suggestive of data dredging. What this study adds is a test in data the papers could not have seen and a check of what their analyses computed. A cycle the papers could not have analyzed tests the second concern, where it has the power, since an association produced by choosing a date range or a subset has no reason to reappear in new data. Most of the informative tests found the association again, at a median {{R1.replication.ratio_informative.median}} times its published size, though that share rests on few tests; the other associations remain untested. A replication cannot answer the first concern: an association that reappears says nothing about how many others were tried, or about the factors a single-factor design leaves out. Nor is an estimate fixed by the data alone: [Patel et al. (2015)](doi:10.1016/j.jclinepi.2015.05.029) fitted 8,192 sets of adjustments to the association of each of 417 NHANES variables with mortality, and for 31% of the variables the direction differed between the 1st and 99th percentiles of those analyses. Several departures recorded here, such as a covariate the text lists but the model leaves out, are choices of that kind, made and not reported. Testing many factors at once, correcting for multiple comparisons, and validating findings in other NHANES cohorts, as an environment-wide association study does ([Patel et al. (2010)](doi:10.1371/journal.pone.0010746)), addresses false discovery and selective use of cycles alike. None of this bears on how or why any paper was written, about which this study claims nothing.

The three estimates that differ from their published values each have explanations other than an error in the original paper. In all three, our analysis meets the reproduction criterion on the paper's own cycles, though for the diabetes paper at {{R2.row303.ratio.original}} times the published effect, and leaving out what 2021–2023 lacks changed it little (the second factor in Table 1), so most of each gap lies between cycles, though none of the three between-cycle differences is significant (corrected p {{R2.row284.heterogeneity_q}} each): it may come from sampling error, from a change in the population or in how the cycle measured it, or from an original estimate that was inflated or confounded. For albumin, the laboratory analyzer changed between the two cycles the paper pooled, and short sleep was more common in the second, so the published estimate may mix the association with a measurement change. A term for the cycle shrinks the estimate from {{R2.row284.published.estimate}} to {{R2.row284.variants.9.estimate}} g/L, but it absorbs every difference between the two cycles; NCHS's bridging equation for the analyzers, which NCHS recommends when combining the cycles, adjusts for the difference NCHS estimated, and with it the paper's own model gives {{R2.row284.variants.10.estimate}} g/L, treating the equation as exact; the analyzer change may account for part of the published decrement, though not all of the gap to 2021–2023. For the immune-inflammation index and diabetes, the paper's analysis, and ours in both cycles, is unweighted, so it describes the sample rather than the population; with the survey weights and design, the estimate is no longer significant on the paper's own cycles, though its interval includes the published one, and in 2021–2023 it differs from the published estimate by the z test but not the t test, so the published association may hold for the unweighted sample more than for the population. The depression estimate rests on {{R2.row311.replication.n}} people, and its difference is not significant by the t test, so a later cycle may well find the published sign again. Only further cycles can tell sampling error from real change.

How unusual these departures are is not known: the papers come from a literature already singled out as formulaic, and no comparable audit exists of other NHANES papers or of observational studies generally. The same re-implementation applied to a random sample of other NHANES papers would tell whether these papers reflect this literature or the field; until then, the departures speak to these papers, and the replication rate to these associations.

For anyone who relies on these papers, the results point three ways. Their estimates are the outputs of computations that may differ from the ones described, so anyone who cites, pools, or builds on one should reproduce it from the public files before relying on its methods, which every reader of an NHANES paper can do. Their effects may be inflated, as effects published for crossing a significance threshold are expected to be, but the median ratio's interval here includes no change, so these data give no factor by which to discount them; and those that ignore the survey design describe their sample rather than the population, and fewer of them replicate when the design is used. And an association this cycle could not test is untested rather than refuted: this study neither establishes nor rules out most of them.

For journals and reviewers, the departures in {{R1.reproduction.departures.shown_by_paper.k}} papers could be seen in the paper alone, in two of its accounts that disagree or a number that cannot follow from the computation described, yet all were published in peer-reviewed journals. The zinc paper is one: its Table 1 counts 208 participants with asthma among 1,141 in the lowest quartile of zinc intake and 272 among 1,153 in the highest, a crude odds ratio of asthma of 1.38, while its Table 3 prints the crude odds ratio as 0.72, the reciprocal, beside the counts of those without asthma ([Cheng et al. (2024b)](doi:10.1016/j.waojou.2024.100900)). A published letter re-analyzed the paper with the survey design and still read its estimates as the odds of asthma ([Lin et al. (2025)](doi:10.1016/j.waojou.2025.101044)), so scrutiny after publication can miss what the tables show too. Most of the others needed only the public files, which reviewers have as well. Asking for the analysis code, and checking that the stated weights, design, and sample give the reported numbers, is the closer attention to analytic methods that [West et al. (2016)](doi:10.1371/journal.pone.0158120) call for, and with public data it is a check that can be made before publication rather than after. The re-implementations here were done by AI agents from each paper and the public files, which suggests such checks can be made routine rather than left to a reader's spare time. For anyone pooling cycles, the albumin estimate (C10) shows a hazard: a change of laboratory method between cycles that coincides with a change in the exposure enters the pooled estimate unless the analysis applies NCHS's bridging equations, which here move it from {{R2.row284.published.estimate}} to {{R2.row284.variants.10.estimate}} g/L. And the tests that lacked power stay open: within NHANES, only pooling 2021–2023 with later cycles, once they are released, can give them the power the plan required, and registering those tests before the data exist would keep them as clean as this one.

## Limitations

The main limitation is power: one two-year cycle has a fraction of the participants most papers pooled, so most tests lacked the planned power, and their failure to replicate says little. Power was registered at the nominal level, which overstates the power of the corrected decision; the count at the Bonferroni level bounds it from below. The effect ratios carry the most information, and their intervals are wide. The intervals for shares and medians treat the 40 associations as independent, but several share participants, cycles, exposures, or outcomes, so their errors may be correlated and the intervals are approximate. The tests against published estimates take the published standard error from a rounded interval and treat it as known. The simulation of how many replications the tests could find takes the tests as independent, their standard errors as estimated, and every effect as the same fraction of its published size, and with the weighted fits it takes a published, often unweighted, effect as the population's. The registered replication follows each paper's computation, so the analyses that ignore the survey design are tested as if NHANES were a simple random sample; the count with the survey weights and design was registered, though first reported in this version, while the design-based variance was added, like this version's other sensitivity analyses, after the results were known. The 2021–2023 cycle also differs from its predecessors: response fell, the first dietary recall moved from an in-person interview to the telephone, blood pressure is measured by an oscillometric device, and the cycle came after the COVID-19 pandemic, any of which could change an association; a significant difference between cycles is therefore not by itself evidence that a published estimate was wrong. The harmonized versions leave out what 2021–2023 cannot build, which changed some estimates on the original cycles (the harmonization factor in `results/associations.csv`).

Following each paper's computation, rather than its text, was a judgment made from its own numbers, and open choices were settled partly by closeness to the published estimate, so a reproduced estimate is compatible with the published one rather than proof of the authors' computation; settling them before 2021–2023 was seen keeps that choice from favoring the replication. The departures recorded are those the coding check found while reproducing the headline estimate; papers' other tables were checked unevenly, so departures that do not bear on the headline are listed but not counted. Departures identified only by recomputation rest on a reconstruction matching the published numbers. How far they move the estimates was measured for each analysis as a whole, after the results were known, from the lead agent's records of what each text says, read under written rules by agents of the same model family and checked by two more blind to the results; four of those rules were settled after the first fits, three after a mock review; ties go to the reading that changes the followed computation less, which leans the described versions toward the published estimate; and a different reading of an ambiguous text could move an estimate across its interval. Reporting departures are left out of it, so it understates how far some descriptions mislead (the caffeine paper's per-milligram estimate, labeled as a contrast between quartiles, is one), and where the Methods misprint a formula, which reading counts as described is a judgment the rules settle by the smaller change. Their kinds are the investigators' classification, and how each is established was classified after registration, in answer to review, by the lead agent and a second agent of the same model family working blind to the first, not by an outside adjudicator. A third agent of the same family checked {{R1.reproduction.departures.validation.drawn}} records, drawn at random, against the papers, for what each says of its paper rather than what recomputation gives, and found {{R1.reproduction.departures.validation.confirmed.k}} correct and {{R1.reproduction.departures.validation.partly.k}} misstating a detail, so a record that misdescribes its paper is uncommon, but one that misstates something is not, and the others were not checked. Where a paper's computation could not be identified, as for the sample of the gallstones paper or the antioxidant index of the cardiovascular paper, our estimate may differ from what its authors computed for reasons we cannot see. Multiple imputation was replaced by complete cases. The sample covers papers eligible under the plan's rules, and excludes papers without full text in Europe PMC (53 of the 206 judged), which may favor open-access journals; 85 papers were judged ineligible from Table A's labels and abstract alone. Extraction, coding, and the classification of departures were done by AI agents of one model family, reviewed by its lead agent but not by people, and every judgment is recorded, with its reason, in the files a reader can check. The study tests associations, not causal effects, and none of the associations, replicated or not, should be read as causal.

## Provenance

The study was planned, run, and written by AI agents of the Claude family (claude-opus-5-5). A lead agent searched prior work, wrote the plan, the eligibility and headline rules, and the shared R library, coordinated agents working under written briefs who screened papers, extracted each paper with verbatim quotes, coded each association, and recorded its departures, reviewed and corrected their work, registered the plan, ran the analysis, and wrote the claims and this paper. For the second version, the lead agent took up the first version's reviews, a second agent coded how each departure is established without seeing the lead agent's coding, and another agent searched and checked the earlier work the Discussion cites. The first version is `sha256:cb94970fd5f3f4762171afbe3674a7ec107c2a503dfc5c16e0c1bdc3274d608f` and the second `sha256:93dfe41114a7c06e6dbbbfae2aac2152fd937279b8d13275a8e834e078dc8c6a`. For this third version, the lead agent took up the second version's reviews, checked each finding against the bundle's files, and added the analyses and changes they called for; four agents wrote each analysis as its paper describes it under the lead agent's written rules, two more checked them blind to their results, and another checked a random sample of the departure records against the papers. Before it was submitted, the person had an agent of another model family, Grok, review a draft of this version as a reviewer would; the lead agent checked each of its findings against the bundle's files and took up those that held, as `deviations.json` records. The tools were R 4.6.1 with the survey, jsonlite, and foreign packages in a pinned Docker image, Python's standard library for the plan's tables, and Europe PMC's and Crossref's public APIs. The data are the NHANES public-use files, Table A of S1 Data of [Suchak et al. (2025)](doi:10.1371/journal.pbio.3003152) under CC BY 4.0, and the sampled papers' full texts, which the bundle does not carry. A person asked for studies outside mathematics; the lead agent proposed this one among others, and the person chose it. The person saw a summary of the plan before registration and approved it unchanged, asked that the report highlight where the 2021–2023 results differ from the published ones and what those analyses computed, and, on reading a summary of the findings, said the bottom line read too softly, after which the agent rewrote the first version's title and Summary. After the first version's reviews, the person asked whether its title was accurate and, when the agent judged that it overstated the findings, asked for a correction of the title and of what the reviews found, the second version. After the second version's reviews, the person asked whether the agent agreed with them and whether a new version was worth making; the agent judged that the methods review's main finding held and recommended one, and the person asked for this third version. Every other proposal, from the plan's rules to the record of departures and each version's changes, was the agents', and the person accepted them. No person reviewed the extractions, the code, the results, or this paper. `provenance.json` says the same.