The paper

Methods

Design. A replication of published estimates in a later, independent national sample, analyzed as the papers analyzed theirs. The plan, its eligibility rules, the extraction of every sampled paper, the analysis code, and the coding check on the papers' own cycles were fixed and registered on 2026-10-06, before any file of NHANES August 2021–August 2023 was downloaded. The registered files are in plan/, unchanged; the plan says what was settled when, and deviations.json lists what was done after registration. To judge which variables 2021–2023 has, we read only its documentation, whose codebook pages show each variable's marginal counts but no association between variables.

Sample. The frame is Table A of S1 Data of Suchak et al. (2025): 341 papers, published from 2014 to 2024, each relating one NHANES predictor to one health condition (data/frame.csv). Its rows were shuffled once in R 4.6.1 with set.seed(20261006, kind = "Mersenne-Twister", normal.kind = "Inversion", sample.kind = "Rejection") (code/sample.R, which writes results/order.csv), and papers were judged in that order until 40 were eligible. A paper was eligible if it was not retracted (E0), its full text was in Europe PMC (E1), it analyzed continuous NHANES (E2), its headline association was cross-sectional (E3), the headline was a significant whole-population estimate from a generalized linear model with a 95% confidence interval (E4), and its exposure, outcome, and defining population could be built from 2021–2023's public files with the same measurement or question (E5). The headline is the first significant estimate the abstract reports for the paper's predictor and condition, its most-adjusted model, and for ordered categories its highest against its lowest; the plan gives the rules in full. Judging ran through rank 206: 2 papers were retracted, 53 had no full text in Europe PMC, 3 did not analyze continuous NHANES, 4 had no cross-sectional headline, 8 had no qualifying headline, and 96 could not be built from 2021–2023 (85 judged from Table A's labels and abstract, 11 from the full text). Each decision is in plan/eligibility.csv.

The 40 sampled associations, by Table A's labels, are: serum vitamin D concentrations and osteoarthritis (Yu et al. (2023)); systemic immune-inflammation index and stroke (Liu et al. (2024a)); dietary inflammatory index and stroke (Mao et al. (2024)); the non-HDL to HDL cholesterol ratio and gallstones (Cheng et al. (2024a)); blood pressure and depression (Zhang et al. (2024a)); non-HDL cholesterol and depression (Zhu et al. (2023)); dietary inflammatory index and hyperuricemia (Wang et al. (2023)); caffeine intake and obesity (Liu and Cui (2024)); sleep health and blood pressure (Su et al. (2022)); red blood cell distribution width and coronary heart disease (Zhang et al. (2024b)); heavy metals exposure and metabolic-associated fatty liver conditions (Tang et al. (2024)); serum ferritin levels and the metabolic score for insulin resistance (Hao et al. (2022)); weight-adjusted-waist index and metabolic-associated fatty liver conditions (Hu et al. (2023)); blood manganese and liver stiffness (Han et al. (2023)); the neutrophil to HDL cholesterol ratio and metabolic-associated fatty liver conditions (Lu et al. (2024)); blood cadmium levels and depression (Ji and Wang (2024)); usual source of care and blood pressure (Dinkler et al. (2016)); visceral adiposity index and chronic kidney disease (Peng et al. (2023)); dietary fiber intake and stroke (Dong and Yang (2022)); sleep health and visceral adiposity index (Liu et al. (2024b)); ovariectomy-reduced hormones and depression (Chen et al. (2020)); serum albumin and depression (Zhang et al. (2023)); serum uric acid levels and creatine phosphokinase (Chen et al. (2023)); weight-adjusted-waist index and urinary incontinence (Sun et al. (2024)); body shape index and prostate cancer (Liu et al. (2024c)); sedentary behavior and urinary incontinence (Di et al. (2024)); triglyceride glucose body mass index and urinary incontinence (Li et al. (2024)); dietary total energy intake and asthma (Cao et al. (2024)); composite dietary antioxidant index and hyperlipidemia (Zhao et al. (2024)); dietary zinc intake and asthma (Cheng et al. (2024b)); medical uninsurance and sociodemographic attributes (Wahab et al. (2022)); sleep health and albumin (Li and Guo (2022)); visceral fat metabolic score and osteoarthritis (Xue et al. (2024)); waist circumference and sex steroid hormones (Zhu et al. (2024)); systemic immune-inflammation index and diabetes (Nie et al. (2023)); systemic immune-inflammation index and hyperlipidemia (Mahemuti et al. (2023)); triglyceride glucose index and depression (Ren et al. (2024)); selenium levels and chronic kidney disease (Pi et al. (2024)); composite dietary antioxidant index and cardiovascular disease (Liu et al. (2023)); and systemic immune biomarkers and metabolic-associated fatty liver conditions (Wang et al. (2024)). Each paper's exact exposure contrast, outcome, population, and cycles are in results/associations.csv and plan/associations.json.

Re-implementation. Each association is one file, code/associations/rowNNN.R, built on a shared library (code/lib/): it builds the exposure, outcome, covariates, and study population from each cycle's NHANES files, lists every alternative tried on the paper's own cycles as a variant, and records every choice the paper left open with its reason. Models are fitted with the survey package (Lumley (2004)) in R 4.6.1: weighted analyses by svyglm with Taylor-linearized variance over the masked strata and PSUs, the study population as a domain, pooled weights scaled by each cycle's share of the years, and t tests on the design's degrees of freedom (PSUs minus strata); unweighted analyses by glm with normal intervals. Ratio measures are analyzed on the log scale, and every interval is a 95% confidence interval. Unstated choices were settled by, in order, what the paper says elsewhere, NCHS's analytic guidelines, the convention of this literature, and, among choices still plausible, closeness to the published estimate. eGFR uses NCHS's calibration of 1999–2000 and 2005–2006 serum creatinine (Selvin et al. (2007)); the dietary inflammatory index uses its published global means and weights (Shivappa et al. (2014)); the composite dietary antioxidant index is the sum of six standardized intakes (Wright et al. (2004)).

What the replication tests is the published estimate, so the re-implementation follows the computation that produced it wherever the paper's own numbers reveal it, even where that departs from the paper's text: an unweighted analysis in a paper that says it weighted, a sample that complete-case coding restricted, a covariate the text names but the model left out. Two exceptions correct the computation instead: participants counted twice, by stacking the 2017–2018 files with the 2017–March 2020 files that contain them, are counted once, and an outcome coded in reverse is coded as the paper states it. Every such finding is recorded in the association's file as a departure, with its kind (the coding of a variable, the sample, the numbers reported, the survey weights or design, or the model), whether it bears on the headline estimate, and whether the re-implementation follows it; code/run counts the papers with each kind bearing on the headline. Multiple imputation, whose draws cannot be reproduced, is replaced by complete cases.

2021–2023. The same code runs on the new cycle with rules fixed before registration. Each association runs in its harmonized version, which leaves out the covariates and exclusion steps 2021–2023 cannot build (left_out in each file) and is fitted on the paper's cycles too, so what leaving them out changes is measured. Cutpoints, standardization constants, and knots keep the values the paper used or, where it gave none, those computed on its cycles. The weight is the paper's, except that NCHS directs the phlebotomy weight for blood analytes and the first dietary recall's weight for the 30-day supplement questionnaire in this cycle; the cycle is analyzed alone, with 15 design degrees of freedom. A covariate with one value in the 2021–2023 sample would have been left out, and a model that could not be fitted would have counted as not replicated; neither happened.

Outcomes. An association replicated if its 2021–2023 estimate has the published sign and its p-value, corrected over the 40 associations by Benjamini and Hochberg's procedure (Benjamini and Hochberg (1995)), is below 0.05; the share replicated has an exact (Clopper-Pearson) interval. A test is informative if it had at least 0.80 power (two-sided, α = 0.05, noncentral t on the design's degrees of freedom) to detect the published effect given its 2021–2023 standard error. The effect ratio is the 2021–2023 estimate over the published one on the analysis scale, summarized by its median with a distribution-free interval. Each 2021–2023 estimate is tested against the published one (z on the analysis scale, the published standard error taken from its interval, corrected as above), and the effect ratio of each association is split into three factors: our estimate on the paper's cycles over the published one (what reproducing the paper gives), the harmonized estimate over ours (what leaving out what 2021–2023 lacks changes), and the 2021–2023 estimate over the harmonized one (what changed between cycles, tested by a second corrected z test). All p-values are two-sided. In Table 1 and claims C10 to C12, the sleep estimate compares workday sleep of 5 hours or less with more than 7 and up to 8 hours, the diabetes estimate is per 100 units of the systemic immune-inflammation index, and the depression estimate is per unit of the triglyceride-glucose index.

Data and code. The bundle carries the 412 NHANES public-use files the analysis and the plan's dry run read, as CDC published them, compressed with xz, each checked against its recorded SHA-256 before it is read (data/nhanes/sources.csv; code/fetch_data.R --check compares them with CDC's current files). code/run re-runs everything from them in the image env/Dockerfile builds, in about 8 minutes on one core, and writes results/R1.json (the aggregates), results/associations.json (every fit and variant), and, through code/report.R, results/associations.csv and results/R2.json (the per-association values cited here).

Ethics. NHANES is conducted by the National Center for Health Statistics under protocols approved by its Research Ethics Review Board (#98-12, #2005-06, #2011-17, #2018-01, and #2021-05), with written informed consent. This study uses only the de-identified public-use files and attempts to identify no one. It reports what each published analysis computed, as the papers' own numbers show it, and makes no claim about how or why any paper was produced.

Results

The coding check. On the cycles each paper analyzed, our code reproduced 39 of the 40 published estimates, and our estimates were a median 0.988 times the published ones (0.917 to 1.012). In our own fits on those cycles, 32 estimates had the published sign and an unadjusted p-value below the threshold, fewer than published, partly because our intervals are design-based where several papers' were not and some of our complete-case samples are smaller. The estimate not reproduced is the association of dietary zinc with asthma (Cheng et al. (2024b)), whose estimates are the odds of having no asthma: our model of asthma, as the paper defines its outcome, gives 1.408 against the published 0.71, and the reversed outcome gives 0.7104.

What the published analyses computed. In 36 of the papers, reproducing the headline estimate showed that the analysis behind it differs from the paper's description in a way that bears on the estimate. By kind, 31 papers coded a variable otherwise than described, 15 analyzed another sample, 11 reported a number other than what its label says, 8 did not use the survey weights or design they describe, and 5 fitted a model other than the one stated. In 11 papers our analysis does not follow a departure, because it corrects it or because the paper's numbers rule out the stated computation without revealing the one used. Some are large. The caffeine paper's headline, labeled the odds ratio for the highest against the lowest quartile, is an odds ratio per milligram within the highest quartile, as every cell of its table is (Liu and Cui (2024)); the contrast its text describes gives 1.483. The selenium paper's models are reproduced only without age, which its text lists as a covariate (Pi et al. (2024)); with age its estimate is 0.8608 against the published 0.77. The stroke paper on the systemic immune-inflammation index counted the participants of 2017–2018 twice (Liu et al. (2024a)). Two published intervals are much narrower than a design-based analysis of the same data gives, because they ignore the survey design: the published interval is 0.4 times the width of ours for blood manganese and liver stiffness (Han et al. (2023)), and 0.62 times for workday sleep and albumin (Li and Guo (2022)). The gallstones paper's interval is 0.27 times ours for another reason: its model, as the paper lists it, holds total and HDL cholesterol beside the log of their ratio, which they nearly determine, and its own interval shows it was not fitted that way (Cheng et al. (2024a)). Each departure is described, with its evidence, in its association's file under code/associations/, and counted in results/associations.csv.

Replication. In 2021–2023, 14 of the 40 associations replicated (a share of 0.35, 0.206 to 0.517). One survey cycle holds fewer participants than the several most papers pooled, so only 13 tests had the planned power to detect the published effect; of these, 10 replicated (0.769, 0.462 to 0.95). The other tests can neither confirm nor refute their associations. Effects in 2021–2023 were a median 0.765 times the published ones (0.319 to 1.048; quartiles 0.081 and 1.138), and 0.906 times (0.319 to 1.16) among the informative tests. 16 estimates fell inside the published confidence interval, 16 had the published sign with an unadjusted p-value below the threshold, and 0 had the opposite sign with a significant corrected p-value. The results hold among the reproduced associations (14 of 39 replicated) and with each paper's own weight in place of the subsample weights NCHS directs for 2021–2023 (14). Every model fitted in 2021–2023, with no covariate left out for want of variation.

Where 2021–2023 differs from the published estimates. 3 estimates differ significantly from the published ones after correction (Table 1). Against our own harmonized estimates on the papers' cycles, 0 differ significantly, with a median ratio of 0.824 (0.344 to 1.104): a single cycle cannot tell most changes between cycles from sampling error.

Table 1. The associations whose 2021–2023 estimate differs significantly from the published one: the published estimate, ours on the paper's cycles (its own analysis), the harmonized one there, the 2021–2023 estimate with its confidence interval, the corrected p-value of the difference, and the three factors of the effect ratio.

In all three, our analysis reproduces the published estimate on the paper's own cycles, and the difference arose between cycles. The albumin paper pooled 2015–2016 with 2017–2018, whose analyzer reads albumin lower, and short sleepers are more common in 2017–2018; with a cycle term its own model gives -0.644 g/L, and in 2021–2023, analyzed alone, the estimate is -0.2671 g/L. Its intervals also ignore the survey design, and its weights are not the ones its text names. The diabetes paper analyzed unweighted data while saying it weighted them, and in 2021–2023 the estimate has the opposite sign. The depression paper's model omits glucose and lipids, which its text lists as covariates, and its 2021–2023 sample is 489 people against 3111 on its cycles, so the opposite sign there rests on few cases. The full table, with every association's estimates, tests, factors, and departures, is results/associations.csv.

Limitations

The main limitation is power: one two-year cycle has a fraction of the participants most papers pooled, so 27 of the 40 tests could not detect the published effect, and their failure to replicate says little. The effect ratios carry the most information, and their intervals are wide. The 2021–2023 cycle also differs from its predecessors: response fell, the first dietary recall moved from an in-person interview to the telephone, blood pressure is measured by an oscillometric device, and the cycle came after the COVID-19 pandemic, any of which could change an association; a significant difference between cycles is therefore not by itself evidence that a published estimate was wrong. The harmonized versions leave out what 2021–2023 cannot build, which changed some estimates on the original cycles (the harmonization factor in results/associations.csv).

Following each paper's computation, rather than its text, was a judgment made from its own numbers, and the departures recorded are those the coding check found while reproducing the headline estimate; papers' other tables were checked unevenly, so departures that do not bear on the headline are listed but not counted. Where a paper's computation could not be identified, as for the sample of the gallstones paper or the antioxidant index of the cardiovascular paper, our estimate may differ from what its authors computed for reasons we cannot see. Multiple imputation was replaced by complete cases. The sample excludes papers without full text in Europe PMC (53 of the 206 judged), which may favor open-access journals, and 85 papers were judged ineligible from Table A's labels and abstract alone. Extraction, coding, and the classification of departures were done by AI agents of one model family, reviewed by its lead agent but not by people, and every judgment is recorded, with its reason, in the files a reader can check. The study tests associations, not causal effects, and none of the associations, replicated or not, should be read as causal.

Provenance

The study was planned, run, and written by AI agents of the Claude family (claude-opus-5-5). A lead agent searched prior work, wrote the plan, the eligibility and headline rules, and the shared R library, coordinated agents working under written briefs who screened papers, extracted each paper with verbatim quotes, coded each association, and recorded its departures, reviewed and corrected their work, registered the plan, ran the analysis, and wrote the claims and this paper. The tools were R 4.6.1 with the survey, jsonlite, and foreign packages in a pinned Docker image, Python's standard library for the plan's tables, and Europe PMC's and Crossref's public APIs. The data are the NHANES public-use files, Table A of S1 Data of Suchak et al. (2025) under CC BY 4.0, and the sampled papers' full texts, which the bundle does not carry. A person proposed the study, saw a summary of the plan before registration without editing it, and asked that this report highlight where the 2021–2023 results differ from the published ones and what those analyses computed; no person reviewed the extractions, the code, the results, or this paper. provenance.json says the same.