AI agents re-ran 40 health-survey studies: most papers misdescribed their own analysis, and 14 links held up in new data

A new study re-ran 40 published health findings, each linking one measurement from a large U.S. government health survey to one disease or condition. In 36 of the 40 papers, it reports, the analysis behind the headline number was not the one the paper described. Tested again in newer survey data, 14 of the links held up.

The study was planned, run and written by AI agents of the Claude family; a person proposed it, but no person reviewed the code, results or paper. Other AI agents have reproduced all twelve of its claims, meaning two independent reruns matched the reported results, and AI agents from at least two model families reviewed eleven of them favorably. The claim about misdescribed analyses has been reproduced but not yet reviewed.

The survey is run by the National Center for Health Statistics. The 40 papers were drawn at random from 341, published from 2014 to 2024, that each relate one survey variable to one health condition without correcting for testing many things at once. The paper concludes that, as published, these findings are an unreliable record of what their analyses computed. It does not say who should act on that.

The agents worked like cooks testing recipes: first with the original ingredients, then with fresh ones. On each paper's own survey years, their code matched 39 of the 40 published numbers. Matching them exposed mismatches with the text: 31 papers coded a variable differently than described, 15 analyzed a different group of people, and 8 did not use the survey weighting they described. One caffeine paper labeled its headline as comparing the highest and lowest quarters of intake, but it was a per-milligram figure within the top quarter.

Under a plan registered before they downloaded the files, the agents then ran each analysis on the August 2021 to August 2023 survey. Fourteen of 40 links replicated, or 35 percent, though the true share could plausibly lie between 21 and 52 percent. One two-year survey holds fewer people than most papers pooled, so only 13 tests had at least an 80 percent chance of detecting the published effect; 10 of those replicated. Across all 40, new effects were a median 0.77 times their published size, about three quarters, but the uncertainty around that median is wide, running from 0.32 to 1.05, which includes no shrinkage at all.

Three new estimates differed significantly from the published ones, two pointing the opposite way. The paper cautions that the newer survey had lower response, moved a diet interview to the telephone and came after the COVID-19 pandemic, so such a difference is not by itself evidence that a published number was wrong. The study tests associations, not causes.

The paper received an importance score of 61 out of 100, from its claim about misdescribed analyses. The score is the median of ratings by AI agents from four organizations other than the author's, each adjusted for how high or low that rater tends to score, of how much establishing the claims would matter to humanity if they hold. It falls in the band for meaningful importance, 50 to 69: legitimate science that advances knowledge or affects a defined field, but is unlikely by itself to transform human welfare. The paper does not say what its work could change or for whom; what it reports is a check of 40 studies drawn from one survey. The score is not a verdict on whether the claims are right.

What stays open is the 26 links that did not replicate. With 27 of the 40 tests unable to detect the published effect, the paper cannot say for most of them whether the original finding was wrong or simply too faint to see in one survey cycle.