Reviews and checks

Methods review

Independent check. From the bundle's own results/outcome_counts.csv, I recomputed the stratified Mantel–Haenszel risk ratios with a Greenland–Robins variance in a separate script. They match the paper for bacterial infections A18 (0.878, 0.831 to 0.927), pneumonia A21 (0.868, 0.809 to 0.931), acute pancreatitis A49 (1.128, 0.923 to 1.380), death B12 (0.872, 0.829 to 0.918), any serious adverse event B13 (0.961, 0.944 to 0.978), thyroid cancer B05 (1.611, 1.067 to 2.432) and ileus B02 (1.028, 0.825 to 1.281). The full pipeline also reproduces exactly in a container (a separate reproduction job).

Design strengths. The plan was registered before arms were compared. Screening and term coding were done blind to counts, with two coders plus an adjudicator and high kappa. It uses complete registry serious-adverse-event tables instead of publications, and the analysis is stratified by comparison. It includes positive controls, a correction for multiple testing within each family, a robustness rule fixed in advance, a random-halves replication, and power computed blind to arm. Deviations are disclosed (only the display-rounding script was added).

Issues across claims:

1. Differential ascertainment window. Where adverse events are collected only on treatment, more discontinuation on the drug (from gastrointestinal effects) shortens observation in that arm. That biases every "lower on drug" estimate downward. The Limitations name this, but the claims that rest on it (C2, C6) don't say so, and no analysis bounds it. A sensitivity analysis restricted to trials that record events through the end of the trial regardless of discontinuation (the record's time-frame field often says this), or a negative-control outcome analysis reported as numbers, would address it. Any serious adverse event at 0.96 is itself consistent with a small observation deficit.
2. Term summing. The conservative variance handles the double counting, but the point estimates are still term counts, not participant counts. The paper is open that the A18 composite isn't significant for its most frequent term.
3. Coders. All the coders are instances of one model family, and the authors acknowledge it. A small human-coded audit sample would strengthen the coding.
4. Hazard ratio vs risk ratio and outcome definitions. For C5, the atlas's estimate is for drug-induced acute pancreatitis, while the trials count acute pancreatitis of any cause. The Discussion explains this well, but "excludes the atlas's estimate" compares different outcome definitions.

Per claim:

• C1: sound. The positive controls behave as expected, and the counts are traceable to the screening files.
• C2: minor_issues. The results are robust under the registered rules and replicate across halves, and they agree with published meta-analyses. Issue 1 should be stated in the claim, or tested in trials with complete follow-up, before the causal wording "lower the risk" stands without qualification.
• C3: sound. The discovery and replication split across random halves is appropriate, and the scan follows from the registered method.
• C4: minor_issues. The categories are applied as registered. "Inconclusive" and "too few events" carry most of the weight (37 of 44), and the claim's power framing is the right one. "Confirm" here means q < 0.05 in the same direction, not agreement in size, which should be stated.
• C5: minor_issues. The computation is correct, but the comparison is any-cause risk ratio vs drug-induced hazard ratio (issue 4). The claim should name that difference.
• C6: minor_issues. Deaths are limited to comparisons that report them for every group, which is a possible selection, and the result is subject to issue 1. Death is the outcome least affected by ascertainment, which supports it.
• C7: sound. Reporting the thyroid signal as nominal and not robust is careful.
• C8: minor_issues. "Not more frequent" for ileus is supported only as no detected increase. The interval allows up to 1.28, so the claim should say the data exclude an increase above about 28%.

Reproducibility. The data, both codebooks, all codings and the seeded code are in the bundle. The run is deterministic and took about 4 s in Docker. The materials resolve (Python, NumPy, SciPy RRIDs).

Significance. The C2 infection reduction is consistent with prior meta-analyses (Han 2025, Yang 2026). The registry-complete, blind-coded confirmation is useful, which I rate moderate. C4 and C5, as a systematic randomized test of an observational atlas, are moderate. C1 is known (positive controls). C3 is minor. C6 is known; lower mortality is established in the outcome trials. C7 and C8 are minor.

Reproduction report

Made by sj-harness 0.6.0 for job job:88407d878ffecf98acfcf0b4ae73a2ea, on bundle sha256:eb8f71de6dd7a26acb758afdc9fad72338405cf0489a62461fa9fe09b4df6cdb, whose verification inputs are sha256:ced832c560ebe8be4131d92c5cac13121f546d97bff094a1a79784f90401ad0d.

How it ran

• Engine: docker 29.4.0, on darwin arm64 with Node v26.10.0.
• Image: sj-harness:812009c2ad79fa18, env/requirements.txt installed with pip on public.ecr.aws/docker/library/python:3.12-slim (built before from the same inputs, and used again). Image ID sha256:ac0b7eaeadb2e056839feefb1ceeac11c90ffa93f54d8ba23dc212903447a072.
• Command: sh code/run, from the bundle's code/run, run from the bundle's root.
• Limits: no network, every capability dropped, no new privileges, at most 4096 processes, 12030m of memory, 12 CPUs, and 7.5 minutes (1.5 times the 5 minutes the bundle declares).
• Outcome: exit code 0 after 4.08 s. Started 2026-10-12T18:50:04.184Z, finished 2026-10-12T18:50:08.269Z.

Verdicts

Claim IDs: C1 is claim:cf22781ed6f3524b5be231efc7129faab11feb0c53acc86645de9f658879a8d7; C2 is claim:b6e575472f620a2e50b72a6f649969ae2def61c71af18200a9afbd443d9d06cf; C3 is claim:70fab88d29c1fe055fc6504ec67780a5b3a01a028dd861b6b0efce3be2058053; C4 is claim:bf4636e8fb9dc1ccc97cd9d033debab9fb5237048a2bcfae929d12528ead5a7f; C5 is claim:7fca5e4a9907984359ec9a59ef37d973e85eb8f01eedaf148032c109d997e84d; C6 is claim:a38d2107c0d790a3c5624868ecdd48d7d7bb171f2df75e2eebb34d37d15f54b9; C7 is claim:e08ef7977c9117522a6df0b3f5ca9781f8f81b59754c49fd030fcd8565189231; C8 is claim:56c1ee2f2b437cdac68aa1a900733e4f0d6ece8eee3bd048f6de5d963855927b.

Results

A number agrees when it lands within its tolerance of the declared value, compared as the decimals canonical JSON writes; anything else must be equal.

Hidden content

Before any model read the bundle, the harness's scan found nothing hidden in its 587 text files.

Files

• No run.log: the run printed nothing.
• environment.json: the machine, engine, image, command, limits, and outcome.
• results/: the 10 files the run wrote under results/.