Study
Opening the study…
Study
Opening the study…
Where the work did something other than what it follows, its registered plan or the claim it replicates, as its author says.
From its pre-registered plan, under plan/
Done differently
The registered code/analyze.py recomputed the structured filter (at least two adverse-event groups) from every one of the 874 records found and checked that the screened set is what it keeps. This bundle carries only the 514 eligible trials' records, so that a verifier's download stays under 100 MB, and its code/analyze.py reads the filter's registered result, data/structured.json (byte for byte the registered plan/data/structured.json), and makes the same check against it. Nothing else in the file differs from plan/code/analyze.py, and results/R1.json to R6.json and S.json are byte for byte what the registered file writes when given all 874 records. code/checks.py confirms that each record carried has the SHA-256 registered for it (data/ctgov_digests.json, a copy of plan/ctgov_digests.json; the coders' agreement files are copied to data/ likewise, since a re-run sees only code/, env/ and data/).
Bears on C1, C2, C3, C4, C5, C6, C7, C8, C9, C10, C11, C12, and C13
Done differently
Added code/checks.py, which code/run calls after the registered analysis. It writes the registered estimates as tables (results/strata.csv, named.csv, organ_systems.csv), counts which pre-specified kinds of trial and vaccine have intervals inside the primary margins (which the plan said the paper would report), and writes results/T.json, the values the paper and claims cite, formatted (counts with thousands grouped) under short names; T.json computes nothing that R1 to R6, S or X don't hold. It also holds the published-count check and the analyses not planned in advance that the entries below describe, and writes their tables (results/published.csv, follow_up.csv).
Bears on C1, C2, C3, C4, C5, C6, C7, C8, C9, C10, C11, C12, and C13
Not stated
The plan foresaw comparing the registry's counts with published ones for the 15 largest main-set comparisons, with code to be written afterwards, and left the details open. Three readers (instances of the model that designed the study, with web access, five comparisons each) were given each trial's titles, the registry's adverse-event time frame, and the names and descriptions of the compared groups, and told not to open the registry. They took, from the main publication, a regulator's review or the sponsor's report, the safety population, participants with at least one serious adverse event, and deaths in each arm over the placebo-controlled period (data/published.json, with source, period, quote and note). They report that search-engine summaries displayed registry counts to them for four trials, which they did not use. A published count is used only if a source states it (or a percentage with its denominator), it counts the same outcome, and it covers the same window in both arms; the file records, per outcome, the counts, their own denominators, whether each is usable and why not. This rule was applied after reviewers of a first version found a substitution that counted a different outcome and one with the wrong denominators; the first version had used every count found. For death, counts are swapped where the registry record has a deaths table, and, reported apart, comparisons whose record has none are also added.
Bears on C13
Done differently
Not planned. The registered estimates showed more serious adverse events on vaccine in trials spanning children and adults; looking at those trials' counts showed records whose vaccine groups cover open-label follow-up after placebo recipients were unblinded or vaccinated. A codebook was then written (data/follow_up_codebook.md) and the two coders of the screening read, without counts, the 60 main-set trials whose arm or adverse-event text matches crossover, unblinding, open-label, extension or booster, coding each as equal, vaccine longer, placebo longer or unclear (data/follow_up.json holds both codings and reasons; no third reader, and nothing is decided between them). code/checks.py re-estimates the primary outcomes, and any serious adverse event and death in each pre-specified kind of trial and vaccine, without the 7 trials both coders coded as not equal and again without the 13 either did, and gives the 6 both coded as vaccine longer alone. The codebook and the choice of which trials to read were made after seeing results by arm, the search words mostly mark designs that lengthen the vaccine groups' follow-up, and leaving trials out moves the estimates toward the vaccine, so this is exploratory. A first version left out all 13 under a rule that took the coding that excluded more; reviewers asked for the version on which both coders agree.
Bears on C12
Done differently
Not planned: code/checks.py splits the live vaccines' estimate for infections other than the vaccine's target by target disease, and gives it without rotavirus vaccine trials, to show where the registered contrast between live and non-live vaccines comes from. The paper reports the by-target ratios in its Results as exploratory; the claim states only the registered contrasts.
Bears on C11
Not stated
The plan says participants at risk are each group's seriousNumAtRisk. The registered code also covers what the plan didn't state: where a group gives none (or zero), it takes the largest number at risk among the group's serious-event rows, else the group's other-event or death denominator (this decides the denominators of two small trials, NCT00420745 and NCT01059617), and it leaves out a comparison with no participants at risk in an arm (one, NCT02415166, so the main set analysed is 501 comparisons from 491 trials where the screening gives 502 from 492). An independent audit of the finished analysis, which recomputed the first two primary estimates from the plan's text alone, found these conventions; with them it reproduces the registered numbers exactly.
Bears on C1 and C3
Done differently
Not planned. After the analysis was finished, an independent audit by a fresh instance of the designing model read the terms tagged for the cardiovascular outcome and listed six untagged terms that the third reader's notes would tag (cardiac death; cerebral artery embolism, thrombosis and occlusion; cerebral thrombosis; basilar artery thrombosis: both coders had left them untagged, so the third reader never saw them) and eight tagged terms it doubted (spinal cord infarction, spinal stroke, intracranial tumour haemorrhage, three forms of cor pulmonale, cardiopulmonary failure, cardiac ischemia/infarction). The registered coding is unchanged and the claim reports it; code/checks.py re-estimates P3 with the six added, the eight dropped, and both (data/term_sensitivity.json), and the paper gives the last beside the registered estimate.
Bears on C4
Done differently
Reviewers of a first version showed that two registered named outcomes take in more than the names the plan gave them: the outcome the plan calls Guillain-Barré syndrome (tag gbs) also counts chronic inflammatory and unspecified demyelinating polyneuropathies, as its codebook definition says, and the one it calls preterm birth (tag preterm) also counts premature labour and preterm premature rupture of membranes. The registered outcomes, tags, estimates and Benjamini-Hochberg family are unchanged. The claims, the paper and results/named.csv now name them for what their terms cover (data/narrow_terms.json, labels), and code/checks.py adds, outside the registered family and not planned, each outcome's estimate with only the terms that name the diagnosis itself (Guillain-Barré or Miller Fisher syndrome; premature delivery or medically induced preterm birth).
Bears on C8