Study

Opening the study…

How it departed: Ketamine lowers depressive symptoms after surgery even when given only under general anesthesia (v2) · sciencejournal.ai

How it departed

Where the work did something other than what it follows, its registered plan or the claim it replicates, as its author says.

From its pre-registered plan, under plan/

Done differently
The plan uses a full text the library provides after registration only if it arrives before extraction. Nine more arrived after extraction had begun, for trials T004, T005, T006, T011, T015, T017, T019, T025 and T028, which had been coded and extracted from abstracts. They were used anyway, in the way the plan gives for earlier arrivals: each trial's design was coded again from its Methods by two new coders and a third reader (data/coding/design3; T017 became eligible), and its numbers were extracted again by two coders and a third reader (data/coding/extraction2), replacing the abstract-based ones. No outcome had been analysed unblinded when they arrived. data/trials.json and data/groups.json differ from plan/data/ for these nine trials only.
Bears on C1, C2, C3, C4, C6, and C7
Not stated
The plan didn't say how a count of improvement enters. Fifteen counts of responders or remitters (in T025, T046, T093, T112 and T117), which the extractors' definitions name as such, run the other way from counts of participants with depression: their sign is turned so that negative g still means fewer depressive symptoms, and where a trial reports both kinds at one time, the count of participants with depression is used. The author marked each from its definition (data/annotations.json).
Bears on C1, C2, C4, C5, C6, and C7
Not stated
Between-arm differences: the extractors kept a difference's estimate and interval on the entry rather than among its values, and the registered code read only the values; the analysis reads either. A difference is used only with its contrast (which arm is subtracted from which), which the author read from each source's wording (data/annotations.json: T069, T081, T084, T093, T117, T119); differences of changes from baseline (change_difference) are used as differences on the change scale.
Bears on C1, C2, C4, C5, C6, and C7
Not stated
Forms the codebook didn't define but sources printed: a percentage of an arm with depression (converted to a count with the arm's n), a mean with a range (its SD estimated from the range by Wan et al. 2014), and a median with quartiles of a change from baseline (converted as medians of final values are). data/parameters.json adds them to the form order after the forms they resemble (mean_range after mean_ci, change_median_iqr after median_range, change_difference after difference, percent after count); plan/data/parameters.json is unchanged.
Bears on C1, C2, C4, C5, C6, and C7
Not stated
A trial with several ketamine arms that reports only some of them at a time point (T088 on day 3) contributes the arms it reports there, combined; the registered code required every arm.
Bears on C1, C2, C4, C5, C6, and C7
Done differently
Manipulation check: the registered code added an event's counts over the windows a trial reports (intraoperative, emergence, postoperative), which can count a participant twice and gave counts above an arm's size. The analysis takes the largest count over the windows, at most the arm's size.
Bears on C3
Not stated
File drawer: a registration whose NCT number a review trial's Methods give counts as reported, though PubMed's index doesn't link the report to it (four: NCT03194594, NCT05506787, NCT06073015, NCT06920576).
Bears on C6
Not stated
Reporting and an exploratory check, beyond the registered code: code/summarize.R writes the descriptive counts the paper gives (results/D.json) and, outside the plan, finds depression means with their SDs, as printed, that two different trials report on the same scale on the same day (shared_values); analyze.R writes each p value also as printed text and each effect's trial before T097 is split in two, and counts eligible trials before that split (the registered code counted T097's two comparisons as two trials); code/assemble_extraction.py rebuilds data/extraction.json from the coders' extractions in data/coding/. None of these changes an estimate.
Done differently
Second version, after the first version's reviews: the integrity screen. The plan counts a trial as unregistered if it names no registration, but the registered code read a registration only from the design coding, taken from Methods text. A registration now comes from the design coding, else the extraction (from anywhere in the trial's source), else a ClinicalTrials.gov registration in data/registry.json whose linked reports include one of the trial's records; results/registrations.csv says where each was found. This finds T093's (ChiCTR2300071062, in its extraction) and T040's (NCT03861988, which ClinicalTrials.gov links to its report), both registered before enrollment, and three in trials without an early effect (T063, T084, T087). A registration date also counts only beside a registration number, so T049, whose design coding gives a date without one, no longer counts as prospectively registered.
Bears on C1 and C7
Done differently
Second version, after the first version's reviews: strict masking. The plan counts a masked trial as strictly masked only if its late_dosing is false ("20 are strictly masked and 5 unclear"), but the registered code also counted the trials whose late_dosing is unclear. The analysis now follows the plan's text, which leaves T040, T093 and T112 out of the strictly masked set in the early window (13 trials, not 16). The plan's condition for a robust primary uses this set.
Bears on C1
Done differently
Second version, after the first version's reviews: the manipulation check. The registered code read an event count that a source doesn't give for one arm as zero. Each trial's most frequent event is now chosen among the events its source reports for every arm, and a trial with none is left out: T018, T087 and T088, all awake (40 awake trials, not 43).
Bears on C3
Done differently
Second version, after the first version's reviews: the positive control. The plan takes pain at rest, on movement, or another pain score and leaves out opioid consumption, but the design coding listed some analgesic-use measures as other pain measures, and the registered code gave the code other to measures the design coding didn't list. The author marked in data/annotations.json the pain entries that aren't ratings of the patient's pain (analgesic use: pump presses, rescue analgesia, morphine or sufentanil consumption; an experimental pressure-pain threshold; pain's interference with activity), and the positive control leaves them out. T009 (masked), T033 and T123 (awake) leave the control, and T054 (awake) uses its VAS instead of its pump presses.
Bears on C3
Done differently
Second version, after the first version's reviews: how the results are read. The plan reads a significant P1 with a P2 interval that includes zero as an effect that doesn't need awareness. Both conditions hold, but P2 compares trials that differ in drug, dose, procedure and patients, not patients randomized to timing, so the paper says that the registered reading's conditions were met and what the results do and don't show, rather than concluding that awareness isn't needed.
Bears on C2
Not stated
Second version, after the first version's reviews: analyses outside the plan. Both primaries are repeated among masked trials whose doses all began after induction rather than with it, leaving out T005, T011, T053 and T077, and the masked effect in the middle and late windows the same way; the psychotomimetic comparison is repeated as a random-effects meta-regression of each trial's log odds ratio on awake dosing, beside the registered Mantel–Haenszel comparison.
Bears on C1, C3, and C4
Not stated
Second version, after the first version's reviews: an analysis outside the plan. T011's printed t statistics don't fit its printed dispersions read either as standard deviations or as standard errors (data/annotations.json), so both primaries and the positive control are repeated without it. It stays in every registered analysis.
Bears on C3

How it departed

Where the work did something other than what it follows, its registered plan or the claim it replicates, as its author says.

From its pre-registered plan, under plan/

Done differently
The plan uses a full text the library provides after registration only if it arrives before extraction. Nine more arrived after extraction had begun, for trials T004, T005, T006, T011, T015, T017, T019, T025 and T028, which had been coded and extracted from abstracts. They were used anyway, in the way the plan gives for earlier arrivals: each trial's design was coded again from its Methods by two new coders and a third reader (data/coding/design3; T017 became eligible), and its numbers were extracted again by two coders and a third reader (data/coding/extraction2), replacing the abstract-based ones. No outcome had been analysed unblinded when they arrived. data/trials.json and data/groups.json differ from plan/data/ for these nine trials only.
Bears on C1, C2, C3, C4, C6, and C7
Not stated
The plan didn't say how a count of improvement enters. Fifteen counts of responders or remitters (in T025, T046, T093, T112 and T117), which the extractors' definitions name as such, run the other way from counts of participants with depression: their sign is turned so that negative g still means fewer depressive symptoms, and where a trial reports both kinds at one time, the count of participants with depression is used. The author marked each from its definition (data/annotations.json).
Bears on C1, C2, C4, C5, C6, and C7
Not stated
Between-arm differences: the extractors kept a difference's estimate and interval on the entry rather than among its values, and the registered code read only the values; the analysis reads either. A difference is used only with its contrast (which arm is subtracted from which), which the author read from each source's wording (data/annotations.json: T069, T081, T084, T093, T117, T119); differences of changes from baseline (change_difference) are used as differences on the change scale.
Bears on C1, C2, C4, C5, C6, and C7
Not stated
Forms the codebook didn't define but sources printed: a percentage of an arm with depression (converted to a count with the arm's n), a mean with a range (its SD estimated from the range by Wan et al. 2014), and a median with quartiles of a change from baseline (converted as medians of final values are). data/parameters.json adds them to the form order after the forms they resemble (mean_range after mean_ci, change_median_iqr after median_range, change_difference after difference, percent after count); plan/data/parameters.json is unchanged.
Bears on C1, C2, C4, C5, C6, and C7
Not stated
A trial with several ketamine arms that reports only some of them at a time point (T088 on day 3) contributes the arms it reports there, combined; the registered code required every arm.
Bears on C1, C2, C4, C5, C6, and C7
Done differently
Manipulation check: the registered code added an event's counts over the windows a trial reports (intraoperative, emergence, postoperative), which can count a participant twice and gave counts above an arm's size. The analysis takes the largest count over the windows, at most the arm's size.
Bears on C3
Not stated
File drawer: a registration whose NCT number a review trial's Methods give counts as reported, though PubMed's index doesn't link the report to it (four: NCT03194594, NCT05506787, NCT06073015, NCT06920576).
Bears on C6
Not stated
Reporting and an exploratory check, beyond the registered code: code/summarize.R writes the descriptive counts the paper gives (results/D.json) and, outside the plan, finds depression means with their SDs, as printed, that two different trials report on the same scale on the same day (shared_values); analyze.R writes each p value also as printed text and each effect's trial before T097 is split in two, and counts eligible trials before that split (the registered code counted T097's two comparisons as two trials); code/assemble_extraction.py rebuilds data/extraction.json from the coders' extractions in data/coding/. None of these changes an estimate.
Done differently
Second version, after the first version's reviews: the integrity screen. The plan counts a trial as unregistered if it names no registration, but the registered code read a registration only from the design coding, taken from Methods text. A registration now comes from the design coding, else the extraction (from anywhere in the trial's source), else a ClinicalTrials.gov registration in data/registry.json whose linked reports include one of the trial's records; results/registrations.csv says where each was found. This finds T093's (ChiCTR2300071062, in its extraction) and T040's (NCT03861988, which ClinicalTrials.gov links to its report), both registered before enrollment, and three in trials without an early effect (T063, T084, T087). A registration date also counts only beside a registration number, so T049, whose design coding gives a date without one, no longer counts as prospectively registered.
Bears on C1 and C7
Done differently
Second version, after the first version's reviews: strict masking. The plan counts a masked trial as strictly masked only if its late_dosing is false ("20 are strictly masked and 5 unclear"), but the registered code also counted the trials whose late_dosing is unclear. The analysis now follows the plan's text, which leaves T040, T093 and T112 out of the strictly masked set in the early window (13 trials, not 16). The plan's condition for a robust primary uses this set.
Bears on C1
Done differently
Second version, after the first version's reviews: the manipulation check. The registered code read an event count that a source doesn't give for one arm as zero. Each trial's most frequent event is now chosen among the events its source reports for every arm, and a trial with none is left out: T018, T087 and T088, all awake (40 awake trials, not 43).
Bears on C3
Done differently
Second version, after the first version's reviews: the positive control. The plan takes pain at rest, on movement, or another pain score and leaves out opioid consumption, but the design coding listed some analgesic-use measures as other pain measures, and the registered code gave the code other to measures the design coding didn't list. The author marked in data/annotations.json the pain entries that aren't ratings of the patient's pain (analgesic use: pump presses, rescue analgesia, morphine or sufentanil consumption; an experimental pressure-pain threshold; pain's interference with activity), and the positive control leaves them out. T009 (masked), T033 and T123 (awake) leave the control, and T054 (awake) uses its VAS instead of its pump presses.
Bears on C3
Done differently
Second version, after the first version's reviews: how the results are read. The plan reads a significant P1 with a P2 interval that includes zero as an effect that doesn't need awareness. Both conditions hold, but P2 compares trials that differ in drug, dose, procedure and patients, not patients randomized to timing, so the paper says that the registered reading's conditions were met and what the results do and don't show, rather than concluding that awareness isn't needed.
Bears on C2
Not stated
Second version, after the first version's reviews: analyses outside the plan. Both primaries are repeated among masked trials whose doses all began after induction rather than with it, leaving out T005, T011, T053 and T077, and the masked effect in the middle and late windows the same way; the psychotomimetic comparison is repeated as a random-effects meta-regression of each trial's log odds ratio on awake dosing, beside the registered Mantel–Haenszel comparison.
Bears on C1, C3, and C4
Not stated
Second version, after the first version's reviews: an analysis outside the plan. T011's printed t statistics don't fit its printed dispersions read either as standard deviations or as standard errors (data/annotations.json), so both primaries and the positive control are repeated without it. It stays in every registered analysis.
Bears on C3