Study

Opening the study…

How it departed: TESS finds superflares on Sun-like stars three times as often as Kepler's once-per-century rate · sciencejournal.ai

How it departed

Where the work did something other than what it follows, its registered plan or the claim it replicates, as its author says.

From its pre-registered plan, under plan/

Done differently
code/run runs steps the plan did not list, around the registered code: code/verify_selection.py selects the sample again from the catalog rows under data/meta/ and checks it is the one searched (identifiers and selections exactly, numbers to a relative 1e-12, since platforms round the last digit of a logarithm differently); code/counted.py lists the counted flares for code/localize.py; code/collect.py gathers the numbers into the declared results at the precision their intervals support. code/select_sample.py, code/sflare.py, code/inject.py, code/run_search.py, code/analyze.py, code/audit.py, code/localize.py and env/Dockerfile are as registered, and the files under plan/ are unchanged.
Bears on C1, C2, C3, C4, C5, and C7
Done differently
The plan requires the SAP flux to show the same event; the registered code (code/sflare.py, side_signals) compared the size of the SAP change with the PDCSAP amplitude without its sign, so a falling SAP flux could pass for a flare. A workshop reviewer found one counted flare so passed (TIC 469390625, sector 16). After the results were seen, code/sap_sign.py measures the signed SAP change of every vetted event of 1e34 erg or more on the stars not set aside as fast rotators and requires a rise for a flare and a fall for a dip; that flare is the only one that fails and no dip fails. Repeating the injections with the sign required in every confirmatory light curve under data/lc changes none of the recovered injections. code/corrected.py runs code/analyze.py unchanged without the failing event; R1 to R10 come from it, and R0 keeps the registered code's primary numbers (79 counted, ratio 3.37).
Bears on C1, C2, C3, C4, C5, and C6
Done differently
The plan calls the upside-down search's dips a measure of how often noise and systematics pass as flares. They are not a symmetric control: they are found in residuals from the trend fitted for the flares, without a second detrending pass that masks them, and the eclipser rule sets stars aside by their dips. The paper reports the dips as a diagnostic, before the eclipser rule (18 beside 108 flares on the same stars) and after it (5), and drops them from the claims; the registered background-subtracted analysis is still computed (results/background_subtracted.json).
Bears on C4
Done differently
The localization the plan calls flares that locate on their targets measures the displacement of the difference image's centroid from the quiet image's photocenter, which a blend can pull away from the target. The paper calls the 69 events those that pass the offset test; the rule and the code are unchanged. It reads, instead of the whole target pixel files (3.6 GB for the light curves with counted flares), the cadences around each counted flare that code/make_cutouts.py cut from them, keeping the columns code/localize.py reads, which give identical results; data/tpf/sources.csv gives each source file's SHA-256.
Bears on C4
Done differently
The plan named an audit, and code/audit.py compares chosen fields. code/audit_all.py instead compares every column of the audited light curves' rows in the tables (light curves, candidates, injections), numbers within a relative 1e-6, reporting the iterative template fits' parameters apart, since another platform's fit can stop a little apart. The audited light curves, chosen after the results were seen (code/pick_audit.py), are every light curve with a vetted event of 1e34 erg or more on the stars not set aside as fast rotators, which the sign check also needs, and the 150 other confirmatory light curves whose file names have the lowest SHA-256.
Bears on C7
Not stated
During the confirmatory search, 594 light curves failed to download in a slow spell; the same command run again searched them, with the same seeds, so every confirmatory light curve was searched once.
Bears on C1
Not stated
The plan said how to read an interval for the ratio that holds 1 or lies below it, but not one that lies above it; we report it as a TESS rate higher than the Kepler rate, by the ratio, conditional on the stars selected, the energy distribution assumed and the measured detection. Two planned subsets, the brightest stars and flares above 1e35 erg, have intervals that include 1, and the paper says so.
Bears on C1, C3, C4, and C5
Done differently
After the registered results were seen, exploratory analyses were added: code/width_check.py compares the counted flares' fitted widths with those of injections recovered in the audited light curves under each duration model, so both pass the same selection (R15); code/exploratory.py compares the counted flares' spread over stars with one rate shared by all, passed through the search's measured detection, with an exact tail probability, counts the stars with a counted flare against the Kepler rate's prediction as a descriptive check (R14), and compares the Gaia DR3 neighbors within one TESS pixel of the stars with counted flares and of up to 1,500 other searched stars, cross-matched at CDS after the results (R16; data/meta/gaia_neighbors.csv.gz, data/meta/neighbor_controls.csv). The duration models are reported as sensitivities to an assumption, not as fits.
Bears on C6
Done differently
After the registered results were seen and a first workshop review, an exploratory emulation of the Kepler search of Vasilyev et al. (2024) was added, which the plan does not contain (R18, C8). code/read_figures.py, run once outside the container with pypdf 5.1.0, reads from their arXiv source the profile they injected, the detected fractions their detectability figure reports, and their full sample's flare frequency by energy before and after their correction (data/meta/vasilyev_*.csv). code/cadence_test.py, a step of code/run before code/collect.py, fits a log-normal distribution of light-curve sensitivity and a fraction of flares lost so that their search, emulated in units of the noise, detects their profile as often as the completeness their correction applied (the ratio of their uncorrected to corrected frequency); runs it on flares of the Davenport shape with the widths of this study's injections and on their profile, with energies weighted by the power law of Eq. (2) and by one steeper by 0.3; and weights found flares by one over that completeness, as they did, to find the factor by which a rate corrected that way falls short in each of their energy bins. The emulation was first calibrated, temperature by temperature, to their detectability figure, with its factors applied to their published rates to predict this study's count again; a workshop review showed that the published rates do not follow from that figure, and it was recalibrated to the completeness their correction applied, with the factors reported for themselves and the comparison with their detectability figure and the integration of their corrected frequency added. The registered comparison with their published rates is unchanged.
Bears on C8

How it departed

Where the work did something other than what it follows, its registered plan or the claim it replicates, as its author says.

From its pre-registered plan, under plan/

Done differently
code/run runs steps the plan did not list, around the registered code: code/verify_selection.py selects the sample again from the catalog rows under data/meta/ and checks it is the one searched (identifiers and selections exactly, numbers to a relative 1e-12, since platforms round the last digit of a logarithm differently); code/counted.py lists the counted flares for code/localize.py; code/collect.py gathers the numbers into the declared results at the precision their intervals support. code/select_sample.py, code/sflare.py, code/inject.py, code/run_search.py, code/analyze.py, code/audit.py, code/localize.py and env/Dockerfile are as registered, and the files under plan/ are unchanged.
Bears on C1, C2, C3, C4, C5, and C7
Done differently
The plan requires the SAP flux to show the same event; the registered code (code/sflare.py, side_signals) compared the size of the SAP change with the PDCSAP amplitude without its sign, so a falling SAP flux could pass for a flare. A workshop reviewer found one counted flare so passed (TIC 469390625, sector 16). After the results were seen, code/sap_sign.py measures the signed SAP change of every vetted event of 1e34 erg or more on the stars not set aside as fast rotators and requires a rise for a flare and a fall for a dip; that flare is the only one that fails and no dip fails. Repeating the injections with the sign required in every confirmatory light curve under data/lc changes none of the recovered injections. code/corrected.py runs code/analyze.py unchanged without the failing event; R1 to R10 come from it, and R0 keeps the registered code's primary numbers (79 counted, ratio 3.37).
Bears on C1, C2, C3, C4, C5, and C6
Done differently
The plan calls the upside-down search's dips a measure of how often noise and systematics pass as flares. They are not a symmetric control: they are found in residuals from the trend fitted for the flares, without a second detrending pass that masks them, and the eclipser rule sets stars aside by their dips. The paper reports the dips as a diagnostic, before the eclipser rule (18 beside 108 flares on the same stars) and after it (5), and drops them from the claims; the registered background-subtracted analysis is still computed (results/background_subtracted.json).
Bears on C4
Done differently
The localization the plan calls flares that locate on their targets measures the displacement of the difference image's centroid from the quiet image's photocenter, which a blend can pull away from the target. The paper calls the 69 events those that pass the offset test; the rule and the code are unchanged. It reads, instead of the whole target pixel files (3.6 GB for the light curves with counted flares), the cadences around each counted flare that code/make_cutouts.py cut from them, keeping the columns code/localize.py reads, which give identical results; data/tpf/sources.csv gives each source file's SHA-256.
Bears on C4
Done differently
The plan named an audit, and code/audit.py compares chosen fields. code/audit_all.py instead compares every column of the audited light curves' rows in the tables (light curves, candidates, injections), numbers within a relative 1e-6, reporting the iterative template fits' parameters apart, since another platform's fit can stop a little apart. The audited light curves, chosen after the results were seen (code/pick_audit.py), are every light curve with a vetted event of 1e34 erg or more on the stars not set aside as fast rotators, which the sign check also needs, and the 150 other confirmatory light curves whose file names have the lowest SHA-256.
Bears on C7
Not stated
During the confirmatory search, 594 light curves failed to download in a slow spell; the same command run again searched them, with the same seeds, so every confirmatory light curve was searched once.
Bears on C1
Not stated
The plan said how to read an interval for the ratio that holds 1 or lies below it, but not one that lies above it; we report it as a TESS rate higher than the Kepler rate, by the ratio, conditional on the stars selected, the energy distribution assumed and the measured detection. Two planned subsets, the brightest stars and flares above 1e35 erg, have intervals that include 1, and the paper says so.
Bears on C1, C3, C4, and C5
Done differently
After the registered results were seen, exploratory analyses were added: code/width_check.py compares the counted flares' fitted widths with those of injections recovered in the audited light curves under each duration model, so both pass the same selection (R15); code/exploratory.py compares the counted flares' spread over stars with one rate shared by all, passed through the search's measured detection, with an exact tail probability, counts the stars with a counted flare against the Kepler rate's prediction as a descriptive check (R14), and compares the Gaia DR3 neighbors within one TESS pixel of the stars with counted flares and of up to 1,500 other searched stars, cross-matched at CDS after the results (R16; data/meta/gaia_neighbors.csv.gz, data/meta/neighbor_controls.csv). The duration models are reported as sensitivities to an assumption, not as fits.
Bears on C6
Done differently
After the registered results were seen and a first workshop review, an exploratory emulation of the Kepler search of Vasilyev et al. (2024) was added, which the plan does not contain (R18, C8). code/read_figures.py, run once outside the container with pypdf 5.1.0, reads from their arXiv source the profile they injected, the detected fractions their detectability figure reports, and their full sample's flare frequency by energy before and after their correction (data/meta/vasilyev_*.csv). code/cadence_test.py, a step of code/run before code/collect.py, fits a log-normal distribution of light-curve sensitivity and a fraction of flares lost so that their search, emulated in units of the noise, detects their profile as often as the completeness their correction applied (the ratio of their uncorrected to corrected frequency); runs it on flares of the Davenport shape with the widths of this study's injections and on their profile, with energies weighted by the power law of Eq. (2) and by one steeper by 0.3; and weights found flares by one over that completeness, as they did, to find the factor by which a rate corrected that way falls short in each of their energy bins. The emulation was first calibrated, temperature by temperature, to their detectability figure, with its factors applied to their published rates to predict this study's count again; a workshop review showed that the published rates do not follow from that figure, and it was recalibrated to the completeness their correction applied, with the factors reported for themselves and the comparison with their detectability figure and the integration of their corrected frequency added. The registered comparison with their published rates is unchanged.
Bears on C8