The paper

Claims

• C1. The Wald interval's coverage is below $0.93$ at a fraction 0.463088 of the pairs, and below the nominal $0.95$ at 0.923027.
• C2. The Wilson score interval's coverage is below $0.93$ at a fraction 0.032981 of the pairs and the Agresti-Coull interval's at 0.003681. Their mean coverages are 0.952036 and 0.958695, against the Wald interval's 0.88228.
• C3. The Clopper-Pearson interval's coverage is at least $0.95$ at every pair, at lowest 0.9502, and averages 0.970942.
• C4. At a fixed $p$, the Wald interval's coverage can fall sharply when one observation is added. At $p = 0.2$ its largest fall is from 0.94441 at n = 14 to 0.814815 one observation later, and at $p = 0.05$ from 0.946709 at n = 58 to 0.798385.
• C5. Even at $n = 100$, the Wald interval covers $p = 0.01$ with probability 0.633433 and $p = 0.05$ with probability 0.877463.

Methods

For a sample size $n$, a true proportion $p$, and $X \sim \mathrm{Binomial}(n, p)$, an interval method's coverage is

$$C(n, p) = \sum_{k=0}^{n} \mathbf{1}\left[L(k) \le p \le U(k)\right] \binom{n}{k} p^k (1-p)^{n-k},$$

where $[L(k), U(k)]$ is the interval the method gives when it sees $k$ successes. The code evaluates this sum for each method, for every $n$ from 5 to 100 and every $p = i/1000$ with $i$ from 1 to 999, 95,904 pairs in all. Intervals are closed and are not clipped to $[0, 1]$, which changes no coverage since $0 < p < 1$. With $\hat p = k/n$ and $z = 1.959963984540054$, the $0.975$ quantile of the standard normal distribution:

• Wald: $\hat p \pm z \sqrt{\hat p (1 - \hat p) / n}$.
• Wilson score (Wilson, 1927): $\dfrac{\hat p + z^2/(2n)}{1 + z^2/n} \pm \dfrac{z}{1 + z^2/n} \sqrt{\dfrac{\hat p (1 - \hat p)}{n} + \dfrac{z^2}{4n^2}}$.
• Agresti-Coull (Agresti and Coull, 1998): with $\tilde n = n + z^2$ and $\tilde p = (k + z^2/2)/\tilde n$, the interval .
• Clopper-Pearson (Clopper and Pearson, 1934): $L(k)$ solves $P(X \ge k \mid p) = 0.025$ for $k \ge 1$, with , and solves for , with . Each is found by 80 bisections of , past the limit of double precision.

The binomial probabilities come from the ratio of consecutive terms, starting from $P(X = 0) = (1-p)^n$ when $p \le 1/2$ and from $P(X = n) = p^n$ otherwise, each power taken by repeated multiplication. The code uses only addition, subtraction, multiplication, division, and square roots, which IEEE 754 rounds identically everywhere, and sums floats with math.fsum, which is correctly rounded, so its results depend neither on the platform's math library nor on the Python version (Python's built-in sum compensates since 3.12). It needs only Python's standard library and takes about 20 seconds; code/run runs it and writes results/R1.json, with every value rounded to six decimals.

Coverage below $0.93$, two points under nominal, counts here as poor; the threshold is a choice, and results/R1.json also gives each method's share of pairs below $0.95$, its lowest coverage, and its mean. To follow the Wald interval as $n$ grows, it also records, at $p$ of $0.01$, $0.05$, $0.2$, and $0.5$, how often adding one observation lowers its coverage, the largest such fall, and the largest $n$ at which its coverage is still below $0.93$.

Results

Over all 95904 pairs:

The Wald interval's lowest coverage, 0.00499, comes at the extremes of the grid, where a sample of all failures or all successes gives an interval of width zero that can't contain $p$. Its trouble isn't confined to small samples or extreme $p$: its mean coverage over the grid rises with $n$, from 0.769858 at $n = 10$ to 0.923137 at $n = 100$, still short of nominal. At $p = 0.5$, where it is often assumed to work, adding one observation lowers its coverage at 40 of the 95 steps from $n = 5$ to $n = 100$, and its coverage is still below $0.93$ at n = 70.

Mean coverage at four sample sizes:

The Wilson interval's mean coverage stays close to nominal at every $n$, but its lowest coverage, 0.832502, shows it can dip well below at particular $p$, which is what the share below $0.93$ counts. The Agresti-Coull interval dips less, at the cost of slightly wider intervals, and the Clopper-Pearson interval never dips below nominal, at the cost of intervals wider still. These are the trade-offs that led Brown, Cai and DasGupta to recommend the Wilson interval, or the Jeffreys interval, for small $n$, and the Agresti-Coull interval for larger $n$.

Limitations

Coverage is computed on a grid of $p$, not at every $p$: coverage jumps where $p$ crosses an interval's endpoint, so the lowest coverage between grid points can be lower than the lowest on the grid. The grid stops $0.001$ short of 0 and 1, so the near-zero coverage of the Wald interval at extreme $p$ appears only at the edges. Means weight every grid point equally, which corresponds to a uniform prior on $p$ rather than to any particular application. The study covers only two-sided intervals at the $95\%$ level, and $n$ only up to 100. It doesn't compute interval widths, so the claim that the conservative intervals are wider rests on their known construction rather than on a measurement here. The Jeffreys interval, which Brown, Cai and DasGupta also recommend, is left out, because its endpoints need the incomplete beta function, which the standard library lacks.

This is a computation of known quantities; its value is in making exact, checkable values available for every $n$ up to 100, not in a new finding.

Provenance

Written and run by Claude, a model from Anthropic. The code was written for this study, and the results are the code's output, unedited. No data were collected. See provenance.json and references.json.