Three Methods, Three Different Answers
When you observe x successes in n trials, you are estimating an unknown proportion p. The point estimate is p̂ = x/n. What comes next — the confidence interval around that estimate — depends entirely on which method you use. Feed the same data to three different tools and you can get three materially different intervals.
Normal Approximation (Wald)
The textbook formula: p̂ ± z√(p̂(1−p̂)/n). Adequate when n ≥ 1,000 and 0.2 < p̂ < 0.8. Breaks down everywhere else, sometimes producing bounds outside [0,1].
Wilson Score Interval
Inverts the score test. Keeps bounds inside [0,1]. Near-nominal coverage for all sample sizes and proportions. The modern standard in A/B testing, product analytics, and NLP evaluation.
Clopper-Pearson (Exact)
Uses exact binomial tail probabilities. Guarantees coverage of at least 1−α. Conservative by design — deliberately wide — making it the standard for FDA submissions and clinical safety reporting.
Master Comparison Table
The table below maps the five dimensions that matter most when choosing a method in practice. Read it top-to-bottom for a quick decision, or use the decision framework in Section 6.
| Normal Approximation (Wald Interval) |
Wilson Score (Recommended) |
Clopper-Pearson (Exact) |
|
|---|---|---|---|
| Mathematical foundation | Large-sample Gaussian approximation to p̂ | Inverted score test (chi-square basis) | Exact binomial tail probabilities via Beta / F-distribution |
| Coverage probability behavior | Oscillates chaotically — the "sawtooth effect" — often well below 95% | Near-nominal; slight conservatism keeps coverage close to 1−α | Always ≥ 1−α — guaranteed but often substantially above the target |
| Interval width / precision | Unreliable: too narrow near boundaries, can be negative or > 1 | Near-optimal: tightest interval that meets nominal coverage on average | Conservative: always wider than Wilson; statistical power is reduced |
| Boundary handling (p̂ near 0 or 1) | Can produce [0, 0] or intervals crossing 0 / 1 — logically impossible | Strictly within (0, 1); no zero-width edge artifacts | Strictly within (0, 1); always returns a proper interval |
| Primary use case | Legacy code, teaching examples, large-n interior proportions | A/B testing, product analytics, CTR, conversion rates, NLP evaluation | Clinical trials, FDA/regulatory submissions, safety-critical decisions |
Numerical Benchmarks: Three Scenarios
Words describe behavior in general. Numbers describe it exactly. The table below shows what each method actually computes for three scenarios that probe the most problematic regions: zero successes, a moderate proportion in a small sample, and a rare event in a large sample. All intervals use 95% confidence (z = 1.96).
Each cell shows (Lower bound, Upper bound). A "−" prefix on a lower bound means the Wald interval has produced a mathematically impossible negative probability.
| Scenario | p̂ = x/n | Normal Approximation (Wald) | Wilson Score | Clopper-Pearson |
|---|---|---|---|---|
| Small n, extreme p̂ x = 0, n = 20 |
0.000 | (0.000, 0.000) | (0.000, 0.161) | (0.000, 0.168) |
| Small n, balanced x = 5, n = 20 |
0.250 | (0.060, 0.440) | (0.099, 0.477) | (0.087, 0.491) |
| Large n, rare event x = 10, n = 10,000 |
0.001 | (0.000, 0.002) | (0.0005, 0.0018) | (0.0005, 0.0018) |
The first scenario shows the most common failure: when x = 0, the Wald interval collapses to a single point (0.000, 0.000), providing zero information. Wilson and Clopper-Pearson both correctly return an upper bound near 16–17%, which is how much probability the data genuinely allows given the small sample. The practical implication is substantial: a product manager who reports a Wald-based zero interval for a zero-defect sample is systematically understating risk.
In the second scenario with n = 20 and a balanced proportion, the Wald interval is narrower than both alternatives — but that apparent precision is false. Its actual coverage probability at these parameters is closer to 91% than 95%. The Wilson interval is wider because it honestly reflects what the data can support.
Only in the third scenario, with n = 10,000 and p̂ = 0.001, do the three methods converge. This is the regime where the Wald interval is appropriate: large sample, interior proportion. Even then, Wilson is preferred because it requires no condition checking and performs identically.
Deep Dive: The Three Methods
The Normal Approximation (Wald Interval)
p̂ = x / n (observed proportion)
z = 1.96 for 95% CI
n = sample size
The Wald interval is the oldest and most widely taught method. It follows directly from the Central Limit Theorem: for large n, the sampling distribution of p̂ is approximately normal with mean p and standard deviation √(p(1−p)/n). Plug in p̂ as an estimate of p, multiply by z, add and subtract. Done in three arithmetic steps.
The problem is structural, not computational. The approximation requires p̂ to be far enough from 0 and 1 that the binomial distribution looks roughly bell-shaped. Near the boundaries, the distribution is skewed and the CLT has not had enough space to kick in — regardless of n. Newcombe (1998) showed that the Wald interval achieves its nominal 95% coverage only sporadically, and that actual coverage oscillates as both n and p change, creating what researchers call the sawtooth effect.
There is a second failure mode: boundary crashes. When p̂ = 0, the margin of error is exactly zero, so the interval is (0, 0). When p̂ = 1, it is (1, 1). When p̂ is very small but nonzero in a tiny sample, subtracting the margin of error can produce a negative lower bound — a probability below zero, which is impossible. These are not edge cases; they occur regularly in any application involving rare events, early-stage products, or small clinical cohorts.
Zero or very small x (even at large n), any sample with n < 30, any p̂ outside the range [0.15, 0.85]. In these cases, reported Wald intervals have genuine coverage probability well below 95%.
The Wilson Score Interval
p̂ = x / n
z = 1.96 for 95%
n = sample size
Center shifts toward 0.5
E.B. Wilson derived this interval in 1927 by inverting the score test. Instead of starting with p̂ and building an interval outward, Wilson found the set of all values of p that would not be rejected at the α significance level if they were the true parameter. That inversion produces an interval that stays strictly inside [0, 1] and whose center is pulled slightly toward 0.5 compared to p̂ — a shrinkage effect that turns out to be statistically beneficial.
The landmark paper by Brown, Cai, and DasGupta (2001) in Statistical Science settled the modern debate. They compared nine methods across a wide range of n and p values and concluded that the Wilson interval (and the closely related Agresti-Coull interval) should replace the Wald interval as the standard recommendation for general use. The Wilson interval has near-nominal coverage for all sample sizes, including small ones, and its slight conservatism is stable rather than oscillating.
Default choice for A/B testing, product analytics, conversion rate reporting, NLP model evaluation, and any application where n < 1,000 or p̂ may be near 0 or 1. Google, Netflix, Airbnb, and Spotify's experimentation platforms have all documented Wilson or Jeffreys as their standard.
The Clopper-Pearson (Exact) Interval
Upper = B(1−α/2; x+1, n−x)
B(q; a, b) = q-quantile of Beta(a,b)
x = observed successes
n = total trials
"Exact" = exact binomial math
Clopper and Pearson (1934) constructed their interval by finding, for each endpoint, the value of p at which the observed x successes falls exactly in the tail of the binomial distribution. The result uses the quantiles of the Beta distribution — which is the conjugate prior for the binomial — and requires no asymptotic approximation whatsoever.
A common misreading is that "exact" means "exact 95% coverage." It does not. It means the mathematics is exact binomial, with no approximation error. The consequence is that coverage is always at least 1−α but routinely higher — sometimes reaching 99% or above when the nominal target is 95%. This conservatism makes Clopper-Pearson wider than Wilson for the same data. The interval never lies; it just overstates how uncertain you should be.
In clinical trials and regulatory submissions, this over-conservatism is a deliberate feature. FDA guidelines for proportion-based endpoints in medical device and pharmaceutical studies frequently require Clopper-Pearson precisely because it provides a coverage guarantee rather than an average. When the consequence of a Type I error is patient harm or product liability, the extra width is worth the loss of statistical power.
"Exact" refers to the binomial arithmetic, not to the coverage target. Clopper-Pearson guarantees coverage of at least 95% — not exactly 95%. The actual coverage is almost always above 95%, often substantially so, which is why the intervals are wider than Wilson.
Edge Cases and Advanced Alternatives
The Zero-Count Problem (x = 0 or x = n)
When every trial either succeeds or fails — when x = 0 or x = n — the Wald interval collapses to a single point. Wilson returns a proper one-sided interval. Clopper-Pearson does the same. But there is a quicker rule of thumb that statisticians use when they need a rapid upper bound without software.
The Rule of Three
Quick upper bound when x = 0
When no events are observed in n trials, the upper 95% confidence limit for the true event rate is approximately 3/n. If 100 patients receive a drug and none has a serious adverse event, the Rule of Three gives an upper bound of 3%. This is a rougher version of Clopper-Pearson's exact result, but it can be computed mentally and is widely cited in medical and reliability engineering contexts.
For the full derivation, see: Hanley, J.A. & Lippman-Hand, A. (1983). JAMA, 249(13), 1743–1745.
Modern Alternatives Worth Knowing
Two additional methods appear regularly in modern experimentation literature and are worth a brief mention.
Agresti-Coull Interval: Lawrence Agresti and Brent Coull proposed a simple approximation to Wilson in 1998: add 2 pseudo-successes and 2 pseudo-failures to the data before applying the Wald formula. With ñ = n + 4 and p̃ = (x + 2)/ñ, the formula becomes p̃ ± z√(p̃(1−p̃)/ñ). This produces intervals nearly identical to Wilson and is especially easy to explain and implement. Brown, Cai, and DasGupta endorsed it alongside Wilson as a practical default.
Jeffreys Interval: This is the Bayesian approach using a non-informative Beta(0.5, 0.5) prior — the Jeffreys prior for the binomial. The posterior is Beta(x + 0.5, n − x + 0.5), and the 95% credible interval is its 2.5th and 97.5th percentiles. Jeffreys often produces tighter intervals than Clopper-Pearson while maintaining near-nominal coverage, and it handles zero-count data gracefully. Several large-scale experimentation platforms, including those documented by Netflix and Microsoft, use Jeffreys or Wilson as their default.
Industry Mapping: Which Method for Which Job
| Domain | Typical use case | Recommended method | Why |
|---|---|---|---|
| Product analytics / A/B testing | CTR, conversion rate, feature adoption | Wilson Score (or Jeffreys) | Near-nominal coverage at all sample sizes; handles low base-rate experiments without boundary artifacts |
| Clinical trials (exploratory) | Response rate in Phase II | Wilson Score | Efficient use of limited sample; adequate for non-regulatory reporting |
| Regulatory / FDA submission | Primary endpoint in Phase III, device approval | Clopper-Pearson | Guaranteed coverage satisfies regulatory requirements; conservatism is a feature, not a bug |
| Safety monitoring | Adverse event rate, defect detection | Clopper-Pearson | Overstating uncertainty is preferable to understating it when the consequence is patient or operational risk |
| Quality control / reliability | Pass/fail rates, failure probability | Clopper-Pearson (or Rule of Three for x = 0) | Regulatory and standards-body requirements; SPC charts require guaranteed coverage |
| Large-n analytics (n ≥ 1,000, 0.2 < p̂ < 0.8) | Population surveys, web analytics at scale | Normal Approximation acceptable | Three methods converge; Wald is computationally simplest and sufficiently accurate in this restricted regime |
| Machine learning / NLP evaluation | F1 score CI, classification accuracy | Wilson Score | Evaluation sets often small; boundary effects common; Wilson is now standard in ML evaluation literature |
Decision Framework: Which Method to Choose
Binomial CI Method Selector
Interactive Calculator: All Three Methods
Enter your data below. The calculator computes all three intervals simultaneously and flags any boundary issues with the Wald interval.
Binomial Confidence Interval Calculator
Coverage Probability: Visualizing the Sawtooth Effect
The chart below plots actual coverage probability against sample size n for p = 0.1 (a common low base rate in A/B tests). The Wald interval oscillates unpredictably. Wilson and Clopper-Pearson stay close to or above the 95% nominal target.
Actual Coverage Probability vs. n (p = 0.10, nominal = 95%)
Software Defaults and Code Implementation
The method you get depends on the function you call. Python and R give different defaults, and some functions apply continuity corrections that further change the output. The snippets below use the worked example of x = 2 successes in n = 15 trials at 95% confidence.
Python (statsmodels)
from statsmodels.stats.proportion import proportion_confint x = 2 # successes n = 15 # trials alpha = 0.05 # 95% CI # Normal Approximation (Wald) — default, but not recommended wald = proportion_confint(x, n, alpha=alpha, method='normal') print(f"Wald: ({wald[0]:.4f}, {wald[1]:.4f})") # Wilson Score — recommended default wilson = proportion_confint(x, n, alpha=alpha, method='wilson') print(f"Wilson: ({wilson[0]:.4f}, {wilson[1]:.4f})") # Clopper-Pearson (Exact) — for regulatory use exact = proportion_confint(x, n, alpha=alpha, method='beta') print(f"Clopper-Pearson: ({exact[0]:.4f}, {exact[1]:.4f})") # Expected output: # Wald: (−0.0101, 0.2768) ← negative lower bound: logically impossible # Wilson: (0.0369, 0.3967) # Clopper-Pearson:(0.0173, 0.4516)
R (base and binom package)
# prop.test() uses Wilson with continuity correction by default prop.test(x = 2, n = 15, conf.level = 0.95, correct = FALSE) # correct = FALSE removes the Yates continuity correction → pure Wilson # Result: 95% CI (0.0369, 0.3967) # binom.test() uses Clopper-Pearson (exact) — no continuity correction binom.test(x = 2, n = 15, conf.level = 0.95) # Result: 95% CI (0.0173, 0.4516) # For full control, use the binom package # install.packages("binom") library(binom) binom.confint(2, 15, conf.level = 0.95, methods = c("exact","wilson","asymptotic")) # Returns all three simultaneously in a single data.frame
When correct = TRUE (the default), prop.test() adds a Yates continuity correction to the Wilson formula. This makes the interval slightly wider and more conservative. To get the standard Wilson interval described in Brown et al. (2001), set correct = FALSE.
The key takeaway on software: never use a default without checking which method it implements. statsmodels defaults to Wald (method='normal'). prop.test() defaults to Wilson with continuity correction. binom.test() uses Clopper-Pearson. The same p̂ and n can produce three different intervals depending on which function you call.
Frequently Asked Questions
The Wilson Score Interval is a confidence interval for a binomial proportion, derived by inverting the score test. It shifts the center slightly toward 0.5 and keeps bounds strictly within [0, 1]. Brown, Cai, and DasGupta (2001) recommended it as the default for general use because its coverage probability stays close to the nominal level across all sample sizes and proportions.
Use Clopper-Pearson when regulatory or safety requirements mandate guaranteed at-least-nominal coverage, such as FDA submissions, clinical trial primary endpoints, or quality-control pass/fail decisions. It is always conservative and typically wider than necessary, but that conservatism helps prevent understating uncertainty. For most other applications, Wilson is often preferred because it is narrower while maintaining accurate coverage.
The Wald interval assumes that the sampling distribution of p̂ is approximately normal, which can break down near the boundaries, when p is close to 0 or 1, and for small sample sizes. Its actual coverage probability can be substantially below the nominal 95% level in many cases. It can also produce lower bounds below 0 or upper bounds above 1, which are impossible for a probability.
When x = 0 successes are observed in n trials, the Rule of Three gives a quick approximate upper 95% confidence bound of 3/n. For example, if no defects appear in 100 items, the upper bound is approximately 3/100 = 3%. This is a rough approximation to the Clopper-Pearson result, but it is easy to calculate and is widely used in medical and reliability contexts.
The Jeffreys interval is a Bayesian credible interval for a binomial proportion using a Beta(0.5, 0.5) prior, known as the Jeffreys prior. The posterior distribution is Beta(x + 0.5, n − x + 0.5), and the interval is obtained from its 2.5th and 97.5th percentiles. It behaves similarly to the Wilson interval and handles zero-count data gracefully. It is also used in applications where frequentist and Bayesian methods are considered together.
statsmodels.stats.proportion.proportion_confint defaults to the Normal approximation using method='normal'. To calculate a Wilson interval, use method='wilson'. For the Clopper-Pearson exact interval, use method='beta' or method='exact' where supported by the installed version. It is best to specify the method explicitly rather than relying on the default and unintentionally using the Wald interval.
Quick Reference Cheat Sheet
- Default for almost everything: Wilson Score. Handles n from 5 to 5,000,000; handles p̂ from 0.001 to 0.999; never produces impossible bounds.
- Regulatory submissions (FDA, EMA): Clopper-Pearson. Coverage guaranteed at ≥ 1−α. Required by many agency guidelines for primary endpoints.
- Quick mental calculation when x = 0: Rule of Three — upper bound ≈ 3/n.
- Acceptable Normal Approximation conditions: n ≥ 1,000 AND 0.2 < p̂ < 0.8 AND result is not safety-critical or regulatory.
- Python default: method='normal' (Wald) — override with method='wilson'.
- R prop.test() default: Wilson with continuity correction — set correct=FALSE for pure Wilson.
- R binom.test() default: Clopper-Pearson (exact).
- Agresti-Coull shortcut: Add 2 successes and 2 failures, then apply Wald formula. Nearly identical to Wilson.
Related Pages on Statistics Fundamentals
This guide is part of the broader confidence intervals topic area at Statistics Fundamentals. The pages listed below build directly on the material covered here.
References and External Resources
The analysis on this page draws on the following primary sources. Each is cited inline above where relevant.
- Brown, L.D., Cai, T.T., & DasGupta, A. (2001). Interval Estimation for a Binomial Proportion. Statistical Science, 16(2), 101–133. doi:10.1214/ss/1009213286 — The definitive modern reference recommending Wilson and Agresti-Coull over Wald.
- Wilson, E.B. (1927). Probable Inference, the Law of Succession, and Statistical Inference. Journal of the American Statistical Association, 22(158), 209–212. Original derivation of the score interval.
- Clopper, C.J. & Pearson, E.S. (1934). The Use of Confidence or Fiducial Limits Illustrated in the Case of the Binomial. Biometrika, 26(4), 404–413. Original derivation of the exact interval.
- Agresti, A. & Coull, B.A. (1998). Approximate is Better than "Exact" for Interval Estimation of Binomial Proportions. The American Statistician, 52(2), 119–126. Introduces the add-2/add-2 approximation to Wilson.
- NIST Engineering Statistics Handbook. Section 2.4.1 — Confidence Intervals for the Proportion. U.S. National Institute of Standards and Technology.
- Hanley, J.A. & Lippman-Hand, A. (1983). If Nothing Goes Wrong, Is Everything All Right? Interpreting Zero Numerators. JAMA, 249(13), 1743–1745. Original derivation of the Rule of Three.