Bonferroni Correction: Controlling Multiple Comparisons

What Is the Bonferroni Correction?

Named after Italian mathematician Carlo Emilio Bonferroni (1892–1960), whose probability inequalities provided the theoretical foundation, the correction addresses a problem that arises the moment you run more than one test on the same data: each test carries its own chance of a false positive, and those chances accumulate.

The logic is straightforward. If you run 20 independent tests each at α = 0.05, you expect roughly one false positive even when all null hypotheses are true. Divide α by 20 and each test is judged against 0.0025 instead of 0.05. Now the chance of any false positive across all 20 tests remains bounded at 5%.

The correction belongs to the broader topic of hypothesis testing and is most often encountered after an ANOVA that returns a significant overall F-test, prompting researchers to determine which specific group pairs actually differ. It appears across medicine, psychology, genetics, and data science wherever multiple tests need to be run together and a clear answer is required about whether any single one is credible.

The Multiple Comparisons Problem

Before the correction makes sense, you need to understand what it's fixing. The multiple comparisons problem arises because every hypothesis test has a probability α of producing a false positive — rejecting a true null hypothesis. When tests are independent, these probabilities combine multiplicatively.

The probability of obtaining at least one false positive across m independent tests is:

The numbers tell the story. As more tests run at α = 0.05 without any correction, the FWER rises quickly:

At 20 tests without correction, you have a 64% chance of at least one false positive. At 100 tests, you are practically guaranteed one. This is sometimes called the "alpha inflation" or "multiplicity problem." The Bonferroni correction holds the FWER at α regardless of m by requiring each test to clear a much higher bar.

The Bonferroni Correction Formula

The adjustment is the simplest possible: divide your target FWER by the number of tests. The result is the per-test significance threshold each individual p-value must clear.

Equivalently, you can multiply each raw p-value by m and compare the result to the original α. This gives a "Bonferroni-adjusted p-value":

Both approaches give the same decision. Adjusting α* downward is mathematically equivalent to adjusting p-values upward. The choice between them is presentational: reporting adjusted p-values is convenient when results are tabulated, because readers can compare each p-value directly against the familiar 0.05 threshold.

How to Apply the Bonferroni Correction — 4 Steps

Interactive Bonferroni Correction Calculator

Enter your original significance level (α) and the number of comparisons (m) to get the adjusted alpha threshold instantly. You can also enter individual raw p-values to see whether each test survives Bonferroni correction.

Bonferroni Correction — 3 Fully Worked Examples

Each example follows the four-step procedure. All arithmetic is shown in full with realistic data. Formulas use standard statistical notation.

Example 1 — ANOVA Post Hoc: Comparing Four Teaching Methods

Example 2 — Clinical Trial: Testing Three Drug Endpoints

Example 3 — Why Bonferroni Fails at Scale: A Genomics Illustration

Limitations of the Bonferroni Correction

The Bonferroni correction has two genuine weaknesses that researchers have to weigh before applying it. These are not reasons to avoid it entirely, but they do determine when it's the right tool and when it isn't.

1. It Reduces Statistical Power

Power is the probability of correctly rejecting a false null hypothesis. By demanding a smaller p-value from each test, the Bonferroni correction makes it harder to detect real effects — Type II errors (false negatives) become more common. This trade-off is acceptable when the cost of a false positive is high and the number of comparisons is moderate. It becomes problematic when m is large and missing true positives is costly, as in early-stage drug discovery or hypothesis-generating research.

2. It Assumes Independence

The derivation of FWER = 1 − (1 − α)ᵐ uses independent tests. When tests are positively correlated — sharing the same subjects, outcomes, or underlying genetic variants — the actual FWER is lower than the formula assumes. Bonferroni then over-corrects, discarding real effects unnecessarily. The Šidák correction handles independent tests marginally more precisely, while permutation methods can account for arbitrary correlation structures.

Alternatives to the Bonferroni Correction

Four methods are most commonly used alongside or instead of Bonferroni. All address the multiple comparisons problem; they differ in what they control and how conservative they are.

Holm-Bonferroni Method (Step-Down)

Proposed by Sture Holm in 1979, this sequential procedure controls FWER at the same level as Bonferroni but is strictly more powerful. Sort your m p-values from smallest to largest. Compare the smallest to α/m, the second smallest to α/(m-1), the third to α/(m-2), and so on. Stop at the first non-significant result — all subsequent hypotheses also fail to reject. Because the denominator gets smaller as you move down the ranked list, the threshold loosens, making it easier for larger p-values to survive.

Šidák Correction

When all tests are truly independent, the Šidák correction gives a slightly less conservative threshold than Bonferroni:

For α = 0.05 and m = 10: α_Šidák = 1 − (0.95)^(1/10) = 0.00512 versus α_Bonferroni = 0.0050. The difference is negligible for small m but grows with m. Use Šidák only when independence is guaranteed.

Benjamini-Hochberg Procedure (FDR Control)

This 1995 method by Yoav Benjamini and Yosef Hochberg shifts the target from FWER to the false discovery rate (FDR): the expected proportion of significant results that are false positives. At FDR = 5%, you accept that roughly 5% of your declared findings will be wrong — in exchange for much greater power to detect real effects.

The procedure: sort p-values from smallest to largest. For each p(k) at rank k, check whether p(k) ≤ (k/m) × q, where q is the desired FDR. Find the largest k that satisfies this inequality and declare all tests with p ≤ p(k) significant.

Benjamini-Hochberg is the standard in genomics, neuroimaging, and large-scale observational studies. It is covered in depth in the hypothesis testing guide on Statistics Fundamentals.

Side-by-Side Comparison

Decision Guide: Choosing the Right Method

The decision between Bonferroni, Holm-Bonferroni, and Benjamini-Hochberg comes down to two questions: how many tests are you running, and what are the consequences of each error type?

Real-World Applications

Entity and Formula Glossary

Running Bonferroni Correction in Software

Every major statistics package implements the Bonferroni correction and its alternatives. The code below applies correction to a vector of raw p-values in three environments.

R

# Raw p-values from 6 pairwise comparisons
p_values <- c(0.161, 0.001, 0.010, 0.024, 0.203, 0.290)

# Bonferroni-adjusted p-values
p.adjust(p_values, method = "bonferroni")

# Holm-Bonferroni adjusted p-values
p.adjust(p_values, method = "holm")

# Benjamini-Hochberg (FDR)
p.adjust(p_values, method = "BH")

Python (scipy)

from statsmodels.stats.multitest import multipletests
import numpy as np

p_values = [0.161, 0.001, 0.010, 0.024, 0.203, 0.290]

# Bonferroni correction
reject, p_adj, _, _ = multipletests(p_values, alpha=0.05, method='bonferroni')

# Holm-Bonferroni
reject_holm, p_adj_holm, _, _ = multipletests(p_values, alpha=0.05, method='holm')

print("Bonferroni adjusted p-values:", p_adj)
print("Reject H0 (Bonferroni):", reject)

SPSS and SAS

In SPSS, Bonferroni correction is available under Analyze → General Linear Model → Univariate → Post Hoc. In SAS PROC GLM, use MEANS / ADJUST=BONFERRONI; after the model statement. Both produce adjusted p-values and simultaneous confidence intervals.

Common Misconceptions

Frequently Asked Questions

Sources and References

This guide draws on the following primary and secondary sources. Formulas and methods are cross-referenced against the sources listed below, which are all freely accessible for verification.

• Miller, R.G. (1981). Simultaneous Statistical Inference, 2nd ed. Springer, New York. The standard graduate-level text on multiple comparisons methods.
• Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6(2), 65–70. Original paper for the Holm-Bonferroni method.
• Benjamini, Y. & Hochberg, Y. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society, Series B, 57(1), 289–300.
• NIST Engineering Statistics Handbook — Multiple Comparisons. National Institute of Standards and Technology. itl.nist.gov
• Penn State STAT 500 — Applied Statistics: Multiple Comparisons. online.stat.psu.edu
• OpenStax Introductory Statistics — Ch. 12: F-Distribution and One-Way ANOVA. Rice University. openstax.org
• ICH E9 Statistical Principles for Clinical Trials — Multiple Testing. European Medicines Agency. ema.europa.eu
• R Documentation — p.adjust function. The Comprehensive R Archive Network. stat.ethz.ch