What Is the Tukey HSD Test?
The test was developed by John Tukey in 1949 and published in his landmark paper on comparing treatment means. The name "Honestly Significant Difference" reflects the key guarantee: unlike running a separate t-test for each pair, the Tukey method ensures the overall probability of making even a single false rejection across all comparisons stays at α, not αk for k comparisons.
To see why that matters, consider an experiment with five groups. There are C(5,2) = 10 possible pairs of means to compare. If you ran 10 separate t-tests each at α = 0.05, the probability of at least one false positive climbs to roughly 1 − (0.95)¹⁰ ≈ 0.40. The Tukey HSD test holds that probability at exactly 0.05 by using a wider critical threshold based on the studentized range distribution rather than the t-distribution.
The Tukey HSD test is a post-hoc comparison test run after a significant ANOVA result. It compares all possible pairs of group means while controlling the family-wise error rate at α. A pair is significantly different if their mean difference exceeds the HSD critical value: q × √(MSE / n), where q is from the studentized range distribution.
Why You Need a Post-Hoc Test After ANOVA
A significant ANOVA F-test answers one specific question: "Are all group means equal?" When the answer is no, ANOVA stops there. It does not tell you which groups differ or by how much — the F-test is an omnibus test of the null hypothesis H₀: μ₁ = μ₂ = ··· = μₖ.
Post-hoc tests exist to answer the follow-up questions that researchers actually care about. If three fertilizer treatments produce different crop yields on average, which pairs of fertilizers are genuinely different? Is treatment A better than B? Better than C? The answers require additional tests — and those tests must be designed to avoid inflating the Type I error rate through repeated testing.
The ANOVA → Tukey HSD Decision Flow
The gating rule is firm: if the overall ANOVA is not significant, you should not run Tukey HSD or any other post-hoc test. Doing so removes the protection that the omnibus F-test provides and inflates the probability of false discoveries.
Assumptions of the Tukey HSD Test
Because Tukey HSD is built on ANOVA's framework, it inherits the same underlying assumptions. Violating them can produce unreliable conclusions.
Independence of Observations
Each observation must be independent of every other. This is primarily a study design requirement — achieved through random sampling and random assignment to groups. Repeated measures designs violate this assumption and require a different approach (repeated measures ANOVA with its own post-hoc procedures).
Normality Within Groups
Each group's data should follow a normal distribution. With sample sizes above roughly 15–20 per group, the Central Limit Theorem provides adequate robustness against moderate departures from normality. Check with a normality test (Shapiro-Wilk, Q-Q plot) when group sizes are small.
Homogeneity of Variance (Homoscedasticity)
All groups should have approximately equal population variances. This is the most important assumption for Tukey HSD specifically. Test it with Levene's test or Bartlett's test before choosing between Tukey HSD (equal variances) and Games-Howell (unequal variances). Find the Levene's test critical values in the reference tables.
Significant ANOVA Result
The omnibus F-test from one-way ANOVA must be statistically significant (p < α) before running Tukey HSD. This is a procedural assumption, not a statistical one. Running post-hoc tests after a non-significant F-test inflates the Type I error rate and produces misleading findings.
Equal or Near-Equal Group Sample Sizes
The standard HSD formula assumes equal sample sizes per group (n). When group sizes differ slightly, the Tukey-Kramer adjustment — replacing n with the harmonic mean of group sizes — provides adequate coverage. For substantially unequal groups with unequal variances, Games-Howell is more appropriate.
The Tukey HSD Formula
The test computes a single critical difference value (HSD) against which every pairwise mean difference is evaluated. If a pair's absolute mean difference exceeds HSD, the pair is declared significantly different.
q = studentized range critical value
α = significance level (e.g., 0.05)
k = number of groups
dferror = N − k
MSE = mean square error from ANOVA
n = sample size per group
x̄ᵢ = mean of group i
x̄ⱼ = mean of group j
The Studentized Range Statistic (q)
The critical value q comes from the studentized range distribution, also called the q-distribution. Unlike the t-distribution, which describes the range of a single mean, the studentized range distribution models the range of the largest minus the smallest of k sample means drawn from populations with equal variances. Using q rather than a t-critical value is what gives Tukey HSD its FWER control.
You look up q in the Tukey q-table using three values: your significance level α, the number of groups k, and the error degrees of freedom df_error = N − k. A partial table is shown below for α = 0.05.
| dferror \ k | 2 | 3 | 4 | 5 | 6 | 8 | 10 |
|---|---|---|---|---|---|---|---|
| 5 | 3.64 | 4.60 | 5.22 | 5.67 | 6.03 | 6.63 | 7.08 |
| 6 | 3.46 | 4.34 | 4.90 | 5.30 | 5.63 | 6.18 | 6.58 |
| 8 | 3.26 | 4.04 | 4.53 | 4.89 | 5.17 | 5.64 | 6.00 |
| 10 | 3.15 | 3.88 | 4.33 | 4.65 | 4.91 | 5.35 | 5.67 |
| 12 | 3.08 | 3.77 | 4.20 | 4.51 | 4.75 | 5.17 | 5.46 |
| 20 | 2.95 | 3.58 | 3.96 | 4.23 | 4.45 | 4.82 | 5.08 |
| 30 | 2.89 | 3.49 | 3.85 | 4.10 | 4.30 | 4.64 | 4.89 |
| ∞ | 2.77 | 3.31 | 3.63 | 3.86 | 4.03 | 4.33 | 4.54 |
Critical values of q at α = 0.05 (two-tailed). Source: Tukey's Q Table. Download the full PDF version.
Tukey HSD Confidence Intervals
Beyond the yes/no significance decision, Tukey HSD also provides simultaneous confidence intervals for every pairwise mean difference. These show the size and direction of each difference, not just whether it clears a threshold.
The family of all Tukey HSD confidence intervals carries a simultaneous coverage guarantee: all intervals contain their true parameter values with probability 1 − α. This is stronger than the marginal guarantee from individual t-intervals, and it is one reason Tukey HSD is preferred over running separate t-tests for planned comparisons when all pairs are of interest.
The 5-Step Tukey HSD Procedure
Step 1: Verify ANOVA is significant. Step 2: Record MSE and df_error from the ANOVA table. Step 3: Look up q (α, k, df_error). Step 4: Calculate HSD = q × √(MSE/n). Step 5: Compare all |x̄ᵢ − x̄ⱼ| pairs to HSD and report.
Confirm the ANOVA F-test is Significant
Run one-way ANOVA. Check that the F-statistic yields p < α. The ANOVA table provides the values you'll need: MSE (also written MS_within or MS_error) and the error degrees of freedom (df_error = N − k, where N is the total sample size and k is the number of groups). Do not proceed if the F-test is not significant. Review the full one-way ANOVA guide for the ANOVA procedure.
Identify the Q Critical Value
Look up q in the studentized range table. You need three values: (1) your significance level α (usually 0.05), (2) the number of groups k, and (3) df_error = N − k. The table entry at (k, df_error) gives the critical q. Use the Tukey q-table or the calculator below. For unequal group sizes, use the Tukey-Kramer adjustment: replace n with the harmonic mean of n_i and n_j: n_h = 2 / (1/n_i + 1/n_j).
Calculate the HSD Critical Difference
Apply the formula: HSD = q × √(MSE / n). This is a single number. Every pair whose absolute mean difference exceeds this value will be declared significantly different. It scales with MSE (more error → wider threshold) and shrinks as n grows (more data → more precise means → smaller threshold needed).
Test All Pairwise Differences
Compute the absolute difference |x̄ᵢ − x̄ⱼ| for every pair. With k groups there are k(k−1)/2 pairs. Compare each difference to HSD. Record which pairs exceed HSD (significant) and which do not. Organize the results in a pairwise comparison matrix — typically a triangular table showing each mean difference and its significance status.
Report the Results
State which specific group pairs differ, report the mean difference and the 95% CI for each significant pair, note the q and HSD values, and mention that FWER was controlled at α. A standard APA-style report reads: "A one-way ANOVA revealed a significant effect of teaching method on test scores, F(2, 27) = 8.14, p = .002. Post-hoc comparisons using the Tukey HSD test indicated that the mean score for Method A (M = 85.3) was significantly higher than Method C (M = 74.1), p = .003, 95% CI [4.6, 17.8]."
Worked Examples
Example 1 — Teaching Methods and Test Scores
Problem: A researcher compares three teaching methods (A, B, C) on a statistics exam. Each group has n = 10 students. ANOVA is significant. Given the data below, which method pairs differ?
| Group | Method | n | Mean (x̄) | Variance |
|---|---|---|---|---|
| A | Lecture | 10 | 85.3 | 42.1 |
| B | Active Learning | 10 | 81.7 | 38.5 |
| C | Online Only | 10 | 74.1 | 51.3 |
From ANOVA output: F(2, 27) = 8.14, p = .002, MSE = 43.97, df_error = 27, k = 3, α = 0.05.
ANOVA confirmed significant: F(2, 27) = 8.14, p = .002 < .05. Proceed with Tukey HSD.
Look up q: α = 0.05, k = 3, df_error = 27. From the Tukey q-table, q₀.₀₅,₃,₂₇ ≈ 3.51 (interpolating between df = 20 and df = 30).
Calculate HSD:
√(MSE / n) = √(43.97 / 10) = √4.397 = 2.097
HSD = 3.51 × 2.097 = 7.36
Compare all pairs to HSD = 7.36:
| Pair | |Mean Diff| | vs HSD (7.36) | Result |
|---|---|---|---|
| A vs B | |85.3 − 81.7| = 3.6 | 3.6 < 7.36 | Not significant |
| A vs C | |85.3 − 74.1| = 11.2 | 11.2 > 7.36 | Significant ✓ |
| B vs C | |81.7 − 74.1| = 7.6 | 7.6 > 7.36 | Significant ✓ |
✅ Conclusion: Methods A and C differ significantly (diff = 11.2, p < .05). Methods B and C differ significantly (diff = 7.6, p < .05). Methods A and B do not differ significantly (diff = 3.6, p > .05). The Online Only method produced reliably lower scores than both Lecture and Active Learning.
Reporting the 95% CI for A vs C: (11.2) ± 7.36 → [3.84, 18.56]. Since this interval excludes 0, the difference is significant — and it estimates the true difference as somewhere between 3.8 and 18.6 points.
Example 2 — Drug Dosage on Blood Pressure
Problem: A pharmacologist tests four dosage levels of a blood pressure drug (Placebo, Low, Medium, High) on n = 8 patients each. ANOVA gives F(3, 28) = 6.42, p = .002, MSE = 18.4. Which dosages differ significantly?
| Group | n | Mean BP Reduction (mmHg) |
|---|---|---|
| Placebo | 8 | 2.1 |
| Low Dose | 8 | 5.8 |
| Medium Dose | 8 | 9.4 |
| High Dose | 8 | 13.2 |
ANOVA confirmed significant: F(3, 28) = 6.42, p = .002. Proceed.
Look up q: α = 0.05, k = 4, df_error = 28. q₀.₀₅,₄,₂₈ ≈ 3.86 (interpolating from table; values at df=20: 3.96, df=30: 3.85).
Calculate HSD:
√(MSE / n) = √(18.4 / 8) = √2.30 = 1.517
HSD = 3.86 × 1.517 = 5.86
Compare all C(4,2) = 6 pairs to HSD = 5.86:
| Pair | |Diff| | Significant? |
|---|---|---|
| Placebo vs Low | 3.7 | No (3.7 < 5.86) |
| Placebo vs Medium | 7.3 | Yes ✓ (7.3 > 5.86) |
| Placebo vs High | 11.1 | Yes ✓ (11.1 > 5.86) |
| Low vs Medium | 3.6 | No (3.6 < 5.86) |
| Low vs High | 7.4 | Yes ✓ (7.4 > 5.86) |
| Medium vs High | 3.8 | No (3.8 < 5.86) |
✅ Conclusion: The Medium and High doses both produce significantly greater BP reduction than Placebo. High dose also significantly outperforms Low dose. Adjacent dosage levels (Placebo-Low, Low-Medium, Medium-High) did not differ significantly from each other, suggesting a stepped dose-response pattern.
Example 3 — Fertilizer on Crop Yield
Problem: An agronomist tests five fertilizer types on crop yield (kg/plot), with n = 6 plots per type. ANOVA: F(4, 25) = 9.31, p < .001, MSE = 12.6. Group means: F1 = 28.4, F2 = 34.1, F3 = 41.7, F4 = 38.9, F5 = 29.8. Which fertilizers differ?
ANOVA confirmed: F(4, 25) = 9.31, p < .001. Proceed.
Look up q: α = 0.05, k = 5, df_error = 25. q₀.₀₅,₅,₂₅ ≈ 4.17 (interpolating between df=20: 4.23 and df=30: 4.10).
Calculate HSD:
√(MSE / n) = √(12.6 / 6) = √2.10 = 1.449
HSD = 4.17 × 1.449 = 6.04
Key pairwise results (C(5,2) = 10 pairs total):
| Comparison | |Diff| | Sig? |
|---|---|---|
| F3 vs F1 | 13.3 | Yes ✓ |
| F3 vs F5 | 11.9 | Yes ✓ |
| F4 vs F1 | 10.5 | Yes ✓ |
| F4 vs F5 | 9.1 | Yes ✓ |
| F2 vs F1 | 5.7 | No |
| F3 vs F4 | 2.8 | No |
| F2 vs F5 | 4.3 | No |
✅ Conclusion: Fertilizers F3 and F4 produce significantly higher yields than F1 and F5. F3 and F4 do not differ from each other or from F2. F1 and F5 are similar to each other and to F2 but consistently outperformed by the top two fertilizers. The practical recommendation is to use F3 or F4.
Running Tukey HSD in Statistical Software
Modern software performs Tukey HSD automatically. The output format varies, but every package produces the same three pieces of information for each pair: the mean difference, the adjusted p-value, and a 95% confidence interval.
R
TukeyHSD(aov(score ~ method, data)). Output gives diff, lwr, upr, p adj for each pair. Use plot(TukeyHSD(...)) for a visual CI plot.
Python (statsmodels)
from statsmodels.stats.multicomp import pairwise_tukeyhsdpairwise_tukeyhsd(data, groups). Returns a table with meandiff, p-adj, lower, upper, reject.
SPSS
Analyze → Compare Means → One-Way ANOVA → Post Hoc → Tukey. Output appears in the "Multiple Comparisons" table with Sig. column showing adjusted p-values.
SAS
PROC GLM; CLASS group; MODEL score=group; MEANS group / TUKEY; RUN; Output includes grouping letters (a, b, ab) showing which groups share a significance homogeneous subset.
Minitab
Stat → ANOVA → One-Way → Comparisons → Tukey. Produces individual CIs, grouping information, and a comparison of means chart.
Excel
Excel's Data Analysis ToolPak does not include Tukey HSD natively. Use the calculator below or import output from R or Python for the Tukey step.
How to Read Tukey HSD Output
All software output shares the same logic. For each pair (i, j), you get a mean difference (x̄ᵢ − x̄ⱼ) and an adjusted p-value. The adjusted p-value already accounts for all the comparisons you're running — you compare it directly to your original α without any further correction. A confidence interval that excludes zero corresponds exactly to a significant adjusted p-value.
The "p adj" column in Tukey HSD output is already adjusted for multiple comparisons. Do not apply an additional Bonferroni correction to it. That would be double-correcting and would make your test far too conservative, missing real effects.
Tukey HSD vs Other Post-Hoc Tests
Tukey HSD is the most widely used post-hoc test for all-pairs comparisons with equal variances, but it is not the only option. The right choice depends on your research design, variance structure, and what comparisons matter.
| Feature | Tukey HSD | Bonferroni | Scheffé |
|---|---|---|---|
| Best for | All pairwise comparisons | Specific, pre-planned comparisons | Complex contrasts, unplanned |
| Error control | FWER at α (exact) | FWER ≤ α (conservative) | FWER at α (most conservative) |
| Statistical power | High (for pairwise) | Moderate | Lowest of the three |
| Requires equal n? | No (Tukey-Kramer adjustment) | No | No |
| Requires equal variance? | Yes | Technically yes | Yes |
| Handles complex contrasts? | No | Yes | Yes |
| Use when | Comparing all pairs, equal variance | Few pre-specified comparisons | All contrasts, maximum protection |
| Feature | Tukey HSD | Games-Howell | Dunnett |
|---|---|---|---|
| Best for | All pairwise, equal variances | All pairwise, unequal variances | Each group vs a control only |
| Variance assumption | Homogeneous required | Heterogeneous OK | Homogeneous required |
| Sample sizes | Equal preferred | Handles unequal | Equal preferred |
| Power vs Tukey HSD | — | Similar | Higher (fewer comparisons) |
| Use when | Standard all-pairs analysis | Levene's test is significant | Only comparing to a reference group |
For three groups (3 pairs), Tukey HSD and Bonferroni give very similar results. As the number of groups increases, the gap widens: Tukey HSD becomes meaningfully more powerful because the Bonferroni correction divides α by the number of comparisons while Tukey's critical value grows more slowly. For 10 groups (45 pairs), Bonferroni is noticeably more conservative. See the full Bonferroni correction guide for details.
Where Tukey HSD Is Used in Practice
Clinical Trials
Comparing mean outcomes across multiple treatment arms — e.g., three drug doses vs placebo. Required reporting standard in many pharmaceutical journals.
Agriculture
Evaluating crop yields across fertilizer types or irrigation methods. Tukey HSD appears routinely in agricultural experiment reports published by university extension programs.
Education Research
Comparing student outcomes across instructional methods, school types, or curriculum designs. A standard part of psychology and education graduate statistics courses.
Quality Engineering
Identifying which manufacturing process settings produce different mean defect rates or dimensions. Used in Six Sigma and design-of-experiments contexts.
Psychology
Behavioral experiments with multiple conditions (e.g., cognitive load levels, social primes). Required by APA publication guidelines when post-hoc tests are performed.
Marketing Analytics
A/B/n testing with more than two variants — comparing conversion rates or purchase amounts across multiple ad versions, pricing tiers, or email subject lines.
Real-World Case Study
Pain Management: Three Physical Therapy Protocols
A rehabilitation hospital assigned 45 chronic pain patients (n = 15 per group) to three physical therapy protocols: Standard Care, Targeted Exercise, and Combined Modality. After 8 weeks, pain scores (0–10 scale) were: Standard = 5.8, Targeted = 4.2, Combined = 3.1.
One-way ANOVA yielded F(2, 42) = 12.7, p < .001, MSE = 1.84. With q₀.₀₅,₃,₄₂ ≈ 3.43 and HSD = 3.43 × √(1.84/15) = 3.43 × 0.350 = 1.20, the pairwise comparisons revealed: Standard vs Combined (diff = 2.7, sig), Targeted vs Combined (diff = 1.1, ns), Standard vs Targeted (diff = 1.6, sig). The Combined protocol outperformed Standard Care, but its advantage over Targeted Exercise alone was not statistically detectable with 15 patients per group — a finding relevant to both clinical decision-making and power planning for future trials.
This example uses synthetic data to illustrate the method. For real clinical trial reporting standards, see the CONSORT reporting guidelines.
Interactive Tukey HSD Calculator
Enter the values from your ANOVA output and group means below. The calculator runs the full Tukey HSD procedure: it computes HSD, tests every pair, and reports which groups differ significantly at your chosen α.
Tukey HSD Calculator
Common Mistakes When Using Tukey HSD
| Mistake | What Goes Wrong | Correct Approach |
|---|---|---|
| Running Tukey HSD before checking ANOVA significance | Inflates overall Type I error rate; results may be meaningless | Always confirm F-test p < α first |
| Using Tukey HSD with unequal variances | Under-estimates variance for large-variance groups; inflates false positives | Run Levene's test first; switch to Games-Howell if violated |
| Applying extra Bonferroni correction to adjusted p-values | Double-corrects; makes test far too conservative, misses real effects | Compare p adj directly to α; no additional correction needed |
| Using raw q-statistic from the test as a t-statistic | Wrong distribution; incorrect degrees of freedom | Report q with its df pair: q(k, df_error) |
| Running Tukey HSD on two groups (k = 2) | Tukey HSD reduces to a t-test when k = 2; use a t-test directly | Post-hoc tests apply only when k ≥ 3 |
| Ignoring effect size when reporting significance | Significant p-value with trivial effect size misleads practical decisions | Report Cohen's d or η² alongside p-values |
Key Terms Glossary
| Term | Symbol | Definition |
|---|---|---|
| Tukey HSD | — | Honestly Significant Difference test; post-hoc pairwise comparison after ANOVA controlling FWER at α |
| Family-Wise Error Rate | FWER | Probability of making at least one Type I error across all comparisons in a family of tests |
| Studentized Range Statistic | q | Critical value from the studentized range distribution, indexed by k and df_error, used to set the Tukey HSD threshold |
| Mean Square Error | MSE | Within-groups variance estimate from ANOVA; the pooled estimate of σ² across all groups |
| Error Degrees of Freedom | df_error | N − k, where N is total sample size and k is the number of groups |
| Post-Hoc Test | — | A statistical test conducted after a significant omnibus test to identify specific differences; "after the fact" |
| Pairwise Comparison | — | A direct statistical comparison between two specific group means |
| Omnibus Test | — | A single test of the overall null hypothesis that all group means are equal (i.e., the ANOVA F-test) |
| ANOVA | F | Analysis of Variance; partitions total variance into between-group and within-group components to test for mean differences |
| Tukey-Kramer Adjustment | — | Modified HSD formula for unequal sample sizes, replacing n with the harmonic mean of the two group sizes being compared |
| Games-Howell Test | — | Alternative to Tukey HSD when the homogeneity of variance assumption is violated |
| Bonferroni Correction | α/m | Controls FWER by dividing α by the number of comparisons m; more conservative than Tukey HSD for all-pairs comparisons |
| Type I Error | α | False positive — incorrectly rejecting a true null hypothesis |
| Homogeneity of Variance | — | The assumption that all groups have equal population variances; tested with Levene's or Bartlett's test |
Frequently Asked Questions
Related Guides and Resources
The Tukey HSD test sits within a broader framework of hypothesis testing and analysis of variance. These pages from Statistics Fundamentals provide the supporting knowledge you need.
One-Way ANOVA
The complete guide to running and interpreting one-way ANOVA — the required first step before Tukey HSD.
ANOVA Calculator
Enter raw data and get a complete ANOVA table with F-statistic, p-value, and MSE — ready to plug into Tukey HSD.
Bonferroni Correction
When Bonferroni is the right choice over Tukey HSD — and how to apply it correctly for pre-specified comparisons.
Tukey Q Critical Value Table
The complete studentized range distribution table at α = 0.05 and α = 0.01. Download as PDF.
Understanding P-Values
What adjusted p-values mean in post-hoc testing and why the Tukey p-value is directly comparable to α.
Normality Tests
How to check the normality assumption before running ANOVA and Tukey HSD — Shapiro-Wilk, Q-Q plots, and more.
Tukey, J.W. (1949). Comparing individual means in the analysis of variance. Biometrics, 5(2), 99–114. | NIST Engineering Statistics Handbook | SciPy pairwise_tukeyhsd documentation | Montgomery, D.C. (2017). Design and Analysis of Experiments, 9th Ed. Wiley. | R multcomp package