What is hypothesis testing in statistics?

Hypothesis testing is a statistical procedure used to decide whether sample data provides enough evidence to reject a claim about a population. It compares an observed result against what would be expected by chance alone, using probability to guide the decision.

What is the p-value in hypothesis testing?

The p-value is the probability of obtaining a test statistic at least as extreme as the one observed, assuming the null hypothesis is true. A p-value ≤ 0.05 is typically used as the threshold to reject the null hypothesis.

What are Type I and Type II errors in hypothesis testing?

A Type I error (false positive) occurs when you reject a true null hypothesis. Its probability equals α. A Type II error (false negative) occurs when you fail to reject a false null hypothesis. Its probability is β, and statistical power equals 1 − β.

What is the difference between a one-tailed and two-tailed hypothesis test?

A two-tailed test checks for a difference in either direction (H₁: μ ≠ μ₀). A one-tailed test checks for a difference in one specific direction — either greater than (right-tailed) or less than (left-tailed). Two-tailed tests are more conservative and more commonly used.

What is statistical power in hypothesis testing?

Statistical power (1 − β) is the probability that a test correctly rejects a false null hypothesis — i.e., detects a real effect when one exists. A power of 0.80 (80%) is the conventional minimum. Power increases with larger sample sizes, larger effect sizes, and higher α levels.

How do you choose between a z-test and a t-test?

Use a z-test when the population standard deviation is known and the sample is large (n ≥ 30). Use a t-test when the population standard deviation is unknown and must be estimated from the sample — which is the case in most real-world research.

What is Tukey's HSD test?

Tukey's Honestly Significant Difference (HSD) test is a post-hoc test run after a significant ANOVA result. It compares all possible pairs of group means to determine which specific pairs are statistically different, while controlling the family-wise error rate. The HSD formula is: HSD = q × √(MSW / n), where q is the studentized range statistic, MSW is the mean square within groups, and n is the group sample size.

What is the difference between statistical significance and practical significance?

Statistical significance (p ≤ α) means the result is unlikely under the null hypothesis. Practical significance means the effect is large enough to matter in the real world. With large samples, even a tiny, trivial difference can be statistically significant. Always report an effect size (Cohen's d, η², r) alongside the p-value to convey practical importance.

Hypothesis Testing: 5 Steps, P-Value, Types & Worked Examples

Q: What is the p-value formula?

For a z-test, the two-tailed p-value = 2 × P(Z > |z|), where Z is the standard normal variable. For a t-test, p = 2 × P(T > |t|) with degrees of freedom = n − 1. In Python: from scipy import stats; p = 2*(1 - stats.norm.cdf(abs(z))) for a z-test, or stats.t.sf(abs(t), df)*2 for a t-test.

Q: What is the difference between hypothesis testing and confidence intervals?

Hypothesis testing gives a binary decision (reject or fail to reject H₀). A confidence interval gives a range of plausible values for the population parameter. They are mathematically equivalent: if a 95% CI excludes the null value, the two-tailed test at α = 0.05 will reject H₀.

What Is Hypothesis Testing?

Hypothesis testing is a formal statistical procedure for evaluating whether sample data supports or contradicts a claim about a population. You start with a specific claim — the null hypothesis — and use probability to decide whether the data is consistent with that claim or whether the evidence against it is strong enough to reject it.

Think of it like a trial. The null hypothesis is the presumption of innocence: the defendant is innocent until the evidence proves otherwise beyond a reasonable doubt. In hypothesis testing, "beyond a reasonable doubt" is defined by a probability threshold called the significance level.

💡

The Core Question

Every hypothesis test answers the same question: "If the null hypothesis were true, how likely is it that we would observe data at least this extreme?" The p-value gives that probability. The reference distribution — the probability distribution of the test statistic under H₀ — is what makes this calculation possible.

🔑 Key Takeaways

The most important ideas every student and analyst should understand before running a hypothesis test.

●

The null hypothesis is always what you test. You never directly test the alternative hypothesis — you test whether the null is consistent with your data.

●

You never "accept" the null hypothesis. You either reject it or "fail to reject" it — absence of evidence is not evidence of absence.

●

Statistical significance is not practical significance. A result can be statistically significant with a tiny, meaningless effect size when the sample is very large. Always report effect size alongside the p-value.

●

Set α before collecting data. Choosing the significance level after seeing the results — "p-hacking" — inflates false positive rates.

●

The test statistic quantifies the gap between observed and expected. The larger it is, the less likely the data fits the null hypothesis.

●

Choosing the wrong test type invalidates results. The choice between z, t, chi-square, and ANOVA depends on data type, sample size, and number of groups.

Null and Alternative Hypotheses

Every hypothesis test begins with two competing statements about a population parameter.

The Null Hypothesis (H₀)

The null hypothesis is the default claim — it asserts that there is no effect, no difference, or no relationship in the population. It is the statement you are testing. Common formulations:

H₀: μ = μ₀ (the population mean equals a specific value)
H₀: μ₁ = μ₂ (two group means are equal)
H₀: p = p₀ (a proportion equals a specific value)
H₀: variables X and Y are independent

⚠️

Critical wording distinction

Say "fail to reject H₀," never "accept H₀." When a test finds no significant result, it means the data didn't provide enough evidence to rule out the null — not that the null is proven true. The difference matters in research and in exams.

The Alternative Hypothesis (H₁ or Hₐ)

The alternative hypothesis is the research claim — the effect or difference you expect to find. It is what you gather evidence for. It must be the logical complement of H₀, and it can be directional or non-directional:

⬅️➡️

Two-tailed

H₁: μ ≠ μ₀
Tests for any difference, in either direction. Use this as your default unless you have a strong directional prediction made before data collection.

➡️

Right-tailed

H₁: μ > μ₀
Tests specifically for an increase above the null value.

⬅️

Left-tailed

H₁: μ < μ₀
Tests specifically for a decrease below the null value.

📐

One-tailed vs. two-tailed: the practical difference

A two-tailed test at α = 0.05 splits the rejection region between both tails (2.5% each), requiring a critical z of ±1.96. A one-tailed test concentrates all 5% in one tail, requiring only z = 1.645. One-tailed tests are more powerful in the predicted direction but give no protection against an effect in the opposite direction — which is why two-tailed tests are the standard default.

→ Deep dive: Null and Alternative Hypotheses → Deep dive: One- vs. Two-Tailed Tests

Real-World Example

Testing a New Blood Pressure Drug

A pharmaceutical company wants to know whether a new drug reduces systolic blood pressure compared to a placebo. The hypotheses are:

H₀: The drug has no effect on blood pressure (μ_drug = μ_placebo)
H₁: The drug reduces blood pressure (μ_drug < μ_placebo) — left-tailed

A two-sample t-test is run. If p < 0.05, the evidence is strong enough to reject H₀ and conclude the drug has a statistically significant effect.

The 5 Steps of Hypothesis Testing

Every hypothesis test — regardless of the test statistic used — follows the same five-step procedure. The arithmetic changes; the logic does not.

State the null and alternative hypotheses

Write H₀ (the claim to be tested, containing equality) and H₁ (the research claim, specifying direction if appropriate). Both must be stated before looking at any data.

Choose the significance level (α)

Set α before data collection — typically 0.05 for most research, 0.01 for medical or safety-critical decisions. This is the maximum tolerable probability of a false positive (Type I error).

Select the appropriate test and compute the test statistic

Choose the test based on data type, number of groups, and whether population parameters are known (see Types of Tests below). Calculate the test statistic (z, t, F, or χ²) from your sample data using the relevant formula.

Find the p-value or compare to the critical value

P-value approach: Calculate the probability of observing a test statistic at least as extreme as yours, assuming H₀ is true. If p ≤ α, reject H₀.
Critical value approach: Find the boundary value at α from statistical tables. If your test statistic exceeds it, reject H₀.

Draw a conclusion in context — and report effect size

State whether you reject or fail to reject H₀, then translate that decision into a plain-language conclusion about the original research question. Report the test statistic, degrees of freedom, p-value, and an effect size measure (Cohen's d, η², or r) in your write-up.

The P-value Explained

The p-value is the probability of obtaining a test statistic at least as extreme as the one calculated from your sample, assuming the null hypothesis is true. It is the most widely reported measure of evidence against H₀.

The P-value Formula

The p-value is derived from the reference distribution — the probability distribution of the test statistic under H₀. For each test type:

P-value formulas by test type

Two-tailed z-test:  p = 2 × P(Z > |z|)           where Z ~ N(0,1)
One-tailed (right): p = P(Z > z)
One-tailed (left):  p = P(Z < z)

Two-tailed t-test:  p = 2 × P(T > |t|)  with df = n − 1
Chi-square:         p = P(χ² > χ²_observed)  with df = (r−1)(c−1)
F-test (ANOVA):     p = P(F > F_observed)  with df₁, df₂

Python: calculating p-values with scipy.stats

from scipy import stats

# Two-tailed z-test p-value
z = 1.848
p_z = 2 * (1 - stats.norm.cdf(abs(z)))   # → 0.0647

# Two-tailed t-test p-value
t, df = 1.848, 24
p_t = stats.t.sf(abs(t), df) * 2           # → 0.077

# Chi-square p-value
chi2_stat, df_chi = 12.0, 1
p_chi = 1 - stats.chi2.cdf(chi2_stat, df_chi)  # → 0.0005

How to Interpret the P-value

P-value	Evidence Against H₀	Decision at α = 0.05
p > 0.10	Weak or none	Fail to reject H₀
0.05 < p ≤ 0.10	Marginal / suggestive	Fail to reject H₀
0.01 < p ≤ 0.05	Moderate	Reject H₀
0.001 < p ≤ 0.01	Strong	Reject H₀
p ≤ 0.001	Very strong	Reject H₀

⚠️

What the p-value does NOT mean

The p-value is not the probability that H₀ is true. It is not the probability that the result was due to chance. And p < 0.05 does not prove that your hypothesis is correct — it only means the data is unlikely under the null. Effect size and confidence intervals carry equally important information.

Significance Level (α) vs P-value

The significance level α is chosen before the test. The p-value is calculated after the test. The decision rule is simple: if the p-value falls at or below α, reject H₀. If not, fail to reject it.

0.05

Standard α in most research disciplines

0.01

Stricter α used in medical and safety contexts

5 × 10⁻⁷

α threshold used in particle physics (five-sigma rule)

→ Deep dive: P-values in Detail → Deep dive: Significance Level

Test Statistic and Critical Value

The test statistic is a single number calculated from sample data that measures how far the observed result deviates from what H₀ predicts. The further it falls from the center of the null distribution, the stronger the evidence against H₀.

General form of a test statistic Test Statistic = (Observed Value − Null Value) ÷ Standard Error

The critical value is the threshold the test statistic must exceed to reject H₀ at the chosen α level. It is read from a statistical table (z-table, t-table, chi-square table, or F-table) based on α and the degrees of freedom — the number of values in the data that are free to vary once certain constraints are applied.

Test Statistic	Distribution Used	Common Use Case	Key Input
Z	Standard normal	Mean or proportion, large n or known σ	Population σ known
t	t-distribution	Mean, small n or unknown σ	Degrees of freedom = n − 1
χ² (chi-square)	Chi-square distribution	Categorical data, goodness-of-fit, independence	df = (rows−1)(cols−1)
F	F-distribution	Comparing 3+ group means (ANOVA), variances	df between and within groups

For one- and two-tailed z-tests, the most commonly referenced critical values at α = 0.05 are ±1.96 (two-tailed) and 1.645 (one-tailed right) or −1.645 (one-tailed left). See the Z-table and T-distribution table for full critical value reference.

→ Deep dive: Degrees of Freedom → Deep dive: Decision Rule

Types of Hypothesis Tests

Choosing the right test is as important as running it correctly. The decision depends on: how many groups you are comparing, the data type (continuous vs. categorical), whether the population variance is known, and whether the samples are independent.

Z-test

Used to test a population mean or proportion when the population standard deviation (σ) is known, or when the sample size is large enough (n ≥ 30) for the central limit theorem to apply.

Z-test for a mean z = (x̄ − μ₀) ÷ (σ ÷ √n)

→ Deep dive: One-Sample Z-test → Z-test vs. T-test

T-test

The most common test for comparing means. Used when the population standard deviation is unknown and estimated from the sample. Three main forms:

One-sample t-test

When to use

Comparing a single sample mean to a known or hypothesized population value. Example: does the average delivery time differ from the advertised 5 days?

Two-sample (independent) t-test

When to use

Comparing means from two separate, independent groups. Example: do men and women differ in average resting heart rate?

Paired t-test

When to use

Comparing means from matched pairs or the same subjects measured twice. Example: does a training program improve test scores (before vs. after)?

One-sample t-test statistic t = (x̄ − μ₀) ÷ (s ÷ √n) df = n − 1

→ One-sample vs. two-sample t-test → Equal vs. unequal variance

Chi-Square Test

Used for categorical data. Two main applications: the goodness-of-fit test (does an observed frequency distribution match an expected one?) and the test of independence (are two categorical variables related?).

Chi-square test statistic χ² = Σ [(Observed − Expected)² ÷ Expected] df = (rows − 1)(cols − 1)

The null hypothesis for a chi-square test of independence states that the two variables are not related. The null hypothesis for a goodness-of-fit test states that the observed frequencies match the expected distribution. Reference the chi-square table for critical values.

→ Deep dive: Chi-Square Test

ANOVA (Analysis of Variance)

ANOVA tests whether three or more group means differ, using the F-statistic. It partitions total variance into between-group variance and within-group variance. A significant F-statistic tells you that at least one group differs — but it does not tell you which pairs differ. That requires a post-hoc test.

F-statistic for one-way ANOVA

F = Mean Square Between (MSB) ÷ Mean Square Within (MSW)

MSB = SS_between ÷ df_between     df_between = k − 1
MSW = SS_within  ÷ df_within      df_within  = N − k

The null hypothesis for ANOVA is H₀: μ₁ = μ₂ = ... = μₖ (all group means are equal). See the F-table and the F-distribution guide for critical values.

Tukey's HSD Post-Hoc Test

🔍 When ANOVA is significant — which groups actually differ?

A significant ANOVA result only confirms that at least one group mean is different. To identify which specific pairs of groups differ while controlling the family-wise error rate, run a post-hoc test. Tukey's Honestly Significant Difference (HSD) is the most widely used choice when group sizes are equal or approximately equal.

Tukey's HSD formula

HSD = q × √(MSW ÷ n)

where: q   = studentized range statistic (from Tukey's q-table, df_within, k groups)
       MSW = mean square within (from the ANOVA table)
       n   = sample size per group

Two group means are significantly different if their absolute difference exceeds the HSD value. Tukey's test is more powerful than Bonferroni when you need all pairwise comparisons, and more conservative than Fisher's LSD to guard against inflated false positives.

Example: An ANOVA comparing test scores across three teaching methods (A, B, C) yields F(2, 27) = 5.4, p = 0.011. Tukey HSD reveals Method A vs. C is significant (p = 0.009) while A vs. B and B vs. C are not — the significant F was driven entirely by the A–C gap.

→ Deep dive: Tukey HSD Test (with worked example and q-table) → Deep dive: ANOVA

Proportion Test (Z-test for Proportions)

Tests whether a sample proportion differs from a hypothesized value, or whether two sample proportions differ from each other. Based on the normal approximation to the binomial, valid when np ≥ 5 and n(1−p) ≥ 5.

Z-test for one proportion z = (p̂ − p₀) ÷ √[p₀(1 − p₀) ÷ n]

→ Deep dive: Proportion Hypothesis Testing

Which Test Should You Use?

Scenario	Data Type	Groups	Use This Test
Compare one mean to a value, σ known or n ≥ 30	Continuous	1	Z-test
Compare one mean to a value, σ unknown	Continuous	1	One-sample t-test
Compare two independent group means	Continuous	2	Two-sample t-test
Compare before/after or matched pairs	Continuous	2 (matched)	Paired t-test
Compare 3+ group means	Continuous	3+	One-way ANOVA
Identify which pairs differ after significant ANOVA	Continuous	3+	Tukey's HSD (post-hoc)
Test association between two categorical variables	Categorical	—	Chi-square independence test
Test whether frequencies match a distribution	Categorical	—	Chi-square goodness-of-fit
Compare one proportion to a value	Proportion	1	One-sample z-test for proportions
Compare two proportions	Proportion	2	Two-sample z-test for proportions

→ Interactive: Statistical Test Selector

Type I and Type II Errors

Because hypothesis tests use probability, they can lead to two kinds of incorrect conclusions. Understanding these errors — and what controls them — is central to interpreting statistical results correctly.

A quick way to keep them straight: Type I is a false alarm (you cried wolf when there was none). Type II is a missed signal (the wolf came but you didn't notice).

Type I Error (False Positive)

Rejecting H₀ when H₀ is actually true. You conclude there is an effect, but in reality there isn't one.

Probability: Equal to the significance level α.

Example: Concluding a drug works when the improvement was due to chance.

Controlled by setting α before the test

Type II Error (False Negative)

Failing to reject H₀ when H₀ is actually false. You miss a real effect because the evidence wasn't strong enough to detect it.

Probability: Called β. Reduced by increasing sample size and effect size.

Example: Concluding a drug doesn't work when it actually does — the study was just underpowered.

Controlled by increasing statistical power (1 − β)

Decision \ Reality	H₀ is True	H₀ is False
Reject H₀	Type I Error (α)	Correct (Power = 1 − β)
Fail to reject H₀	Correct (1 − α)	Type II Error (β)

⚖️

The Error Trade-off

Reducing α (making it harder to reject H₀) decreases Type I errors but increases Type II errors, because a stricter threshold makes it harder to detect real effects. The two error types are inversely related at a fixed sample size. The solution is to increase sample size — this reduces both errors simultaneously.

→ Deep dive: Type I and Type II Errors

Statistical Power of a Hypothesis Test

Statistical power is the probability that a test correctly rejects a false null hypothesis. It equals 1 − β. A power of 0.80 means an 80% chance of detecting a real effect when one exists — the conventional minimum in behavioral and social research.

Statistical power formula

Power = 1 − β = P(Reject H₀ | H₀ is false)

For a one-sample z-test:
Power = P(Z > z_crit − δ) + P(Z < −z_crit − δ)   [two-tailed]

where δ = (μ₁ − μ₀) ÷ (σ ÷ √n)  is the non-centrality parameter

Power must be calculated before data collection, not after. Running a study without a power calculation risks wasting resources on a test too weak to detect the effect of interest. A "non-significant" result in an underpowered study cannot be interpreted as evidence of no effect.

📏

Sample size

The single most controllable driver of power. Doubling n substantially increases power. Calculate the minimum n required before starting.

📐

Effect size

Larger true differences are easier to detect. Small effects require much larger samples. Use Cohen's d for means, r for correlations, and Cohen's w for chi-square tests.

🎯

Significance level (α)

A higher α (e.g., 0.10 vs. 0.05) increases power but also raises the false positive rate. This tradeoff must be justified by the context.

📊

Variability

Less variability in the outcome makes it easier to detect true effects. Tighter measurement instruments and standardized conditions improve power.

0.80

Conventional minimum power threshold in most research fields

>50%

Type II error rate in many underpowered clinical trials

4×

Approximate increase in n needed to halve the effect size detectable

→ Deep dive: Power of a Test → Sample Size Calculator

Statistical Significance vs. Practical Significance

A statistically significant result only means the observed effect is unlikely to have arisen by chance. It says nothing about whether the effect is large enough to matter. With a sufficiently large sample, even a trivially small difference will yield p < 0.05.

Statistical Significance

Determined by the p-value relative to α. A result is statistically significant if p ≤ α — meaning the data is unlikely under H₀.

Driven by: Sample size, effect size, variability, and α level.

Can be misleading when: n is very large — even a 0.3-point difference on a 100-point scale becomes "significant."

Practical Significance

Determined by effect size — the magnitude of the difference in real-world units. An effect can be statistically significant but practically irrelevant.

Measured by: Cohen's d (means), η² (ANOVA), r (correlations), odds ratio (proportions).

Cohen's d benchmarks: Small ≈ 0.2, Medium ≈ 0.5, Large ≈ 0.8.

Illustrative Example

When statistical significance misleads

A study of 50,000 employees finds that workers who drink coffee score 0.4 points higher on a 100-point productivity scale than those who don't (p < 0.0001). The result is highly statistically significant, but Cohen's d = 0.04 — a negligible effect. No manager should restructure the office coffee policy on this basis. Always ask: even if this is real, is it large enough to act on?

✅

Best Practice

Report the test result (p-value, test statistic, df), the effect size (Cohen's d, η², or r), and the confidence interval. Together these tell readers whether an effect was detected, how large it is, and how precisely it has been estimated — far more informative than a p-value alone.

→ Deep dive: Effect Size → Deep dive: Cohen's d

Confidence Intervals and Hypothesis Testing

Hypothesis tests and confidence intervals are two sides of the same coin — they draw on the same mathematical framework and reach equivalent conclusions.

A 95% confidence interval gives the range of population parameter values consistent with your sample at α = 0.05. The hypothesis test at α = 0.05 asks whether the null value falls inside that range:

If the null hypothesis value falls outside the 95% CI → the two-tailed test rejects H₀ at α = 0.05.
If the null hypothesis value falls inside the 95% CI → the test fails to reject H₀ at α = 0.05.

Feature	Hypothesis Test	Confidence Interval
Output	Binary: reject or fail to reject H₀	Range of plausible parameter values
Information provided	Whether an effect exists	Direction and magnitude of the effect
Linked to α	Directly — α sets the decision boundary	Confidence level = 1 − α (e.g., 95% CI for α = 0.05)
Preferred when	Simple yes/no decision needed	Effect size and precision are important
Equivalent?	Yes — at α = 0.05, a two-tailed test and 95% CI always agree on the reject/fail-to-reject decision

→ Deep dive: Confidence Intervals

Hypothesis Testing: Worked Examples

Example 1: One-Sample T-test

Worked Example

Do students score differently from the national average?

A school reports that the national average on a standardized exam is 75 points. A sample of 25 students from one school scored a mean of 78.4 with a standard deviation of 9.2. Test at α = 0.05 whether this school's students perform differently from the national average.

State hypotheses: H₀: μ = 75 (school mean equals national average) | H₁: μ ≠ 75 (two-tailed, because "differently" means either direction)

Significance level: α = 0.05. Two-tailed test → each tail = 0.025.

Calculate t: t = (78.4 − 75) ÷ (9.2 ÷ √25) = 3.4 ÷ 1.84 = 1.848. Degrees of freedom = 25 − 1 = 24.

Critical value: From the t-distribution table at df = 24 and α/2 = 0.025, the critical value is ±2.064. Since |1.848| < 2.064, the test statistic does not fall in the rejection region. The p-value ≈ 0.077.

Decision: p = 0.077 > α = 0.05. Fail to reject H₀. Effect size: Cohen's d = (78.4 − 75) ÷ 9.2 = 0.37 (small-to-medium).

At the 5% significance level, there is not enough evidence to conclude that students at this school perform differently from the national average (t(24) = 1.848, p = 0.077, d = 0.37). Note: the effect size is not negligible — a larger sample might detect a real difference.

Python implementation

from scipy import stats
import numpy as np

sample_mean = 78.4
pop_mean    = 75
std_dev     = 9.2
n           = 25

t_stat = (sample_mean - pop_mean) / (std_dev / np.sqrt(n))
p_val  = stats.t.sf(abs(t_stat), df=n-1) * 2    # two-tailed
cohens_d = (sample_mean - pop_mean) / std_dev

print(f"t = {t_stat:.3f}, p = {p_val:.3f}, d = {cohens_d:.2f}")
# → t = 1.848, p = 0.077, d = 0.37  (fail to reject H₀)

Example 2: Chi-Square Test of Independence

Worked Example

Is there an association between exercise frequency and sleep quality?

A survey of 200 adults cross-tabulates exercise frequency (regular / irregular) against reported sleep quality (good / poor). The observed counts are: regular/good = 72, regular/poor = 28, irregular/good = 48, irregular/poor = 52. Test at α = 0.05 whether exercise and sleep quality are independent.

State hypotheses: H₀: Exercise frequency and sleep quality are independent. H₁: They are not independent.

Significance level: α = 0.05. df = (2−1)(2−1) = 1. Critical value from the chi-square table = 3.841.

Calculate expected frequencies: Row totals: regular = 100, irregular = 100. Column totals: good = 120, poor = 80. Expected: reg/good = 60, reg/poor = 40, irreg/good = 60, irreg/poor = 40.

Calculate χ²: = (72−60)²/60 + (28−40)²/40 + (48−60)²/60 + (52−40)²/40 = 2.4 + 3.6 + 2.4 + 3.6 = 12.0. p-value < 0.001.

Decision: χ² = 12.0 > critical value 3.841, and p < 0.001 < α. Reject H₀. Effect size (Cramér's V) = √(12.0 / 200) = 0.245 (small-to-medium).

There is statistically significant evidence that exercise frequency and sleep quality are associated (χ²(1) = 12.0, p < 0.001, V = 0.245). Regular exercisers reported better sleep more often than would be expected by chance. This is an association, not causation.

Python implementation

from scipy.stats import chi2_contingency
import numpy as np

observed = np.array([[72, 28],
                     [48, 52]])

chi2, p, dof, expected = chi2_contingency(observed)
cramers_v = np.sqrt(chi2 / (observed.sum() * (min(observed.shape)-1)))

print(f"chi2 = {chi2:.1f}, df = {dof}, p = {p:.4f}, V = {cramers_v:.3f}")
# → chi2 = 12.0, df = 1, p = 0.0005, V = 0.245

Example 3: Z-test for Proportions

Worked Example

Did a new website design improve the conversion rate?

A company's old website had a 12% conversion rate. After redesigning the site, 84 out of 600 visitors converted (14%). Test at α = 0.05 whether the new design has a higher conversion rate than the old one.

State hypotheses: H₀: p = 0.12 (conversion rate is unchanged). H₁: p > 0.12 (conversion rate increased) — right-tailed test.

Significance level: α = 0.05. Critical z = 1.645 (right-tailed).

Calculate z: p̂ = 84/600 = 0.14. z = (0.14 − 0.12) ÷ √[0.12 × 0.88 ÷ 600] = 0.02 ÷ 0.01321 = 1.514.

Compare: z = 1.514 < 1.645. p-value ≈ 0.065.

Decision: p = 0.065 > α = 0.05. Fail to reject H₀.

At the 5% level, the data does not provide sufficient evidence to conclude that the new design increased the conversion rate (z = 1.514, p = 0.065). The observed 2-percentage-point gain is not statistically significant at this sample size. A power calculation suggests approximately 900 visitors per variant would be needed to detect a 2pp lift with 80% power.

Python implementation

from statsmodels.stats.proportion import proportions_ztest
import numpy as np

conversions = 84
n           = 600
p0          = 0.12

z_stat, p_val = proportions_ztest(conversions, n, value=p0, alternative='larger')
print(f"z = {z_stat:.3f}, p = {p_val:.3f}")
# → z = 1.514, p = 0.065  (fail to reject H₀)

→ More worked examples: Hypothesis Testing Examples & Solutions

Advanced Topics in Hypothesis Testing

Multiple Hypothesis Testing

When you run many tests simultaneously — for example, comparing 10 biomarkers across two groups — the probability of at least one false positive increases rapidly. With 20 independent tests at α = 0.05, you expect at least one false positive by chance alone.

The Bonferroni correction addresses this by dividing α by the number of tests. If you run k tests, use α* = α/k as the threshold for each individual test.

Bonferroni correction

Adjusted significance level:  α* = α ÷ k

Example: 5 simultaneous tests at α = 0.05
α* = 0.05 ÷ 5 = 0.01   →  each test must reach p ≤ 0.01 to be declared significant

The Bonferroni correction is conservative. More powerful alternatives include the Benjamini-Hochberg procedure, which controls the false discovery rate (the expected proportion of false positives among rejected tests) rather than the family-wise error rate. It is preferred when many simultaneous tests are run — for example, in genomics or neuroimaging studies.

→ Deep dive: Bonferroni Correction

Bayesian Hypothesis Testing

Classical (frequentist) hypothesis testing answers: "If H₀ is true, how likely is this data?" Bayesian hypothesis testing answers a different question: "Given this data, how likely is H₀ relative to H₁?" The Bayes factor is the ratio of these likelihoods.

Bayesian methods require specifying prior beliefs about the parameters, which can be an advantage (they incorporate existing knowledge) or a subject of debate (the prior choice influences conclusions). For routine research, frequentist methods remain standard; Bayesian approaches are particularly useful when prior information is strong or sequential testing is needed.

Hypothesis Testing in Regression Analysis

In linear regression, hypothesis tests evaluate whether individual predictors significantly explain variance in the outcome. The null hypothesis for each coefficient is H₀: β = 0 (the predictor has no effect on Y, holding others constant). The t-statistic = coefficient ÷ standard error, compared against a t-distribution. The overall model fit is tested with an F-test (H₀: all slopes = 0 simultaneously).

Bootstrap Hypothesis Testing

Bootstrap tests make no distributional assumptions. They estimate the null distribution by resampling from the observed data thousands of times and measuring how often the resampled statistic exceeds the observed one. Useful when sample sizes are small or the test statistic does not follow a known distribution.

Limitations of Hypothesis Testing

Hypothesis testing is one of the most widely used tools in statistics, but it has real limitations that every analyst should understand before interpreting results.

Binary decisions oversimplify

Reducing a continuous body of evidence to "reject / fail to reject" discards nuance. A p-value of 0.049 and one of 0.051 represent nearly identical evidence — yet one crosses the threshold and one doesn't.

P-hacking inflates false positives

Choosing α after seeing results, running multiple comparisons without adjustment, or stopping data collection when p crosses 0.05 all inflate the false positive rate well beyond the nominal α level.

Garbage in, garbage out

Results are only as valid as the data quality. Poor sampling, measurement error, and confounding variables invalidate conclusions regardless of how correctly the test is applied.

Significance ≠ causation

A statistically significant association between X and Y does not mean X causes Y. Establishing causation requires experimental design (randomisation, controls), not just a p-value.

Publication bias

Studies that fail to reject H₀ are less likely to be published. This "file drawer problem" distorts the literature toward apparent effects that may not replicate.

Assumptions are rarely checked

Every parametric test assumes something — normality, equal variances, independence. Violating these assumptions invalidates the p-value. Always check assumptions before interpreting results.

🔬

The solution is not to abandon hypothesis testing

These limitations point toward better practice: pre-register your hypotheses and α, report effect sizes and confidence intervals alongside p-values, perform power calculations before data collection, and check test assumptions explicitly. Used carefully, hypothesis testing remains indispensable for making principled decisions from data.

→ Statistical Test Assumptions

Hypothesis Testing at a Glance

Concept	Definition	Key Point
H₀ (Null hypothesis)	The claim of no effect or no difference	Always contains equality; never "accepted," only failed to reject
H₁ (Alternative)	The research claim — what you expect to find	Can be one-tailed or two-tailed; default is two-tailed
Significance level (α)	Threshold for rejecting H₀	Set before data collection; usually 0.05
Test statistic	Measured deviation from the null value	Z, t, χ², or F depending on test type
P-value formula	Probability of this data (or more extreme) if H₀ were true	Two-tailed z: p = 2 × P(Z > \|z\|)
Type I error	Rejecting a true H₀	Probability = α
Type II error	Failing to reject a false H₀	Probability = β; reduced by larger n
Statistical power	Probability of detecting a real effect	Power = 1 − β; target ≥ 0.80
Effect size	Magnitude of the difference in real-world terms	Cohen's d, η², r; required alongside p-value
Tukey's HSD	Post-hoc test for pairwise comparisons after ANOVA	HSD = q × √(MSW/n); controls family-wise error rate

Conclusion

Hypothesis testing gives you a principled, repeatable method for making decisions from data under uncertainty. The procedure is consistent across every test type: state the hypotheses, set α, calculate the test statistic, find the p-value, and draw a conclusion in context.

Getting it right means more than following the steps. It means choosing the correct test for your data, setting α before you look at results, running Tukey HSD (or another post-hoc test) when ANOVA is significant, reporting effect sizes alongside p-values, and interpreting "fail to reject" as absence of evidence — not as proof the null is true.

Statistical significance and practical significance are different things. A large sample can make a trivial difference statistically significant. Always pair your test result with a confidence interval and a measure of effect size — Cohen's d for means, Cramér's V for chi-square, η² for ANOVA. That combination tells readers what they need to know: whether an effect was detected, and how large it might plausibly be.

FAQs About Hypothesis Testing

Hypothesis testing is a statistical procedure used to decide whether sample data provides enough evidence to reject a claim about a population parameter. You compare observed data to what would be expected if the null hypothesis were true, using the p-value or a critical value to guide the decision.

The p-value is the probability of observing a test statistic at least as extreme as the one calculated from your sample, assuming the null hypothesis is true. It is not the probability that H₀ is true. A p-value ≤ α (typically 0.05) leads to rejecting H₀.

The p-value is calculated from the reference distribution of the test statistic under H₀. For a two-tailed z-test: p = 2 × P(Z > |z|), where Z follows the standard normal distribution. For a two-tailed t-test: p = 2 × P(T > |t|) with degrees of freedom = n − 1. In Python: from scipy import stats; p = stats.t.sf(abs(t), df)*2 for a t-test.

Reject the null hypothesis when the p-value is less than or equal to the pre-set significance level α (usually 0.05). Equivalently, reject H₀ when the calculated test statistic exceeds the critical value at the chosen α. Both approaches give the same decision.

A Type I error (false positive) is rejecting a null hypothesis that is actually true. Its probability equals α, the significance level. A Type II error (false negative) is failing to reject a null hypothesis that is actually false. Its probability is β. Statistical power = 1 − β. Reducing α increases β, and vice versa, at a fixed sample size.

A two-tailed test checks for a difference in either direction (H₁: μ ≠ μ₀) and splits α between both tails of the distribution. A one-tailed test checks only one direction — either H₁: μ > μ₀ (right-tailed) or H₁: μ < μ₀ (left-tailed) — and concentrates all of α in one tail. Two-tailed tests are the default unless you have a strong directional prediction established before data collection.

Use a z-test when the population standard deviation (σ) is known, or when the sample is large (n ≥ 30) and the central limit theorem applies. Use a t-test when σ is unknown and must be estimated from the sample — which is the situation in the vast majority of real research. As sample size grows, the t-distribution approaches the normal distribution, so the distinction matters most in small samples.

Statistical power (1 − β) is the probability that a test will detect a real effect when one exists. A power of 0.80 means an 80% chance of a true positive. Underpowered studies — those with insufficient sample sizes — frequently produce false negatives and waste research resources. Power should be calculated before data collection to determine the minimum sample size required.

Statistical significance (p ≤ α) means the observed result is unlikely under H₀. Practical significance means the effect is large enough to matter in the real world. With large samples, even tiny, trivial differences can be statistically significant. Always report an effect size (Cohen's d, η², Cramér's V) alongside the p-value to convey practical importance. A result that is statistically significant but has a negligible effect size may not justify any real-world action.

Tukey's Honestly Significant Difference (HSD) test is a post-hoc test run after a statistically significant one-way ANOVA. It compares all possible pairs of group means to determine which specific groups differ from each other, while controlling the family-wise error rate. The formula is HSD = q × √(MSW / n), where q is the studentized range statistic from the Tukey q-table. Use Tukey's HSD when group sizes are equal or approximately equal and you want to make all pairwise comparisons.

Hypothesis testing produces a binary decision: reject or fail to reject H₀. A confidence interval gives a range of plausible values for the population parameter, providing information about the magnitude and direction of the effect. The two are mathematically equivalent at the same α level: if a 95% CI excludes the null value, the two-tailed test at α = 0.05 will reject H₀, and vice versa. Reporting both is best practice.

What Is Hypothesis Testing?

🔑 Key Takeaways

Null and Alternative Hypotheses

The Null Hypothesis (H₀)

The Alternative Hypothesis (H₁ or Hₐ)

Two-tailed

Right-tailed

Left-tailed

Real-World Example

Testing a New Blood Pressure Drug

The 5 Steps of Hypothesis Testing

State the null and alternative hypotheses

Choose the significance level (α)

Select the appropriate test and compute the test statistic

Find the p-value or compare to the critical value

Draw a conclusion in context — and report effect size

The P-value Explained

The P-value Formula

How to Interpret the P-value

Significance Level (α) vs P-value

Test Statistic and Critical Value

Types of Hypothesis Tests

Z-test

T-test

One-sample t-test

Two-sample (independent) t-test

Paired t-test

Chi-Square Test

ANOVA (Analysis of Variance)

Tukey's HSD Post-Hoc Test

🔍 When ANOVA is significant — which groups actually differ?

Proportion Test (Z-test for Proportions)

Which Test Should You Use?

Type I and Type II Errors

Type I Error (False Positive)

Type II Error (False Negative)

Statistical Power of a Hypothesis Test

Sample size

Effect size

Significance level (α)

Variability

Statistical Significance vs. Practical Significance

Statistical Significance

Practical Significance

Illustrative Example

When statistical significance misleads

Confidence Intervals and Hypothesis Testing

Hypothesis Testing: Worked Examples

Example 1: One-Sample T-test

Worked Example

Do students score differently from the national average?

Example 2: Chi-Square Test of Independence

Worked Example

Is there an association between exercise frequency and sleep quality?

Example 3: Z-test for Proportions

Worked Example

Did a new website design improve the conversion rate?

Advanced Topics in Hypothesis Testing

Multiple Hypothesis Testing

Bayesian Hypothesis Testing

Hypothesis Testing in Regression Analysis

Bootstrap Hypothesis Testing

Limitations of Hypothesis Testing

Binary decisions oversimplify

P-hacking inflates false positives

Garbage in, garbage out

Significance ≠ causation

Publication bias

Assumptions are rarely checked

Hypothesis Testing at a Glance

Conclusion

Read More on Statistics Fundamentals

Null and Alternative Hypotheses

P-values in Detail

Type I and Type II Errors

Tukey's HSD Test

Effect Size (Cohen's d, η², r)

Statistical Power of a Test

Z-table (Standard Normal)

Hypothesis Testing Examples & Solutions