Data Science Statistics Hypothesis Testing 9 min read November 18, 2026
BY: Statistics Fundamentals Team
Reviewed By: Minsa A (Senior Statistics Editor)

Is Your A/B Test Winner Real? The Two-Proportion Z-Test, Step by Step

A product team runs an A/B test on a signup page. Version A converts 400 of 10,000 visitors (4.0%). Version B converts 460 of 10,000 (4.6%). B looks 15% better. Before anyone ships it, one question needs an answer: could a gap this size appear by chance if the two pages were identical?

That question is a hypothesis test, and for conversion rates the standard tool is the two-proportion z-test. This guide walks through it by hand, then covers the three mistakes that make a "significant" result worthless: peeking, a broken traffic split and an underpowered test.

Step 1: state the hypotheses

The null hypothesis says both versions share the same true conversion rate, so any observed gap is noise. The alternative says the rates differ. Most product tests use a two-sided alternative, because a change can hurt as easily as it helps.

Pick the significance level before looking at the data. At 95% confidence, alpha is 0.05: you accept a 5% chance of calling a difference real when it is not.

Step 2: pool the conversion rate

Under the null hypothesis both groups come from one population, so the best estimate of the shared rate pools them:

Formula
p = (conversions A + conversions B) / (visitors A + visitors B)

Here p = (400 + 460) / 20,000 = 0.043.

Step 3: compute the standard error and z

The standard error of the difference between two proportions, under the null, is:

Formula
SE = √( p(1 − p) × (1/nA + 1/nB) )

SE = √(0.043 × 0.957 × (1/10,000 + 1/10,000)) ≈ 0.00287.

The z statistic is the observed difference divided by that standard error:

Formula
z = (rate B − rate A) / SE = (0.046 − 0.040) / 0.00287 ≈ 2.09

Step 4: turn z into a p-value

For a two-sided test, the p-value is twice the area of the standard normal curve beyond |z|. For z = 2.09 that gives p ≈ 0.036. Because 0.036 is below 0.05, the result is significant at 95% confidence: if the pages were truly identical, a gap this large would show up in fewer than 4 of 100 repeated tests.

A p-value says nothing about how big the effect is, so report a confidence interval next to it. The 95% interval for the difference uses the unpooled standard error, √(0.04 × 0.96 / 10,000 + 0.046 × 0.954 / 10,000) ≈ 0.00287, so the interval is 0.006 ± 1.96 × 0.00287, or about 0.04 to 1.16 percentage points. The lift is probably real, but it could be almost nothing or more than a full point.

Doing this by hand once is the best way to understand it. For day-to-day work, a free A/B test calculator runs the same z-test and also returns the confidence interval, the probability that B beats A and a sample ratio check, which is useful for verifying your own arithmetic.

Mistake 1: peeking and stopping early

The 5% false positive rate holds only if you look at the result once, at a sample size fixed in advance. If you check every morning and stop the first time p dips below 0.05, you give chance many attempts to cross the line, and the real false positive rate climbs well above 5%. Decide the sample size first, run the test to that size and read it once.

Mistake 2: a broken traffic split

If you planned a 50/50 split and got 10,000 visitors in A but 9,400 in B, something in the assignment is broken: a redirect that drops some users, a bot filter applied to one side, a caching bug. This is called sample ratio mismatch (SRM). A chi-square goodness-of-fit test compares the observed counts with the planned split. Microsoft's experimentation team flags a mismatch when that test's p-value falls below 0.0005. When SRM appears, fix the assignment before trusting any significance result, because the groups are no longer comparable.

Mistake 3: an underpowered test

Power is the probability that a test detects an effect of a given size when it exists. The usual target is 80%. The sample size needed per variant for two proportions is approximately:

Formula
n = (zα/2 + zβ)² × (p₁(1 − p₁) + p₂(1 − p₂)) / (p₂ − p₁)²

To detect a move from 4.0% to 4.6% at 95% confidence (z = 1.96) and 80% power (z = 0.84):

⚠️
Sample size calculation

n = (1.96 + 0.84)² × (0.0384 + 0.0439) / 0.006² ≈ 17,900 visitors per variant.

The example test stopped at 10,000 per variant, little more than half of that. It still came out significant, but significant results from underpowered tests tend to overstate the true effect, because only the lucky, inflated estimates clear the bar. A fair reading of the example: B probably helps, the 15% lift is likely an overestimate and a properly sized follow-up test is worth running.

A short checklist

  • Write down the hypothesis, the primary metric and alpha before the test starts.
  • Compute the sample size from the baseline rate and the smallest lift worth detecting.
  • Run whole weeks, so weekday and weekend visitors are both represented.
  • Check the traffic split for sample ratio mismatch before reading the result.
  • Report the p-value together with the confidence interval, never the p-value alone.
  • If you test several variants against one control, tighten alpha, for example with a Bonferroni correction (alpha divided by the number of comparisons).

The z-test itself is four lines of arithmetic. The discipline around it, fixing the sample size, checking the split and reading the interval, is what separates a real improvement from a coin flip that happened to land on B.