Free A/B Test Calculator

An A/B test calculator compares two versions of a page, email, or product experience by running a statistical test on conversion data. It tells you whether the difference between the two versions is large enough to be real, rather than the result of random visitor variation. This tool runs a two-proportion z-test and returns conversion rates, lift, p-value, z-score, and a confidence interval.

Calculate the conversion rate for each variant, find the pooled standard error, divide the difference in rates by that standard error to get a z-score, and convert the z-score to a p-value using the standard normal distribution. If the p-value is below your significance threshold — commonly 0.05 — the result is statistically significant. Use the Significance Test tab above to run this calculation automatically.

Relative lift is (Conversion Rate B − Conversion Rate A) ÷ Conversion Rate A, expressed as a percentage. Absolute lift is simply Conversion Rate B minus Conversion Rate A, in percentage points. Both numbers matter: relative lift is easier to compare across tests, while absolute lift shows the raw size of the effect.

Required sample size depends on your baseline conversion rate, the minimum lift you want to detect, your confidence level, and your statistical power. As a rough reference, detecting a 20% relative lift on a 5% baseline rate at 95% confidence and 80% power requires roughly 7,260 visitors per variant. Use the Sample Size tab above to calculate the exact number for your situation.

95% confidence is the standard choice for most A/B tests and corresponds to a 0.05 significance threshold. Use 99% confidence for high-stakes decisions, such as changes to a checkout flow or pricing page, where a false positive is costly. Lower thresholds like 90% are sometimes used for low-risk, exploratory tests where speed matters more than certainty.

An A/B test should run until it reaches its planned sample size, and for at least one to two full business cycles, typically one to two weeks, so day-of-week effects average out across both variants. Stopping a test the moment it first looks significant, before the planned sample size, inflates the false-positive rate.

Statistical power is the probability of correctly detecting a real difference between variants when one truly exists. It is calculated as 1 minus the Type II error rate (β). The standard target is 80% power; some teams use 90% power for higher-stakes tests, which requires a larger sample size.

The minimum detectable effect is the smallest relative lift you want the test to be able to catch reliably. Setting a smaller MDE makes a test more sensitive but requires a much larger sample size; setting a larger MDE requires fewer visitors but will miss smaller real improvements.

A p-value below 0.05 is the conventional threshold for statistical significance, corresponding to a 95% confidence level. Some teams use a stricter 0.01 threshold for high-stakes decisions, while exploratory tests sometimes use 0.10. The threshold should be set before the test starts, not chosen after looking at the data.

The confidence interval for the difference between two proportions is (p̂_B − p̂_A) ± z* × SE, where SE uses the unpooled standard error and z* is the critical value for your chosen confidence level. If this interval does not contain zero, the result is statistically significant at that confidence level.

It takes visitor and conversion counts for two groups, computes each group's conversion rate, calculates the pooled standard error under the assumption of no difference, derives a z-score, and converts that z-score into a p-value using the cumulative standard normal distribution. The calculator on this page performs all of these steps and shows the full working in the Step-by-Step tab.

A two-proportion z-test is a statistical test used to determine whether two population proportions — such as two conversion rates — differ significantly. It assumes large enough samples for the normal approximation to the binomial distribution to hold, and it is the standard method behind most A/B testing tools, including this calculator.

Frequentist A/B testing, used by this calculator, computes a p-value and asks how surprising the observed data would be if there were truly no difference between variants. Bayesian A/B testing instead estimates the probability that one variant beats the other, given the data and a prior belief, and can update continuously as data arrives. Frequentist tests require a fixed sample size set in advance; Bayesian methods are generally more tolerant of early peeking, though both approaches can be misused.

Stop an A/B test when it reaches the sample size calculated before the test began, and after it has run for at least one full business cycle. Stopping earlier because the result happens to look significant on a given day is a common source of false positives, often called peeking bias.

Results from a small sample can still be statistically valid if the calculated p-value is significant and both np̂ and n(1−p̂) are at least 5 for each variant, but small samples make it harder to detect real, smaller effects and produce wider confidence intervals. Checking the confidence interval, not just the p-value, gives a more complete picture of the uncertainty involved.

The most common causes are stopping a test early after repeated peeking, testing multiple metrics or variants without correcting the significance threshold, and running a test too briefly to account for day-of-week or seasonal variation. Setting a sample size and significance threshold in advance, and sticking to them, addresses most of these causes directly.

A valid A/B test uses random, independent assignment of visitors to each variant, reaches a sample size calculated before the test began, runs for at least one full business cycle, and tests one primary metric defined in advance. Checking for sample ratio mismatch — an unexpected split between variants — is also part of validating results before drawing conclusions.