The same gap between measured and real lift shows up in commercial testing programs, usually a quarter later, when finance asks where the promised revenue went.
Nobody lied in most of these cases. The test was run, the p-value came in under 0.05, the variant shipped. What failed was the step from "statistically significant" to "worth this much money," which depends on power, metric choice, timing and traffic mix far more than dashboards admit.
A Worked Example: 12% Lift, Revenue Within the Noise
Take a checkout test with 20,000 visitors in each arm. Control converts 600 visitors (3.00%) and the variant converts 672 (3.36%). That is a 12% relative lift, a two-proportion z statistic of about 2.05, and a two-sided p-value of 0.04. The 95% confidence interval on the relative lift runs from about 0.5% to 23.5%. Read that interval honestly and the test says the change probably helps, by somewhere between almost nothing and a lot.
Now look at the metric the business actually cares about. Suppose average order value is $80 with a standard deviation of $120, plausible for a store selling both $20 accessories and $400 bundles. Revenue per visitor goes from $2.40 to $2.69, a difference of $0.29. But per-visitor revenue is mostly zeros with a long right tail, so its standard deviation is about $25. The standard error on the difference comes to roughly $0.26, giving a z of 1.13 and an interval from minus $0.21 to plus $0.79 per visitor. The same experiment that "won" on conversion is inconclusive on revenue.
Why Conversion Rate Is the Wrong Scoreboard
Conversion rate is a proportion, so its variance is bounded and tests reach significance quickly. Revenue has no such ceiling. A single $2,000 order moves an arm's total as much as 25 typical ones, which is why revenue per visitor in this example needs roughly three times the sample to reach the same relative precision. Teams pick conversion as the primary metric because it settles faster, then forecast revenue from it as if the two were interchangeable. They often are not. A free shipping threshold, a discount banner or a simplified cart can raise conversion while pulling buyers toward cheaper baskets. In the example above, if the variant's average order value slips from $80 to $74, the revenue lift shrinks to 3.6% and its interval spans minus $0.41 to plus $0.58. The conversion chart still looks great.
Traffic mix adds a second layer. If a test ran during a week when paid social sent an unusual share of mobile visitors, and mobile users responded more strongly to the change, the measured average reflects that week's blend. After launch, the blend drifts back. The effect within each segment never changed, but the blended number did, and revenue follows the blend that actually arrives.
Peeking, Novelty, and the Calendar
Many of these failures trace back to decisions made before the test started. Was the stopping rule fixed in advance, or did someone check the dashboard every morning and stop the first time p dipped under 0.05? Was revenue per visitor declared as a guardrail, with a minimum sample sized for it? Whether the program lives in-house or runs through an outside CRO service, those questions belong in a written plan signed off before launch, because daily peeking with a fixed 0.05 threshold pushes the real false positive rate far above 5%, and no amount of careful reporting can undo that later.
Time works against winners too. A new layout gets attention because it is new, and returning visitors click on it out of curiosity before settling into old habits. The reverse also happens: some users resist a change at first and warm to it later. Research by Henning Hohnhold, Deirdre O'Brien and Diane Tang at Google found that users change behavior in response to the quality of ads they have seen before, which means the short-term effect of a change is not always predictive of its long-term effect. A two-week test captures the first reaction, not the settled one.
Shrink the Estimate Before You Forecast With It
The deepest problem is selection. Significance acts as a filter, and only estimates that land far enough from zero get through. When power is low, the estimates that pass are mostly the lucky high draws. Andrew Gelman and John Carlin formalized this in 2014 as Type M (magnitude) error, showing how significant results from underpowered studies inflate estimates of the true effect. At 80% power their expected exaggeration factor is a modest 1.12. Below 50% power it grows quickly, and in their design analysis of one published study, where power came out at 0.06, the expected exaggeration was 9.7.
Apply that to the checkout test. If the true lift were 3%, a 20,000-per-arm design has only about 8% power to detect it. A simulation of that setup shows the tests that do reach significance report an average lift near 15%, about five times the truth. The 12% the team celebrated is exactly the kind of number this design produces when the real effect is small. Forecasting next year's revenue from 12% is a reliable way to disappoint.
Holdouts Are How Revenue Gets Its Say
The practical fix is a long-running holdout. Keep 5% to 10% of traffic on the old experience for a full quarter after launch, measure revenue per visitor in both groups, and treat that comparison as the real verdict. It is slower and costs a little revenue, but it captures novelty decay, mix drift and order value shifts in one number that no two-week test can produce.
Holdouts work best paired with honest planning on the front end. Before the next test goes live, choose revenue per visitor or a variance-reduced version of it as the primary metric, set a minimum detectable effect the business would actually act on, and size the test for 80% power against that effect. Anyone unsure of the mechanics can learn to calculate power from a worked two-sample example in an afternoon.
Then shrink. Reporting a winner as "12% measured, probably a fraction of that in reality, given the low power" is less exciting in a meeting. It is also closer to what finance will see.
A significant test result tells you something changed. It says far less about how much money the change will make. The distance between the two is predictable: low power inflates the estimate, conversion rate hides order value, the calendar flatters novelty and traffic mix moves underneath it all. Plan for that distance before launch, verify it with a holdout after, and the next winner will be one that revenue actually agrees with.