A successful A/B test gives you evidence that one version performed better than another under the conditions of that test.
That is a valuable result. It is also not the end of the story.
You still need to decide whether the change should be rolled out, whether the expected value holds up in practice, and whether the result is strong enough to inform a wider business decision.
I saw this play out with a rent-to-own ecommerce retailer. A four-week A/B test produced a 14% relative increase in checkout completion. The improvement was valued at around £20 million over a year.
That was a significant result. The next question was whether the improvement would continue once the winning experience was live.
A test result is evidence, not the whole business case
The test gave us a controlled comparison. It showed that the new experience increased checkout completion during the experiment.
The annual value estimate translated that improvement into a business outcome. That estimate was useful for making the decision, but it still depended on the uplift carrying through after launch.
This is where it helps to keep three things separate:
– The test result: what changed during the experiment.
– The value estimate: what that change could mean for the business over a longer period.
– The realised outcome: what happened after the change went live.
They are connected, but they are not interchangeable. A projected annual value is not automatically the same as a realised annual impact.
Check what happens after launch
After the change went live, performance was checked against the test result. The improvement was confirmed in the year that followed. The company also referred to improved online conversion in its year-end prospectus as one contributor to its year-on-year improvements in revenue and profit.
That does not mean one test explains every change in a company’s financial performance. It does mean the result had evidence beyond the original experiment.
The follow-up mattered because it tested whether the decision held up in the live business, where customer behaviour and commercial conditions continue to move.
Why post-launch validation matters
A test can be run correctly and still produce an outcome that changes after rollout. The live population may differ. Other changes may affect the same journey. The impact may fade, or a guardrail may move in the wrong direction.
Post-launch checks help you spot those differences. They also give you a chance to revisit the original assumptions and improve the next decision.
That does not mean waiting for perfect certainty before acting. It means being clear about what the test established, what the value estimate assumes, and what you will check once the change is live.
Treat the decision as part of the experiment
When a test wins, the next steps should be clear:
1. Record the result and the conditions under which it was measured.
2. Explain how the projected value was calculated.
3. Decide what to monitor after rollout, including relevant guardrails.
4. Set a point to review the live outcome.
5. Revisit the decision if the evidence changes.
The test is one part of the evidence. The post-launch result adds another.
A result is more useful when you can explain what it showed, how you translated it into business value, and what happened after you acted on it.
A winning experiment earns a decision. The follow-up tells you whether that decision stood up.