From an A/B testing tool to a £50m global experimentation programme

The business was already doing CRO, but there was no single owner bringing the work together. It had built a substantial backlog of ideas without an agreed way of deciding what to do first.

The A/B testing platform was sometimes being used like a shadow content management system to put short-term promotions live. That made it easy to launch activity, but launching activity was not the same as generating reliable evidence.

There were also gaps in experimental practice. Sample sizes and test durations were not consistently planned in advance, tests could be stopped early, and checks such as sample ratio mismatch (SRM) were not embedded in the process.

The programme covered 43 countries, so running every experiment independently everywhere would have been impractical. We organised the work across three broad regions: North America, Europe and Asia.

Six priority markets were tested individually. For the remaining countries, we used results from the relevant leading market to inform rollout decisions. That was a pragmatic deployment model, not a claim that identical effects had been experimentally proven in all 43 countries.

The country marketing teams played an active role. They suggested opportunities, reviewed ideas for local relevance and helped translate experiment content. Experimental standards remained centralised; local knowledge influenced what was useful and appropriate to test.

Partway through the engagement, an external management consultancy assessed the programme's contribution. Based on my recollection, its work drew on projected experiment uplifts and year-end business performance, and attributed approximately £50m in additional annual revenue to the experimentation programme.

This is a historical programme-level attribution, rather than an independently re-calculated figure from experiment-level data available today.

There was a further commercial effect. Improvements in conversion rate, average order value and margins helped support increased programmatic advertising investment in selected markets. The business reported around £50m of wider ecommerce revenue growth associated with that expansion.

That second figure should not be presented as another £50m directly caused by experimentation, or automatically added to the headline as a £100m experimentation outcome. Paid media investment was another contributor, and the figures may not be wholly independent.

There was no shortage of enthusiasm for testing at the start. The difference came from putting a repeatable system around it: a way to challenge priorities, plan experiments before they launched, protect against unreliable conclusions and share what was learned.

The international model mattered too. Central governance created consistency, but local teams helped ensure the programme remained connected to actual market conditions.

Perhaps most importantly, improved conversion performance had consequences beyond the experimentation team. It helped inform where the wider business could afford to invest for growth.