Experimentation isn't about running as many A/B tests as possible. It's about using evidence to make better decisions about what you change, what you don't and where you invest.
I help organisations build commercially focused experimentation and conversion rate optimisation programmes that connect customer behaviour, rigorous testing and business outcomes.
I'd call that a pretty good place to start a conversation.
Running more experiments isn't success. Increasing conversion rate isn't automatically success either. If fewer people convert but the customers you acquire are more valuable, revenue increases and profit improves, the business may be in a much better position.
That's why I start with the business outcome, understand where value is actually created, then work backwards to the decisions and customer behaviours that could change it.
The question isn't simply “what can we test?” It's “what decisions does the business need to make, and where could experimentation give us better evidence to make them?”
The A/B test is only one part of the process. The work starts before a variant is built and continues after the results are in.
What business result are we actually trying to change?
What do customers do, why do they do it and what happens beyond the website?
Turn evidence into a clear explanation, intervention and expected outcome.
Focus effort where impact, evidence and practical delivery justify it.
Define success, risk, sample size, duration and analysis before launch.
Run with discipline. Don't rewrite the rules because you like the result.
Understand what happened, what didn't and what the evidence changes.
Get valuable changes into production and feed the learning into the next decision.
Your customers use competitors, read reviews, speak to people, compare propositions and bring previous experiences into every decision. Looking only at the funnel means looking at only part of the problem.
Analytics, funnels, session recordings and heatmaps to understand what people actually do.
Surveys, user testing, interviews and reviews to understand motivations, doubts and perceived risk.
Competitor research and wider journeys to understand why customers choose you for one thing and somebody else for another.
Customer-service logs, shadowing and conversations with the people who hear the real problems every day.
Previous experiments, historic research and the knowledge sitting inside the organisation. Sometimes the useful signal is in the analytics. Sometimes it's what the team has believed for years but nobody has properly investigated.
A useful hypothesis connects an observation to an explanation, an intervention and a meaningful business outcome.
Based on the observation that customers choose Product B elsewhere because a better returns policy reduces perceived purchase risk, we believe that introducing a comparable policy and clearly communicating why B is the better choice will increase sales and market share of B.
I use Impact, Confidence and Ease, but each needs to mean something useful if the score is going to influence where you invest experimentation capacity.
Quantify the commercial opportunity where possible. Traffic, sales, margin, customer value, affected population and plausible movement are more useful than “this feels like a nine.”
Score the strength of evidence behind the idea. Opinion is not the same as behavioural data, customer research or evidence from previous experiments.
Consider the whole route to value: build, QA, legal, compliance, dependencies and the effort required to deploy permanently if the experiment wins.
The important decisions are recorded before launch, not selected afterwards because they make the experiment look better.
A negative or inconclusive experiment may have discovered one way that didn't work. Interrogate the data, revisit the research and use credible segment signals to decide whether the concept deserves another implementation or whether the evidence now says to move on.
The result needs interpretation in the context of the original evidence, commercial objective and risks you agreed before launch.
Look at sample sizes, primary and secondary outcomes, guardrails and credible segment differences. Check whether aggregate results could be masking important effects and whether unexpected results could be explained by implementation or QA issues.
The experiment supplies evidence. It doesn't make the decision for you. You may deploy everywhere, deploy to part of the audience, iterate, investigate further or accept a known risk because the commercial trade-off justifies it.
A/B testing is valuable when you're uncertain and having a control gives you useful confidence about causality.
If something is simply broken, fix it and monitor. If you don't have enough traffic or stable enough data to generate a reliable answer, use another method. If the additional confidence isn't worth the time and cost of testing, don't test for the sake of saying you did.
As the obvious opportunities are exhausted, experimentation should often get harder, not easier. A programme boasting ever-increasing volume and win rates is something I'd want to understand, not simply applaud.
It's to find out whether you are. Test velocity and win rate are poor substitutes for commercially important questions, sound methodology and useful learning.
Button copy and micro-interactions have their place. But if the business needs material changes in profit, retention, market share or customer value, the experimentation roadmap needs a credible route to those outcomes.
Engagements can focus on a specific problem or help build the operating model, standards and capability around experimentation.
Connect experimentation priorities to business objectives and decisions.
Build commercially focused optimisation programmes around evidence and learning.
Combine behavioural, qualitative, competitive and organisational evidence.
Turn observations into testable explanations and interventions.
Use quantified impact, evidence-based confidence and realistic ease.
KPIs, MDEs, sample sizes, duration, guardrails and analysis planning.
Interrogate results, risks, segments and the next questions created.
Build standards, train teams and scale experimentation without losing rigour.
I generally favour starting centrally, embedding capable people into the teams making the decisions, then distributing ownership as those teams mature.
Establish process, statistical standards, governance and ways of working.
Develop key people inside Product, Marketing, UX and other relevant teams.
Move experimentation closer to the teams making the decisions.
Retain enough oversight to share learning and stop standards drifting.
I've spent more than 19 years working across analytics, experimentation and optimisation, with experience spanning large organisations, digital businesses and complex regulated environments.
The person understanding the commercial objective is also involved in the research, challenging the hypothesis, thinking about measurement, understanding the statistics, interpreting the result and helping decide what happens next.
There isn't a chain of hand-offs where context gets diluted between strategist, researcher, analyst and account manager. You get one person who understands how the moving parts fit together and takes ownership of helping the programme navigate them.
The goal isn't to make you dependent on me. It's to leave your teams with stronger decisions, stronger experimentation capability and standards that continue after I'm gone.
CRO is concerned with improving customer experiences and commercial performance. Experimentation is one way of establishing whether a change actually caused an improvement. I use experimentation within a broader decision-optimisation approach rather than treating conversion rate as the objective.
No. A/B testing is useful when uncertainty is meaningful and a control provides valuable confidence. Research, analytics, usability work, monitoring and other forms of evidence may be more appropriate when traffic is limited or an experiment isn't needed.
Yes. That can include objectives, research, hypothesis development, prioritisation, statistical standards, QA, governance, reporting, learning repositories, team structure and capability development.
Yes. I can review how experiments are selected, designed, run, analysed and deployed, including KPI choice, sample-size planning, MDEs, test duration, early stopping, guardrails, prioritisation and how programme performance is being judged.
Then I won't recommend forcing an A/B testing programme onto the problem. We can use the available evidence and choose methods that better match the traffic, decision and level of confidence the business actually needs.
Yes. Experimentation works best when it connects the people who understand customers, data, technology and the commercial objective. I can lead the programme, work alongside existing specialists or help build capability into those teams.
Good. Start with the business problem instead. We'll work out what evidence you need, whether experimentation is the right tool and what a useful answer would actually look like.
