Stage 01 / Define the question
What decision are you trying to make?
Start with evidence of a problem, a plausible mechanism and an outcome that matters to the business.
Write the experiment brief
e.g. Explain delivery costs before checkout
Because [evidence], we believe [change] for [audience] will improve [outcome], by [mechanism].
e.g. Users purchasing within 7 days / all eligible randomised users
e.g. +0.5 percentage points, provided contribution margin does not deteriorate
e.g. New UK customers, randomised by persistent user ID. Exclude staff and known bots before assignment.
e.g. Monitor margin per eligible user, payment errors and refunds. Specify acceptable harm, alert threshold, owner and rollback action.
Design before you build
Use concurrent, random assignment and keep the assignment stable. Choose a user, account or other unit that avoids people crossing between variants. Analyse at that unit; repeated sessions from one user are not independent observations.
Define eligibility and exclusions before assignment. Use intention-to-treat as the default: analyse eligible randomised units in their assigned groups, including those who did not engage. A triggered analysis needs a treatment-independent eligibility rule applied consistently to both groups.
Fix obvious defects. If traffic is too low, data is unreliable or the cost of waiting exceeds the value of learning, use research, usability work or another suitable design.
Stage 02 / Plan the evidence
How much data will you need?
Choose an effect worth detecting. Then check whether you can collect enough independent participants in a sensible time.
Sample-size planner
Binary outcome · two independent groups · two-sided fixed-horizon test · normal approximation
Agree the stopping rule
MDE is a planning assumption, not a promise that smaller effects cannot occur. Power is the probability of detecting the specified effect if it exists under the assumptions. Use a representative baseline and plan sensitivity scenarios.
For a fixed-horizon test, set sample targets and a minimum calendar duration in advance. Include relevant weekly or business cycles and allow outcomes to mature. Do not extend an inconclusive experiment until it becomes significant. If continuous decisions are necessary, choose a valid sequential design before launch.
The planner allows unequal allocation and applies Bonferroni α / comparisons. This is conservative; it is not a multi-arm optimiser. Rare outcomes, clustering, revenue, ratios, interference and high variance need a different power calculation.
Record A/B sample targets, minimum full business cycles, maximum feasible duration, maturation window and what happens if traffic falls short.
Stage 03 / Trust the measurement
Can you trust what you will count?
A precise answer from broken tracking is still a broken answer.
Instrument the experiment lifecycle
| Record | What to validate |
|---|---|
| Assignment | Experiment ID, version, variant, stable randomisation-unit ID, timestamp and eligibility. One assignment per unit; no personal details in browser events. |
| Exposure | When the assigned experience was actually available. Log rendering failures and latency separately from assignment. |
| Outcome | Event definition, deduplication key, time window, identity join and attribution rule. Count each participant once for this binary calculator. |
| Diagnostics | Pre-treatment device, channel and geography; consent coverage, errors, assignment conflicts and missing joins. |
Compare assignment → exposure → outcome joins across variants. Validate against a trusted backend source where possible. Consent and browser restrictions can change who is observable; never infer that tracked users represent everyone without checking coverage.
QA the experience and the data
- Test control and treatment on real devices, browsers, screen sizes, keyboard navigation and assistive technology.
- Check flicker, loading performance, error states, payment paths, accessibility and consent behaviour.
- Verify returning users keep their variant, users do not switch on login and experiment IDs survive redirects and cross-domain journeys.
- Test event firing, duplicates, time zones, bot filters, refunds and late outcomes. Use an A/A test where useful to validate the pipeline; passing one does not prove all tracking is correct.
Stage 04 / Launch with control
Make the launch reversible.
Agree who is watching, what would trigger a rollback and how the final analysis population will be defined.
A controlled launch sequence
- Freeze the design. Record the experiment version, allocation, exclusion rules and overlapping experiments. Check interactions and use mutually exclusive layers where needed.
- Verify in production. Start with a controlled QA audience. Keep staff and preview traffic out of the pre-specified analysis population.
- Ramp with a plan. Monitor technical health and guardrails. Record every allocation or version change; avoid changes motivated by promising outcome data.
- Confirm the start. Record the analysis start, owners and rollback route. If allocation varies by time, use time-stratified diagnostics and an appropriate analysis; a single pooled SRM test can mislead.
Include experiment/version ID, start time and time zone, traffic allocation, ramp plan, owners and rollback steps.
Stage 05 / Check the experiment
Is the comparison still fair?
Monitor for harm and broken measurement. Keep the urge to call a winner separate from operational monitoring.
Sample ratio mismatch (SRM)
Do the observed assignment counts fit the planned split? Use unique randomised units, not conversions or sessions. Unequal group sizes are fine when they match the plan.
Pearson χ² goodness-of-fit, 1 degree of freedom. One diagnostic snapshot; repeated monitoring raises false-alert risk. Agree monitoring rules. No alert is not proof of unbiased assignment.
Common checks that change the interpretation
Unequal exposure or missing outcomes
Compare exposure / assigned, logging success and outcome maturity by group, device and time. Similar assignment counts can hide treatment-specific rendering failures. Do not remove unexposed users after assignment simply to make the result look cleaner: treatment may cause the missing exposure. Investigate the mechanism and preserve the pre-specified population.
Peeking and early stopping
Looking at error rates and serious harm is sensible. Repeatedly testing the primary metric and stopping when p < 0.05 inflates false positives in a fixed-horizon test. Use the agreed horizon or a properly designed sequential method. If safety forces an early stop, document it and treat ordinary fixed-horizon inference with caution.
Guardrails and hidden costs
A conversion increase can coexist with lower margin, more cancellations, slower pages or worse accessibility. Set acceptable harm bounds and escalation rules in advance. A non-significant guardrail result does not prove safety: use a suitably powered non-inferiority assessment when ruling out harm matters.
Novelty, seasonality and concurrent changes
Inspect trends by assignment cohort, day and relevant business cycles. Promotion changes, releases, outages or channel shifts can alter the population or interact with treatment. Avoid dropping inconvenient days after seeing results. Document deviations and consider a new clean experiment if comparability is compromised.
Record assignment/exposure differences, missing data, guardrail alerts and any departure from the plan.
Stage 06 / Understand the result
Estimate the effect. Keep the uncertainty.
Use the agreed analysis window and mature outcomes. The size and plausible range of the effect matter as much as a threshold.
A/B significance & confidence intervals
For one binary outcome per independent participant. “Converters” means unique participants with at least one qualifying conversion, not total orders or events.
How to read the numbers
Absolute difference: B − A in percentage points. Relative lift: (B − A) / A. A move from 5% to 6% is +1 percentage point or +20% relative.
Confidence interval: a range from a procedure with the stated long-run coverage under its assumptions. It is not a probability that this particular interval contains the true effect. The difference interval is in percentage points; no interval for relative lift is calculated.
P-value: assuming no difference and the test assumptions, how often a result at least this extreme would occur. It is not the probability that B is better, nor the probability the result is a false positive.
Rate intervals use Wilson score; the difference uses Newcombe's hybrid score interval. The p-value uses a pooled two-proportion z-test without continuity correction. These methods are not exact inverses and can disagree near a boundary. Bonferroni adjusts the comparison α and intervals; rate intervals are descriptive, not simultaneous coverage for every rate shown.
Segment imbalance & Simpson’s paradox
A different mix of high- and low-converting users can reverse the pooled result. The example below has B performing worse in each segment but better overall. Edit the counts to explore your own two mutually exclusive, exhaustive, pre-treatment segments.
Independent of the main A/B calculator; totals are not automatically linked. The standardised difference uses the combined sample’s segment weights in both groups. It is descriptive, with no CI or hypothesis test. No reversal does not rule out confounding.
What to do about an imbalance
Inspect pre-treatment device, channel, geography and customer mix by variant and time. Chance imbalance is possible; also check targeting, randomisation, consent and logging. Do not balance groups by deleting users after seeing outcomes. Use pre-specified stratification or covariate adjustment where appropriate, and report the original randomised estimate.
Multiple comparisons and segment fishing
Extra variants, metrics and segment analyses create more opportunities for false positives. Define the confirmatory family in advance. This guide offers a conservative Bonferroni correction for a fixed family; it does not correct optional stopping. Label post-hoc findings exploratory and validate them in a new test. A significant result in one segment and a non-significant result in another is not evidence of an interaction; test the difference in effects.
Stage 07 / Turn evidence into action
Make a decision you can explain.
A calculator cannot decide whether to ship. Combine a valid comparison, meaningful impact, guardrails and the cost of being wrong.
Use the range, not just the headline
| Evidence | Practical interpretation |
|---|---|
| Quality checks fail | Investigate before interpreting the effect. Repairing a pipeline may require a new clean test; do not simply rescale counts. |
| Interval includes meaningful harm and benefit | Inconclusive. Weigh the risk, improve the design or plan a new test. “Not significant” does not mean “no effect”. |
| Positive effect, interval includes trivial gains | Evidence of improvement may still leave commercial value uncertain. Compare implementation cost and the smallest worthwhile effect. |
| Plausible effects are worthwhile, guardrails acceptable | Consider a staged rollout with monitoring. Document residual uncertainty and check whether the effect persists. |
| Interval is narrow around zero | Large benefits may be ruled out. Claiming equivalence requires pre-specified equivalence margins and a suitable test. |
| Meaningful harm or guardrail breach | Follow the agreed safety action. Learn from the mechanism; an apparent primary-metric win does not override material harm. |
Name · DD/MM/YYYY
What did we learn? Include absolute effect and CI, practical value, data-quality findings, guardrails, deviations and what happens next.
Reference / Methods & boundaries
Know what the tools assume.
Useful calculations depend on a suitable design. Keep the method consistent with your platform and analysis plan.
Scope of these calculators
These tools assume independently randomised groups, stable assignment, a binary outcome and one observation per randomisation unit. They do not analyse average revenue, counts of orders, clustered accounts, repeated measures, switchbacks, time-to-event outcomes, network effects or Bayesian posteriors.
The sample-size calculation uses a two-sided normal approximation with a pooled null variance and alternative variance, allowing a fixed unequal allocation. No continuity correction is applied. It is an estimate, not an exact guarantee of power. Very rare outcomes require specialist planning.
The significance test suppresses a significance verdict when any expected success/failure cell is below 5; use an exact method such as Fisher’s test or a suitable specialist analysis. Score intervals are still displayed as approximate descriptive estimates. Different statistical engines can legitimately return different values.
Bonferroni covers the specified fixed family of confirmatory comparisons. It does not repair post-hoc selection, biased assignment, missing data or repeated looks. Data-quality and operational safety checks remain necessary.
Sources & further tools
- NIST: comparing two proportions — pooled z-test, assumptions and exact-test alternative.
- Newcombe (1998): interval estimation for the difference between independent proportions — source for the hybrid score difference interval.
- Wilson (1927): probable inference and the law of succession — score intervals for individual conversion rates.
- Evan Miller: sample-size calculator — alternative planner; align assumptions before comparing.
- Evan Miller: A/B chi-squared calculator — independent calculation reference.
- Microsoft Research: diagnosing sample ratio mismatch — causes and investigation of allocation problems.
- Evan Miller: How Not To Run an A/B Test — why fixed-horizon peeking changes error rates.
- Simpson (1951): interpretation of interaction in contingency tables — the aggregation paradox.
- NIST: Bonferroni’s method — controlling a family of comparisons.
Guide version 1.0 · 10 November 2026. Sources describe methods; they do not endorse this implementation.
