When a conversion uplift hides a worse outcome

Getting people to start isn't the whole job

The homepage was an important entry point into an application journey. Increasing the proportion of people who progressed from it was a sensible opportunity, but progression alone could not establish whether a change improved the business outcome.

The archived experiment compared several approaches to homepage messaging, including the addition of a five-star rating, an “Apply Today” prompt and a “No Fees” message.

Three variants, three views of performance

The original report tracked progression to the first application step, completed application rate and successful application rate. Its headline comparisons were:

The report shows approximately 784 to 825 visits in each arm. All percentage changes above are relative movements compared with the control, not percentage-point differences.

These are reported point estimates. The source does not provide confidence intervals, statistical significance or the final implementation decision, so the figures should not be presented as conclusive evidence that any variant was better or worse.

A stronger top-of-funnel result can tell the wrong story

If the team had judged the five-star-rating-only variant by application progression, it would have looked encouraging. Completed applications also moved in a positive direction.

Successful applications, however, moved sharply in the opposite direction in the reported results. That doesn't establish why the outcome changed. It does show why an upstream metric should not be treated as a substitute for a downstream business outcome.

The alternative variants reinforce the point: different messages produced different patterns across the journey. The variant with the largest reported progression uplift also had positive reported movements in both downstream measures, although the archive is insufficient to determine whether those differences were statistically reliable.

Define success before looking at the results

Experiments are easier to interpret when teams agree, before launch, which metric determines the decision and which downstream measures must be protected. That means defining a primary KPI, secondary diagnostics and guardrails, and planning the sample size and duration accordingly.

The archived report illustrates the need for that discipline. It does not document whether guardrails were formally pre-registered in this particular test. The lesson is drawn from the contrast between the reported outcomes, rather than from a documented implementation decision.

A variant that gets more people to start an application is not automatically a better variant. The quality and ultimate outcome of those applications matter too.

What the original report does and doesn't establish