This is a fictional case created to show how the SHERLOCK problem-solving framework can be used. It is not a client case study.
The Adventure of the Untrustworthy Winner: What Sherlock Holmes Can Teach Us About Analytics
There are few things more suspicious than a result everyone is keen to believe.
Holmes was at the window when Amrdeep “Ammo” Athwal arrived from Leeds. A digital analytics and experimentation consultant, he had been asked to look at an online retailer’s latest checkout test. The dashboard showed an eighteen per cent increase in completed orders. The directors wanted to roll out the new version across every market.
There was one awkward detail. Revenue had barely moved.
“The test says the new checkout is a winner,” I said.
“The test says something,” Ammo replied. “First we need to establish what.”
Holmes turned from the window. “An excellent distinction. Let us begin with the case, rather than the conclusion.”
The case of the rising conversion rate
The retailer’s team had run the experiment for four weeks. The sample looked substantial. The dashboard showed a clear lift, and the confidence interval appeared reassuringly narrow.
Ammo asked for the event definitions, the implementation notes and the order records from the payment system. He wanted to know what counted as a completed order in each version, where that information came from, and whether the test had changed anything besides the checkout itself.
The answer was in the event setup. In the original checkout, a conversion was recorded when the customer reached the confirmation page. In the new checkout, the event fired as soon as the customer clicked “Place order”. A failed payment still counted as a conversion.
“The two versions are not measuring the same thing,” said Ammo. “One counts confirmed orders. The other counts attempts.”
The developer had made the change to work around a confirmation page that sometimes failed to load on mobile. It was a reasonable fix for one problem, but the change had quietly altered the definition of success in the experiment.
Holmes examined the release notes. “No attempt was made to conceal it. The clue was present, but no one asked what consequence it might have.”
Ammo compared the events with the payment records. Once the team counted confirmed orders in both versions, the apparent uplift disappeared. That did not prove the new checkout was worse. It meant the original result could not support the decision to roll it out.
The directors postponed the rollout. The team corrected the tracking, checked the payment journey and planned a new test with one consistent definition of conversion.
How SHERLOCK helped solve the case
The investigation followed the eight steps in the SHERLOCK problem-solving framework. The framework helps teams define the problem, test the evidence and challenge explanations before deciding what to investigate, test or fix.
- State the case. The reported checkout conversion rate had increased, but revenue had not. The decision was whether to roll out the new checkout.
- Hunt for evidence. Ammo gathered the event definitions, release notes, analytics data and confirmed payment records.
- Evaluate the evidence. The team checked how each version recorded a conversion and whether the underlying events could support the claim.
- Raise rival explanations. The new checkout might have improved genuine orders. The change might instead reflect traffic mix, tracking behaviour or a difference in how the versions counted success.
- Look for contradictions. The reported conversion lift did not fit comfortably with the flat revenue trend. That did not settle the case, but it was a reason to investigate.
- Obtain the missing evidence. Payment records showed which attempts became confirmed orders and allowed the team to compare like with like.
- Choose the next action. The evidence did not justify a full rollout. The team fixed the measurement and prepared a valid retest.
- Keep learning. The release process could now include a check that experiment events and conversion definitions stay consistent across versions.
A dashboard is a clue, not a verdict
The first report was not useless. It was a clue. The mistake was treating it as a verdict before checking what had been counted.
That is the value of an investigation framework. It gives people a way to move from “the number changed” to “what could explain the change, what evidence would separate those explanations, and what should we do next?”
SHERLOCK is inspired by Holmesian investigation, but the eight steps are a contemporary framework for analytics, experimentation and problem-solving. The point is not to imitate a detective. It is to make the reasoning visible, challenge the first explanation and choose an action the evidence can support.
When a test looks like a winner, ask one more question before celebrating: what would we expect to see if that explanation were true?
