What Would Sherlock Holmes Do?
A framework for gathering evidence, challenging assumptions and solving problems.
The SHERLOCK problem solving framework helps you define the case, gather reliable evidence, consider competing explanations and decide what to investigate, test or fix.
Because having a plausible explanation is easy. Working out whether it holds up takes a bit more effort.
A dashboard tells you something happened. Now what?
Conversion is down. Someone blames the checkout. Someone else blames the traffic. The person who wanted a redesign last month has already opened their presentation.
Everyone has a theory. The question is what evidence would help us tell those theories apart.
Inspired by Sherlock Holmes, SHERLOCK gives that investigation a structure. Start with the problem, check the clues, challenge your explanation and choose an action you can justify.
You can use it in digital analytics, experimentation, user research or any situation where a confident story is getting ahead of the evidence.
The SHERLOCK framework
Work through the steps, then revisit them when new evidence changes the case. You do not have to finish every possible analysis before making a useful decision.
-
State the case
What is happening, to whom, and why does it matter?
Describe the observation without sneaking your preferred explanation into it. “Mobile checkout completion has fallen” opens an investigation. “We need a better checkout” has already picked a solution.
Define the affected audience, period, comparison and business consequence. Be clear about the decision this work needs to support.
Leave with: A problem statement, its scope and the decision to be made.
-
Hunt for evidence
What do we already know, and where could we look?
Bring together relevant analytics, user research, customer feedback, operational records and recent changes. A release log may explain something another dashboard cannot.
Record where each clue came from. Keep what you observed separate from what you think it means.
Leave with: An evidence log showing observations, sources and gaps.
-
Evaluate the evidence
Can we trust these clues enough to use them?
Check definitions, tracking, coverage, timing and comparability. Look for missing users, biased samples and changes in how the data was collected.
Three dashboards using the same broken event are still one broken source. Check whether apparently separate clues are actually independent.
Leave with: A view of what is reliable, what is uncertain and what needs checking.
-
Raise rival explanations
What else could explain what we have observed?
Write down credible alternatives before one explanation becomes the official story. A fall in completion might involve a technical fault, different traffic, measurement changes or genuine usability problems.
Include the possibility that the apparent change is ordinary variation. Focus on explanations that fit the case and would change your next action.
Leave with: A shortlist of competing explanations worth investigating.
-
Look for contradictions
What should we see if each explanation is right?
Make predictions for each explanation, then compare them with the evidence. Look for findings that challenge your favourite theory as actively as those that support it.
An absent clue only helps if your measurement was capable of finding it. “We did not record any errors” means very little if errors were never logged.
Leave with: Predictions, contradictions and an updated view of which explanations hold up.
-
Obtain the missing evidence
What investigation would best separate the remaining explanations?
Choose the next useful check. That might be a tracking audit, a segmented analysis, a usability session, a reproduced error or a controlled experiment.
Consider the time, cost and likelihood that the result will change your decision. More research is only useful when it helps resolve something that matters.
Leave with: A targeted evidence plan, an owner and a decision it will inform.
-
Choose the next action
Do we know enough to investigate, test, fix or do nothing?
Match the action to the evidence, the stakes and how easily it can be reversed. A confirmed payment fault and an unproven redesign idea need different responses.
This is the handover to “Should We Actually Test This?” SHERLOCK builds the case. That decision framework helps choose what to do with it.
Leave with: A decision, its rationale, remaining uncertainty and a review point.
-
Keep learning
What happened after the decision, and what does it change?
Check the outcome against your prediction. Monitor the measures that matter and any consequences elsewhere. Record what would trigger another investigation.
An improvement after a change can support an explanation, but timing alone does not establish causation. Keep the conclusion proportionate to the evidence.
Leave with: An updated case record and the next question, if one remains.
“Mobile conversion is down. Redesign the checkout.”
That is a proposed solution attached to an observation. We still need to establish what happened and why.
Start by defining the metric. Has purchase conversion fallen across all mobile visits, or has completion fallen among users who started checkout? Those are different cases.
| Possible explanation | What we would look for | A useful next check |
|---|---|---|
| The measurement changed | A change in event collection, definitions or consent coverage around the time of the drop. | Reconcile recorded purchases with order records and validate the checkout events. |
| The audience mix changed | More checkout starters from groups with historically lower completion, even if completion within those groups is stable. | Compare completion within relevant groups and assess how their share of checkout starts changed. |
| Payments are failing | More payment errors, potentially concentrated in a browser, payment method or release. | Inspect payment logs and try to reproduce the failure in the affected conditions. |
| Users are struggling | Repeated difficulty at a specific step, supported by behavioural evidence and user feedback. | Observe relevant users completing the task and investigate the suspected friction. |
| The apparent drop is ordinary variation | A change consistent with the uncertainty and variability in comparable periods. | Check volumes, uncertainty, seasonality and the choice of comparison period. |
Three rules for a better investigation
An observation is not an explanation
“Completion fell” describes a change. It does not tell you why it happened or what will improve it.
Repeated evidence is not always independent
A report, a dashboard and an AI summary can all repeat the same underlying mistake. Trace the source.
Your conclusion remains provisional
Make the best decision the evidence supports. Write down the uncertainty and the conditions that would make you revisit it.
Start with a case note.
You do not need a new platform or a 40-slide presentation. A shared document can hold the investigation.
- The case: What happened, who is affected and what decision is needed?
- The evidence: What do we know, where did it come from and can we trust it?
- The explanations: What could account for this, and what would each predict?
- The challenge: What contradicts each explanation, and what remains unknown?
- The next check: What evidence would most help us decide?
- The decision: What will we do, why, and with what confidence?
- The review: Who checks the outcome, when, and what would reopen the case?
A few loose ends
What is the SHERLOCK problem solving framework?
SHERLOCK is an eight-step framework created by Amrdeep “Ammo” Athwal at Conversions Matter. It covers stating the case, hunting for evidence, evaluating it, raising rival explanations, looking for contradictions, obtaining missing evidence, choosing an action and keeping learning.
Is this a method from the Sherlock Holmes stories?
It is a contemporary framework inspired by Holmesian investigation, developed for analytics, experimentation and business problem solving. The acronym and steps are my own formulation, rather than a method taken directly from the stories.
How is it different from an A/B testing process?
SHERLOCK helps establish what needs investigating and which evidence would help. An A/B test is one possible method within that work. Other cases may call for a tracking check, user research, a technical fix or monitoring.
Do I have to follow every step in order?
Use the sequence as a guide and revisit earlier steps as you learn. A small, reversible decision may need a short investigation. A costly decision with serious consequences needs stronger evidence and more scrutiny.
Where does measurement quality fit?
Evaluate the evidence is the explicit checkpoint, but quality matters throughout the investigation. Before explaining a change, check that the underlying events, definitions and comparisons can support the claim. My Measurement Trust Check is designed to help surface questions about that foundation.
Can I use SHERLOCK with my team?
Use the eight steps to structure a real problem, record your reasoning and review the outcome together. If you share or adapt the framework, please credit Amrdeep “Ammo” Athwal at Conversions Matter and link back to this page.
