Picture a client with a mature CRO programme. A proper test backlog, solid hypotheses grounded in user research, decent traffic volumes, statistical rigour. Tests run for the right duration. Results come back flat. Inconclusive. The next hypothesis goes in. Also flat.
The instinct is to interrogate the hypotheses. Run more research. Sharpen the briefs. Maybe the sample sizes are still too small. Maybe the segment is wrong.
More often than not, the problem is somewhere nobody is looking: the technical foundation the tests are running on.
The stack is the experiment
Jono Alderson wrote recently about how the modern web got rebuilt for developers rather than users. The argument is blunt and correct: around 2010, the industry decided websites should feel like apps, and what followed was an arms race of complexity that never really served the people clicking on things.
The CRO implications of that are direct and largely undiscussed.
When a product page loads via a JavaScript framework with a hydration strategy, a separate API layer, and a build pipeline that takes three steps to update a heading, you are not testing a clean surface. You are testing on top of all of that.
Render delays, content flicker, hydration failures, JS errors that only surface in certain browsers: these do not show up cleanly in your testing tool. They show up as noise in your results.
As Alderson puts it: “Today, we optimise for ‘DX’. Developer experience. Not user experience. Not performance. Not outcomes.”
That is exactly the environment most CRO programmes are running inside. The conversion rate you are trying to improve is already suppressed by the choices that were made before you arrived. When every test comes back flat, it is worth asking whether the signal is even legible against that amount of background drag.
A/B tests get scrapped because the analytics layer is not compatible with the hydration strategy. This is not a theoretical risk. It is a named, documented failure mode from the very teams building these stacks.
Speed is not a separate workstream
There is a habit in organisations of separating performance work from conversion work. CRO sits with marketing or product. Performance sits with engineering. They have different backlogs, different owners, different success metrics.
This separation costs money.
Perceived speed at the moment of decision, specifically the product page, the checkout, the form submission, is one of the most direct variables in whether a user completes a purchase. It does not need an A/B test to validate. The relationship between load time and conversion rate is well-documented. The question is not whether speed matters; it is whether anyone has joined it up to the CRO programme.
Alderson’s piece on speculation rules is technically focused on the mechanics and misuse of browser preloading, but the underlying capability is genuinely useful for conversion work. Speculation rules let browsers prefetch or prerender pages before a user clicks. For a checkout flow, where the next step is predictable, prerendering the shipping page before the user leaves the cart page can make the transition feel instant. That is not a marginal gain. Perceived speed at the point of commitment is a real conversion variable, and it is available without running a single A/B test, if the architecture supports it.
Most architectures do not support it well, because they were not designed with that outcome in mind. They were designed around engineering preferences.
The measurement problem
The feedback loop that CRO depends on is: observe behaviour, form a hypothesis, test, measure the result, improve. It has worked well for a long time because the web was, broadly, observable.
That assumption is cracking.
Alderson’s piece on opaque marketing is written from an SEO and brand perspective, but the argument maps directly onto CRO. Users increasingly arrive not through a traceable paid or organic channel, but via an AI summary, a forum recommendation surfaced by a chatbot, a product mention in a newsletter that your analytics will attribute to direct. The journey before they hit your site is mediated by systems that do not generate the clicks, sessions, and referrer data that your funnel analysis depends on.
“When an AI assistant decides what to recommend based on Reddit sentiment… your analytics stack won’t capture it.” That is Alderson’s framing, and it is not distant-future speculation. It is happening now.
For CRO specifically, this creates a problem with baseline assumptions. If a meaningful portion of your converting users arrived via channels you cannot see, with expectations and mental models formed by sources you cannot audit, your funnel analysis is working with incomplete information. The user who converts after reading three Reddit threads and an AI-generated comparison does not look different in your session data from the one who found you through a branded search. But their journey, their intent, and what convinced them to buy are entirely different.
This is not a reason to abandon testing. It is a reason to stop treating session data as the whole picture.
What this means for CRO programmes in practice
Three things follow from all of this.
Audit the stack before you audit the hypotheses. If Time to Interactive on your product pages is above three seconds, that is almost certainly your biggest conversion lever. Not the button colour. Not the headline variant. The architecture. Any CRO programme that ignores the technical baseline it is operating on is working around a problem rather than addressing it.
Performance optimisation belongs inside the CRO remit. Not as a separate engineering ticket with its own OKRs. The two disciplines are working on the same outcome, user completion of valuable actions, through different mechanisms. Treating them as separate workstreams creates the conditions where a CRO team runs tests on a platform that a performance audit would have condemned. The CRO practitioner does not need to do the engineering work, but they need enough authority and enough relationship with the engineering team to make performance a shared priority.
Broaden the measurement framework. Session data is one signal. Post-purchase surveys tell you what actually influenced the decision. Qualitative research surfaces friction that heatmaps miss. Dark social listening and forum monitoring give you access to the pre-site journey. None of these replace quantitative testing, but together they give a more honest picture of whether your programme is measuring the right things. The funnel you can see is only part of the funnel.
The bottom line
The most expensive CRO problem most organisations have is not a weak test backlog. It is that they are optimising the paint job on a building with structural problems. Fix the foundation, make the measurement honest, and then test. Not the other way around.