So we have all seen a page flicker when a AB test is running but a lot of people don’t realise the magnitude of the effect this can have on accurate test results.
First we need to understand why this is such a issue. A AB Test is a double blind randomised control trial. The gold standard of testing used mainly in the medical field as it produces the most reliable results when used properly.
The issue we have when using client side AB testing tools is there will always be some element of page flicker. This causes two problems. First a user will see the page without the change and then with the change. We are hardwired to notice when things change as it gives us a advantage over anything trying to hunt us and allowsus to notice when pray has moved.
Secondly when we see something new it arouses our curiosity and we are much more likely to interact with it.
So now we are much more likely to notice something and much more likely to interact with it. The test is now biased towards the change vs the control especially when the key KPI mabe interactions with the new element.
The only ways around this are:
- Use server side testing
- Hide all elements until page has fully loaded
- not use interactions with the element as the primary KPI but ones further down the funnel
The latter two options are the ones I would recommend for most organisations simply due to the cost and complexity of running server-side experimentation