Now let me start this by saying that without hypothesis testing you are just shooting in the dark. A solid hypothesis will not only tell what you are hoping to change but why you expect it to change. It should also include some evidence why you believe this will happen as well as the measures you are going to use to validate the test. As well as being able to prove your hypothesis you should also be able to disprove your hypothesis
Now thats how it should be done how is it used in most cases? Well, let’s say you notice the main CTA on the homepage of your site is the same colour as everything else on your site. Instinct and experience tell you that this a quick win and most likely does not even need testing. However your a conscientious CRO pro so you decide to test it with the optimal colour on the colour wheel and some variations.
So you have made your observation, then you find some data that shows that only 10% of users who saw the CTA click on it. Now you right your hypothesis.
“As only 10% of people who see the CTA click on it, I believe that by changing the colour of the CTA to something more prominent we can increase the number of people who click on it”
So this is your hypothesis. You run your test and presto more clicks. Job done, right?
If only it was so simple. Have the increased clicks led to any more conversions? no. You have optimised the page but not the decisions and created a shift in user behaviour. Instead of everyone dropping out on the homepage they drop out on the next step.
What would have happened if the test lost does that mean colour contrast is not important on your homepage? Again no.
The simple fact is you noticed something that didn’t look right, you found some data that supported your pre-determined conclusion and you ran a test.
The simple truth is your hypothesis was weak. A much better one would have looked like the below.
“Based on the poor colour contrast I believe that by changing the CTA colour to something more prominent that we shall see an increase in the primary metric of users who see the button clicking on it we may also see an increase small increase in conversions but don’t expect this to be significant. If we see no difference in CTR then we believe that the colour of the CTA is the main problem affecting CTR.
You see we have made it abundantly clear what we expect to happen and why. I have left out the data because it wasn’t really strong enough but when something is so black and white as poor CTA colour contrast it doesn’t really require it
Would love to hear your thoughts on hypotheses