The problem with Hypotheses

Now let me start this by saying that without hypothesis testing you are just shooting in the dark. A solid hypothesis will not only tell what you are hoping to change but why you expect it to change. It should also include some evidence why you believe this will happen as well as the measures you are going to use to validate the test. As well as being able to prove your hypothesis you should also be able to disprove your hypothesis

Now thats how it should be done how is it used in most cases? Well, let’s say you notice the main CTA on the homepage of your site is the same colour as everything else on your site. Instinct and experience tell you that this a quick win and most likely does not even need testing. However your a conscientious CRO pro so you decide to test it with the optimal colour on the colour wheel and some variations.

Image result for colour wheel

So you have made your observation, then you find some data that shows that only 10% of users who saw the CTA click on it. Now you right your hypothesis.

As only 10% of people who see the CTA click on it, I believe that by changing the colour of the CTA to something more prominent we can increase the number of people who click on it”

So this is your hypothesis. You run your test and presto more clicks. Job done, right?

If only it was so simple. Have the increased clicks led to any more conversions? no. You have optimised the page but not the decisions and created a shift in user behaviour. Instead of everyone dropping out on the homepage they drop out on the next step.

What would have happened if the test lost does that mean colour contrast is not important on your homepage? Again no.

The simple fact is you noticed something that didn’t look right, you found some data that supported your pre-determined conclusion and you ran a test.

The simple truth is your hypothesis was weak. A much better one would have looked like the below.

Based on the poor colour contrast I believe that by changing the CTA colour to something more prominent that we shall see an increase in the primary metric of users who see the button clicking on it we may also see an increase small increase in conversions but don’t expect this to be significant. If we see no difference in CTR then we believe that the colour of the CTA is the main problem affecting CTR.

You see we have made it abundantly clear what we expect to happen and why. I have left out the data because it wasn’t really strong enough but when something is so black and white as poor CTA colour contrast it doesn’t really require it

Would love to hear your thoughts on hypotheses

17 CRO Tools to optimise your CRO campaigns

What are the top CRO tools in 2015

Like any facet of digital marketing CRO has its fair share of tools that are available to make your job a little easier and to make you look good to your client/Boss.

However with so many tools out there which do you try? well hopefully this post will help you decide and prevent you from being just another fool with another tool.

If you have any tools that I have missed just add them to the comments and I will update the article once I have given them the once over.


  1. Page monitor – This plugin allows you to “monitor” a page and it will tell you every time the page changes. Great for monitoring competitors landing pages or your own site to make sure that you are always aware of what is happening
  2. Ghostery  – Allows you to check what trackers are on a site e.g A/b testing platforms, analytics platforms etc… Great for some corporate espionage, checking that your devs have correctly installed a tracker and for blocking said trackers with ease
  3. Awesome screenshot – For full-page screen shots of all your A/B tests or other screen grabs
  4. UA Spoofer – Allows you to spoof you browser client to pretend to be some of the top devices and Operating Systems on the market
  5. Wappalyzer – uncovers the technologies used on websites. It detects content management systems, eCommerce platforms, web servers, JavaScript frameworks, analytics tools and more.
  6. Page load time – measures page load time and displays it in the toolbar. Great for showing your boss/client how long it takes your site/pages to load.


7. Formisimo – Analyses what users are doing on your forms and tells you drop out rates by field, time taken and most corrected fields
8. Hotjar for surveys, heatmaps, screen recordings – unlimited visitors for flat fee
9. Decibel insight – is a one stop shop for analytics featuring advanced heat maps, form analytics, session recording and a lot more
10. VWO – For low to mid end optimisation needs with WYSIWYG editor, heatmaps included as well as mobile testing and advanced segmentation
11. Optimizely – For mid to high-end optimisation needs with WYSIWYG editor, advanced segmentation and more API integrations then you can shake a stick at
12. Monetate – Personalisation and a/b testing platform that uses all your data plus their own to optimise and personalise user journeys
13. Unbounce – Create landing pages without IT support and host and test them against existing pages with an easy to use WYSIWYG editor
14. Survey monkey – Free basic surveys and more advanced features for paid accounts including white labeled surveys
15. Attensee – Blurs out designs/pages forcing people to use the mouse to see a small unblurred window. Basically its eye tracking done via mouse behaviour for a fraction of the cost
16. – Want to edit images but don’t have the funds or time to source Photoshop or other image editing software then this free tool is a life saver.
17. Notepad++ – Great free tool for editing code has libraries that will highlight code in multiple languages

10 more things you never thought about split testing

10 more things you never thought about split testing but you should probably test


Everyone loves a listicle so here goes with a follow-up to a popular one where we go into some less heard of ideas on what to split test in e-commerce.


1 – Most sites offer two flavours of free shipping Free for everyone or over £xx but how many people have actually tested this? I would recommend doing this as a priority but remember to measure AOV as well as conversions and if possible profit as well. Try testing on high profit and high volume items first.
2. Promo and voucher codes are ubiquitous on Ecommerce sites but have you tested including them vs not. Or displaying them as a small text link vs applying any relevant discounts automatically as above measure sales, AOV and profit for the full picture
3. Free returns or free same day shipping which works best or does offering both give you the biggest uplift
4. Web chat vs CS call centre test one vs the other or try both at the same time. What do your customers prefer
5. Type of packaging can be an unusual thing to split test but can be done send it a standard box or a shiny hamper
6. Showing most profitable products on homepage vs most popular can give big uplifts in profits and revenue, measure revenue and profit
7. Extra photos vs 3D images which convert better also make sure you measure returns as these can cost you big time.
8. Test affiliates A/B who gives you the best customer. try one for a month then the other for a month what are their relevant conversion rates and return rates
9. Test position 1 of adwords vs 2 or 3 I have seen some good results where position 2 gives similar traffic but lower cost per customer
10. Test bidding higher on mobile in the evenings and early mornings and daytime as people will tend to research on mobile and evenings then convert on tablet or desktop during the latter parts of the evening when they are home.

%d bloggers like this: