CRO research and testing leads to big wins for this B2B manufacturer.
Improving the marketing funnel approach for this property investment business drives record growth.
Discover your marketing performance issues.
Take our 5 minute assessment to find out & receive your free report.
Discover your marketing performance issues.
Has your test reached statistical significance?
Converted’s CRO Test Significance Calculator is a fast and easy way to check whether a test is at a point where you can effectively draw a conclusion. Testing software should offer this, but just in case you’ve run your own!
Statistical significance is all about the probability of whether the results you see in a test are a fluke or not. So if you have a significance score of 95%, it is in fact saying that there is only a 5% chance that your Null Hypothesis is true and that you can be 95% certain that the data you are seeing from the test is not random. Statistical significance is used to help you understand the confidence you should have is a set of test results.
Null Hypothesis can be most easily explained as the following: ‘If I run this CRO A/B test then the new variation will not deliver a significant change.’ It means the hypothesis of your test is that there won’t be a significant change. By adopting the Null Hypothesis it means you set out in every test to prove yourself wrong. Not prove that your new variation is right. This is a key difference in mindset and approach. The P-Value is the figure that will help disprove your null hypothesis.
The easiest way we can explain the P Value is that if it is converted into a percentage (a 0.25 P Value becomes 25%), this is the percentage chance that your test is a fluke and your Null Hypothesis is true. P-Value runs between 0-1 with a low P Value number (say 0.05 for example) indicating a strong probability that your null hypothesis is false and the variant has a high statistically significant probability of being the winner if you were to run the test again and again. A P-Value close to 0.05, say 0.06-0.07 is considered marginal.
When we look at the P-Value you have to consider the scale. So as we mentioned a 0.05 P-Value indicates that there is only a 5% chance that the variant would not win if the test was run again or a 5% chance the result of the test was a fluke. However a 0.95 P-Value is the opposite. It is saying there is a 95% chance that the variant won’t win if the test is to be run again. Either way the result has reached statistical significance.
