CRO Test Significance Calculator

Statistical significance is all about the probability of whether the results you see in a test are a fluke or not. So if you have a significance score of 95%, it is in fact saying that there is only a 5% chance that your Null Hypothesis is true and that you can be 95% certain that the data you are seeing from the test is not random. Statistical significance is used to help you understand the confidence you should have is a set of test results.

Null Hypothesis can be most easily explained as the following: ‘If I run this CRO A/B test then the new variation will not deliver a significant change.’ It means the hypothesis of your test is that there won’t be a significant change. By adopting the Null Hypothesis it means you set out in every test to prove yourself wrong. Not prove that your new variation is right. This is a key difference in mindset and approach. The P-Value is the figure that will help disprove your null hypothesis.

The easiest way we can explain the P Value is that if it is converted into a percentage (a 0.25 P Value becomes 25%), this is the percentage chance that your test is a fluke and your Null Hypothesis is true. P-Value runs between 0-1 with a low P Value number (say 0.05 for example) indicating a strong probability that your null hypothesis is false and the variant has a high statistically significant probability of being the winner if you were to run the test again and again. A P-Value close to 0.05, say 0.06-0.07 is considered marginal.

When we look at the P-Value you have to consider the scale. So as we mentioned a 0.05 P-Value indicates that there is only a 5% chance that the variant would not win if the test was run again or a 5% chance the result of the test was a fluke. However a 0.95 P-Value is the opposite. It is saying there is a 95% chance that the variant won’t win if the test is to be run again. Either way the result has reached statistical significance.