Flip a fair coin 10 times and you might get 7 heads. Nothing’s wrong with the coin. That’s just luck. A/B tests work the same way. Even if your new variation makes no difference at all, one version will usually edge ahead purely by chance.
Statistical significance is the point where the gap between your original and variation is too big to put down to luck. Most CRO tests set that bar at 95%.
Here’s what 95% actually means. Imagine running 100 tests where the change did absolutely nothing. At 95%, around 95 of them would correctly show no winner, and about 5 would throw up a false winner from random noise alone. So 95% isn’t “we’re 95% sure this variation is better”. It’s “a change that does nothing will only fool us about 1 time in 20.”
That’s also why research matters. The better your ideas are to begin with, the more of your winners will be real.
And one rule worth tattooing somewhere visible is – don’t stop a test the moment it turns significant. Check often enough, and you’ll catch luck at its peak, and that 1 in 20 becomes a lot more than 1 in 20. Work out how long your test needs first (our test duration calculator will help), then let it run.
