How does the a/b significance calculator work?
Borrowed from Clinical trials: Two-proportion z-test
Clinical trials settled this question long before websites existed: compare two rates, work out how often a gap that size would appear if the treatment did nothing, and call the result real only when that chance is small enough.
Enter the visitors and conversions for each variant. The test pools both rates to estimate the noise, measures how many standard errors apart they sit, and converts that distance into a p-value.
Worked example
A US landing page converts 100 of 1,000 visitors, and the variant converts 130 of 1,000. That is 10% against 13%, a z of 2.10 and a p-value of 0.036, so the result clears 95% confidence and the variant is genuinely ahead.
How should I read my result?
Significant at 95%. The gap is larger than noise would usually produce, so you can act on it.
Suggestive and not conclusive. Keep the test running rather than calling it either way.
No result yet. Either the true difference is small, or you need considerably more traffic to see it.
What the p-value does and does not tell you
The p-value is the chance of seeing a gap at least this large if the two variants were in truth identical. It is not the chance that your variant is better, and the difference between those two statements is where most A/B testing goes wrong.
A p-value of 0.04 does not mean a 96% chance the variant wins. It means that if the variants were the same, you would see a gap this big about 4 times in 100. That is good enough to act on in most commercial decisions, and it is not proof.
Why stopping early breaks the test
Checking a running test and stopping the moment it turns significant is the most common way to get a false result. Each look is another chance for noise to cross the line, so a test peeked at daily will eventually show significance whether or not any real difference exists.
Decide the sample size before you start, with the A/B test sample size calculator, then run to that number and read the result once. If you cannot resist looking, at least do not act on what you see until the planned visitors have arrived.
Read the interval, not just the verdict
A significant result whose confidence interval runs from 0.1 to 6 percentage points tells you the direction and almost nothing about the size. Planning a rollout on the assumption of 6 points when 0.1 is equally consistent with your data is how a win becomes a disappointment.
The interval is reported below the verdict for that reason. When it is wide, the honest summary is that the variant is probably better by an amount you have not measured yet.
Embed this calculator
Questions about a/b significance calculator
What counts as statistically significant?
By convention a p-value below 0.05, which is 95% confidence. The threshold is a choice rather than a law, and a costly change deserves a stricter one.
Can I stop the test as soon as it turns significant?
No. Checking repeatedly and stopping at the first good reading inflates false positives badly. Decide the sample size first with the sample size calculator, then run to it.
What does the p-value actually mean?
The chance of seeing a gap at least this large if the two variants were in truth identical. It is not the chance that your variant is better.
Is this the same as a statistical significance calculator for any metric?
It covers conversion rates, where each visitor either converts or does not. Revenue per visitor needs a different test, because its variance is not set by the rate.
Why does the confidence interval matter more than the verdict?
A significant result with an interval running from 0.1 to 6 points tells you the direction but not the size. The interval is what you plan with.