How does the a/b test planner work?
Borrowed from Clinical trials: Power analysis
Clinical trials use power analysis to decide how many patients are needed before a trial starts. The same math prevents the common mistake of stopping a web test as soon as one version looks ahead.
The tool uses a two-sided test on two proportions, with 95% confidence and 80% power by default.
Worked example
A US landing page converts at 3%. To detect a 20% relative lift (3% to 3.6%), each version needs about 13,900 visitors. At 1,000 visitors a day split across two versions, that is 28 days.
How should I read my result?
Run at least one full week to cover weekday effects, even if traffic is high.
A workable range. Run the test to its planned size and do not stop early.
The test is too slow. Test a bolder change or a higher-traffic page.
Embed this calculator
Questions about a/b test planner
Why not stop when it looks significant?
Checking repeatedly and stopping at the first good result inflates false positives.
What lift should I target?
Use the smallest lift that would be worth acting on.
Does this work for revenue per visitor?
No. It covers conversion rates. Revenue metrics need a different variance estimate.