Key takeaways
- Decide the sample size before you start.
- Smaller lifts need far more visitors.
- Do not stop at the first significant result.
Inputs you need
You need the current conversion rate, the relative lift you care about and your traffic. A baseline of 3% and a target lift of 20% means detecting a move from 3.0% to 3.6%.
The result
That test needs about 13,900 visitors per variant, or 27,800 in total. At 1,000 visitors a day, plan for 28 days. Halving the lift to 10% raises the requirement to about 53,200 per variant.
Protect the result
Stopping early inflates false positives. Decide the end date in advance, run whole weeks and change only one thing at a time.
Free tool / Clinical trials
A/B Test Planner How many visitors before a result is valid?
Questions about Unit Economics and Measurement
What if I lack the traffic?
Test bigger changes, test higher-traffic pages or use a less strict confidence level with care.
What is statistical power?
It is the chance of detecting a real effect of the size you chose.
Key terms
Keep reading