How much traffic do you need for A/B testing?

For a straightforward comparison of two versions, a widely used approximation puts the visitors needed per variant at sixteen times the baseline rate multiplied by one minus the baseline rate, divided by the square of the absolute difference you want to detect. That gives roughly eighty percent power at the usual significance level.

The important part is the square in the denominator. Halving the change you want to detect quadruples the traffic you need. This is why a store can measure a redesign of a product page and cannot measure a new button colour, and why an ambitious test is often the only kind worth running at moderate volume.

Be careful to express the difference in absolute terms. Moving from two percent to two and a half percent is a half point difference in absolute terms, and putting the relative figure into the formula understates the requirement badly.

Run the calculation before designing the test. If the answer is more visitors than you will see this quarter, you have learned something useful before spending any development time.

Reaching the visitor count quickly does not make a test valid. Buying behaviour varies by day of week, by payday, by weather and by campaign schedule, so a test that runs for four days measures those four days rather than your store.

Run whole weeks, and at least two of them. That covers the weekday and weekend pattern and gives the traffic mix time to look like normal traffic rather than like whatever campaign happened to be live.

Allow for the gap between visit and purchase. In categories where people consider for several days, a visitor who saw the test on day one may buy on day nine, and stopping the test on day seven attributes their eventual order to nobody.

Avoid running a test across an event that changes behaviour. A sale, a stock-out on a hero product, a public holiday or a large campaign launch will move the numbers more than the change you are testing, and no amount of statistics will separate them afterwards.

Test bigger things. A store that cannot detect a small refinement can often detect a wholesale change to a page: a different structure, a different offer, a different set of information. Those are also the changes most likely to matter.

Move up the funnel. Add to cart happens far more often than purchase, so a test measured on add-to-cart rate needs far less traffic to read. Treat it as a proxy, confirm afterwards that orders moved too, and be alert to changes that raise adds and lower purchases.

Use before-and-after measurement carefully where a split test is impossible. It is weaker because the world changes between periods, and it is not worthless if you segment by traffic source, compare like periods and avoid confounding events.

And spend the effort on research instead of on statistics. Session recordings, on-site surveys, support tickets and watching five real people use the checkout will tell a low-traffic store more in a week than an underpowered test will tell it in a quarter.

Stopping when the result looks good is the biggest one. Checking daily and ending the test the moment significance appears converts a rigorous method into a search for a favourable moment, and it produces winners that do not repeat.

Decide the sample size and the end date before you start, then look at the result once. If you need to monitor a test in flight, monitor it for breakage rather than for outcome.

Check that traffic actually split as intended. A large imbalance between the groups points at a technical problem with the assignment, and any result from an unbalanced test should be discarded rather than interpreted.

Run one test at a time on the same journey. Two overlapping tests on the product page and the cart interact, and the reported results for each will include the effect of the other. And record what you tested and what happened, including the failures, because a store's own testing history is the most valuable evidence it will ever have.