A/B testing looks simple: show two versions, count the winners, ship the better one. In practice, most tests are quietly invalid. The test ended too early, the sample was too small, the change was tested at the wrong time of year, or the "winner" was declared from noise that a fair coin would have produced. The result is a roadmap full of confident decisions built on nothing.
The good news is that the common mistakes are all avoidable with a few disciplines. This guide covers the seven that invalidate most tests and the habits that replace them.
Mistake 1: Ending the test too early
The most common error is declaring a winner the moment the variant looks ahead. Early results are noisy; a variant can be up by 20 percent after a hundred visitors and down by 10 percent after a thousand. If you stop at the first exciting number, you are shipping noise.
The fix: decide the sample size before the test starts (a calculator will tell you how many visitors you need for a meaningful result) and do not peek. Let the test run to its planned end.
Mistake 2: Ignoring statistical significance
Significance is the probability that your result is not luck. The convention is 95 percent: a result with 95 percent confidence has about one chance in twenty of being a fluke. Testing below that and shipping the winner is how teams accumulate a graveyard of "improvements" that did nothing. Run tests until the confidence level reaches your threshold, and treat anything below it as inconclusive, not as a loss.
Mistake 3: Testing too many things at once
If you change the headline, the image and the button color in one variant, you will know the combination "won" but not which element did the work. Isolated variables give clean answers; compound changes give marketing theater. Test one element at a time, or use proper multivariate methods, and you will build a library of knowledge instead of a collection of guesses.
Mistake 4: Testing during abnormal periods
Holidays, launches, email campaigns and site outages all distort behavior. A test run during Black Friday tells you about Black Friday visitors, not about your normal traffic. Check your calendar before starting a test, keep an eye on external events while it runs, and be ready to pause and restart when the world does something unusual.
Mistake 5: Small sample sizes
With a low-traffic page, reaching statistical significance can take months, and running a two week test on a page that gets fifty visitors a day is a coin flip with extra steps. If your traffic cannot support reliable tests, change your approach: test higher-traffic pages, use larger changes that produce bigger signals, or focus on qualitative feedback until traffic grows.
Mistake 6: Ignoring segments
An overall "no winner" can hide a strong segment result. The variant that is flat for everyone might be up 30 percent for returning visitors, or for mobile users, or for visitors from a specific campaign. Analyze your results by traffic source, device and new versus returning before you bury the test. Segmenting after the fact is not cherry picking when you decide the segments before you look at the data.
Mistake 7: Never testing again after a loss
A losing test is not a failure, it is a data point. Knowing that "urgency copy does not move this audience" saves you from repeating the experiment and points you toward what might work instead. Teams that treat every test as a verdict stop testing after a few losses; teams that treat tests as experiments keep learning and compound their wins.
The discipline that makes testing reliable
The reliable testing loop looks like this: form a hypothesis with a reason ("visitors hesitate at checkout, so a guarantee line will lift completion"), decide the sample size and success metric in advance, run the test clean, read the result with significance and segments, and ship or discard based on the data, not the hope. Run this loop consistently and the mistakes above stop being risks and start being lessons you have already learned.
When should you skip the A/B test?
Testing is not always the right tool. If the change is a fix (a broken button, an unreadable font), just ship it. If the sample would take months to fill, skip the test and use session recordings, surveys or customer feedback instead. And if the decision is strategic (a new pricing model, a repositioned landing page), testing a subtle variant will not answer it; the strategy needs a launch and a measurement plan, not a button experiment.
FAQ
Q: How long should an A/B test run? A: Long enough to reach statistical significance with your planned sample, and at least one full business cycle (a week) to cover weekday and weekend behavior. Never stop a test purely because a date arrived if the sample is not complete.
Q: What is a good sample size for A/B testing? A: It depends on the size of the effect you expect. A calculator will tell you: for a typical 10 percent lift with 95 percent confidence, you usually need thousands of visitors per variant. Small pages cannot support reliable tests.
Q: Can I test with very little traffic? A: Yes, but change what you test: bigger, more obvious changes produce bigger signals, and you can test on your highest-traffic pages only. Below a few thousand visitors per variant per month, rely more on qualitative research.
Q: What should I test first on my website? A: The highest-leverage elements: the headline, the core offer, the CTA copy and the page structure. Those move conversion far more than button colors or font sizes.
Q: What does 95 percent confidence actually mean? A: It means that if the true effect were zero, you would see a result this extreme about 5 percent of the time by chance. It is a guardrail against noise, not a guarantee, which is why replicating important wins matters.