A/B testing basics: hypotheses, sample size and significance

An A/B test shows two versions of something to comparable groups of people at the same time and measures which one does better. It is the most reliable way to learn whether a change to a website or app caused an improvement, rather than just happening alongside one. It is also easy to get wrong in ways that produce confident, false answers.

This guide covers the basics: writing a hypothesis, working out how many visitors you need, reading significance, and the traps that catch most teams. It is written for founders and analysts running tests on modest traffic, not for data science teams with dedicated platforms.

What an A/B test is

Harvard Business Review describes A/B testing as one of the simplest kinds of randomised controlled experiment. Visitors are randomly assigned to version A (usually the current one) or version B (the change), and you compare a success metric between the two groups. Wikipedia notes that Google ran its first such test in 2000, to decide how many search results to show, and that by 2011 it was running more than 7,000 a year.

Random assignment is the whole point. Because the only systematic difference between the groups is the version they saw, a real difference in results can be put down to the change. Comparing this month with last month cannot do that, because too many other things change at the same time.

Start with a hypothesis

A test without a hypothesis is a fishing trip. Write it down before you build anything, in a form like this:

Because [evidence], we believe that [change] for [audience] will [effect on metric]. We will know when [metric] moves by [amount].

Choose one primary metric in advance. Wikipedia gives the reason: a variant that wins on clicks may not win on sales, so the goal has to be defined before the test, not picked afterwards. You can watch a few secondary metrics as guardrails (refunds, support contacts, page speed), but the decision rests on the primary one.

Work out the sample size first

The size of the test depends on two things: how often the outcome happens now (the baseline rate) and the smallest change you care about detecting (the minimum detectable effect). Small effects need very large samples.

Evan Miller, in his widely cited article How Not To Run an A/B Test, gives a rule of thumb: n = 16σ²/δ², where δ is the minimum effect you want to detect and σ² is the expected variance. For a conversion rate p, the variance is p × (1 − p).

An example (illustration only). An online shop converts 4% of visitors. The owner wants to detect a rise to 5%, a change of one percentage point.

• σ² = 0.04 × 0.96 = 0.0384
• δ = 0.01, so δ² = 0.0001
• n = 16 × 0.0384 ÷ 0.0001 = 6,144

That is roughly 6,100 visitors in each version, or about 12,300 in total. If the shop gets 1,500 visitors a week, the test needs around eight weeks. To detect a smaller lift, from 4% to 4.5%, the number roughly quadruples. Many small businesses discover at this step that they do not have enough traffic to test small tweaks, which is useful to know before spending a month on it. Use a proper calculator, such as the one on Miller’s site, to set your exact significance level and power.

Reading significance

Statistical significance answers a narrow question: if there were really no difference between A and B, how likely is a result at least this large? A common threshold is 5%. Passing it means the result is unlikely to be pure chance. It does not mean the effect is large, important or certain to last.

Wikipedia notes that different data need different tests. Rates such as conversion are often compared with tests like Fisher’s exact or chi-squared, while averages such as revenue per user often use Welch’s t-test. Most testing tools choose for you, but check what yours does.

The traps

Peeking. This is the big one. Miller shows that checking results repeatedly and stopping as soon as they look significant inflates false positives badly. In his worst-case example, a test with no real effect was called significant 26.1% of the time instead of the expected 5%. His fix: fix the sample size in advance and stick to it “with near-religious discipline”. HBR lists reacting to early data as a common mistake for the same reason.

Sample ratio mismatch. If you split traffic 50/50 but end up with 52,000 in one group and 48,000 in the other, something is broken. Microsoft’s experimentation team, writing on sample ratio mismatch, reports that about 6% of its A/B tests have this problem, and warns not to trust results until the cause is found. Common causes include redirects, bots and tracking that fires differently in each version.

Too many metrics. Check twenty metrics and one will look significant by chance. HBR also warns against tracking too many measures instead of the few that matter.

Slicing after the fact. Finding that B won “for mobile users in Scotland on Tuesdays” is usually noise. Wikipedia notes that segment analysis works best when it is designed into the test from the start.

Stopping at a partial week. Behaviour differs between weekdays and weekends. Run tests in whole weeks.

Never retesting. HBR recommends repeating important tests to rule out false positives before you build on them.

A short checklist for each test

1. Write the hypothesis and pick one primary metric.
2. Calculate the sample size and the number of weeks it needs.
3. Check that both versions track correctly before launch.
4. Launch, and do not stop early.
5. When the sample is reached, check the traffic split first, then the primary metric.
6. Write up the result, including tests that showed no difference.

When not to A/B test

Do not test when traffic is too low to reach a sensible sample in a few weeks. Make the change if the reasoning is strong, and watch the numbers. Do not test fixes for clear bugs or accessibility problems; just fix them. Do not test changes that need weeks of learning to show value, such as a new workflow, because short tests favour novelty. And do not use an A/B test to settle a question that five customer interviews could answer faster. Testing tells you which version performed better. It rarely tells you why.