Activity⚡ Medium

A/B Testing

Let real user behaviour settle design debates — not opinion. A/B testing shows two versions to different groups of real users simultaneously and measures which produces better outcomes with statistical confidence.

Conversion OptimisationOnboarding FlowsCTA DesignPricing PagesNavigation TestingEmail Campaigns

01 — TL;DR

Two sentences.

A/B testing is the practice of simultaneously showing two versions of a design — the control and the variant — to randomly assigned segments of real users and measuring which produces better outcomes on a defined metric, with the result valid only at statistical significance. It is the only method that tells you with quantifiable confidence whether a specific design change improves a specific metric in real-world conditions.

The method originated in agricultural experiments and was formalised by Ronald Fisher, becoming widespread in digital product design in the 2000s. For product designers, A/B testing makes design decisions empirical. A debate that has run for three weeks in Slack can be resolved in three days of live testing, with a result that is reproducible and not subject to the seniority of whoever argues most persuasively.

02 — When to Use

Apply this when…

→You have sufficient traffic to detect a meaningful difference within a reasonable timeframe

→The change is isolated enough to test one variable — multiple changes make attribution impossible

→The outcome metric is clearly defined and measurable before the test begins

→A design decision has been contested internally and empirical data would settle it

→A significant change to a high-traffic flow needs confidence before full rollout

When NOT to apply it

Skip it when traffic is too low for significance, when the change is too fundamental for controlled comparison, when the question is why (use usability testing), or when the team lacks engineering infrastructure for clean variant serving and reliable tracking.

03 — How It Works

The mechanism

A/B testing isolates a single variable, randomly assigns users to control or variant, measures a defined metric simultaneously, and evaluates whether the difference is statistically significant. Every element — isolation, randomisation, simultaneity, statistical threshold — is load-bearing.

Statistical significance prevents false conclusions

A result with p < 0.05 means less than 5% probability the difference is random. Reaching this requires sufficient sample size for the effect being measured. Running a test until it shows significance — rather than to the pre-calculated sample size — produces false positives at dramatically higher rates than the 5% the p-value implies.

One variable, one metric, one decision

Testing a page where the variant changes headline, button colour, layout, and image simultaneously tells you which combination won — not which element drove it. One variable per test is the condition that makes learning possible. The metric should also be singular and pre-defined.

A winning test can still be the wrong decision

A variant that increases clicks while reducing downstream conversion has won the test and lost the product. Teams that optimise CTA clicks without measuring what users do after clicking are making local improvements that compound into global degradation. The metric choice is a design decision.

Pre-calculate sample size before starting

Calculate required sample size using baseline rate, minimum detectable effect, 80% power, and 95% confidence. This determines how long the test must run. Running shorter produces underpowered results. Running longer after peeking produces false positives. The result is read exactly once — at the end.

A null result is a result

A test showing no significant difference is not a failure — it prevents shipping a change that requires engineering effort, increases complexity, and produces no measurable benefit. The goal is not to confirm every change works — it is to find out which ones do and which do not, and act on both answers equally.

04 — Real Example

Booking.com's culture of radical experimentation

Booking.com runs over 1,000 simultaneous A/B tests at any given time. Their culture assumes no design decision should ship without a test, and every result — including null results — is treated as signal. Their conversion improvements over twenty years are the compound effect of thousands of small, validated changes.

The deeper insight is institutional: A/B testing at scale requires a culture where shipping untested changes is the exception, and where null results are valued as contributions that prevent unvalidated code accumulation. Most teams test occasionally; Booking.com tests as the default mode of making any testable decision.

Booking.com · Experimentation Culture

Testing as default mode rather than occasional validation

1000+ concurrent tests → compound improvement

05 — In the Wild

Test yourself & see real examples

No examples yet — be the first.

Spotted a product that clearly runs disciplined A/B testing — or one where untested changes have degraded the experience? Submit what you observed.

✓ Reviewed before publishing✓ Your name on every example you submit✓ Violation or fix — both welcome

Seen A/B Testing misused in a real product? Help grow the evidence base.

06 — Common Mistakes

Where teams go wrong

✗Peeking at results and stopping early. Checking results daily and stopping when significance is reached inflates false positives from 5% to 25-30%. Pre-calculate sample size, then read the result once at the end.

✗Testing multiple variables without factorial design. Changing headline, button, image, and layout simultaneously means if the variant wins, no individual change can be credited. One variable per test is the condition for actionable learning.

✗Choosing the metric to report after seeing results. Measuring twenty metrics and reporting the one that moved is a false positive factory. The primary metric must be chosen before the test begins.

✗Declaring winners based on early data. Early traffic skews toward power users and specific acquisition channels. A test that wins in week one and loses in week two was never converged. Run to the pre-calculated sample size in full.

07 — Variations & Related Principles

Connected ideas

A/B testing sits within a broader experimentation toolkit. Understanding when each method is appropriate prevents applying A/B testing to questions it cannot answer.

The most important pairing is A/B testing with Usability Testing. Usability testing diagnoses what is wrong and why; A/B testing validates whether a fix works at scale. Teams that only A/B test spend months testing variants of a broken design. Teams that only usability test cannot confirm fixes work for the full population. Together they form the complete validation loop.

08 — 10-Min Exercise

Run it right now

⏱ 10 minutes · Solo · No prep

The Test Readiness Audit

1. Pick one design change your team is planning. Write the single metric that would tell you whether it worked. If you cannot name one, the change is not ready to test.

2. Estimate the baseline value of that metric. If you do not know it, write "unknown." An A/B test cannot be designed without a baseline.

3. Write the minimum improvement worth shipping — the smallest effect that justifies the engineering effort. Be specific: "1% absolute improvement in checkout conversion."

4. Use baseline, minimum effect, and standard 80% power / 95% confidence to estimate required sample size. Rough rule: detecting a 10% relative improvement on a 5% rate requires ~35,000 users per variant. If your traffic cannot deliver that in under eight weeks, the decision should be made by another method.

10 minutes