A/B testing: make the comparison trustworthy

A/B testing compares alternatives using a planned experiment. It can help assess whether a change improves a meaningful outcome, but it cannot turn a small or biased comparison into a dependable answer.

Start with a question and a hypothesis

Identify the customer problem, the proposed change and the outcome it is expected to influence. For example, a clearer explanation before an enquiry form may improve the proportion of suitable visitors who complete it.

Choose a primary outcome and relevant guardrails before starting. More button clicks may be unhelpful if they produce more invalid enquiries or make another important task harder.

Compare suitable groups at the same time

For a conventional online controlled experiment, assign eligible users randomly to control and variation during the same period. Keep assignment consistent enough for the journey being tested.

Showing version A for a week and version B the following week is a before-and-after comparison. Seasonal demand, traffic mix or another campaign could explain the difference. It should not be described as the same evidence as a concurrent randomised test.

Microsoft's experimentation research explains the role of randomisation and experimental design in assessing causal effects.

Make the comparison interpretable

Changing one well-defined element can make the result easier to explain. Testing a complete alternative journey can also be valid if that is the decision; it tells you about the package, not the independent contribution of each changed component.

Multivariate designs answer different questions and require appropriate planning and data. Avoid changing several unrelated elements casually and then claiming to know which one caused an effect.

Plan sample size and duration

Consider the baseline outcome rate, the smallest effect worth acting on, the available traffic and the statistical approach. Allow for relevant business cycles and delays between exposure and outcome.

There is no universal minimum number of visitors or a fixed seven-day rule that makes every test valid. A few extra conversions may be noise. Low traffic can leave an experiment inconclusive even when the idea is sensible.

Do not stop simply because a dashboard briefly shows a favourable result. Choose an appropriate stopping and analysis method before the test, or use a method explicitly designed to support sequential decisions.

Check the experiment itself

Verify that assignment, delivery and measurement work as intended. Check for uneven allocation, duplicate events, missing outcomes, device-specific failures and interference from other changes.

Document exclusions and practical limitations. If the variation fails to load for part of the audience, analysing only the successful exposures can create a misleading comparison.

Interpret the result with uncertainty

Report the estimated effect and its uncertainty, not just a winner badge. Statistical evidence does not automatically make an effect commercially useful, and an inconclusive result is not proof that the versions are identical.

Review guardrails and customer feedback before adopting a change. The primary outcome might improve while support needs, accessibility or lead quality deteriorate.

Choose another method when appropriate

Use usability research, interviews, technical diagnostics or expert review when the immediate question is why a task fails or whether a known defect exists. You do not need an experiment to justify repairing a broken form or correcting misleading information.

Testing is most useful when it resolves a meaningful uncertainty and the organisation can support a credible comparison. It is a decision tool, not a requirement to test every colour change or a promise of quick certainty.