A/B Testing Basics: Find Out What Really Works Instead of Guessing

Admin Cream · · upskilling
A/B Testing Basics: Find Out What Really Works Instead of Guessing

What A/B testing is, how to write a hypothesis, what to test first, how long to run it, and the mistakes that make results misleading, with email and landing page examples.

In a meeting, the team argues whether the landing page button should say Sign up free or Try 14 days. One person likes the first because it is short, another prefers the second because it is clear. In the end the manager picks a personal favourite. If you have been in that room, A/B testing is the way to let customers decide instead.

What is A/B testing?

A/B testing randomly splits real users into two groups. One sees version A, the other version B, and the versions differ in only one thing. You then measure which version produces more of the outcome you want, such as clicks, sign-ups or purchases. Because both groups see their version at the same time, any difference in results most likely comes from the change you made, not from a holiday or another promotion.

What you can test

Five steps to a good test

1. Write a clear hypothesis

Use the format: if we change X to Y, metric Z will improve, because of reason W. Example: If we cut the sign-up form from 7 fields to 3, the sign-up rate will rise, because customers complain in chat that the form is long. A hypothesis with a reason lets you learn even when the result surprises you.

2. Choose one primary metric

Decide before you start what you will look at, such as sign-up rate, rather than picking whichever looks good afterwards. Set secondary metrics to watch for side effects, for example whether a shorter form lowers lead quality.

3. Change one thing at a time

If you change the headline, image and button together and results improve, you will not know why. Testing several elements at once is possible but needs more complex methods and many more users.

4. Set duration and sample size in advance

Use a sample size calculator, freely available online, to estimate how many users each group needs, and run the test for at least one full week, since weekday and weekend behaviour usually differ.

5. Read results carefully

Check whether the difference is statistically significant. Most testing tools calculate this for you. If the difference is tiny and not significant, treat it as we do not know yet, not as B wins. Record every result in a shared team document, including tests that did not win.

Mistakes that make results misleading

Example: testing an email subject line

An online baking supplies shop wants more opens on its monthly promotion email. The team hypothesises that a subject line stating a clear benefit will beat one focused on the discount. Half the list gets version A, 20% off everything, this month only, and the other half gets version B, a soft bread recipe that stays fresh for 3 days, plus items on sale. Both go out at the same time, and the team measures open rate and clicks to product pages. Whatever the result, they have data to use in next month's email.

A test that does not win is not a failure. It is learning that costs far less than a year of wrong guesses.

If your traffic is still low

Many small businesses do not yet have enough users for reliable A/B tests. Alternatives: test in higher-volume channels, such as email subject lines to your full list or two ad versions, or talk to 5 to 8 customers to find the big problems first, then test once you have enough traffic.

How to start

  1. Pick the one page or email series with the biggest revenue impact.
  2. Write a hypothesis from real data, such as customer complaints or where people leave the page.
  3. Set the primary metric, sample size and duration before starting.
  4. Let the test run to the end without stopping early.
  5. Record results and lessons, then plan the next test.

Read more on data-driven marketing and marketing KPIs worth tracking, or browse courses from our learning partners on data analysis.