Skip to main content

Quantitative research: A\B tests

CareLogger increased its conversion rate by 34% simply by changing the color of the sign-up button from green to red. Wow, right? Such gems of data could be obtained using quantitative research method called A\B testing. It's designed to learn which of two variants works best. With A\B tests you can compare conversion elements on a website, call2actions, design elements, marketing messages and so on. A\B tests, also called split tests, could be characterized as follows:

Small and simple

A\B tests should be small. The smaller the better. One dimensional is perfect.

More tests instead of complicated tests

Do more simple and quick tests rather than increase the complexity of a test. One piece of knowledge at the time, but often. 

One metric to define success 

The judgment of an A\B test success should be as simple as an A\B test itself. It's better to have one metric that you will use to say what variant has won. 

You can test just about anything 

Just about any element of your website or an app could be A\B tested. Those include text (product offering, C2A labels, Page headers, and titles), forms, images, videos, emails and even full web pages.  

Stages of an A\B test:


What do you want to learn from A\B test? What question you want to answer? The objective should be clear, simple and one dimensional. For example: what converts (number of clicks) better: green checkout button or red checkout button?

Qualification of users

You need to ensure equal distribution of traffic to each test variant. This means: if you want to make an A\B test on 1000 users, you need to distribute equal 500 to one variant and 500 to another. You'll need to come up with a number of participants that needs to be qualified for a test in order to collect "statistically confident" data. To do this you may reach for the help of statistics theory or use one of many available sample size calculators. Also, your traffic should be distributed randomly among the test variants.  

Running the test

Usually, A\B tests are either time based or traffic based. A time-based test will run for a specific period of time and would stop disregard the number of participants that were qualified to each test version. Those kind of tests are good when checking specific time-bound scenarios, like: "if we make "Buy" button more visible on the homepage during lunch time, will it increase the number of orders?" 
Traffic based A\B tests are the most common. In this case, you run the test until you collect results from all planned participants, in other words: when you reach your sample size. You can estimate needed sample size using calculators below.

Analyzing results 

After an A\B test finished you'll need to analyze results and prepare test report. It may include the following key elements:
  • Objective
  • Options that were tested
  • Sample size
  • Test results per a version
  • Summary - overall test outcome corresponding to the objective 
  • Suggestions or list of actions that should\will be taken after the test
When you read about A\B tests you often spot words like: "statistical confidence", "statistical significance" or "confidence interval". While the scientific (statistics) explanation is quite tricky, here is the easiest one I've heard: "Significance is a statistical term that tells how sure you are that a difference or relationship exists." Your A\B results should be statistically significant. You can either calculate results yourself or use one of many available calculators for that. Last, you need to describe actions that you will take as a result of an A\B test. 

Sample size calculator

Popular posts from this blog

Product management and operations tools - Jira Product Discovery review

  JPD is a new player in the market of product management software. Jira (and the whole Atlassian suite) has been one of the most popular tool stacks for teams to deliver software products. Now they're adding a missing piece - product discovery.

Product Vision: an elevator pitch for your product

On this blog, I write a lot about making data-driven decisions . But what if you just starting to think about your product? You have a vague idea and nothing more. No point to go for prototyping or even talking to customers as you don't know yet who to talk to and what to talk about. In such situation - start from creating a product vision.

2 simple but powerful filters for your problem and product ideas

Nowadays lots of people and companies want to innovate. They want to generate new ideas and turn them into profitable products. But how would you separate good ideas from not so good ones? How would you make sure you invest only in good ideas?