Skip to main content

Quantitative research: Multivariate test

A multivariate test is a logical continuation of an A\B test topic. Instead of comparing only two variants (A\B test) you can compare more than two - multivariate test (MVT).

Image credit.

Multivariate testing vs A\B (split) testing ?

The main difference of multivariate tests is that you can simultaneously test several elements of a page instead of testing one thing at the time (A\B test).

Let's imagine you have a headline and a button in two versions. You want to find out which combination of a headline and a button will lead to a higher conversion. With MVT you can compare each version of a button and each version of a headline. This means you will have four variants to test.

What's the difference between multivariate test and pairs of A\B tests ? 

Thing is, you need to run A\B test in isolation. To get clear, representable results you need to run one A\B test at a time on a single page. Which brings in certain limitations.
Multivariate tests normally allow you to test more variants at the same time. That will allow you to get results quicker.

Disadvantages ? 

The more options you'll test the more fragmented your traffic distribution is going to be. In a case of an A\B test your traffic split is 50\50. While if you want to test four options in a multivariate test your traffic split will be 25\25\25\25. To get reliable results - traffic groups should contain enough participants (sample size). In a case of an MVT, this requirement could be fulfilled only by a large amount of traffic.

When do A\B test and when do MVT ?

For major changes and drastically different options - make an A\B test.
For marginal improvements and conversion optimization - make an MVT. 

Have a lot of traffic and a lot of ideas ? - make an MVT. 
Have one metric to improve and best guess how ? - make an A\B test. 

Start with an A\B test, because it's simple. Continue with MVT to reach top conversion. 

An hour long webinar on A\B and MVT: 

Read more about multivariate tests:

Popular posts from this blog

Product management and operations tools - Jira Product Discovery review

  JPD is a new player in the market of product management software. Jira (and the whole Atlassian suite) has been one of the most popular tool stacks for teams to deliver software products. Now they're adding a missing piece - product discovery.

Product Vision: an elevator pitch for your product

On this blog, I write a lot about making data-driven decisions . But what if you just starting to think about your product? You have a vague idea and nothing more. No point to go for prototyping or even talking to customers as you don't know yet who to talk to and what to talk about. In such situation - start from creating a product vision.

2 simple but powerful filters for your problem and product ideas

Nowadays lots of people and companies want to innovate. They want to generate new ideas and turn them into profitable products. But how would you separate good ideas from not so good ones? How would you make sure you invest only in good ideas?