Skip to main content

Quantitative research: Multivariate test

A multivariate test is a logical continuation of an A\B test topic. Instead of comparing only two variants (A\B test) you can compare more than two - multivariate test (MVT).

Image credit.


Multivariate testing vs A\B (split) testing ?

The main difference of multivariate tests is that you can simultaneously test several elements of a page instead of testing one thing at the time (A\B test).

Let's imagine you have a headline and a button in two versions. You want to find out which combination of a headline and a button will lead to a higher conversion. With MVT you can compare each version of a button and each version of a headline. This means you will have four variants to test.

What's the difference between multivariate test and pairs of A\B tests ? 

Thing is, you need to run A\B test in isolation. To get clear, representable results you need to run one A\B test at a time on a single page. Which brings in certain limitations.
Multivariate tests normally allow you to test more variants at the same time. That will allow you to get results quicker.

Disadvantages ? 

The more options you'll test the more fragmented your traffic distribution is going to be. In a case of an A\B test your traffic split is 50\50. While if you want to test four options in a multivariate test your traffic split will be 25\25\25\25. To get reliable results - traffic groups should contain enough participants (sample size). In a case of an MVT, this requirement could be fulfilled only by a large amount of traffic.

When do A\B test and when do MVT ?

For major changes and drastically different options - make an A\B test.
For marginal improvements and conversion optimization - make an MVT. 

Have a lot of traffic and a lot of ideas ? - make an MVT. 
Have one metric to improve and best guess how ? - make an A\B test. 

Start with an A\B test, because it's simple. Continue with MVT to reach top conversion. 

An hour long webinar on A\B and MVT: 

Read more about multivariate tests:

Popular posts from this blog

Product management and operations tools - Jira Product Discovery review

  JPD is a new player in the market of product management software. Jira (and the whole Atlassian suite) has been one of the most popular tool stacks for teams to deliver software products. Now they're adding a missing piece - product discovery.

Product management and operations tools - Productboard review

  The second product management tool I've decided to review is Productboard . It is widely regarded as one of the most popular tools for product teams and the main competitor of Craft.io that I reviewed in the previous post .

Trust - the currency of leadership

  Here's a lesson I learned relatively late in my career - when it comes to leadership there is only one thing that truly matters - do you have the trust?