Skip to main content

Quantitative research: Multivariate test

A multivariate test is a logical continuation of an A\B test topic. Instead of comparing only two variants (A\B test) you can compare more than two - multivariate test (MVT).

Image credit.


Multivariate testing vs A\B (split) testing ?

The main difference of multivariate tests is that you can simultaneously test several elements of a page instead of testing one thing at the time (A\B test).

Let's imagine you have a headline and a button in two versions. You want to find out which combination of a headline and a button will lead to a higher conversion. With MVT you can compare each version of a button and each version of a headline. This means you will have four variants to test.

What's the difference between multivariate test and pairs of A\B tests ? 

Thing is, you need to run A\B test in isolation. To get clear, representable results you need to run one A\B test at a time on a single page. Which brings in certain limitations.
Multivariate tests normally allow you to test more variants at the same time. That will allow you to get results quicker.

Disadvantages ? 

The more options you'll test the more fragmented your traffic distribution is going to be. In a case of an A\B test your traffic split is 50\50. While if you want to test four options in a multivariate test your traffic split will be 25\25\25\25. To get reliable results - traffic groups should contain enough participants (sample size). In a case of an MVT, this requirement could be fulfilled only by a large amount of traffic.

When do A\B test and when do MVT ?

For major changes and drastically different options - make an A\B test.
For marginal improvements and conversion optimization - make an MVT. 

Have a lot of traffic and a lot of ideas ? - make an MVT. 
Have one metric to improve and best guess how ? - make an A\B test. 

Start with an A\B test, because it's simple. Continue with MVT to reach top conversion. 

An hour long webinar on A\B and MVT: 

Read more about multivariate tests:

Comments

Popular posts from this blog

Fogg Behavior Model

Have you ever wondered why you do certain things? Why are some behaviors easy and joy to do while other not so? And your customers - have you ever struggled to understand their behavior?
BJ Fogg, from Stanford University, has created simple and powerful behavioral model for persuasive product design.


Product Vision: an elevator pitch for your product

On this blog, I write a lot about making data-driven decisions. But what if you just starting to think about your product? You have a vague idea and nothing more. No point to go for prototyping or even talking to customers as you don't know yet who to talk to and what to talk about. In such situation - start from creating a product vision.

2017: Less of How, more of What, and a little bit of Why

Focusing less on how (methodologies, frameworks, processes). For me, that was the most important product management trend in 2017. Surely, "Agile" talks are bottomless. Yet, I noticed a certain tiredness and dare I say, boredom among product community with regards to this topic.
- Maybe that's enough? Haven't we discussed that like thousand times? Shouldn't we all have that figured by now?