A multivariate test is a logical continuation of an A\B test topic. Instead of comparing only two variants (A\B test) you can compare more than two - multivariate test (MVT).
Image credit.
Let's imagine you have a headline and a button in two versions. You want to find out which combination of a headline and a button will lead to a higher conversion. With MVT you can compare each version of a button and each version of a headline. This means you will have four variants to test.
Multivariate tests normally allow you to test more variants at the same time. That will allow you to get results quicker.
Multivariate testing vs A\B (split) testing ?
The main difference of multivariate tests is that you can simultaneously test several elements of a page instead of testing one thing at the time (A\B test).Let's imagine you have a headline and a button in two versions. You want to find out which combination of a headline and a button will lead to a higher conversion. With MVT you can compare each version of a button and each version of a headline. This means you will have four variants to test.
What's the difference between multivariate test and pairs of A\B tests ?
Thing is, you need to run A\B test in isolation. To get clear, representable results you need to run one A\B test at a time on a single page. Which brings in certain limitations.Multivariate tests normally allow you to test more variants at the same time. That will allow you to get results quicker.
Disadvantages ?
The more options you'll test the more fragmented your traffic distribution is going to be. In a case of an A\B test your traffic split is 50\50. While if you want to test four options in a multivariate test your traffic split will be 25\25\25\25. To get reliable results - traffic groups should contain enough participants (sample size). In a case of an MVT, this requirement could be fulfilled only by a large amount of traffic.When do A\B test and when do MVT ?
For major changes and drastically different options - make an A\B test.
For marginal improvements and conversion optimization - make an MVT.
Have a lot of traffic and a lot of ideas ? - make an MVT.
Have one metric to improve and best guess how ? - make an A\B test.
Start with an A\B test, because it's simple. Continue with MVT to reach top conversion.
An hour long webinar on A\B and MVT: