Multivariate testing is a powerful method for making data-driven decisions by testing multiple variations within a single experience. Follow these steps to set up a multivariate test:
Step 1:
Create a first variation featuring the change you want to test. Once your first variation is ready, click on "Next" to reach the review page:
Step 2:
On the review page, opt to run an A/B test validation and select 'Create another variant'.
Step 3:
You can either build a new variant from scratch or duplicate the existing variant and make any adjustments.
Monitoring the performance of a multivariate test:
In the reports, you will be able to see the performance of all variations.
Why doesn't the Control line appear in Multi-Variant test report?
In a multi-variant test, the Control represents the original experience with no changes, while each Variant contains a different set of changes. For example:
- Control - Original experience
- Variant 1 - Original experience with Change A
- Variant 2 - Original experience with Change B
For example, the Win Chance graph calculates Odds to Win (OTW) as a relative metric by comparing one experience against another.
Because the Control can have a different OTW value depending on which variant it's being compared against, there isn't a single Control value that accurately represents all comparisons. For example:
- Control may have a 17% OTW when compared to Variant 1.
- The same Control may have a 10% OTW when compared to Variant 2.
Since these values are different, displaying one universal Control line in the graph could be misleading. For this reason, the Win Chance graph does not display a single Control line for experiments with more than two variants.
💡 You can create up to six variations within a single experience
❗ Additional variants may extend the time needed to achieve statistical significance.
For further assistance or questions, reach out on the chat support widget or via support@visually.io.