Once a test reaches statistical significance, you will see the following message displayed on the experience report page:
At this point, we recommend that you update the winner to 100%. If the winner is the control, we recommend that you pause and iterate the experience. Our team is happy to help with suggestions, just reach out on our chat or to your dedicated CRO expert.
How are "odds to win" calculated?
We use a range of statistical methods centered around the primary KPI of the experience.
For goal metrics with a binomial distribution, such as Conversion Rate, we use the binomial Bayesian test. For goal metrics that do not follow a binomial distribution, such as AOV (Average Order Value) and PSV (Per-Session Value), we apply the Gaussian Bayesian test. These varied testing approaches enable us to more accurately assess the statistical significance of the results whilst taking into consideration the unique characteristics of each metric.