Skip to content
  • There are no suggestions because the search field is empty.

How "odds to win" are calculated?

Odds to Win (OTW) represents the probability that a specific variation is performing better than the Control. It measures the area under the probability distribution curve that sits to the right of zero (the point of no difference).

OTW has no minimum sample size threshold. You could have an OTW of 100% after only 10 sessions, but that doesn't mean the result is reliable yet, it just means that, based only on those 10 sessions, the variant is ahead. We recommend combining this with other metrics including confidence interval and statistical significance when analysing your results.

Calculation Methods by Metric

We use different models depending on the data type and based on the type of goal metrics:

  • Binomial Bayesian: Used for binomial distribution goal metrics e.g. "Yes/No" events like Conversion Rate (converted vs. not converted).

  • Gaussian Bayesian: Used for goal metrics that do not follow binomial distribution e.g. continuous numbers like Average Order Value. This accounts for the mean and the standard deviation of order values.

  • Combined Model: For some goal metrics, we combine the above two to provide an accurate estimate.

These tailored testing approaches enable us to accurately assess the statistical significance of the results, taking into consideration the unique characteristics of each metric.​

For multivariate experiences, Each variant is evaluated independently against the baseline: 

E.g. looking at weather in two cities:

  • 80% chance of rain in London
  • 70% chance of rain in Paris

Those percentages don’t total 100%, because one outcome doesn’t exclude the other. Both variants look at "how likely it is this variant to outperform the baseline?" and are not slices of one 100% pie. 

The baseline doesn’t display odds to win because it’s the reference point everyone else is measured against.