Skip to content
  • There are no suggestions because the search field is empty.

Understanding Statistical Significance

While Odd to Win tells you which variant is currently winning, Statistical Significance will tell you if you can trust that win.

In our reports, we indicate Statistical Significance with a green strip once a winner is confirmed. It ensures that the difference between the variant and control is caused by a real phenomenon and is not just a "lucky" random effect.

How long until my experiment will reach Statistical Significance?

When you launch an experience, it takes some time to determine if the results are displaying significant results. The length of time to reach statistical significance can greatly differ from experience to experience. A result is only marked as statistically significant when the Minimum Sample Size specific for that experience is met. We use a statistical power test (set at 80% power) to calculate the required sample size dynamically.

You can have 99% Odds to Win but No Statistical Significance if the sample size is still too small to rule out random chance.

How can I see if my experience has reached Statistical Significance?

The experience dashboard will share insights on the Statistical Significance Status. 

If the Variant is winning, a green strip will appear.

Tip: Sometimes one filter (e.g. New Visitors) may reach statistical significance whilst others need more time. Filter the results to see if any of the segments have reached statistical significance.

What do I do once Statistical Significance is reached?

Once an experience reaches statistical significance, you will see the following message displayed on the experience report page: 

Screenshot 2024-03-22 at 21.11.18

At this point, we recommend that you update the winner to 100%. Our team is happy to help with suggestions on iterations - just reach out on our chat or to your dedicated CRO expert.