Sign-up page A/B test analysis

The brief

A product team changed its sign-up page and conversions went up in the first week. Before rolling it out, the product manager wants to know if the change is real. Analyse the experiment properly: check the sample, test the difference and estimate the effect with its uncertainty. Good looks like: a sample-ratio check, a correct test, a confidence interval and a clear recommendation.

What you will build

1. Clean the experiment data and remove duplicate users
2. Check the groups are balanced (sample ratio mismatch)
3. Test the difference in conversion and compute a confidence interval
4. Work out the sample size the test needed
5. Present the result and a recommendation with clear charts

Free, open data to build it with

• Marketing A/B testing ↗ Kaggle

Check each dataset's licence before you publish your work. Your mentor gives your team its own version of the brief (data, scope or region), so every team's project is its own.