A Jupyter notebook template/tutorial for running and evaluating ABn tests where the KPI of interest is binomial.
Reduce the amount of work required to set up and evaluate website experiments.
- Step by step guide with instructions for setting up an experiment and explanations for how to interpret results
- Deconstructed built-in python functions used to calculate test statistics
- Plug and play format with call-outs to customizable cell blocks to allow the experimenter to quickly load their own data and modify formula default inputs
- Links to additional resources
- Explanation for how to evaluate multiple variation tests
- Set Up: Pre Test
- Sample Size Calculator
- Alpha Cutoff (Type I Error & Bonferroni Adjustment)
- Determining Minimum Detectable Effect Size
- Set Up: Proportional Data
- General Data Set Up (should be applicable to most data for proportion evaluation)
- Test Evaluation
- 2 Sample Testing (z-test)
- 2+ Sample Testing (chi-square)
- Marascuilo Procedure (Multi pairwise z-test)
- Load in a csv file (or query result) that matches the data structure (see section Data Setup: Proportion).
- Any
Cell blockthat requires the experimenter to modify/input new variables is denoted with the color red and the following heading ***~ Set up Variables ~ *** (the rest of the cell blocks should not require any modifications). - The 2 and 2+ Sample Test sections do NOT need to be run sequentially (2+ Sample Testing section will work without first having to run the cell blocks in the 2 Sample Testing).
I wanted to share some of the kind words that my colleagues used to describe the notebook.
- "I used a bunch of your code from the AB testing notebook for proportions in a HP testing notebook. It worked pretty smoothly!"
- "This is a fantastic notebook. It is thorough and comprehensive. You've left no stone unturned."
- "Isn't it great how fast I can asnwer all these questions?! (love this notebook)"
- "Starting to see some positive results... Thanks again for all your help getting the analysis notebooks together"