Wish Tackles Peeking with Always Valid p-values | by Qike (Max) Li | Aug, 2022
Peek and conclude experiments adaptively without inflating the false-positive ratePhoto by Kristi Evans from PexelsDo you estimate the effect size and calculate the sample size before running your A/B test? Do you check the p-value only once when you conclude the A/B test? When the p-value is 5.1% (0.1% higher than the threshold 5%) and the pre-calculated sample size has been reached, instead of running the experiment longer, do you stop the experiment and conclude that the product feature is not improving business? If…