Mintao

A gentle walkthrough of the sample size formula in A/B testsThis article answers two important “how-to” questions centering on the minimum sample size calculation in A/B tests:1) How to derive the formula for the minimum sample size 𝜨?The essential idea behind the formula is to reverse the p-value calculation in hypothesis testing, with a particular focus on statistical power, which is the probability of rejecting the null hypothesis when the null hypothesis is indeed false.Figure 1: Intuition Behind Minimum Sample Size…

A Discussion of the go-to methods for 5 Types of A/B MetricsThis article summarizes A/B test evaluation metrics into 5 categories and outlines the suggested statistics testing for its significance values in the table below.Table by AuthorWhy Should We CareUser Average MetricsUser-level Conversion MetricsPageview-level Conversion MetricsPercentile MetricsSUM MetricsSummary and Practical TipsNoteWhile the t-tests are powerful, they are not universally applicable in the data world that is populated by business metrics with…

Introducing four types of plotting functions and relevant tricks for exploratory data analysis based on SeabornIntroductionViz 1: Double-axis Time Series Plot with Auxiliary Line/BandViz 2: Scatter Plot with Fitted TrendlineViz 3: Distribution Plot with KDE Line (Kernel Density Estimation)Viz 4: Categorical Bar Plot SeriesSummaryInitiatives“Matplotlib and seaborn are ugly, I only use ggplot2 in R”; “The seaborn API is a pain and very rigid to work with”; “The default plots for seaborn and matplotlib are so poor and I have…