Empirical Bayes Estimation of Treatment Effects with Many A/B Tests: An Overview

The use of large-scale experimentation to screen product innovations is increasingly common. This is a practical guide on how to use treatment effect estimates from a large number of experiments to improve estimates of the effects of each experiment. When thousands of new features are A/B tested by internet companies, the winners tend to be a combination of good features and features that got lucky experimental draws. Empirical Bayes methods are a commonly used tool in statistics to separate good features from lucky draws. We give a user-friendly overview of both classic and recent approaches to this problem.