Susan Atheyy Guido W. Imbensz
First Draft: October 2013
This Draft: April 2015
Abstract
In this paper we study the problems of estimating heterogeneity in causal eects in
experimental or observational studies and conducting inference about the magnitude of
the dierences in treatment eects across subsets of the population. In applications, our
method provides a data-driven approach to determine which subpopulations have large or
small treatment eects and to test hypotheses about the dierences in these eects. For
experiments, our method allows researchers to identify heterogeneity in treatment eects
that was not specied in a pre-analysis plan, without concern about invalidating inference
due to multiple testing. In most of the literature on supervised machine learning (e.g.
regression trees, random forests, LASSO, etc.), the goal is to build a model of the relationship
between a unit's attributes and an observed outcome. A prominent role in these methods is
played by cross-validation which compares predictions to actual outcomes in test samples, in
order to select the level of complexity of the model that provides the best predictive power.
Our method is closely related, but it diers in that it is tailored for predicting causal eects
of a treatment rather than a unit's outcome. The challenge is that the \ground truth" for a
causal eect is not observed for any individual unit: we observe the unit with the treatment,
or without the treatment, but not both at the same time. Thus, it is not obvious how to
use cross-validation to determine whether a causal eect has been accurately predicted. We
propose several novel cross-validation criteria for this problem and demonstrate through
simulations the conditions under which they perform better than standard methods for
the problem of causal eects. We then apply the method to a large-scale eld experiment
re-ranking results on a search engine.
Keywords: Potential Outcomes, Heterogeneous Treatment Eects, Causal In-
ference, Supervised Machine Learning, Cross-Validation