Developing Standards for Post-Hoc Weighting in Population-Based Survey Experiments

2017 ◽  
Vol 4 (2) ◽  
pp. 161-172 ◽  
Author(s):  
Annie Franco ◽  
Neil Malhotra ◽  
Gabor Simonovits ◽  
L. J. Zigerell
Keyword(s):  
Treatment Effect ◽  
Methodological Issue ◽  
Sample Selection ◽  
Population Based ◽  
Average Treatment Effect ◽  
Average Treatment ◽  
Sample Average ◽  
Population Average ◽  
Survey Experiments ◽  
Post Hoc

AbstractWeighting techniques are employed to generalize results from survey experiments to populations of theoretical and substantive interest. Although weighting is often viewed as a second-order methodological issue, these adjustment methods invoke untestable assumptions about the nature of sample selection and potential heterogeneity in the treatment effect. Therefore, although weighting is a useful technique in estimating population quantities, it can introduce bias and also be used as a researcher degree of freedom. We review survey experiments published in three major journals from 2000–2015 and find that there are no standard operating procedures for weighting survey experiments. We argue that all survey experiments should report the sample average treatment effect (SATE). Researchers seeking to generalize to a broader population can weight to estimate the population average treatment effect (PATE), but should discuss the construction and application of weights in a detailed and transparent manner given the possibility that weighting can introduce bias.

scholarly journals High-dimensional regression adjustments in randomized experiments

2016 ◽  
Vol 113 (45) ◽  
pp. 12673-12678 ◽  
Author(s):  
Stefan Wager ◽  
Wenfei Du ◽  
Jonathan Taylor ◽  
Robert J. Tibshirani
Keyword(s):  
Treatment Effect ◽  
Average Treatment Effect ◽  
High Dimensional ◽  
Randomized Experiments ◽  
Finite Sample ◽  
Simple Method ◽  
Average Treatment ◽  
Population Average ◽  
High Dimensional Regression ◽  
Effect Estimation

We study the problem of treatment effect estimation in randomized experiments with high-dimensional covariate information and show that essentially any risk-consistent regression adjustment can be used to obtain efficient estimates of the average treatment effect. Our results considerably extend the range of settings where high-dimensional regression adjustments are guaranteed to provide valid inference about the population average treatment effect. We then propose cross-estimation, a simple method for obtaining finite-sample–unbiased treatment effect estimates that leverages high-dimensional regression adjustments. Our method can be used when the regression model is estimated using the lasso, the elastic net, subset selection, etc. Finally, we extend our analysis to allow for adaptive specification search via cross-validation and flexible nonparametric regression adjustments with machine-learning methods such as random forests or neural networks.

Estimating the Population Average Treatment Effect in Observational Studies with Choice-Based Sampling

2019 ◽  
Vol 15 (1) ◽  
Author(s):  
Zhiwei Zhang ◽  
Zonghui Hu ◽  
Chunling Liu
Keyword(s):  
Observational Study ◽  
Treatment Effect ◽  
Observational Studies ◽  
Average Treatment Effect ◽  
Parametric Models ◽  
Average Treatment ◽  
Population Average ◽  
Doubly Robust ◽  
Asymptotically Efficient ◽  
Semiparametric Theory

AbstractWe consider causal inference in observational studies with choice-based sampling, in which subject enrollment is stratified on treatment choice. Choice-based sampling has been considered mainly in the econometrics literature, but it can be useful for biomedical studies as well, especially when one of the treatments being compared is uncommon. We propose new methods for estimating the population average treatment effect under choice-based sampling, including doubly robust methods motivated by semiparametric theory. A doubly robust, locally efficient estimator may be obtained by replacing nuisance functions in the efficient influence function with estimates based on parametric models. The use of machine learning methods to estimate nuisance functions leads to estimators that are consistent and asymptotically efficient under broader conditions. The methods are compared in simulation experiments and illustrated in the context of a large observational study in obstetrics. We also make suggestions on how to choose the target proportion of treated subjects and the sample size in designing a choice-based observational study.

The Sample Average Treatment Effect

2018 ◽  
pp. 175-193
Author(s):  
Laura B. Balzer ◽  
Maya L. Petersen ◽  
Mark J. van der Laan
Keyword(s):  
Treatment Effect ◽  
Average Treatment Effect ◽  
Average Treatment ◽  
Sample Average

scholarly journals Targeted estimation and inference for the sample average treatment effect in trials with and without pair-matching

2016 ◽  
Vol 35 (21) ◽  
pp. 3717-3732 ◽  
Author(s):  
Laura B. Balzer ◽  
Maya L. Petersen ◽  
Mark J. van der Laan ◽  
Keyword(s):  
Treatment Effect ◽  
Average Treatment Effect ◽  
Average Treatment ◽  
Sample Average ◽  
Pair Matching

scholarly journals Worth Weighting? How to Think About and Use Weights in Survey Experiments

2018 ◽  
Vol 26 (3) ◽  
pp. 275-291 ◽  
Author(s):  
Luke W. Miratrix ◽  
Jasjeet S. Sekhon ◽  
Alexander G. Theodoridis ◽  
Luis F. Campos
Keyword(s):  
Statistical Power ◽  
Online Survey ◽  
Treatment Effects ◽  
Average Treatment Effect ◽  
Sampling Weights ◽  
Average Treatment ◽  
Sample Average ◽  
Population Average ◽  
Highly Correlated ◽  
Representative Samples

The popularity of online surveys has increased the prominence of using sampling weights to enhance claims of representativeness. Yet, much uncertainty remains regarding how these weights should be employed in survey experiment analysis: should they be used? If so, which estimators are preferred? We offer practical advice, rooted in the Neyman–Rubin model, for researchers working with survey experimental data. We examine simple, efficient estimators, and give formulas for their biases and variances. We provide simulations that examine these estimators as well as real examples from experiments administered online through YouGov. We find that for examining the existence of population treatment effects using high-quality, broadly representative samples recruited by top online survey firms, sample quantities, which do not rely on weights, are often sufficient. We found that sample average treatment effect (SATE) estimates did not appear to differ substantially from their weighted counterparts, and they avoided the substantial loss of statistical power that accompanies weighting. When precise estimates of population average treatment effects (PATE) are essential, we analytically show poststratifying on survey weights and/or covariates highly correlated with outcomes to be a conservative choice. While we show substantial gains in simulations, we find limited evidence of them in practice.

Sign in / Sign up

Export Citation Format

Share Document