From sample average treatment effect to population average treatment effect on the treated: combining experimental with observational studies to estimate population treatment effects

AbstractWeighting techniques are employed to generalize results from survey experiments to populations of theoretical and substantive interest. Although weighting is often viewed as a second-order methodological issue, these adjustment methods invoke untestable assumptions about the nature of sample selection and potential heterogeneity in the treatment effect. Therefore, although weighting is a useful technique in estimating population quantities, it can introduce bias and also be used as a researcher degree of freedom. We review survey experiments published in three major journals from 2000–2015 and find that there are no standard operating procedures for weighting survey experiments. We argue that all survey experiments should report the sample average treatment effect (SATE). Researchers seeking to generalize to a broader population can weight to estimate the population average treatment effect (PATE), but should discuss the construction and application of weights in a detailed and transparent manner given the possibility that weighting can introduce bias.

Download Full-text

Studentized Sensitivity Analysis for the Sample Average Treatment Effect in Paired Observational Studies

Journal of the American Statistical Association ◽

10.1080/01621459.2019.1632072 ◽

2019 ◽

Vol 115 (531) ◽

pp. 1518-1530 ◽

Cited By ~ 7

Author(s):

Colin B. Fogarty

Keyword(s):

Sensitivity Analysis ◽

Treatment Effect ◽

Observational Studies ◽

Average Treatment Effect ◽

Average Treatment ◽

Sample Average

Download Full-text

Estimating the Population Average Treatment Effect in Observational Studies with Choice-Based Sampling

The International Journal of Biostatistics ◽

10.1515/ijb-2018-0093 ◽

2019 ◽

Vol 15 (1) ◽

Author(s):

Zhiwei Zhang ◽

Zonghui Hu ◽

Chunling Liu

Keyword(s):

Observational Study ◽

Treatment Effect ◽

Observational Studies ◽

Average Treatment Effect ◽

Parametric Models ◽

Average Treatment ◽

Population Average ◽

Doubly Robust ◽

Asymptotically Efficient ◽

Semiparametric Theory

AbstractWe consider causal inference in observational studies with choice-based sampling, in which subject enrollment is stratified on treatment choice. Choice-based sampling has been considered mainly in the econometrics literature, but it can be useful for biomedical studies as well, especially when one of the treatments being compared is uncommon. We propose new methods for estimating the population average treatment effect under choice-based sampling, including doubly robust methods motivated by semiparametric theory. A doubly robust, locally efficient estimator may be obtained by replacing nuisance functions in the efficient influence function with estimates based on parametric models. The use of machine learning methods to estimate nuisance functions leads to estimators that are consistent and asymptotically efficient under broader conditions. The methods are compared in simulation experiments and illustrated in the context of a large observational study in obstetrics. We also make suggestions on how to choose the target proportion of treated subjects and the sample size in designing a choice-based observational study.

Download Full-text

Worth Weighting? How to Think About and Use Weights in Survey Experiments

Political Analysis ◽

10.1017/pan.2018.1 ◽

2018 ◽

Vol 26 (3) ◽

pp. 275-291 ◽

Cited By ~ 25

Author(s):

Luke W. Miratrix ◽

Jasjeet S. Sekhon ◽

Alexander G. Theodoridis ◽

Luis F. Campos

Keyword(s):

Statistical Power ◽

Online Survey ◽

Treatment Effects ◽

Average Treatment Effect ◽

Sampling Weights ◽

Average Treatment ◽

Sample Average ◽

Population Average ◽

Highly Correlated ◽

Representative Samples

The popularity of online surveys has increased the prominence of using sampling weights to enhance claims of representativeness. Yet, much uncertainty remains regarding how these weights should be employed in survey experiment analysis: should they be used? If so, which estimators are preferred? We offer practical advice, rooted in the Neyman–Rubin model, for researchers working with survey experimental data. We examine simple, efficient estimators, and give formulas for their biases and variances. We provide simulations that examine these estimators as well as real examples from experiments administered online through YouGov. We find that for examining the existence of population treatment effects using high-quality, broadly representative samples recruited by top online survey firms, sample quantities, which do not rely on weights, are often sufficient. We found that sample average treatment effect (SATE) estimates did not appear to differ substantially from their weighted counterparts, and they avoided the substantial loss of statistical power that accompanies weighting. When precise estimates of population average treatment effects (PATE) are essential, we analytically show poststratifying on survey weights and/or covariates highly correlated with outcomes to be a conservative choice. While we show substantial gains in simulations, we find limited evidence of them in practice.

Download Full-text

The Effect of Extension Services and Credit on Agricultural Production in Bolivia, Peru, and Colombia

10.18235/0003404 ◽

2021 ◽

Author(s):

Mateus C. R. Neves ◽

Felipe De Figueiredo Silva ◽

Carlos Otávio Freitas

Keyword(s):

Travel Time ◽

Treatment Effect ◽

Agricultural Production ◽

Treatment Effects ◽

Geographic Information ◽

Average Treatment Effect ◽

Extension Services ◽

Average Treatment ◽

Andean Countries ◽

Average Treatment Effects

In this paper we estimate the average treatment effect from access to extension services and credit on agricultural production in selected Andean countries (Bolivia, Peru, and Colombia). More specifically, we want to identify the effect of accessibility, here represented as travel time to the nearest area with 1,500 or more inhabitants per square kilometer or at least 50,000 inhabitants, on the likelihood of accessing extension and credit. To estimate the treatment effect and identify the effect of accessibility on these variables, we use data from the Colombian and Bolivian Agricultural Censuses of 2013 and 2014, respectively; a national agricultural survey from 2017 for Peru; and geographic information on travel time. We find that the average treatment effect for extension is higher compared to that of credit for farms in Bolivia and Peru, and lower for Colombia. The average treatment effects of extension and credit for Peruvian farms are $2,387.45 and $3,583.42 respectively. The average treatment effect for extension and credit are $941.92 and $668.69, respectively, while in Colombia are $1,365.98 and $1,192.51, respectively. We also find that accessibility and the likelihood of accessing these services are nonlinearly related. Results indicate that higher likelihood is associated with lower travel time, especially in the analysis of credit.

Download Full-text

Treatment Interactions with Nonexperimental Data in Stata

The Stata Journal Promoting communications on statistics and Stata ◽

10.1177/1536867x1201100403 ◽

2011 ◽

Vol 11 (4) ◽

pp. 545-555 ◽

Cited By ~ 8

Author(s):

Graham K. Brown ◽

Thanos Mergoupis

Keyword(s):

Experimental Data ◽

Observational Data ◽

Treatment Effect ◽

Treatment Effects ◽

Average Treatment Effect ◽

Interaction Parameters ◽

Average Treatment ◽

Interaction Terms ◽

Treatment Interaction ◽

Predicted Values

Treatment effects may vary with the observed characteristics of the treated, often with important implications. In the context of experimental data, a growing literature deals with the problem of specifying treatment interaction terms that most effectively capture this variation. Some results of this literature are now implemented in Stata. With nonexperimental (observational) data, and in particular when selection into treatment depends on unmeasured factors, treatment effects can be estimated using Stata's treatreg command. Though not originally designed for this purpose, treatreg can be used to consistently estimate treatment interaction parameters. With interactions, however, adjustments are required to generate predicted values and estimate the average treatment effect. In this article, we introduce commands that perform this adjustment for multiplicative interactions, and we show the required adjustment for more complicated interactions.

Download Full-text

High-dimensional regression adjustments in randomized experiments

Proceedings of the National Academy of Sciences ◽

10.1073/pnas.1614732113 ◽

2016 ◽

Vol 113 (45) ◽

pp. 12673-12678 ◽

Cited By ~ 21

Author(s):

Stefan Wager ◽

Wenfei Du ◽

Jonathan Taylor ◽

Robert J. Tibshirani

Keyword(s):

Treatment Effect ◽

Average Treatment Effect ◽

High Dimensional ◽

Randomized Experiments ◽

Finite Sample ◽

Simple Method ◽

Average Treatment ◽

Population Average ◽

High Dimensional Regression ◽

Effect Estimation

We study the problem of treatment effect estimation in randomized experiments with high-dimensional covariate information and show that essentially any risk-consistent regression adjustment can be used to obtain efficient estimates of the average treatment effect. Our results considerably extend the range of settings where high-dimensional regression adjustments are guaranteed to provide valid inference about the population average treatment effect. We then propose cross-estimation, a simple method for obtaining finite-sample–unbiased treatment effect estimates that leverages high-dimensional regression adjustments. Our method can be used when the regression model is estimated using the lasso, the elastic net, subset selection, etc. Finally, we extend our analysis to allow for adaptive specification search via cross-validation and flexible nonparametric regression adjustments with machine-learning methods such as random forests or neural networks.

Download Full-text

Bootstrap bias correction for average treatment effects with inverse propensity weights

10.47302/jsr.2018520205 ◽

2019 ◽

Vol 52 (2) ◽

pp. 187-200

Author(s):

GUBHINDER KUNDHI ◽

MARCEL VOIA

Keyword(s):

Propensity Score ◽

Treatment Effect ◽

Bias Correction ◽

Treatment Effects ◽

Potential Problem ◽

Average Treatment Effect ◽

Omitted Variables ◽

Control Groups ◽

Average Treatment ◽

And Control

The estimated average treatment effect in observational studies is biased if the assumptions of ignorability and overlap are not satisfied. To deal with this potential problem when propensity score weights are used in the estimation of the treatment effects, in this paper we propose a bootstrap bias correction estimator for the average treatment effect (ATE) obtained with the inverse propensity score (BBC-IPS) estimator. We show in simulations that the BBC-IPC performs well when we have misspecifications of the propensity score (PS) due to: omitted variables (ignorability property may not be satisfied), overlap (imbalances in distribution between treatment and control groups) and confounding effects between observables and unobservables (endogeneity). Further refinements in bias reductions of the ATE estimates in smaller samples are attained by iterating the BBC-IPS estimator.

Download Full-text