scholarly journals Treatment effect heterogeneity in the head start impact study: A systematic review of study characteristics and findings

2021 ◽  
pp. 100916
Author(s):  
Sun Yeop Lee ◽  
Rockli Kim ◽  
Justin Rodgers ◽  
S.V. Subramanian
Keyword(s):  
Systematic Review ◽  
Head Start ◽  
Treatment Effect ◽  
Impact Study ◽  
Treatment Effect Heterogeneity ◽  
Study Characteristics ◽  
Effect Heterogeneity

How Much Can We Generalize From Impact Evaluations?

2020 ◽  
Vol 18 (6) ◽  
pp. 3045-3089
Author(s):  
Eva Vivalt
Keyword(s):  
Sample Size ◽  
Treatment Effect ◽  
Impact Evaluation ◽  
Effect Sizes ◽  
Data Set ◽  
Governmental Organization ◽  
Treatment Effect Heterogeneity ◽  
Study Characteristics ◽  
Effect Heterogeneity ◽  
Open Question

Abstract Impact evaluations can help to inform policy decisions, but they are rooted in particular contexts and to what extent they generalize is an open question. I exploit a new data set of impact evaluation results and find a large amount of effect heterogeneity. Effect sizes vary systematically with study characteristics, with government-implemented programs having smaller effect sizes than academic or non-governmental organization-implemented programs, even controlling for sample size. I show that treatment effect heterogeneity can be appreciably reduced by taking study characteristics into account.

scholarly journals Looking Inside the Black Box: The Importance of Causal Mechanism and Treatment Effect Heterogeneity in Experimentally Evaluated Criminal Justice Interventions

2016 ◽  
Vol 31 (1) ◽  
pp. 89-112
Author(s):  
Na Chong Min
Keyword(s):  
Treatment Effect ◽  
Empirical Studies ◽  
Evaluation Research ◽  
Black Box ◽  
Average Treatment Effect ◽  
Causal Mechanism ◽  
Population Average ◽  
Treatment Effect Heterogeneity ◽  
Policies And Programs ◽  
Effect Heterogeneity

This paper discusses limitations of the ???black-box??? experimental archetype by highlighting the narrowness of outcome-focused approaches. For a more complete understanding of the nuanced implications of policies and programs, this study calls for an investigation of causal mechanism and treatment effect heterogeneity in experimentally evaluated interventions. This study draws on two distinct but closely related empirical studies, one undertaken by Na and Paternoster (2012) and the other by Na, Loughran, and Paternoster (2015), that go beyond the estimation of a population average treatment effect by adopting more recent methodological advancements that are still underappreciated and underutilized in evaluation research.

Estimating Treatment Effects in the Presence of Noncompliance and Nonresponse: The Generalized Endogenous Treatment Model

2011 ◽  
Vol 19 (2) ◽  
pp. 205-226 ◽  
Author(s):  
Kevin M. Esterling ◽  
Michael A. Neblo ◽  
David M. J. Lazer
Keyword(s):  
Treatment Effect ◽  
Instrumental Variable ◽  
Treatment Effects ◽  
Mean Squared Error ◽  
Principal Stratification ◽  
Squared Error ◽  
Treatment Effect Heterogeneity ◽  
Causal Treatment ◽  
Effect Heterogeneity ◽  
And Control

If ignored, noncompliance with a treatment or nonresponse on outcome measures can bias estimates of treatment effects in a randomized experiment. To identify and estimate causal treatment effects in the case where compliance and response depend on unobservables, we propose the parametric generalized endogenous treatment (GET) model. GET incorporates behavioral responses within an experiment to measure each subject's latent compliance type and identifies causal effects via principal stratification. Using simulation methods and an application to field experimental data, we show GET has a dramatically lower mean squared error for treatment effect estimates than existing approaches to principal stratification that impute, rather than measure, compliance type. In addition, we show that GET allows one to relax and test the instrumental variable exclusion restriction assumption, to test for the presence of treatment effect heterogeneity across a range of compliance types, and to test for treatment ignorability when treatment and control samples are balanced on observable covariates.

Lessons Learned From Multi-regional Trials With Signals of Treatment Effect Heterogeneity

2020 ◽  
Vol 54 (1) ◽  
pp. 21-31
Author(s):  
Gang Li ◽  
Hui Quan ◽  
Gordon Lan ◽  
Soo Peter Ouyang ◽  
Fei Chen ◽  
...  
Keyword(s):  
Treatment Effect ◽  
Lessons Learned ◽  
Treatment Effect Heterogeneity ◽  
Effect Heterogeneity

scholarly journals Generalizing from Survey Experiments Conducted on Mechanical Turk: A Replication Approach

2018 ◽  
Vol 7 (3) ◽  
pp. 613-628 ◽  
Author(s):  
Alexander Coppock
Keyword(s):  
Treatment Effect ◽  
Experimental Treatment ◽  
Mechanical Turk ◽  
Background Characteristics ◽  
The Public ◽  
Treatment Effect Heterogeneity ◽  
Survey Experiments ◽  
Amazon's Mechanical Turk ◽  
Effect Heterogeneity

To what extent do survey experimental treatment effect estimates generalize to other populations and contexts? Survey experiments conducted on convenience samples have often been criticized on the grounds that subjects are sufficiently different from the public at large to render the results of such experiments uninformative more broadly. In the presence of moderate treatment effect heterogeneity, however, such concerns may be allayed. I provide evidence from a series of 15 replication experiments that results derived from convenience samples like Amazon’s Mechanical Turk are similar to those obtained from national samples. Either the treatments deployed in these experiments cause similar responses for many subject types or convenience and national samples do not differ much with respect to treatment effect moderators. Using evidence of limited within-experiment heterogeneity, I show that the former is likely to be the case. Despite a wide diversity of background characteristics across samples, the effects uncovered in these experiments appear to be relatively homogeneous.

Sign in / Sign up

Export Citation Format

Share Document