scholarly journals A fundamental measure of treatment effect heterogeneity

2021 ◽  
Vol 9 (1) ◽  
pp. 83-108
Author(s):  
Jonathan Levy ◽  
Mark van der Laan ◽  
Alan Hubbard ◽  
Romain Pirracchio
Keyword(s):  
Treatment Effect ◽  
Mean Squared Error ◽  
Specific Treatment ◽  
Random Variable ◽  
Average Treatment Effect ◽  
Effect Function ◽  
True Variance ◽  
Targeted Maximum Likelihood ◽  
Data Adaptive ◽  
Effect Heterogeneity

Abstract The stratum-specific treatment effect function is a random variable giving the average treatment effect (ATE) for a randomly drawn stratum of potential confounders a clinician may use to assign treatment. In addition to the ATE, the variance of the stratum-specific treatment effect function is fundamental in determining the heterogeneity of treatment effect values. We offer a non-parametric plug-in estimator, the targeted maximum likelihood estimator (TMLE) and the cross-validated TMLE (CV-TMLE), to simultaneously estimate both the average and variance of the stratum-specific treatment effect function. The CV-TMLE is preferable because it guarantees asymptotic efficiency under two conditions without needing entropy conditions on the initial fits of the outcome model and treatment mechanism, as required by TMLE. Particularly, in circumstances where data adaptive fitting methods are very important to eliminate bias but hold no guarantee of satisfying the entropy condition, we show that the CV-TMLE sampling distributions maintain normality with a lower mean squared error than TMLE. In addition to verifying the theoretical properties of TMLE and CV-TMLE through simulations, we highlight some of the challenges in estimating the variance of the treatment effect, which lack double robustness and might be biased if the true variance is small and sample size insufficient.

scholarly journals Looking Inside the Black Box: The Importance of Causal Mechanism and Treatment Effect Heterogeneity in Experimentally Evaluated Criminal Justice Interventions

2016 ◽  
Vol 31 (1) ◽  
pp. 89-112
Author(s):  
Na Chong Min
Keyword(s):  
Treatment Effect ◽  
Empirical Studies ◽  
Evaluation Research ◽  
Black Box ◽  
Average Treatment Effect ◽  
Causal Mechanism ◽  
Population Average ◽  
Treatment Effect Heterogeneity ◽  
Policies And Programs ◽  
Effect Heterogeneity

This paper discusses limitations of the ???black-box??? experimental archetype by highlighting the narrowness of outcome-focused approaches. For a more complete understanding of the nuanced implications of policies and programs, this study calls for an investigation of causal mechanism and treatment effect heterogeneity in experimentally evaluated interventions. This study draws on two distinct but closely related empirical studies, one undertaken by Na and Paternoster (2012) and the other by Na, Loughran, and Paternoster (2015), that go beyond the estimation of a population average treatment effect by adopting more recent methodological advancements that are still underappreciated and underutilized in evaluation research.

Estimating Treatment Effects in the Presence of Noncompliance and Nonresponse: The Generalized Endogenous Treatment Model

2011 ◽  
Vol 19 (2) ◽  
pp. 205-226 ◽  
Author(s):  
Kevin M. Esterling ◽  
Michael A. Neblo ◽  
David M. J. Lazer
Keyword(s):  
Treatment Effect ◽  
Instrumental Variable ◽  
Treatment Effects ◽  
Mean Squared Error ◽  
Principal Stratification ◽  
Squared Error ◽  
Treatment Effect Heterogeneity ◽  
Causal Treatment ◽  
Effect Heterogeneity ◽  
And Control

If ignored, noncompliance with a treatment or nonresponse on outcome measures can bias estimates of treatment effects in a randomized experiment. To identify and estimate causal treatment effects in the case where compliance and response depend on unobservables, we propose the parametric generalized endogenous treatment (GET) model. GET incorporates behavioral responses within an experiment to measure each subject's latent compliance type and identifies causal effects via principal stratification. Using simulation methods and an application to field experimental data, we show GET has a dramatically lower mean squared error for treatment effect estimates than existing approaches to principal stratification that impute, rather than measure, compliance type. In addition, we show that GET allows one to relax and test the instrumental variable exclusion restriction assumption, to test for the presence of treatment effect heterogeneity across a range of compliance types, and to test for treatment ignorability when treatment and control samples are balanced on observable covariates.

scholarly journals EXPRESS: Treatment Effect Heterogeneity in Randomized Field Experiments: A Methodological Comparison and Public Policy Implications

2021 ◽  
pp. 074391562110327
Author(s):  
Yixing Chen ◽  
Shrihari Sridhar ◽  
Vikas Mittal
Keyword(s):  
Public Policy ◽  
Treatment Group ◽  
Treatment Effect ◽  
Field Experiments ◽  
Average Treatment Effect ◽  
Policy Implications ◽  
Control Group ◽  
Policy Studies ◽  
Randomized Field Experiment ◽  
Effect Heterogeneity

Many public-policy studies (Martin and Scott 2020) use randomized field experiments for drawing causal conclusions (e.g., Chen et al. 2020). A typical randomized field experiment involves a control group and a treatment group to which individual units (e.g., consumers, patients) are randomly assigned, after which an intervention is implemented in the treatment group. An intervention could be a marketing program to which only units in the treatment group are exposed. To assess the intervention's efficacy, researchers typically estimate the average treatment effect computed as the mean difference in the outcome between the units in the treatment group and the control group. When applying the results of a randomized experiment, it is assumed that the treatment effect within the manipulated condition is the same for all the units assigned to the treatment condition. This may not always be the case, as the effect may differ for subgroups within a treatment (subgroup differences).

scholarly journals On the treatment effect heterogeneity of antidepressants in major depression. A Bayesian meta-analysis

2020 ◽  
Author(s):  
Constantin Volkmann ◽  
Alexander Volkmann ◽  
Christian A. Müller
Keyword(s):  
Major Depression ◽  
Random Effects ◽  
Treatment Effect ◽  
Clinical Relevance ◽  
Meta Analysis ◽  
Placebo Response ◽  
Average Treatment Effect ◽  
Average Treatment ◽  
Treatment Effect Heterogeneity ◽  
Effect Heterogeneity

AbstractBackgroundThe average treatment effect of antidepressants in major depression was found to be about 2 points on the 17-item Hamilton Depression Rating Scale, which lies below clinical relevance. Here, we searched for evidence of a relevant treatment effect heterogeneity that could justify the usage of antidepressants despite their low average treatment effect.MethodsBayesian meta-analysis of 169 randomized, controlled trials including 58,687 patients. We considered the effect sizes log variability ratio (lnVR) and log coefficient of variation ratio (lnCVR) to analyze the difference in variability of active and placebo response. We used Bayesian random-effects meta-analyses (REMA) for lnVR and lnCVR and fitted a random-effects meta-regression (REMR) model to estimate the treatment effect variability between antidepressants and placebo.ResultsThe variability ratio was found to be very close to 1 in the best fitting models (REMR: 95% HPD [0.98, 1.02], REMA: 95% HPD [1.00, 1.02]). The between-study variance τ2 under the REMA was found to be low (95% HPD [0.00, 0.00]). Simulations showed that a large treatment effect heterogeneity is only compatible with the data if a strong correlation between placebo response and individual treatment effect is assumed.ConclusionsThe published data from RCTs on antidepressants for the treatment of major depression is compatible with a near-constant treatment effect. Although it is impossible to rule out a substantial treatment effect heterogeneity, its existence seems rather unlikely. Since the average treatment effect of antidepressants falls short of clinical relevance, the current prescribing practice should be re-evaluated.

scholarly journals Understanding and diagnosing the potential for bias when using machine learning methods with doubly robust causal estimators

2018 ◽  
Vol 28 (6) ◽  
pp. 1637-1650 ◽  
Author(s):  
Asma Bahamyirou ◽  
Lucie Blais ◽  
Amélie Forget ◽  
Mireille E Schnitzer
Keyword(s):  
Propensity Score ◽  
Treatment Effect ◽  
Diagnostic Tool ◽  
Estimation Procedure ◽  
Adaptive Methods ◽  
Outcome Data ◽  
Average Treatment Effect ◽  
Simulation Scenario ◽  
Data Adaptive ◽  
Doubly Robust

Data-adaptive methods have been proposed to estimate nuisance parameters when using doubly robust semiparametric methods for estimating marginal causal effects. However, in the presence of near practical positivity violations, these methods can produce a separation of the two exposure groups in terms of propensity score densities which can lead to biased estimates of the treatment effect. To motivate the problem, we evaluated the Targeted Minimum Loss-based Estimation procedure using a simulation scenario to estimate the average treatment effect. We highlight the divergence in estimates obtained when using parametric and data-adaptive methods to estimate the propensity score. We then adapted an existing diagnostic tool based on a bootstrap resampling of the subjects and simulation of the outcome data in order to show that the estimation using data-adaptive methods for the propensity score in this study may lead to large bias and poor coverage. The adapted bootstrap procedure is able to identify this instability and can be used as a diagnostic tool.

scholarly journals Generalizability of heterogeneous treatment effect estimates across samples

2018 ◽  
Vol 115 (49) ◽  
pp. 12441-12446 ◽  
Author(s):  
Alexander Coppock ◽  
Thomas J. Leeper ◽  
Kevin J. Mullinix
Keyword(s):  
Treatment Effect ◽  
Treatment Effects ◽  
Average Treatment Effect ◽  
Partial Explanation ◽  
Convenience Sample ◽  
Conditional Average ◽  
Replication Studies ◽  
Average Treatment ◽  
Average Treatment Effects ◽  
Effect Heterogeneity

The extent to which survey experiments conducted with nonrepresentative convenience samples are generalizable to target populations depends critically on the degree of treatment effect heterogeneity. Recent inquiries have found a strong correspondence between sample average treatment effects estimated in nationally representative experiments and in replication studies conducted with convenience samples. We consider here two possible explanations: low levels of effect heterogeneity or high levels of effect heterogeneity that are unrelated to selection into the convenience sample. We analyze subgroup conditional average treatment effects using 27 original–replication study pairs (encompassing 101,745 individual survey responses) to assess the extent to which subgroup effect estimates generalize. While there are exceptions, the overwhelming pattern that emerges is one of treatment effect homogeneity, providing a partial explanation for strong correspondence across both unconditional and conditional average treatment effect estimates.

scholarly journals CATE meets ML

2021 ◽  
Author(s):  
Daniel Jacob
Keyword(s):  
Treatment Effect ◽  
Prediction Models ◽  
Pension Plan ◽  
Real Data ◽  
Average Treatment Effect ◽  
Conflicting Evidence ◽  
Positive Treatment ◽  
Doubly Robust ◽  
Effect Heterogeneity ◽  
True Treatment Effect

AbstractFor treatment effects—one of the core issues in modern econometric analysis—prediction and estimation are two sides of the same coin. As it turns out, machine learning methods are the tool for generalized prediction models. Combined with econometric theory, they allow us to estimate not only the average but a personalized treatment effect—the conditional average treatment effect (CATE). In this tutorial, we give an overview of novel methods, explain them in detail, and apply them via Quantlets in real data applications. We study the effect that microcredit availability has on the amount of money borrowed and if 401(k) pension plan eligibility has an impact on net financial assets, as two empirical examples. The presented toolbox of methods contains meta-learners, like the doubly-robust, R-, T- and X-learner, and methods that are specially designed to estimate the CATE like the causal BART and the generalized random forest. In both, the microcredit and 401(k) example, we find a positive treatment effect for all observations but conflicting evidence of treatment effect heterogeneity. An additional simulation study, where the true treatment effect is known, allows us to compare the different methods and to observe patterns and similarities.

Sign in / Sign up

Export Citation Format

Share Document