Accessibility and generalizability: Are social media effects moderated by age or digital literacy?

An emerging empirical regularity suggests that older people use and respond to social media very differently than younger people. Older people are the fastest-growing population of Internet and social media users in the US, and this heterogeneity will soon become central to online politics. However, many important experiments in this field have been conducted on online samples that do not contain enough older people to be useful to generalize to the current population of Internet users; this issue is more pronounced for studies that are even a few years old. In this paper, we report the results of replicating two experiments involving social media (specifically, Facebook) conducted on one such sample lacking older users (Amazon’s Mechanical Turk) using a source of online subjects which does contain sufficient variation in subject age. We add a standard battery of questions designed to explicitly measure digital literacy. We find evidence of significant treatment effect heterogeneity in subject age and digital literacy in the replication of one of the two experiments. This result is an example of limitations to generalizability of research conducted on samples where selection is related to treatment effect heterogeneity; specifically, this result indicates that Mechanical Turk should not be used to recruit subjects when researchers suspect treatment effect heterogeneity in age or digital literacy, as we argue should be the case for research on digital media effects.

Download Full-text

Generalizing from Survey Experiments Conducted on Mechanical Turk: A Replication Approach

Political Science Research and Methods ◽

10.1017/psrm.2018.10 ◽

2018 ◽

Vol 7 (3) ◽

pp. 613-628 ◽

Cited By ~ 78

Author(s):

Alexander Coppock

Keyword(s):

Treatment Effect ◽

Experimental Treatment ◽

Mechanical Turk ◽

Background Characteristics ◽

The Public ◽

Treatment Effect Heterogeneity ◽

Survey Experiments ◽

Amazon's Mechanical Turk ◽

Effect Heterogeneity

To what extent do survey experimental treatment effect estimates generalize to other populations and contexts? Survey experiments conducted on convenience samples have often been criticized on the grounds that subjects are sufficiently different from the public at large to render the results of such experiments uninformative more broadly. In the presence of moderate treatment effect heterogeneity, however, such concerns may be allayed. I provide evidence from a series of 15 replication experiments that results derived from convenience samples like Amazon’s Mechanical Turk are similar to those obtained from national samples. Either the treatments deployed in these experiments cause similar responses for many subject types or convenience and national samples do not differ much with respect to treatment effect moderators. Using evidence of limited within-experiment heterogeneity, I show that the former is likely to be the case. Despite a wide diversity of background characteristics across samples, the effects uncovered in these experiments appear to be relatively homogeneous.

Download Full-text

Broken or Fixed Effects?

Journal of Econometric Methods ◽

10.1515/jem-2017-0002 ◽

2018 ◽

Vol 8 (1) ◽

Cited By ~ 13

Author(s):

Charles E. Gibbons ◽

Juan Carlos Suárez Serrato ◽

Michael B. Urbancic

Keyword(s):

Empirical Evidence ◽

Treatment Effect ◽

Fixed Effects ◽

Treatment Effects ◽

Average Treatment Effect ◽

Significant Treatment Effect ◽

Significant Treatment ◽

Heterogeneous Treatment Effects ◽

Treatment Effect Heterogeneity ◽

Effect Heterogeneity

Abstract We replicate eight influential papers to provide empirical evidence that, in the presence of heterogeneous treatment effects, OLS with fixed effects (FE) is generally not a consistent estimator of the average treatment effect (ATE). We propose two alternative estimators that recover the ATE in the presence of group-specific heterogeneity. We document that heterogeneous treatment effects are common and the ATE is often statistically and economically different from the FE estimate. In all but one of our replications, there is statistically significant treatment effect heterogeneity and, in six, the ATEs are either economically or statistically different from the FE estimates.

Download Full-text

Treatment Effect Heterogeneity in Theory and Practice

SSRN Electronic Journal ◽

10.2139/ssrn.432501 ◽

2003 ◽

Author(s):

Joshua D. Angrist

Keyword(s):

Treatment Effect ◽

Theory And Practice ◽

Treatment Effect Heterogeneity ◽

Effect Heterogeneity

Download Full-text

Looking Inside the Black Box: The Importance of Causal Mechanism and Treatment Effect Heterogeneity in Experimentally Evaluated Criminal Justice Interventions

Korean Journal of Policy Studies ◽

10.52372/kjps31104 ◽

2016 ◽

Vol 31 (1) ◽

pp. 89-112

Author(s):

Na Chong Min

Keyword(s):

Treatment Effect ◽

Empirical Studies ◽

Evaluation Research ◽

Black Box ◽

Average Treatment Effect ◽

Causal Mechanism ◽

Population Average ◽

Treatment Effect Heterogeneity ◽

Policies And Programs ◽

Effect Heterogeneity

This paper discusses limitations of the ???black-box??? experimental archetype by highlighting the narrowness of outcome-focused approaches. For a more complete understanding of the nuanced implications of policies and programs, this study calls for an investigation of causal mechanism and treatment effect heterogeneity in experimentally evaluated interventions. This study draws on two distinct but closely related empirical studies, one undertaken by Na and Paternoster (2012) and the other by Na, Loughran, and Paternoster (2015), that go beyond the estimation of a population average treatment effect by adopting more recent methodological advancements that are still underappreciated and underutilized in evaluation research.

Download Full-text

Estimating Treatment Effects in the Presence of Noncompliance and Nonresponse: The Generalized Endogenous Treatment Model

Political Analysis ◽

10.1093/pan/mpr005 ◽

2011 ◽

Vol 19 (2) ◽

pp. 205-226 ◽

Cited By ~ 8

Author(s):

Kevin M. Esterling ◽

Michael A. Neblo ◽

David M. J. Lazer

Keyword(s):

Treatment Effect ◽

Instrumental Variable ◽

Treatment Effects ◽

Mean Squared Error ◽

Principal Stratification ◽

Squared Error ◽

Treatment Effect Heterogeneity ◽

Causal Treatment ◽

Effect Heterogeneity ◽

And Control

If ignored, noncompliance with a treatment or nonresponse on outcome measures can bias estimates of treatment effects in a randomized experiment. To identify and estimate causal treatment effects in the case where compliance and response depend on unobservables, we propose the parametric generalized endogenous treatment (GET) model. GET incorporates behavioral responses within an experiment to measure each subject's latent compliance type and identifies causal effects via principal stratification. Using simulation methods and an application to field experimental data, we show GET has a dramatically lower mean squared error for treatment effect estimates than existing approaches to principal stratification that impute, rather than measure, compliance type. In addition, we show that GET allows one to relax and test the instrumental variable exclusion restriction assumption, to test for the presence of treatment effect heterogeneity across a range of compliance types, and to test for treatment ignorability when treatment and control samples are balanced on observable covariates.

Download Full-text