scholarly journals The Necessity of Construct and External Validity for Generalized Causal Claims

2021 ◽  
Author(s):  
Kevin Esterling ◽  
David Brady ◽  
Eric Schwitzgebel
Keyword(s):  
Social Sciences ◽  
Construct Validity ◽  
External Validity ◽  
Causal Effect ◽  
Internal Validity ◽  
Causal Effects ◽  
Formal Framework ◽  
Tremendous Progress ◽  
The Social

The credibility revolution has facilitated tremendous progress in the social sciences by advancing design-based strategies that rely on internal validity to deductively identify causal effects. We demonstrate that prioritizing internal validity while neglecting construct and external validity prevents causal generalization and misleadingly converts a deductive claim of causality into a claim based on speculation and exploration -- undermining the very goals of the credibility revolution. We develop a formal framework of causal specification to demonstrate that internal, external and construct validity are jointly necessary for generalized claims regarding a causal effect. If one lacks construct validity, one cannot assign meaningful labels to the cause or to the outcome. If one lacks external validity, one cannot make statements about the conditions required for the cause to occur. Re-balancing considerations of internal, construct and external via causal specification preserves and advances the intent of the credibility revolution to understand causal effects.

Reviewing Qualitative Research: Proposed Criteria for Fairness and Rigor

1994 ◽  
Vol 14 (2) ◽  
pp. 78-92 ◽  
Author(s):  
Jeffrey A. Gliner
Keyword(s):  
Social Sciences ◽  
Qualitative Research ◽  
Occupational Therapy ◽  
External Validity ◽  
Internal Validity ◽  
Case Analysis ◽  
Logical Positivism ◽  
Naturalistic Inquiry ◽  
Negative Case ◽  
The Social

The introduction of qualitative naturalistic research—qualitative research within the paradigm of naturalistic inquiry—into a scholarly tradition that historically has been positivistic has caused concern and controversy among both naturalists and logical positivists in the social sciences. The purpose of this article is to attempt to establish legitimate and fair criteria for the publication of qualitative naturalistic research in occupational therapy. Traditional criteria from the paradigm of logical positivism emphasizing internal validity and external validity are reviewed, and parallel criteria for qualitative naturalistic research, such as credibility and transferability, are examined. Methods such as triangulation, negative case analysis, and testing for rival hypotheses appear to show promise as criteria of fairness and rigor for publication of qualitative naturalistic research in occupational therapy.

scholarly journals Experimental practices and objectivity in the social sciences: re-embedding construct validity in the internal–external validity distinction

Synthese ◽  
2021 ◽  
Author(s):  
María Jiménez-Buedo ◽  
Federica Russo
Keyword(s):  
Social Sciences ◽  
Social Science ◽  
Construct Validity ◽  
External Validity ◽  
Natural Sciences ◽  
Objective Knowledge ◽  
Biomedical Sciences ◽  
The Social ◽  
Experimental Validity ◽  
Internal And External Validity

AbstractThe experimental revolution in the social sciences is one of the most significant methodological shifts undergone by the field since the ‘quantitative revolution’ in the nineteenth century. One of the often valued features of social science experimentation is precisely the fact that there are (alleged) clear methodological rules regarding hypothesis testing that come from the methods of the natural sciences and from the methodology of RCTs in the biomedical sciences, and that allow for the adjudication among contentious causal claims. We examine critically this claim and argue that some current understandings of the practices that surround social science experimentation overestimate the degree to which experiments can actually fulfil this role as “objective” adjudicators, by neglecting the importance of shared background knowledge or assumptions and of consensus regarding the validity of the constructs involved in an experiment. We take issue with the way the distinction between internal and external validity is often used to comment on the inferential import of experiments, used both among practitioners and among philosophers of science. We describe the ways in which the more common (dichotomous) use of the internal/external distinction differs from Cook and Campbell’s original methodological project, in which construct validity and the four-fold validity typology were all important in assessing the inferential import of experiments. We argue that the current uses of the labels internal and external, as applied to experimental validity, help to encroach a simplistic view on the inferential import of experiments that, in turn, misrepresents their capacity to provide objective knowledge about the causal relations between variables.

scholarly journals Multi-Method Research in the Social Sciences: A Review of Recent Frameworks and a Way Forward

2019 ◽  
Vol 55 (1) ◽  
pp. 163-182 ◽  
Author(s):  
Derek Beach
Keyword(s):  
Social Sciences ◽  
Case Studies ◽  
Comparative Politics ◽  
Causal Effects ◽  
Large Set ◽  
Causal Process ◽  
Top Down ◽  
The Social ◽  
Social Scientific ◽  
Case Based

AbstractThis article reviews recent attempts to develop multi-method social scientific frameworks. The article starts by discussing the ontological and epistemological foundations underlying case studies and variance-based approaches, differentiating approaches into bottom-up, case-based and top-down, variance-based approaches. Case-based approaches aim to learn how a causal process works within a case, whereas variance-based approaches assess mean causal effects across a set of cases. However, because of the different fundamental assumptions, it is very difficult for in-depth studies of individual cases to communicate meaningfully with claims about mean causal effects across a large set of cases. The conclusions discuss the broader challenges this distinction has for the study of comparative politics more broadly.

State responses to ‘Ethnic Riots’ in liberal democracies: evidence from Western Europe

2010 ◽  
Vol 2 (2) ◽  
pp. 269-295 ◽  
Author(s):  
Erik Bleich ◽  
Carolina Caeiro ◽  
Sarah Luehrman
Keyword(s):  
Social Sciences ◽  
Social Control ◽  
External Validity ◽  
Western Europe ◽  
Preliminary Test ◽  
Liberal Democracies ◽  
Electoral Incentives ◽  
The Social ◽  
The World ◽  
State Responses

The study of ethnic riots has a substantial pedigree in the social sciences, but so far there has been no systematic attempt to unify insights from scholars working on different areas of the world, nor has there been any extensive application of existing knowledge to the study of Western Europe. We address these two lacunae by drawing on contemporary scholarship to generate testable hypotheses about state responses to ethnic riots in liberal democracies, and by conducting a preliminary test of these hypotheses on four controlled comparison cases from Britain and France. Our cases reveal that states employ a relatively even balance of repression and accommodation in keeping with thesocial controlperspective, but that the precise balance is affected by theelectoral incentivesof the party in power. This evidence suggests the external validity of findings by Fording (2001) – who emphasizes the significance of social control in the American context – and Wilkinson (2004) – who stresses the importance of electoral incentives in the Indian environment – but it implies that these separate insights may be more powerful in combination. Our study also demonstrates the limitations of perspectives that predict either simple repression or accommodation of rioters, and of those that emphasize distinctive national responses to riots.

When Natural Experiments Are Neither Natural nor Experiments

2012 ◽  
Vol 106 (1) ◽  
pp. 35-57 ◽  
Author(s):  
JASJEET S. SEKHON ◽  
ROCÍO TITIUNIK
Keyword(s):  
Social Sciences ◽  
Unbiased Estimate ◽  
Causal Effect ◽  
Natural Experiments ◽  
Treatment Control ◽  
Causal Inferences ◽  
The Social ◽  
Research Designs ◽  
Alternative Research ◽  
Conceptual Difficulties

Natural experiments help to overcome some of the obstacles researchers face when making causal inferences in the social sciences. However, even when natural interventions are randomly assigned, some of the treatment–control comparisons made available by natural experiments may not be valid. We offer a framework for clarifying the issues involved, which are subtle and often overlooked. We illustrate our framework by examining four different natural experiments used in the literature. In each case, random assignment of the intervention is not sufficient to provide an unbiased estimate of the causal effect. Additional assumptions are required that are problematic. For some examples, we propose alternative research designs that avoid these conceptual difficulties.

Replicating Experiments Using Aggregate and Survey Data: The Case of Negative Advertising and Turnout

1999 ◽  
Vol 93 (4) ◽  
pp. 901-909 ◽  
Author(s):  
Stephen D. Ansolabehere ◽  
Shanto Iyengar ◽  
Adam Simon
Keyword(s):  
Survey Data ◽  
Instrumental Variables ◽  
External Validity ◽  
External Validation ◽  
Internal Validity ◽  
Aggregate Data ◽  
Causal Effects ◽  
Negative Advertising ◽  
Senate Elections ◽  
Advertising Exposure

Experiments show significant demobilizing and alienating effects of negative advertising. Although internally valid, experiments may have limited external validity. Aggregate and survey data offer two ways of providing external validation for experiments. We show that survey recall measures of advertising exposure suffer from problems of internal validity due to simultaneity and measurement error, which bias estimated effects of ad exposure. We provide valid estimates of the causal effects of ad exposure for the NES surveys using instrumental variables and find that negative advertising causes lower turnout in the NES data. We also provide a careful statistical analysis of aggregate turnout data from the 1992 Senate elections that Wattenberg and Brians (1999) recommend. These aggregate data confirm our original findings. Experiments, surveys, and aggregate data all point to the same conclusion: Negative advertising demobilizes voters.

Testing Social Science Network Theories with Online Network Data: An Evaluation of External Validity

2017 ◽  
Vol 111 (3) ◽  
pp. 502-521 ◽  
Author(s):  
JAMES BISBEE ◽  
JENNIFER M. LARSON
Keyword(s):  
Social Sciences ◽  
External Validity ◽  
Social Ties ◽  
Great Promise ◽  
Experimental Setting ◽  
Network Information ◽  
The Social ◽  
Using Data ◽  
Online Sources

To answer questions about the origins and outcomes of collective action, political scientists increasingly turn to datasets with social network information culled from online sources. However, a fundamental question of external validity remains untested: are the relationships measured between a person and her online peers informative of the kind of offline, “real-world” relationships to which network theories typically speak? This article offers the first direct comparison of the nature and consequences of online and offline social ties, using data collected via a novel network elicitation technique in an experimental setting. We document strong, robust similarity between online and offline relationships. This parity is not driven by sharedidentityof online and offline ties, but a shared nature of relationships in both domains. Our results affirm that online social tie data offer great promise for testing long-standing theories in the social sciences about the role of social networks.

scholarly journals Who’s in and Who’s Out? Selection Bias in Aging Research

2020 ◽  
Vol 4 (Supplement_1) ◽  
pp. 822-822
Author(s):  
Elizabeth Rose Mayeda ◽  
Eleanor Hayes-Larson ◽  
Hailey Banack
Keyword(s):  
Selection Bias ◽  
External Validity ◽  
Sample Selection ◽  
Internal Validity ◽  
Causal Effects ◽  
Aging Research ◽  
Selection Processes ◽  
The People ◽  
Biased Estimates ◽  
Internal And External Validity

Abstract Selection bias presents a major threat to both internal and external validity in aging research. “Selection bias” refers to sample selection processes that lead to statistical associations in the study sample that are biased estimates of causal effects in the population of interest. These processes can lead to: (1) results that do not generalize to the population of interest (threat to external validity) or (2) biased effect estimates (associations that do not represent causal effects for any population, including the people in the sample; a threat to internal validity). In this presentation, we give an overview of selection bias in aging research. We will describe processes that can give rise to selection bias, highlight why they are particularly pervasive in this field, and present several examples of selection bias in aging research. We end with a brief summary of strategies to prevent and correct for selection bias in aging research.

10. Indices of Robustness for Sample Representation

2007 ◽  
Vol 37 (1) ◽  
pp. 349-392 ◽  
Author(s):  
Kenneth Frank ◽  
Kyung-Seok Min
Keyword(s):  
Social Sciences ◽  
External Validity ◽  
Full Range ◽  
Target Population ◽  
Statistical Test ◽  
Causal Inferences ◽  
Social Scientists ◽  
The Social ◽  
Sample Representation ◽  
Positive Effect

Social scientists are rarely able to gather data from the full range of contexts to which they hope to generalize (Shadish, Cook, and Campbell 2002). Here we suggest that debates about the generality of causal inferences in the social sciences can be informed by quantifying the conditions necessary to invalidate an inference. We begin by differentiating the target population into two sub-populations: a potentially observed subpopulation from which all of a sample is drawn and a potentially unobserved subpopulation from which no members of the sample are drawn but which is part of the population to which policymakers seek to generalize. We then quantify the robustness of an inference in terms of the conditions necessary to invalidate an inference if cases from the potentially unobserved subpopulation were included in the sample. We apply the indices to inferences regarding the positive effect of small classes on achievement from the Tennessee class size study and then consider the breadth of external validity. We use the statistical test for whether there is a difference in effects between two subpopulations as a baseline to evaluate robustness, and we consider a Bayesian motivation for the indices and compare the use of the indices with other procedures. In the discussion we emphasize the value of quantifying robustness, consider the value of different quantitative thresholds, and conclude by extending a metaphor linking statistical and causal inferences.

Sign in / Sign up

Export Citation Format

Share Document