A Validity-Based Framework for Understanding Replication in Psychology

2020 ◽  
Vol 24 (4) ◽  
pp. 316-344
Author(s):  
Leandre R. Fabrigar ◽  
Duane T. Wegener ◽  
Richard E. Petty
Keyword(s):  
Construct Validity ◽  
Empirical Evidence ◽  
External Validity ◽  
Internal Validity ◽  
Psychological Research ◽  
Original Study ◽  
Statistical Conclusion ◽  
Conceptual Logic ◽  
Replication Failure

In recent years, psychology has wrestled with the broader implications of disappointing rates of replication of previously demonstrated effects. This article proposes that many aspects of this pattern of results can be understood within the classic framework of four proposed forms of validity: statistical conclusion validity, internal validity, construct validity, and external validity. The article explains the conceptual logic for how differences in each type of validity across an original study and a subsequent replication attempt can lead to replication “failure.” Existing themes in the replication literature related to each type of validity are also highlighted. Furthermore, empirical evidence is considered for the role of each type of validity in non-replication. The article concludes with a discussion of broader implications of this classic validity framework for improving replication rates in psychological research.

External Validity

2017 ◽  
Author(s):  
Richard McCleary ◽  
David McDowall ◽  
Bradley J. Bartos
Keyword(s):  
Construct Validity ◽  
Failure Rate ◽  
External Validity ◽  
Replication Study ◽  
Collaborative Group ◽  
Incentive Structure ◽  
Social Scientists ◽  
Time Frames ◽  
Statistical Conclusion ◽  
Mainstream Psychology

A threat to external validity is any factor that limits the generalizability of an observed result. Unlike all threats to statistical conclusion and internal validities and some threats to construct validity, threats to external validity cannot ordinarily be controlled by design. Nor is there any disagreement on how threats to external validity should be controlled. In most instances, it can only be controlled by replication?—across subjects, situations and time frames. This seldom happens, unfortunately, because the academic incentive structure discourages replication. The contemporary “reproducibility crisis” was spurred by a collaborative group of social scientists attempting to replicate one hundred experimental and correlational studies published in three mainstream psychology journals. Sixty percent of replications failed to reproduce the published effect. Failures to control for threats to external validity that stem from uncontrolled variations in persons, situations, and time frames, parsimosniously explain the failure rate in this replication study.

scholarly journals The Necessity of Construct and External Validity for Generalized Causal Claims

2021 ◽  
Author(s):  
Kevin Esterling ◽  
David Brady ◽  
Eric Schwitzgebel
Keyword(s):  
Social Sciences ◽  
Construct Validity ◽  
External Validity ◽  
Causal Effect ◽  
Internal Validity ◽  
Causal Effects ◽  
Formal Framework ◽  
Tremendous Progress ◽  
The Social

The credibility revolution has facilitated tremendous progress in the social sciences by advancing design-based strategies that rely on internal validity to deductively identify causal effects. We demonstrate that prioritizing internal validity while neglecting construct and external validity prevents causal generalization and misleadingly converts a deductive claim of causality into a claim based on speculation and exploration -- undermining the very goals of the credibility revolution. We develop a formal framework of causal specification to demonstrate that internal, external and construct validity are jointly necessary for generalized claims regarding a causal effect. If one lacks construct validity, one cannot assign meaningful labels to the cause or to the outcome. If one lacks external validity, one cannot make statements about the conditions required for the cause to occur. Re-balancing considerations of internal, construct and external via causal specification preserves and advances the intent of the credibility revolution to understand causal effects.

scholarly journals Towards Construct Validity of Relational Aggression: An Examination of the Children's Social Behavior Scale

2021 ◽  
Author(s):  
Cassandra M Brandes ◽  
Kathleen Wade Reardon ◽  
Allison Shields ◽  
Jennifer L Tackett
Keyword(s):  
Social Behavior ◽  
Construct Validity ◽  
Relational Aggression ◽  
External Validity ◽  
Reliability And Validity ◽  
Psychological Research ◽  
Demographic Groups ◽  
Developmental Factors ◽  
Behavior Scale ◽  
Interdisciplinary Study

Relational aggression – or behavior intended to harm the relationships of its victims – has been the focus of interdisciplinary study across developmental, clinical, personality, and social psychology in the last several decades. One of the primary measures used to assess relational aggression in youth is the Children’s Social Behavior Scale (CSBS; Crick & Grotpeter, 1995), but despite its common usage, the construct validity of this measure has not been comprehensively assessed. In the present study, we used a multistage construct validity framework to thoroughly investigate the nature of relational aggression across six community samples totaling 3,102 youth and their caregivers. We used multiple methods to map the reliability, internal or structural validity, and external validity of this scale. Through these analyses, we found that CSBS Relational Aggression demonstrated strong internal consistency, test-retest, and interrater reliability as well as a robust single factor structure and invariance across multiple demographic groups. External validity analyses positioned relational aggression within a theoretically consistent nomological net including psychopathology, personality, and social developmental factors. Contrary to concerns about the validity of self- and parent-reports of relational aggression, both parent- and youth-report forms of the CSBS Relational Aggression scale demonstrated strong reliability and validity. While construct validation has received inadequate attention in the psychological literature to date, through this project, we aimed to demonstrate how this approach may be used to investigate existing measures across psychological research.

L2 access to UG: Now you see it, now you don't

1996 ◽  
Vol 19 (4) ◽  
pp. 731-732
Author(s):  
Michael Harrington
Keyword(s):  
Empirical Evidence ◽  
Internal Validity ◽  
Structural Knowledge ◽  
L2 Proficiency ◽  
L2 Development

AbstractThe confirmatory nature of the empirical evidence used to establish UG effects in L2 development is considered. Specific issues are also raised concerning the internal validity of Epstein et al.'s findings. It is concluded that the role of UG in adult L2 development will only be established when researchers better understand the interaction between the development of UG-constrained structural knowledge and the development of overall L2 proficiency.

scholarly journals Development and Validation of the RAFFLE; A Measure of Reasons and Facilitators for Loot Box Engagement

2021 ◽  
Vol 10 (24) ◽  
pp. 5949
Author(s):  
Joanne Lloyd ◽  
Laura Louise Nicklin ◽  
Stuart Gordon Spicer ◽  
Chris Fullwood ◽  
Maria Uther ◽  
...  
Keyword(s):  
Qualitative Research ◽  
Factor Analysis ◽  
Video Games ◽  
Construct Validity ◽  
Internal Validity ◽  
Theoretical Models ◽  
Cognitive Interviews ◽  
Purchasing Behaviour ◽  
Development And Validation

Qualitative studies have identified a diverse array of motivations for purchasing items within video games through chance-based mechanisms (i.e., “loot boxes”). Given that some individuals—particularly those at risk of disordered gaming and/or gambling—are prone to over-involvement with loot box purchasing, it is important to have a reliable, valid means of measuring the role of different motivations in driving purchasing behaviour. Building on prior qualitative research, this paper reports the development and validation of the “RAFFLE” scale, to measure the Reasons and Facilitators for Loot box Engagement. A 23-item, seven-factor scale was developed through cognitive interviews (n = 25) followed by two surveys of UK-based gamers who purchase loot boxes; analysed via exploratory (n = 503) and confirmatory (n = 1495) factor analysis, respectively. Subscales encompassed “enhancement’; “progression’; “social pressure’; “distraction/compulsion’; “altruism’; “fear of missing out’; and “resale”. The scale showed good criterion and construct validity (correlating well with measures of loot box engagement; the risky loot box index (r = 0.63) and monthly self-reported spend (r = 0.38)), and good internal validity (Cronbach’s alpha = 0.84). Parallels with, and divergence from, motivations for related activities of gaming and gambling, and alignment with broader theoretical models of motivation, are discussed.

Response Interference as a Mechanism Underlying Implicit Measures

2008 ◽  
Vol 24 (4) ◽  
pp. 218-225 ◽  
Author(s):  
Bertram Gawronski ◽  
Roland Deutsch ◽  
Etienne P. LeBel ◽  
Kurt R. Peters
Keyword(s):  
Task Performance ◽  
Psychological Research ◽  
Implicit Measures ◽  
Mediation Model ◽  
Response Interference ◽  
Relevant Evidence ◽  
Moderated Mediation Model ◽  
Stimulus Features ◽  
Underlying Mechanisms

Over the last decade, implicit measures of mental associations (e.g., Implicit Association Test, sequential priming) have become increasingly popular in many areas of psychological research. Even though successful applications provide preliminary support for the validity of these measures, their underlying mechanisms are still controversial. The present article addresses the role of a particular mechanism that is hypothesized to mediate the influence of activated associations on task performance in many implicit measures: response interference (RI). Based on a review of relevant evidence, we argue that RI effects in implicit measures depend on participants’ attention to association-relevant stimulus features, which in turn can influence the reliability and the construct validity of these measures. Drawing on a moderated-mediation model (MMM) of task performance in RI paradigms, we provide several suggestions on how to address these problems in research using implicit measures.

The role of psychological research in the formation of policies affecting children.

1983 ◽  
Vol 38 (1) ◽  
pp. 80-84 ◽  
Author(s):  
Eleanor E. Maccoby ◽  
Alfred J. Kahn ◽  
Barbara A. Everett
Keyword(s):  
Psychological Research

Membangun Fondasi Keilmuan Ekonomi Islam: Telaah Metodologi Muhammad Akram Khan

2017 ◽  
Vol 1 (2) ◽  
pp. 152-170
Author(s):  
Ahmad Ubaidillah
Keyword(s):  
General Theory ◽  
Empirical Evidence ◽  
Inductive Method ◽  
Ethical Values ◽  
Institutional Conditions ◽  
Faith Based ◽  
Behavioral Parameters ◽  
Science Methodology ◽  
Islamic Society

Throughout my experience in tracking down and reading books on faith-based economics, in this case Islam, there are no books that specifically list the title of "Islamic economics". If there is, it is only initiated or introduced. Most books coming down to us still use the titles starting with the word, for example, system, concept, principle, or the doctrine of Islamic economics. Why do the authors of the book Islamic economics seem not dared to give his book title with label "science"? I presume that Islamic economics has not been considered as a science. In building a science, methodology is required. Islamic Economics also requires a well-established methodology to build the foundation of science. The study answers questions; how is methodology which is offered by Muhammad Akram Khan to build Islamic economics. The method used in this research is the study of literature with qualitative approach.The result of study concludes that Khan offers methodology of Islamic economics, if summarized, written as follows: First, Islamic economics uses a framework derived from the texts of divinity (revelation). Second, Islamic economics uses the inductive method, which gives witness to the truth or falsity assumptions and predictions about the two criteria of rationality and empirical evidence. Third, Islamic economy is built on ethical values ​​such as justice, virtue, moderation, sacrifice, caring for others, in the analysis, as behavioral parameters. Fourth, Islamic economics is a normative discipline. Islamic Economics investigates ways and means to change the existing economy with Islamic economy. Fifth, Islamic economics ask different questions with conventional economics. Its attention is on welfare (falah) human and creating social and institutional conditions that maximize falah in society. Clearly, Islamic economics strongly supports research programs that help maximize falah. Furthermore, Khan elaborates several issues related to the methodology that often appears in the forum of Islamic economists. There are some problems that Khan proposes, they are the interaction with modern economics, the role of revelation, assuming ideal Islamic society, and the general theory of Islamic economics.

Sign in / Sign up

Export Citation Format

Share Document