External Validity

2017 ◽  
Author(s):  
Richard McCleary ◽  
David McDowall ◽  
Bradley J. Bartos
Keyword(s):  
Construct Validity ◽  
Failure Rate ◽  
External Validity ◽  
Replication Study ◽  
Collaborative Group ◽  
Incentive Structure ◽  
Social Scientists ◽  
Time Frames ◽  
Statistical Conclusion ◽  
Mainstream Psychology

A threat to external validity is any factor that limits the generalizability of an observed result. Unlike all threats to statistical conclusion and internal validities and some threats to construct validity, threats to external validity cannot ordinarily be controlled by design. Nor is there any disagreement on how threats to external validity should be controlled. In most instances, it can only be controlled by replication?—across subjects, situations and time frames. This seldom happens, unfortunately, because the academic incentive structure discourages replication. The contemporary “reproducibility crisis” was spurred by a collaborative group of social scientists attempting to replicate one hundred experimental and correlational studies published in three mainstream psychology journals. Sixty percent of replications failed to reproduce the published effect. Failures to control for threats to external validity that stem from uncontrolled variations in persons, situations, and time frames, parsimosniously explain the failure rate in this replication study.

A Validity-Based Framework for Understanding Replication in Psychology

2020 ◽  
Vol 24 (4) ◽  
pp. 316-344
Author(s):  
Leandre R. Fabrigar ◽  
Duane T. Wegener ◽  
Richard E. Petty
Keyword(s):  
Construct Validity ◽  
Empirical Evidence ◽  
External Validity ◽  
Internal Validity ◽  
Psychological Research ◽  
Original Study ◽  
Statistical Conclusion ◽  
Conceptual Logic ◽  
Replication Failure

In recent years, psychology has wrestled with the broader implications of disappointing rates of replication of previously demonstrated effects. This article proposes that many aspects of this pattern of results can be understood within the classic framework of four proposed forms of validity: statistical conclusion validity, internal validity, construct validity, and external validity. The article explains the conceptual logic for how differences in each type of validity across an original study and a subsequent replication attempt can lead to replication “failure.” Existing themes in the replication literature related to each type of validity are also highlighted. Furthermore, empirical evidence is considered for the role of each type of validity in non-replication. The article concludes with a discussion of broader implications of this classic validity framework for improving replication rates in psychological research.

Improving the External Validity of Conjoint Analysis: The Essential Role of Profile Distribution

2021 ◽  
pp. 1-27 ◽  
Author(s):  
Brandon de la Cuesta ◽  
Naoki Egami ◽  
Kosuke Imai
Keyword(s):  
Conjoint Analysis ◽  
Real World ◽  
External Validity ◽  
Causal Effect ◽  
The Other ◽  
Estimation Methods ◽  
Social Scientists ◽  
Actual Distribution ◽  
Open Source Software Package ◽  
Profile Distribution

Abstract Conjoint analysis has become popular among social scientists for measuring multidimensional preferences. When analyzing such experiments, researchers often focus on the average marginal component effect (AMCE), which represents the causal effect of a single profile attribute while averaging over the remaining attributes. What has been overlooked, however, is the fact that the AMCE critically relies upon the distribution of the other attributes used for the averaging. Although most experiments employ the uniform distribution, which equally weights each profile, both the actual distribution of profiles in the real world and the distribution of theoretical interest are often far from uniform. This mismatch can severely compromise the external validity of conjoint analysis. We empirically demonstrate that estimates of the AMCE can be substantially different when averaging over the target profile distribution instead of uniform. We propose new experimental designs and estimation methods that incorporate substantive knowledge about the profile distribution. We illustrate our methodology through two empirical applications, one using a real-world distribution and the other based on a counterfactual distribution motivated by a theoretical consideration. The proposed methodology is implemented through an open-source software package.

Reliability and Validity A Quick Review

2007 ◽  
Vol 33 (5) ◽  
pp. 775-780 ◽  
Author(s):  
Karen Fitzner
Keyword(s):  
Construct Validity ◽  
External Validity ◽  
Diabetes Education ◽  
Reliability And Validity ◽  
Management Training ◽  
Self Management ◽  
Validity Testing ◽  
Education Practice ◽  
Rigorous Testing ◽  
Diabetes Educators

The purpose of this article is to provide a brief review of reliability and validity testing. These concepts are important to researchers who are choosing techniques and/or developing tools that will be applied and evaluated in diabetes education practice. Several types of reliability and validity testing are defined, and an easy-to-use check sheet is provided for research purposes. Following testing for the basic aspects of reliability and validity such as face and construct validity, a tool may be appropriate for use in practice settings. Those conducting comprehensive outcomes evaluations, however, may desire additional validation such as testing for external validity. Diabetes educators can and should incorporate rigorous testing for these important aspects when conducting assessments of techniques and tools relating to diabetes self-management training.

scholarly journals Organizational commitment, psychometric qualities and invariance of the Meyer and Allen Questionnaire for Portuguese Nurses

2018 ◽  
Vol 26 (0) ◽  
Author(s):  
Teresa Neves ◽  
João Graveto ◽  
Victor Rodrigues ◽  
João Marôco ◽  
Pedro Parreira
Keyword(s):  
Organizational Commitment ◽  
Construct Validity ◽  
Internal Consistency ◽  
External Validity ◽  
Goodness Of Fit ◽  
Factorial Structure ◽  
Portuguese Version ◽  
Confirmatory Factor ◽  
Specification Search ◽  
Confirmatory Analysis

ABSTRACT Objective: to evaluate the psychometric qualities of the Portuguese version of the Organizational Commitment Questionnaire for the nursing context, through confirmatory analysis and invariance, aiming to evaluate the reliability, internal consistency, construct validity and external validity of the instrument. Method: confirmatory factor analysis of the Portuguese version of the questionnaire was carried out with a sample of 850 nurses, in hospital context. The analysis was complemented using specification search. Goodness of fit was evaluated through different indices. Reliability, internal consistency and construct validity were estimated. The invariance of the model was evaluated in two subsamples of the same sample, in order to confirm the external validity of the factorial solution. Results: the refined model demonstrated good overall fit (χ2/df=6.37; CFI=0.91; GFI=0.92; RMSEA=0.08; MECVI=0.62). The factorial structure was stable (λ:Δχ2(14)=18.31; p=0;193; Intercepts: Δχ2(14)=22.29; p=0.073; Covariance: Δχ2(3)=6.01; p=0.111; Residuals: Δχ2(15)=22.44; p=0.097). Conclusion: the simplified model of the questionnaire demonstrated adequate goodness of fit, representing a stable factorial solution. The instrument was fit to monitor and evaluate the organizational commitment of Portuguese nurses.

scholarly journals Validez de constructo interna y externa del AGQ y PTEGQ en español

2015 ◽  
Vol 31 (2) ◽  
pp. 626
Author(s):  
Francisco Ruiz-Juan
Keyword(s):  
Physical Education ◽  
Construct Validity ◽  
Intrinsic Motivation ◽  
Achievement Goals ◽  
External Validity ◽  
Leisure Time ◽  
Motivational Climate ◽  
Psychometric Tests ◽  
Four Dimensions ◽  
Perceived Motivational Climate

The objective is to analyze the psychometric properties of Achievement Goals Questionnaire (AGQ) and Perceptions of Teachers' Emphasis on Goals Questionnaire (PTEGQ) in Spanish, determining the reliability and construct validity and external validity by understanding that achievement goals and perceived motivational climate in physical education may predict intrinsic motivation and satisfaction in those subjects who exercise at leisure time regularly. Psychometric tests confirm PTEGQ and AGQ have four dimensions that are hypothesized from the original one. It has proved the structural supporting hypothesis that it is based on the principle of compatibility. It has also been demonstrated construct validity and external validity as achievement goals and perceived motivational climate in physical education may predict intrinsic motivation and satisfaction in active subjects. Its reliability has been acceptable

scholarly journals Construct Validity in Software Engineering

2021 ◽  
Author(s):  
Dag Sjøberg ◽  
Gunnar Bergersen
Keyword(s):  
Quantitative Analysis ◽  
Software Engineering ◽  
Construct Validity ◽  
Common Ground ◽  
Conceptual Level ◽  
High Quality ◽  
Operational Level ◽  
Minimalist Approach ◽  
Unbiased Manner ◽  
Statistical Conclusion

Empirical research aims to establish generalizable claims from data. Such claims involve concepts that often must be measured indirectly by using indicators. Construct validity is concerned with whether one can justifiably make claims at the conceptual level that are supported by results at the operational level. We report a quantitative analysis of the awareness of construct validity in the software engineering literature between 2000 and 2019 and a qualitative review of 83 articles about human-centric experiments published in five high-quality journals between 2015 and 2019. Over the two decades, the appearance in the literature of the term construct validity increased sevenfold. Some of the reviewed articles we reviewed employed various ways to ensure that the indicators span the concept in an unbiased manner. We also found articles that reuse formerly validated constructs. However, the articles disagree about how to define construct validity. Several interpret construct validity excessively by including threats to internal, external, or statistical conclusion validity. A few articles also include fundamental challenges of a study, such as cheating and misunderstandings of experiment material. The diversity of topics discussed makes us recommend a minimalist approach to construct validity. We propose seven guidelines to establish a common ground for addressing construct validity in software engineering.

scholarly journals Measurement Schmeasurement: Questionable Measurement Practices and How to Avoid Them

2019 ◽  
Author(s):  
Jessica Kay Flake ◽  
Eiko I Fried
Keyword(s):  
Construct Validity ◽  
Degrees Of Freedom ◽  
Scientific Research ◽  
Replication Studies ◽  
Psychological Science ◽  
Rigorous Research ◽  
Statistical Conclusion ◽  
Measurement Practices

In this paper, we define questionable measurement practices (QMPs) as decisions researchers make that raise doubts about the validity of the measures, and ultimately the validity of study conclusions. Doubts arise for a host of reasons including a lack of transparency, ignorance, negligence, or misrepresentation of the evidence. We describe the scope of the problem and focus on how transparency is a part of the solution. A lack of measurement transparency makes it impossible to evaluate potential threats to internal, external, statistical conclusion, and construct validity. We demonstrate that psychology is plagued by a measurement schmeasurement attitude: QMPs are common, hide a stunning source of researcher degrees of freedom, pose a serious threat to cumulative psychological science, but are largely ignored. We address these challenges by providing a set of questions that researchers and consumers of scientific research can consider to identify and avoid QMPs. Transparent answers to these measurement questions promote rigorous research, allow for thorough evaluations of a study’s inferences, and are necessary for meaningful replication studies.

scholarly journals Measurement Schmeasurement: Questionable Measurement Practices and How to Avoid Them

2020 ◽  
Vol 3 (4) ◽  
pp. 456-465
Author(s):  
Jessica Kay Flake ◽  
Eiko I. Fried
Keyword(s):  
Construct Validity ◽  
Degrees Of Freedom ◽  
Scientific Research ◽  
Replication Studies ◽  
Psychological Science ◽  
Rigorous Research ◽  
Statistical Conclusion ◽  
Measurement Practices

In this article, we define questionable measurement practices (QMPs) as decisions researchers make that raise doubts about the validity of the measures, and ultimately the validity of study conclusions. Doubts arise for a host of reasons, including a lack of transparency, ignorance, negligence, or misrepresentation of the evidence. We describe the scope of the problem and focus on how transparency is a part of the solution. A lack of measurement transparency makes it impossible to evaluate potential threats to internal, external, statistical-conclusion, and construct validity. We demonstrate that psychology is plagued by a measurement schmeasurement attitude: QMPs are common, hide a stunning source of researcher degrees of freedom, and pose a serious threat to cumulative psychological science, but are largely ignored. We address these challenges by providing a set of questions that researchers and consumers of scientific research can consider to identify and avoid QMPs. Transparent answers to these measurement questions promote rigorous research, allow for thorough evaluations of a study’s inferences, and are necessary for meaningful replication studies.

scholarly journals Criteria of validity in experimental psychopathology: application to models of anxiety and depression

2012 ◽  
Vol 43 (11) ◽  
pp. 2241-2244 ◽  
Author(s):  
B. Vervliet ◽  
F. Raes
Keyword(s):  
Construct Validity ◽  
External Validity ◽  
Normal Subjects ◽  
Abnormal Behavior ◽  
Anxiety And Depression ◽  
Diagnostic Validity ◽  
Psychological Processes ◽  
Behavioral Models ◽  
Experimental Psychopathology ◽  
Novel Drug

The modeling of abnormal behavior in ‘normal’ subjects (often animals) has a long history in pharmacological research for the screening of novel drug compounds. Systematic criteria have been outlined in that literature to estimate the external validity of a model, that is to estimate how closely the model is linked to the disorder of interest. Experimental psychopathology (EPP) also uses behavioral models to study the psychological processes that underlie abnormal behavior. Although EPP researchers may occasionally feel uneasy about the validity of the model that they use, the issue has not received direct attention in this literature. Here, we review the criteria of validity as set out in pharmacology research (face, predictive and construct validity) and discuss their relevance for EPP research. Furthermore, we propose diagnostic validity as an additional criterion of external validity that is relevant to EPP research. We evaluate two models for the study of anxiety and depression, and show that they have good face, diagnostic and construct validity. However, EPP research generally lacks direct tests of predictive validity. We conclude that combined evaluations of predictive, diagnostic and construct validity provide a sound basis to infer the external validity of behavioral models in EPP research.

Sign in / Sign up

Export Citation Format

Share Document