scholarly journals The Validation Crisis in Psychology

2021 ◽  
Vol 5 ◽  
Author(s):  
Ulrich Schimmack
Keyword(s):  
Construct Validity ◽  
Structural Equation ◽  
Structural Equation Models ◽  
Cost Benefit ◽  
Causal Modeling ◽  
Crisis In Psychology ◽  
Multi Method Approach ◽  
Validation Research ◽  
Method Approach

Cronbach and Meehl (1955) introduced the concept of construct validity and described how researchers can demonstrate that their measures have construct validity. Although the term construct validity is widely used, few researchers follow Cronbach and Meehl’s recommendation to quantify construct validity with the help of nomological networks. As a result, the construct validity of many popular measures in psychology is unknown. I call for rigorous tests of construct validity that follow Cronbach and Meehl’s recommendations to improve psychology as a science. Without valid measures even replicable results are uninformative. I suggest that a proper program of validation research requires a multi-method approach and causal modeling of correlations with structural equation models. Construct validity should be quantified to enable cost-benefit analyses and to replace existing measures with better measures that have superior construct validity.

scholarly journals The Validation Crisis in Psychology

2019 ◽  
Author(s):  
Ulrich Schimmack
Keyword(s):  
Construct Validity ◽  
Structural Equation Model ◽  
Structural Equation ◽  
Construct Validation ◽  
Measurement Model ◽  
Equation Model ◽  
The State ◽  
Crisis In Psychology ◽  
Seminal Article ◽  
Validation Research

In this commentary on the state of validation research in psychology, I review Cronbach and Fiske’s (1955) seminal article and point out that the term is widely used, but researchers rarely follow their recommendations. Most important, construct validation requires specification of a nomological net, which could be done with a structural equation model and construct validity should be quantified, which could be done by means of factor loadings in an SEM measurement model.

scholarly journals Assessing the dimensionality and educational impacts of integrated ICT literacy in the higher education context

2018 ◽  
Vol 34 (1) ◽  
Author(s):  
Tefera Tadesse ◽  
Robyn M Gillies ◽  
Chris Campbell
Keyword(s):  
Higher Education ◽  
Conceptual Model ◽  
Learning Outcomes ◽  
Undergraduate Students ◽  
Structural Equation ◽  
Factor Model ◽  
Ict Literacy ◽  
Information And Communication ◽  
Multi Method Approach ◽  
Method Approach

The purpose of this paper is threefold: first, to introduce a conceptual model for assessing undergraduate students’ integrated information and communication technology (ICT) literacy capacity that involves 12 items generated from the modified version of the Australasian Survey of Student Engagement (AUSSE) questionnaire (Coates, 2010); second, to illustrate the construct validity and internal consistency of the model as implemented in a sample of undergraduate students (n = 536) enrolled in two colleges within a large Ethiopian university; and third, to further demonstrate the criterion validity of the model by examining predictive validity of the identified ICT literacy factors on student learning outcomes. A multi-method approach is used, which comprises correlation analysis, multiple regression analysis and structural equation modelling (SEM) techniques. The main finding is the support found for the 4-factor model consisting of ICT use, cognitive process, reading task and writing task. Results of the multi-method approach provide specific guidelines to higher education (HE) institutions using this approach to evaluate ICT literacy capacity and the resultant learning outcomes among their undergraduate students. The paper provides a conceptual model and supporting tools that can be used by other HE institutions to assist in the evaluation of students’ ICT literacy capacities.

Construct Validity of the Questionnaire on Experience with Skin Complaints (Short Form)

2007 ◽  
Vol 23 (3) ◽  
pp. 176-184 ◽  
Author(s):  
Horst Müller ◽  
Thomas Brockow ◽  
Annegret Franke ◽  
Karl-Ludwig Resch ◽  
Iris Tatjana Calliess ◽  
...  
Keyword(s):  
Construct Validity ◽  
Structural Equation ◽  
Structural Equation Models ◽  
Skin Diseases ◽  
Short Form ◽  
Treatment Success ◽  
Likelihood Estimation ◽  
Self Esteem ◽  
Factor Analyses ◽  
Chronic Skin

Abstract. Introduction: Feelings of stigmatization can strongly influence quality of life in individuals with chronic skin diseases. The Short-Form of the Questionnaire on Experience with Skin Complaints (SF-QES) differentiates four factors of stigmatization: self-esteem and retreat, experienced refusal, concealment, and composure. The current study aimed to investigate the construct validity of the SF-QES. Method: The analysis was based on the complete SF-QES records of a clinical psoriasis trial, which yielded 1,005 records at baseline, 1,010 records at the end of therapy, and 885 and 827 records, respectively, at two follow-ups. Factor analyses and corresponding structural equation models (SEMs) using robust maximum likelihood estimation (RML) were applied. Additionally, the responsiveness of the scales to judgments of treatment success and two different interventions were compared. Results: The factor analyses provided results that widely agreed with the supposed four factors. SEM, however, showed moderately insufficient fit for this model, but supported five factors and the discrimination between affected self-esteem and retreat. The retreat scale was clearly most responsive to treatment success and intervention type. Conclusion: The construct validity of the SF-QES was largely supported, but the factor retreat should be separated from affected self-esteem and is most sensitive to physical intervention.

scholarly journals Causality on longitudinal data: Stable specification search in constrained structural equation modeling

2017 ◽  
Vol 27 (12) ◽  
pp. 3814-3834 ◽  
Author(s):  
Ridho Rahmadi ◽  
Perry Groot ◽  
Marieke HC van Rijn ◽  
Jan AJG van den Brand ◽  
Marianne Heins ◽  
...  
Keyword(s):  
Longitudinal Data ◽  
Structural Equation ◽  
Structural Equation Models ◽  
Simulated Data ◽  
Chronic Fatigue ◽  
Causal Modeling ◽  
Model Complexity ◽  
Equation Modeling ◽  
Data Set ◽  
Finite Data

A typical problem in causal modeling is the instability of model structure learning, i.e., small changes in finite data can result in completely different optimal models. The present work introduces a novel causal modeling algorithm for longitudinal data, that is robust for finite samples based on recent advances in stability selection using subsampling and selection algorithms. Our approach uses exploratory search but allows incorporation of prior knowledge, e.g., the absence of a particular causal relationship between two specific variables. We represent causal relationships using structural equation models. Models are scored along two objectives: the model fit and the model complexity. Since both objectives are often conflicting, we apply a multi-objective evolutionary algorithm to search for Pareto optimal models. To handle the instability of small finite data samples, we repeatedly subsample the data and select those substructures (from the optimal models) that are both stable and parsimonious. These substructures can be visualized through a causal graph. Our more exploratory approach achieves at least comparable performance as, but often a significant improvement over state-of-the-art alternative approaches on a simulated data set with a known ground truth. We also present the results of our method on three real-world longitudinal data sets on chronic fatigue syndrome, Alzheimer disease, and chronic kidney disease. The findings obtained with our approach are generally in line with results from more hypothesis-driven analyses in earlier studies and suggest some novel relationships that deserve further research.

Measuring the Big Five in Self-Report and Other Ratings

2000 ◽  
Vol 16 (1) ◽  
pp. 31-43 ◽  
Author(s):  
Claudio Barbaranelli ◽  
Gian Vittorio Caprara
Keyword(s):  
Big Five ◽  
Structural Equation ◽  
Structural Equation Models ◽  
Error Variance ◽  
The Self ◽  
Self Report ◽  
Sources Of Information ◽  
Third Person ◽  
Fit Indexes ◽  
Response Modes

Summary: The aim of the study is to assess the construct validity of two different measures of the Big Five, matching two “response modes” (phrase-questionnaire and list of adjectives) and two sources of information or raters (self-report and other ratings). Two-hundred subjects, equally divided in males and females, were administered the self-report versions of the Big Five Questionnaire (BFQ) and the Big Five Observer (BFO), a list of bipolar pairs of adjectives ( Caprara, Barbaranelli, & Borgogni, 1993 , 1994 ). Every subject was rated by six acquaintances, then aggregated by means of the same instruments used for the self-report, but worded in a third-person format. The multitrait-multimethod matrix derived from these measures was then analyzed via Structural Equation Models according to the criteria proposed by Widaman (1985) , Marsh (1989) , and Bagozzi (1994) . In particular, four different models were compared. While the global fit indexes of the models were only moderate, convergent and discriminant validities were clearly supported, and method and error variance were moderate or low.

Collective Efficacy and Organizational Commitment in an Italian City Hall

2009 ◽  
Vol 14 (4) ◽  
pp. 363-371 ◽  
Author(s):  
Laura Borgogni ◽  
Silvia Dello Russo ◽  
Laura Petitta ◽  
Gary P. Latham
Keyword(s):  
Organizational Commitment ◽  
Collective Efficacy ◽  
Structural Equation ◽  
Structural Equation Models ◽  
Practical Significance ◽  
City Hall ◽  
Italian City ◽  
Public Sector Managers ◽  
The Individual ◽  
Level Group

Employees (N = 170) of a City Hall in Italy were administered a questionnaire measuring collective efficacy (CE), perceptions of context (PoC), and organizational commitment (OC). Two facets of collective efficacy were identified, namely group and organizational. Structural equation models revealed that perceptions of top management display a stronger relationship with organizational collective efficacy, whereas employees’ perceptions of their colleagues and their direct superior are related to collective efficacy at the group level. Group collective efficacy had a stronger relationship with affective organizational commitment than did organizational collective efficacy. The theoretical significance of this study is in showing that CE is two-dimensional rather than unidimensional. The practical significance of this finding is that the PoC model provides a framework that public sector managers can use to increase the efficacy of the organization as a whole as well as the individual groups that compose it.

A Cautionary Note on Incremental Fit Indices Reported by LISREL

Methodology ◽  
2005 ◽  
Vol 1 (2) ◽  
pp. 81-85 ◽  
Author(s):  
Stefan C. Schmukle ◽  
Jochen Hardt
Keyword(s):  
Factor Analysis ◽  
Maximum Likelihood ◽  
Structural Equation ◽  
Structural Equation Models ◽  
Null Model ◽  
Likelihood Estimation ◽  
Parameter Estimates ◽  
Fit Indices ◽  
Cautionary Note ◽  
Confirmatory Factor

Abstract. Incremental fit indices (IFIs) are regularly used when assessing the fit of structural equation models. IFIs are based on the comparison of the fit of a target model with that of a null model. For maximum-likelihood estimation, IFIs are usually computed by using the χ2 statistics of the maximum-likelihood fitting function (ML-χ2). However, LISREL recently changed the computation of IFIs. Since version 8.52, IFIs reported by LISREL are based on the χ2 statistics of the reweighted least squares fitting function (RLS-χ2). Although both functions lead to the same maximum-likelihood parameter estimates, the two χ2 statistics reach different values. Because these differences are especially large for null models, IFIs are affected in particular. Consequently, RLS-χ2 based IFIs in combination with conventional cut-off values explored for ML-χ2 based IFIs may lead to a wrong acceptance of models. We demonstrate this point by a confirmatory factor analysis in a sample of 2449 subjects.

Forced Zero Cross-Loading Misspecifications in Measurement Component of Structural Equation Models

Methodology ◽  
2014 ◽  
Vol 10 (4) ◽  
pp. 138-152 ◽  
Author(s):  
Hsien-Yuan Hsu ◽  
Susan Troncoso Skidmore ◽  
Yan Li ◽  
Bruce Thompson
Keyword(s):  
Structural Equation ◽  
Structural Model ◽  
Structural Equation Models ◽  
Equation Model ◽  
Model Fit ◽  
Model Specification ◽  
Simulation Design ◽  
Measurement Models ◽  
Fit Statistics ◽  
Path Coefficients

The purpose of the present paper was to evaluate the effect of constraining near-zero parameter cross-loadings to zero in the measurement component of a structural equation model. A Monte Carlo 3 × 5 × 2 simulation design was conducted (i.e., sample sizes of 200, 600, and 1,000; parameter cross-loadings of 0.07, 0.10, 0.13, 0.16, and 0.19 misspecified to be zero; and parameter path coefficients in the structural model of either 0.50 or 0.70). Results indicated that factor pattern coefficients and factor covariances were overestimated in measurement models when near-zero parameter cross-loadings constrained to zero were higher than 0.13 in the population. Moreover, the path coefficients between factors were misestimated when the near-zero parameter cross-loadings constrained to zero were noteworthy. Our results add to the literature detailing the importance of testing individual model specification decisions, and not simply evaluating omnibus model fit statistics.

Sign in / Sign up

Export Citation Format

Share Document