On the Pitfalls of Estimating and Using Standardized Reliability Coefficients

2020 ◽  
pp. 001316442093734
Author(s):  
Tenko Raykov ◽  
George A. Marcoulides
Keyword(s):  
Empirical Research ◽  
Latent Variable ◽  
Reliability Coefficient ◽  
Latent Variable Modeling ◽  
Measuring Instruments ◽  
Modeling Framework ◽  
Scale Reliability ◽  
Reliability Coefficients

The population discrepancy between unstandardized and standardized reliability of homogeneous multicomponent measuring instruments is examined. Within a latent variable modeling framework, it is shown that the standardized reliability coefficient for unidimensional scales can be markedly higher than the corresponding unstandardized reliability coefficient, or alternatively substantially lower than the latter. Based on these findings, it is recommended that scholars avoid estimating, reporting, interpreting, or using standardized scale reliability coefficients in empirical research, unless they have strong reasons to consider standardizing the original components of utilized scales.

scholarly journals Don’t Forget the Model in Your Model-based Reliability Coefficients: A Reply to McNeish (2018)

2019 ◽  
Vol 5 (1) ◽  
Author(s):  
Victoria Savalei ◽  
Steven P. Reise
Keyword(s):  
Lower Bound ◽  
Latent Variable ◽  
Reliability Coefficient ◽  
Latent Variable Model ◽  
Reliability Measures ◽  
Variable Model ◽  
Large Samples ◽  
Model Based ◽  
Coefficient Omega ◽  
Reliability Coefficients

McNeish (2018) advocates that researchers abandon coefficient alpha in favor of alternative reliability measures, such as the 1-factor reliability (coefficient omega), a total reliability coefficient based on an exploratory bifactor solution (“Revelle’s omega total”), and the glb (“greatest lower bound”). McNeish supports this argument by demonstrating that these coefficients produce higher sample values in several examples. We express three main disagreements with this article. First, we show that McNeish exaggerates the extent to which alpha is different from omega when unidimensionality holds. Second, we argue that, when unidimensionality is violated, most alternative reliability coefficients are model-based, and it is critical to carefully select the underlying latent variable model rather than relying on software defaults. Third, we point out that higher sample reliability values do not necessarily capture population reliability better: many alternative reliability coefficients are upwardly biased except in very large samples. We conclude with a set of alternative recommendations for researchers.

Examining Multidimensional Measuring Instruments for Proximity to Unidimensional Structure Using Latent Variable Modeling

2020 ◽  
pp. 001316442094076
Author(s):  
Tenko Raykov ◽  
Matthias Bluemke
Keyword(s):  
Latent Variable ◽  
Social Research ◽  
General Structure ◽  
Latent Variable Modeling ◽  
Measuring Instruments ◽  
Interval Estimate ◽  
Measuring Instrument ◽  
Scale Test ◽  
Explained Variance ◽  
Unidimensional Structure

A widely applicable procedure of examining proximity to unidimensionality for multicomponent measuring instruments with multidimensional structure is discussed. The method is developed within the framework of latent variable modeling and allows one to point and interval estimate an explained variance proportion-based index that may be considered a measure of proximity to unidimensional structure. The approach is readily utilized in educational, behavioral, and social research when it is of interest to evaluate whether a more general structure scale, test, or measuring instrument could be treated as being associated with an approximately unidimensional latent structure for some empirical purposes.

scholarly journals A Method for Examining the Equating of Psychometric Scales and Tests: An Application Using Dementia Screening Test Batteries

2018 ◽  
Vol 80 (1) ◽  
pp. 199-209
Author(s):  
N. Maritza Dowling ◽  
Tenko Raykov ◽  
George A. Marcoulides
Keyword(s):  
Clinical Research ◽  
Empirical Research ◽  
Screening Test ◽  
Latent Variable ◽  
Necessary Condition ◽  
Measuring Instruments ◽  
Psychometric Scales ◽  
Latent Constructs ◽  
Dementia Screening ◽  
Wide Range

Equating of psychometric scales and tests is frequently required and conducted in educational, behavioral, and clinical research. Construct comparability or equivalence between measuring instruments is a necessary condition for making decisions about linking and equating resulting scores. This article is concerned with a widely applicable method for examining if two scales or tests cannot be equated. A latent variable modeling method is discussed that can be used to evaluate whether the tests or parts thereof measure latent constructs that are distinct from each other. The approach can be routinely used before an equating procedure is undertaken, in order to assess whether equating could be meaningfully carried out to begin with. The procedure is readily applicable in empirical research using popular software. The method is illustrated with data from dementia screening test batteries administered as part of two studies designed to evaluate a wide range of biomarkers throughout the process of normal aging to dementia or Alzheimer’s disease.

scholarly journals Examining Population Differences in Within-Person Variability in Longitudinal Designs Using Latent Variable Modeling: An Application to the Study of Cognitive Functioning of Older Adults

2018 ◽  
Vol 79 (3) ◽  
pp. 598-609 ◽  
Author(s):  
N. Maritza Dowling ◽  
Tenko Raykov ◽  
George A. Marcoulides
Keyword(s):  
Repeated Measures ◽  
Latent Variable ◽  
Cognitive Test ◽  
Latent Variable Modeling ◽  
Population Differences ◽  
Modeling Framework ◽  
Interval Estimates ◽  
Modeling Methodology ◽  
Multiple Dimensions ◽  
Verbal Episodic Memory

Longitudinal studies have steadily grown in popularity across the educational and behavioral sciences, particularly with the increased availability of technological devices that allow the easy collection of repeated measures on multiple dimensions of substantive relevance. This article discusses a procedure that can be used to evaluate population differences in within-person (intraindividual) variability in such longitudinal investigations. The method is based on an application of the latent variable modeling methodology within a two-level modeling framework. The approach is used to obtain point and interval estimates of the differences in within-person variance and in the strength of correlative effects of repeated measures between normal and very mildly demented persons in a longitudinal study of a diagnostic cognitive test assessing verbal episodic memory.

scholarly journals Flow With an Intelligent Tutor: A Latent Variable Modeling Approach to Tracking Flow During Artificial Tutoring

2021 ◽  
Author(s):  
Hyeon-Ah Kang ◽  
Adam Sales ◽  
Tiffany A. Whittaker
Keyword(s):  
Latent Variable ◽  
Latent Class ◽  
Intelligent Tutoring System ◽  
Latent Class Model ◽  
Model Fitting ◽  
Latent Variable Modeling ◽  
Flow State ◽  
Modeling Framework ◽  
Modeling Approach ◽  
Comprehensive Survey

Increasing use of intelligent tutoring system (ITS) in education calls for analytic methods that can unravel students' learning behaviors. In this study we suggest a latent variable modeling approach to tracking flow during artificial tutoring. Flow is a mental state a student achieves when immersed in deep learning. Modeling latent flow helps identify when and how students flow during tutoring. The result of the model can also inform the functioning of ITS and provide instrumental information for designing interventions. Three latent variable models are considered to draw discrete inference on the flow state: the (i) latent class model, (ii) latent transition model, and (iii) hidden Markov model. For each of the models, we suggest practical model-fitting strategies, addressing the assumptions and estimation constraints. Using example data from Cognitive Tutor Algebra I, we show that each model provides unique and meaningful information about student's learning process. Through comprehensive survey of the models, we evaluate merits and drawbacks of each modeling framework and illuminate areas that need future development.

scholarly journals Studying Latent Criterion Validity for Complex Structure Measuring Instruments Using Latent Variable Modeling

2017 ◽  
Vol 78 (5) ◽  
pp. 905-917 ◽  
Author(s):  
Tenko Raykov ◽  
Natalja Menold ◽  
George A. Marcoulides
Keyword(s):  
Measurement Error ◽  
Latent Variable ◽  
Statistical Tests ◽  
Complex Structure ◽  
Second Order ◽  
Latent Variable Modeling ◽  
Measuring Instruments ◽  
Population Parameter ◽  
Criterion Variable ◽  
The Relationship

Validity coefficients for multicomponent measuring instruments are known to be affected by measurement error that attenuates them, affects associated standard errors, and influences results of statistical tests with respect to population parameter values. To account for measurement error, a latent variable modeling approach is discussed that allows point and interval estimation of the relationship of an underlying latent factor to a criterion variable in a setting that is more general than the commonly considered homogeneous psychometric test case. The method is particularly helpful in validity studies for scales with a second-order factorial structure, by allowing evaluation of the relationship between the second-order factor and a criterion variable. The procedure is similarly useful in studies of discriminant, convergent, concurrent, and predictive validity of measuring instruments with complex latent structure, and is readily applicable when measuring interrelated traits that share a common variance source. The outlined approach is illustrated using data from an authoritarianism study.

scholarly journals Reliability of Scales With Second-Order Structure: Evaluation of Coefficient Alpha’s Population Slippage Using Latent Variable Modeling

2017 ◽  
Vol 78 (6) ◽  
pp. 1123-1135
Author(s):  
Tenko Raykov ◽  
Philippe Goldammer ◽  
George A. Marcoulides ◽  
Tatyana Li ◽  
Natalja Menold
Keyword(s):  
Latent Variable ◽  
Complex Structure ◽  
Interval Estimation ◽  
Psychological Research ◽  
Second Order ◽  
Coefficient Alpha ◽  
Latent Variable Modeling ◽  
Order Structure ◽  
Measuring Instruments ◽  
Structure Evaluation

A readily applicable procedure is discussed that allows evaluation of the discrepancy between the popular coefficient alpha and the reliability coefficient of a scale with second-order factorial structure that is frequently of relevance in empirical educational and psychological research. The approach is developed within the framework of the widely used latent variable modeling methodology and permits point and interval estimation of the slippage of alpha from scale reliability in a population under investigation. The method is useful when examining the consistency of complex structure measuring instruments assessing higher order latent constructs and, under its assumptions, represents a generally recommendable alternative to coefficient alpha. The outlined procedure is illustrated using data from an authoritarianism study.

Sign in / Sign up

Export Citation Format

Share Document