Reliability Generalization: Exploring Variance in Measurement Error Affecting Score Reliability Across Studies

2011 ◽  
pp. 202-218
Author(s):  
Vacha-Haase Tammi
Keyword(s):  
Measurement Error ◽  
Reliability Generalization ◽  
Score Reliability

A Meta-Analytical Multilevel Reliability Generalization of Situational Judgment Tests (SJTs)

2016 ◽  
Vol 32 (3) ◽  
pp. 230-240 ◽  
Author(s):  
Nadine Kasten ◽  
Philipp Alexander Freund
Keyword(s):  
Personnel Selection ◽  
Hierarchical Linear Model ◽  
Future Research ◽  
Reliability Generalization ◽  
High Stakes ◽  
Situational Judgment ◽  
Situational Judgment Tests ◽  
Dependent Observations ◽  
Score Reliability ◽  
Study Characteristics

Abstract. During the past 20 years, Situational Judgment Tests (SJTs) have developed into a viable tool in personnel selection. Despite their growing popularity, research examining the extent of measurement error is widely lacking. Using reliability generalization, the aim of this article was twofold: (1) establish an estimate for an average coefficient alpha of SJT scores across studies and (2) examine the influence of essential SJT features and selected study variables on score reliability. To handle potential dependent observations a three-level hierarchical linear model was used. The results indicate that the reliability of SJT scores is typically rather low and below recommended levels for high-stakes applications. Additionally, both SJT and study characteristics affect score reliability. Implications for practitioners and researchers are provided to guide an appropriate use of SJTs and to initiate future research.

scholarly journals At What Sample Size Do Latent Variable Correlations Stabilize?

2018 ◽  
Author(s):  
Andre Kretzschmar ◽  
Gilles Gignac
Keyword(s):  
Monte Carlo Simulation ◽  
Monte Carlo ◽  
Measurement Error ◽  
Latent Variable ◽  
Composite Score ◽  
Sample Sizes ◽  
Population Correlation ◽  
Score Reliability ◽  
Coefficient Omega ◽  
Stable Measurement

We conducted a Monte-Carlo simulation within a latent variable framework by varying the following characteristics: population correlation (ρ = .10, .20, .30, .40, .50, .60, .70, .80, .90, and 1.00) and composite score reliability (coefficient omega: ω = .40, .50, .60, .70, .80, and .90). The sample sizes required to estimate stable measurement-error-free correlations were found to approach N = 490 for typical research scenarios (population correlation ρ = .20; composite score reliability ω = .70) and as high as N = 1,000+ for data associated with lower, but still sometimes observed, reliabilities (ω = .40 to .50). We encourage researchers to take into consideration reliability, when evaluating the sample sizes required to produce stable measurement-error-free correlations.

Sign in / Sign up

Export Citation Format

Share Document