Putting the individual into reliability: Bayesian testing of homogeneous within-person variance in hierarchical models

Measurement reliability is a fundamental concept in psychology. It is traditionally considered a stable property of a questionnaire, measurement device, or experimental task. Although intraclass correlation coefficients (ICC) are often used to assess reliability in repeated measure designs, their descriptive nature depends upon the assumption of a common within-person variance. This work focuses on the presumption that each individual is adequately described by the average within-person variance in hierarchical models. And thus whether reliability generalizes to the individual level, which leads directly into the notion of individually varying ICCs. In particular, we introduce a novel approach, using the Bayes factor, wherein a researcher can directly test for homogeneous within-person variance in hierarchical models. Additionally, we introduce a membership model that allows for classifying which (and how many) individuals belong to the common variance model. The utility of our methodology is demonstrated on cognitive inhibition tasks. We find that heterogeneous within-person variance is a defining feature of these tasks, and in one case, the ratio between the largest to smallest within-person variance exceeded 20. This translates into a tenfold difference in person-specific reliability! We also find that few individuals belong to the common variance model, and thus traditional reliability indices are potentially masking important individual variation. We discuss the implications of our findings and possible future directions. The methods are implemented in the R package vICC

Dimensionality and Reliability of the Intentions to Seeking Counseling Inventory with International Students

To date, the Intentions to Seek Counseling Inventory (ISCI; Cash et al., 1975) is the most widely used instrument used to measure psychological help-seeking intentions. However, the ISCI has yet to be validated with international students. In this study, we examined the dimensionality and reliability of the Intentions to Seek Counseling Inventory by performing exploratory factor analysis, Cronbach alpha reliability analysis, and split-half reliability analysis with a sample of international students (N = 183). We found a two-factor, 15-item assessment that explained 49.92% of the common variance. Our findings provide support for the use of ISCI with international students, albeit with different dimensionality. Implications of the findings for both research and practice are discussed.

Discriminant analysis of the equicorrelated Gaussian observations

In this paper the problem of classification of an observation into one of two Gaussian populations with different means and common variance is considered in the case when equicorrelated training sample is given. Unknown means and common variance are estimated from training sample and these estimators are pluged in the Bayes discriminant function. The maximum likelihood estimators are used. The approximation of the expected error rate associated with Bayes plug-in discriminant function is derived. Numerical analysis of the accuracy of that approximation for various values of correlation is presented.

Factor Structure of the Scales for Assessing Emotional Disturbance : 3 Rating Scale for Students Identified With Emotional Disturbance

This study examined the internal structure of the Scales for Assessing Emotional Disturbance-3 Rating Scale (RS), a teacher-completed RS developed to measure emotional disturbance (ED). As defined in U.S. law and regulations, ED involves five characteristics or patterns of behavioral and emotional maladaptation. RS data obtained on a sample of students with ED were used to examine validity evidence based on the internal structure of the assessment. Of particular interest was the extent to which multivariate factors derived from the RS data conform to the five characteristics of ED stated in the definition. Results indicate that the RS data fit a 5-factor model reasonably well. A subsequent bifactor analysis identified a considerable proportion of common variance across factors, suggesting the presence of a strong general ED factor, two distinct group factors (Inability to Learn and Inappropriate Behavior), and three weak group factors. The findings provided evidence of the validity of the SAED-3 RS based on internal structure and pointed to support for use of the RS in contributing to the process of determining whether a student qualifies for the ED education disability. Implications for improved research on the nature of ED and how students with ED can be better served are discussed.

Entwicklung und Validierung eines Kurzfragebogens zur Erfassung eines motivierenden Unterrichtsstils

Zusammenfassung. Die vorliegende Studie beschreibt die Entwicklung und Validierung eines Kurzfragebogens zur Erfassung eines motivierenden Unterrichtsstils für Schülerinnen und Schüler der fünften bis zwölften Schulstufe (KEMU 5–12) im Sinne der Selbstbestimmungstheorie ( Ryan & Deci, 2020 ). Ein motivierender Unterrichtstil ist durch Autonomieunterstützung und Struktur der Lehrperson gekennzeichnet und kann das Engagement und das psychische Wohlbefinden von Schülerinnen und Schülern steigern. Anhand zweier heterogener Stichproben ( N 1 = 1,155, N 2 = 1,686) wurden die Faktorenstruktur, die Reliabilität und die valide Interpretierbarkeit der Testwerte des Fragebogens für den Einsatz in Forschungsarbeiten zur Beurteilung von Auswirkungen eines motivierenden Unterrichtsstils sowie zu Gruppenvergleichen untersucht. Indizes zur Beurteilung der Dimensionalität (Percentage of Uncontamined Correlations, Explained Common Variance, Omega Hierarchical) weisen auf einen starken Generalfaktor hin und sprechen dafür, dass eine eindimensionale Modellierung des KEMU 5–12 unverzerrte Schätzungen für Strukturparameter liefert. Die Reliabilität der Skala ist in beiden Stichproben ausgezeichnet (>ω t = .88 und .85). Außerdem konnte skalare Messinvarianz für Geschlecht, Schulstufe, Fach und Erhebungsmethode belegt werden. Die theoriekonformen Korrelationen mit selbstbestimmter Motivation, Flowerleben, schulischem Wohlbefinden, den unterrichtsbezogenen Emotionen Freude, Angst, Ärger und Langeweile sowie mit Noten sprechen für eine valide Interpretierbarkeit der Testwerte in den genannten Einsatzbereichen. Mit dem KEMU 5–12 kann somit ein motivierender Unterrichtsstil reliabel, valide und ökonomisch erfasst werden.

Office Clutter: Comparing Lower and Upper-Level Employees on Work-related Criteria

Office clutter might significantly impact productivity, yet no study examined workers differences across upper and lower employee status. The present study surveyed 202 U.S. on-site workers on work-related variables, including office clutter. Job classifications were aggregated, creating two groups: upper- and lower-level employees. A significant difference in office clutter impacted worker-levels: upper-level workers compared to lower-level workers had higher office clutter scores. Exploratory factor analysis created a two-factor solution (explaining 62.6% of the common variance): satisfaction/pleasure from one’s work and risk for work-related burnout/tension. There was a significant difference in office clutter perception: upper-level workers were significantly more likely to report clutter and being at risk for burnout/tension than lower-level workers. Office clutter significantly negatively predicted satisfaction with one’s job and positively related with risk for work-related burnout. Frequently reported office clutter items (in order of frequency) were paper, trash (e.g., used coffee cups), and office supplies.

Vertical Jump Testing in Rugby League: A Rationale for Calculating Take-Off Momentum

The purpose of this study was to determine the usefulness of calculating jump take-off momentum in rugby league (RL) by exploring its relationship with sprint momentum, due to the latter being an important attribute of this sport. Twenty-five male RL players performed 3 maximal-effort countermovement jumps on a force platform and 3 maximal effort 20-m sprints (with split times recorded). Jump take-off momentum and sprint momentum (between 0 and 5, 5 and 10, and 10 and 20 m) were calculated (mass multiplied by velocity) and their relationship determined. There was a very large positive relationship between both jump take-off and 0- to 5-m sprint momentum (r = .781, P < .001) and jump take-off and 5- to 10-m sprint momentum (r = .878, P < .001). There was a nearly perfect positive relationship between jump take-off and 10- to 20-m sprint momentum (r = .920, P < .001). Jump take-off and sprint momentum demonstrated good–excellent reliability and very large–nearly perfect associations (61%–85% common variance) in an RL cohort, enabling prediction equations to be created. Thus, it may be practically useful to calculate jump take-off momentum as part of routine countermovement jump testing of RL players and other collision-sport athletes to enable the indirect monitoring of sprint momentum.

Within-person structures of daily cognitive performance differ from between-person structures of cognitive abilities

Over a century of research on between-person differences has resulted in the consensus that human cognitive abilities are hierarchically organized, with a general factor, termed general intelligence or “g,” uppermost. Surprisingly, it is unknown whether this body of evidence is informative about how cognition is structured within individuals. Using data from 101 young adults performing nine cognitive tasks on 100 occasions distributed over six months, we find that the structures of individuals’ cognitive abilities vary among each other, and deviate greatly from the modal between-person structure. Working memory contributes the largest share of common variance to both between- and within-person structures, but the g factor is much less prominent within than between persons. We conclude that between-person structures of cognitive abilities cannot serve as a surrogate for within-person structures. To reveal the development and organization of human intelligence, individuals need to be studied over time.

Factor Structure of the Differential Ability Scales–Second Edition Core Subtests: Standardization Sample Confirmatory Factor Analyses

The present study examined the factor structure of the Differential Ability Scales–Second Edition (DAS-II) core subtests from the standardization sample via confirmatory factor analysis (CFA) using methods (bifactor modeling and variance partitioning) and procedures (robust model estimation due to nonnormal subtest score distributions) recommended but not included in the DAS-II Introductory and Technical Handbook. CFAs were conducted with the three DAS-II standardization sample age groups (lower early years [age = 2:6–3:5 years], upper early years [age = 3:6–6:11 years], school age [7:0–17:11 years]) using standardization sample raw data provided by NCS Pearson, Inc. Although most DAS-II core subtests were properly associated with the theoretically proposed group factors, both the higher order and bifactor models indicated that the g factor accounted for large portions of total and common variance, whereas the group factors (Verbal, Nonverbal, Spatial) accounted for small portions of total and common variance. The DAS-II core battery provides strong measurement of general intelligence, and clinical interpretation should be primarily, if not exclusively, at that level.