Diagnostic Classification Model for Forced-Choice Items and Noncognitive Tests

The forced-choice (FC) item formats used for noncognitive tests typically develop a set of response options that measure different traits and instruct respondents to make judgments among these options in terms of their preference to control the response biases that are commonly observed in normative tests. Diagnostic classification models (DCMs) can provide information regarding the mastery status of test takers on latent discrete variables and are more commonly used for cognitive tests employed in educational settings than for noncognitive tests. The purpose of this study is to develop a new class of DCM for FC items under the higher-order DCM framework to meet the practical demands of simultaneously controlling for response biases and providing diagnostic classification information. By conducting a series of simulations and calibrating the model parameters with a Bayesian estimation, the study shows that, in general, the model parameters can be recovered satisfactorily with the use of long tests and large samples. More attributes improve the precision of the second-order latent trait estimation in a long test, but decrease the classification accuracy and the estimation quality of the structural parameters. When statements are allowed to load on two distinct attributes in paired comparison items, the specific-attribute condition produces better a parameter estimation than the overlap-attribute condition. Finally, an empirical analysis related to work-motivation measures is presented to demonstrate the applications and implications of the new model.

Examining the Robustness of the Graded Response and 2-Parameter Logistic Models to Violations of Construct Normality

When fitting unidimensional item response theory (IRT) models, the population distribution of the latent trait (θ) is often assumed to be normally distributed. However, some psychological theories would suggest a nonnormal θ. For example, some clinical traits (e.g., alcoholism, depression) are believed to follow a positively skewed distribution where the construct is low for most people, medium for some, and high for few. Failure to account for nonnormality may compromise the validity of inferences and conclusions. Although corrections have been developed to account for nonnormality, these methods can be computationally intensive and have not yet been widely adopted. Previous research has recommended implementing nonnormality corrections when θ is not “approximately normal.” This research focused on examining how far θ can deviate from normal before the normality assumption becomes untenable. Specifically, our goal was to identify the type(s) and degree(s) of nonnormality that result in unacceptable parameter recovery for the graded response model (GRM) and 2-parameter logistic model (2PLM).

Are attitudes toward immigration changing in Europe? An analysis based on latent class IRT models

We analyze the changing attitudes toward immigration in EU host countries in the last few years (2010–2018) on the basis of the European Social Survey data. These data are collected by the administration of a questionnaire made of items concerning different aspects related to the immigration phenomenon. For this analysis, we rely on a latent class approach considering a variety of models that allow for: (1) multidimensionality; (2) discreteness of the latent trait distribution; (3) time-constant and time-varying covariates; and (4) sample weights. Through these models we find latent classes of Europeans with similar levels of immigration acceptance and we study the effect of different socio-economic covariates on the probability of belonging to these classes for which we provide a specific interpretation. In this way we show which countries tend to be more or less positive toward immigration and we analyze the temporal dynamics of the phenomenon under study.

Reliability and Validity of the Japanese Version of the 12-Item Psychosocial Safety Climate Scale (PSC-12J)

The 12-item psychosocial safety climate scale (PSC-12) has been used extensively in previous research, but its reliability and validity in a Japanese context are still unknown. We examined the psychometrics of the Japanese version of the PSC-12 (PSC-12J). The PSC-12J and scales on the relevant variables were administered to 2200 employees registered with an online survey company. A follow-up survey with 1400 of the respondents was conducted two weeks later. Internal consistency and test–retest reliability were examined via Cronbach’s alpha and Cohen’s weighted kappa coefficients, respectively. Structural, convergent, and known-group validities were examined using confirmatory factor analysis (CFA) and item response theory (IRT) analysis, correlation analysis, and Kruskal–Wallis test, respectively. Cronbach’s alpha and Cohen’s weighted kappa coefficients were 0.97 and 0.53, respectively. CFA based on the four-factor structure established in the previous literature showed an acceptable model fit. IRT analysis showed that each item was an adequate measure of the respondent’s latent trait. Correlations of the PSC-12J with the relevant variables and distribution of scores by demographic characteristics were also observed in the theoretically expected directions, supporting the construct validity of the PSC-12J. Our findings establish the PSC-12J as a reliable and valid measure of the psychosocial safety climate construct in the Japanese context.

Development of an exercise programme for balance abilities in people with multiple sclerosis: a development of concept study using Rasch analysis

Background People with multiple sclerosis (PwMS) frequently have impaired balance from an early stage of the disease. Balance difficulties can be divided into categories; although, to date, these lack scientific foundation. Impaired balance in PwMS can be addressed using specific and challenging exercises. Such exercises should provide an optimal challenge point; however, the difficulty of balance exercises is often unknown, making it difficult to target the exercises to an individual’s abilities. The aims of this study were: to develop an exercise programme for PwMS relating the exercises to the balance problem categories; to establish the order of difficulty of exercises in each category and; to evaluate the content and structural validity of the exercise programme. Methods A “construct map” approach was used to design and develop an exercise programme for PwMS. Potentially relevant balance exercises were identified, then a framework was set up, comprising four dimensions (subsequently reduced to three dimensions) of balance exercises. The relevance, comprehensibility, and comprehensiveness of the exercise programme were rated by 13 physiotherapists, who also linked 19 key exercises to balance categories. A total of 65 PwMS performed the 19 balance exercises, rated their difficulty and commented on the relevance and comprehensibility of each exercise. A Rasch model was used to evaluate the relative difficulty of the exercises. To assess fit of the data to the Rasch model a rating scale model was used, which is a unidimensional latent trait model for polytomous item responses. Results Evaluation by the physiotherapists and PwMS indicated that the content validity of the exercise programme was adequate. Rasch analysis showed that the latent trait “balance exercises in PwMS” comprised three subdimensions (“stable BOS”, “sway” and “step and walk”). The 19 balance exercises showed adequate fit to the respective dimensions. The difficulties of the balance exercises were adequate to cover the ability spectrum of the PwMS. Conclusion A balance exercise programme for PwMS comprising three dimensions of balance exercises was developed. Difficulty estimates have been established for each of the exercises, which can be used for targeted balance training. Content and structural validity of the programme was adequate.

The Rasch Model Cannot Reveal Systematic Differential Item Functioning in Single Tests: Subset DIF Analysis as an Alternative Methodology

This article demonstrates that the Rasch model cannot reveal systematic differential item functioning (DIF) in single tests. The person total score is the sufficient statistic for the person parameter estimate, eliminating the possibility for residuals at the test level. An alternative approach is to use subset DIF analysis to search for DIF in item subsets that form the components of the broader latent trait. In this methodology, person parameter estimates are initially calculated using all test items. Then, in separate analyses, these person estimates are compared to the observed means in each subset, and the residuals assessed. As such, this methodology tests the assumption that the person locations in each factor group are invariant across subsets. The first objective is to demonstrate that in single tests differences in factor groups will appear as differences in the mean person estimates and the distributions of these estimates. The second objective is to demonstrate how subset DIF analysis reveals differences between person estimates and the observed means in subsets. Implications for practitioners are discussed.

Psychometric evaluation of an Italian custom 4-item short form of the PROMIS anxiety item bank in immune-mediated inflammatory diseases: an item response theory analysis

Background There has recently been growing interest in the roles of inflammation in contributing to the development of anxiety in people with immune-mediated inflammatory diseases (IMID). Patient-reported outcome measures can facilitate the assessment of physical and psychological functioning. The National Institutes of Health (NIH)’s Patient-Reported Outcomes Measurement Information System (PROMIS®) is a set of Patient-Reported Outcomes (PROs) that cover physical appearance, mental health, and social health. The PROMIS has been built through an Item Response Theory approach (IRT), a model-based measurement in which trait level estimates depend on both persons’ responses and on the properties of the items that were administered. The aim of this study is to test the psychometric properties of an Italian custom four-item Short Form of the PROMIS Anxiety item bank in a cohort of outpatients with IMIDs. Methods We selected four items from the Italian standard Short Form Anxiety 8a and administered them to consecutive outpatients affected by Inflammatory Bowel disease (n = 246), rheumatological (n = 100) and dermatological (n = 43) diseases, and healthy volunteers (n = 280). Data was analyzed through an Item Response Theory (IRT) analysis in order to evaluate the psychometric properties of the Italian adaptation of the PROMIS anxiety short form. Results Taken together, Confirmatory Factor Analysis and Exploratory Factor analysis suggest that the unidimensionality assumption of the instrument holds. The instrument has excellent reliability from a Classical Theory of Test (CTT) standpoint (Cronbach’s α = 0.93, McDonald’s ω = 0.92). The 2PL Graded Response Model (GRM) model provided showed a better goodness of fit as compared to the 1PL GRM model, and local independence assumption appears to be met overall. We did not find signs of differential item functioning (DIF) for age and gender, but evidence for uniform (but not non-uniform) DIF was found in three out of four items for the patient vs. control group. Analysis of the test reliability curve suggested that the instrument is most reliable for higher levels of the latent trait of anxiety. The groups of patients exhibited higher levels of anxiety as compared to the control group (ps < 0.001, Bonferroni-corrected). The groups of patients were not different between themselves (p = 1, Bonferroni-corrected). T-scores based on estimated latent trait and raw scores were highly correlated (Pearson’s r = 0.98) and led to similar results. Discussion The Italian custom four-item short form from the PROMIS anxiety form 8a shows acceptable psychometric properties both from a CTT and an IRT standpoint. The Test Reliability Curve shows that this instrument is mostly informative for people with higher levels of anxiety, making it particularly suitable for clinical populations such as IMID patients.

A Lognormal Ipsative Model for Multidimensional Compositional Items

Compositional items – a form of forced-choice items – require respondents to allocate a fixed total number of points to a set of statements. To describe the responses to these items, the Thurstonian item response theory (IRT) model was developed. Despite its prominence, the model requires that items composed of parts of statements result in a factor loading matrix with full rank. Without this requirement, the model cannot be identified, and the latent trait estimates would be seriously biased. Besides, the estimation of the Thurstonian IRT model often results in convergence problems. To address these issues, this study developed a new version of the Thurstonian IRT model for analyzing compositional items – the lognormal ipsative model (LIM) – that would be sufficient for tests using items with all statements positively phrased and with equal factor loadings. We developed an online value test following Schwartz’s values theory using compositional items and collected response data from a sample size of N = 512 participants with ages from 13 to 51 years. The results showed that our LIM had an acceptable fit to the data, and that the reliabilities exceeded 0.85. A simulation study resulted in good parameter recovery, high convergence rate, and the sufficient precision of estimation in the various conditions of covariance matrices between traits, test lengths and sample sizes. Overall, our results indicate that the proposed model can overcome the problems of the Thurstonian IRT model when all statements are positively phrased and factor loadings are similar.