A Motivational-Developmental Free Response Assessment Through a Bifactor Lens

The Motivational-Developmental Assessment (MDA) measures a university student’s motivational and developmental attributes by utilizing overlapping constructs measured across four writing prompts. The MDA’s format may lead to the violation of the local item independence (LII) assumption for unidimensional item response theory (IRT) scoring models, or the uncorrelated errors assumption for scoring models in classical test theory (CTT) due to the measurement of overlapping constructs within a prompt. This assumption violation is known as a testlet effect, which can be viewed as a method effect. The application of a unidimensional IRT or CTT model to score the MDA can result in imprecise parameter estimates when this effect is ignored. To control for this effect in the MDA responses, we first examined the presence of local dependence via a restricted bifactor model and Yen’s Q3 statistic. Second, we applied bifactor models to account for the testlet effect in the responses, as this effect is modeled as an additional latent variable in a factor model. Results support the presence of local dependence in two of the four MDA prompts, and the use of the restricted bifactor model to account for the testlet effect in the responses. Modeling the testlet effect through the restricted bifactor model supports a scoring inference in a validation argument framework. Implications are discussed.

Evaluation of the psychometric properties of the burn specific health scale-brief: A National Institute on Disability, Independent Living, and Rehabilitation Research Burn Model System Study

Objective The Burn Specific Health Scale-Brief (BSHS-B) is a commonly used burn specific health outcome measure that includes 40 items across nine subscales. The objective of this study was to use both classical and modern psychometric methods to evaluate psychometric properties of the BSHS-B. Methods Data were collected post burn injury by a multisite federally funded project tracking long term outcomes. We examined dimensionality, local dependence, item fit, and functioning of response categories, homogeneity, and floor and ceiling effects. Items were fit to Item Response Theory models for evaluation. Results A total of 653 adults with burn injury completed the BSHS-B. Factor analyses supported unidimensionality for all subscales, but not for a total score based on all 40 items. All nine of the subscales had significant ceiling effects. Six item pairs displayed local dependence suggesting redundance and 11 items did not fit the Item Response Theory models. At least 15 items have too many response options. Conclusions Results identified numerous psychometric issues with the BSHS-B. A single summary score should never be used for any purpose. Psychometric properties of the scale need to be improved by removing redundant items, reducing response categories and modifying or deleting problematic items. Additional conceptual work is needed to, at a minimum, revise the work subscale and optimally to revisit and clearly define the constructs measured by all the subscales. Additional items are needed to address ceiling effects.

Development and Validation of the Perceived Stress Scale in Emergency Medical Teams During the Epidemic of COVID-19

BackgroundDuring the epidemic of COVID-19 of China, the emergency medical teams are facing serious stress in the front-line. As far as we know, there are no studies to test the applicability and measurement properties of the 10-item Chinese perceived stress scale (CPSS-10) in the emergency medical team.MethodsFrom March 17 to 27, 2020, an online survey was conducted on the emergency medical teams of Liaoning Province who supporting Wuhan. The CPSS-10 was used to measure the stress of medical workers. Classical test theory (CTT), bifactor model and multidimensional graded response model (MGRM) were used to analyze the measurement characteristics and differential item functioning (DIF) of CPSS-10.ResultsThe Cronbach's alpha coefficient of CPSS-10 was 0.86. Bifactor model confirmed that CPSS-10 was a two-factor structure. MGRM showed ordered response categories of K10. Item 8 could distinguish individual stress, but the slope of this item was very large (slope is 7.97, which was higher than 4), showing local dependence. There was a significant age DIF, but no DIF in gender. After removing the items 2, 5, and 8, the CPSS-7 showed high reliability, without DIF of age and gender, and there was no local dependence.ConclusionsMGRM could provide useful measurement information about CPSS-10 and CPSS-7. MGRM found that CPSS-10 did not fully conform to the item response theory (IRT). CPSS-7 had proved to be a more effective and reliable tool for assessing the perceived stress of emergency medical team.

Use of the Lagrange Multiplier Test for Assessing Measurement Invariance Under Model Misspecification

This article studies the Type I error, false positive rates, and power of four versions of the Lagrange multiplier test to detect measurement noninvariance in item response theory (IRT) models for binary data under model misspecification. The tests considered are the Lagrange multiplier test computed with the Hessian and cross-product approach, the generalized Lagrange multiplier test and the generalized jackknife score test. The two model misspecifications are those of local dependence among items and nonnormal distribution of the latent variable. The power of the tests is computed in two ways, empirically through Monte Carlo simulation methods and asymptotically, using the asymptotic distribution of each test under the alternative hypothesis. The performance of these tests is evaluated by means of a simulation study. The results highlight that, under mild model misspecification, all tests have good performance while, under strong model misspecification, the tests performance deteriorates, especially for false positive rates under local dependence and power for small sample size under misspecification of the latent variable distribution. In general, the Lagrange multiplier test computed with the Hessian approach and the generalized Lagrange multiplier test have better performance in terms of false positive rates while the Lagrange multiplier test computed with the cross-product approach has the highest power for small sample sizes. The asymptotic power turns out to be a good alternative to the classic empirical power because it is less time consuming. The Lagrange tests studied here have been also applied to a real data set.