Psychometric evaluation of the Relative Mastery Scale: An Occupational Adaptation instrument

This study evaluated the psychometric properties of the Relative Mastery Scale (RMS). Valid and reliable client-centered instruments support practice in value-based health care and community-based settings. Participants were 368 community-dwelling adults aged 18 to 95 years. Researchers conducted validity and reliability examinations of the RMS using classical test theory and Rasch measurement model. A partial credit model allowed exploration of individual scale properties. Spearman’s correlation coefficients between items were statistically significant at the .01 level. Cronbach’s alpha coefficient was .94 showing strong internal consistency. In exploratory factor analysis, Factor 1 accounted for 71% of variance with an eigenvalue of 4.26. In Rasch analysis, the 5-point rating scale demonstrated adequate functioning, confirmed unidimensionality, and person/item separation. The RMS instrument demonstrates sound psychometric characteristics. A valid and reliable measure of internal occupational adaptation supports application to monitor progress of internal occupational adaptation across a variety of individuals.

How are the Characteristics of Integrated Assessment Instruments to Measure Critical Thinking Skills and Scientific Attitudes of High School Students?

This research aimed to develop an integrated assessment instrument and to study the instrument’s characteristics. This research used 4-D development model of which the stages included defining, designing, developing, and disseminating. The primary product is validated by expert judgment. This research involved 274 mathematics and natural science eleventh-grade students of Senior High Schools in Sleman district, Yogyakarta. The polytomous data generated by the instrument trial was analyzed using the WINSTEPS 3.73 program and the Partial Credit Model 1-Parameter Logistic (PCM 1-PL) approach. Data collecting techniques included interviews, questionnaires, and tests. Data collecting instruments included interview guidelines, a question item validation sheet, and the integrated assessment instrument an essay test. The result of the research shows that the integrated assessment instrument has 0.93 of Aiken validation value. All tryout test items were stated to fit with the PCM 1-PL based on the criteria for the lowest and the highest limit outfit MNSQ 0.5 and 1.5. Item reliability was 0.96 and person reliability was 0.71. The difficulty level of question items was good, because of the range from -1.05 to +1.01 logit. Therefore, the integrated assessment instrument is suitable to be applied to measure the student’s critical thinking skills and scientific attitudes.

An R toolbox for score-based measurement invariance tests in IRT models

The detection of differential item functioning (DIF) is a central topic in psychometrics and educational measurement. In the past few years, a new family of score-based tests of measurement invariance has been proposed, which allows the detection of DIF along arbitrary person covariates in a variety of item response theory (IRT) models. This paper illustrates the application of these tests within the R system for statistical computing, making them accessible to a broad range of users. This presentation also includes IRT models for which these tests have not previously been investigated, such as the generalized partial credit model. The paper has three goals: First, we review the ideas behind score-based tests of measurement invariance. Second, we describe the implementation of these tests within the R system for statistical computing, which is based on the interaction of the R packages mirt, psychotools and strucchange. Third, we illustrate the application of this software and the interpretation of its output in two empirical datasets. The complete R code for reproducing our results is reported in the paper.

Healthcare Service Utilization Patterns and Patient Experience in Persons With High Healthcare Needs: A Comparison Across 22 Countries

BackgroundPersons with complex health conditions, e.g. spinal cord injury (SCI), frequently visit numerous clinical settings. Their service utilization and patient experience is a comprehensive indicator of how a system is functioning overall. This study compared the patient experience of persons with chronic SCI in relation to healthcare service utilization patterns in 22 countries, hypothesizing that primary-care oriented patterns would offer better experience.MethodsThis study was based on International Spinal Cord Injury Survey with 12 588 participants from 22 countries worldwide. Utilization clusters were identified by cluster analysis, experience score was attained by partial credit model. The association between the two was explored by regression analysis. ResultsHighest share of visits was to primary care physician (18%) and rehabilitation physician (16%). Utilization patterns had diverse orientation: from primary care to specialized and from inpatient to outpatient. The experience was reported as very good and good across different dimensions: 78% reported respectful treatment; 75% – clear explanations; 71% – involvement in decision making; 63% – satisfaction with care. Average experience score on 0-100-point scale was 64, highest – 74 (Brazil) and lowest – 52 (Japan, South Korea). Service utilization patterns were associated with patient experience, but no uniformly better patient experience was found for neither primary or specialized care-oriented systems. ConclusionWhile there are distinct utilization patterns between countries of how persons with chronic SCI use the healthcare system, neither the more primary care oriented nor the specialized care system leads to a uniformly better patient experience.

Are the Items of the Starkstein Apathy Scale Fit for the Purpose of Measuring Apathy Post-stroke?

Importance: Given the importance of apathy for stroke, we felt it was time to scrutinize the psychometric properties of the commonly used Starkstein Apathy Scale (SAS) for this purpose.Objectives: The objectives were to: (i) estimate the extent to which the SAS items fit a hierarchical continuum of the Rasch Model; and (ii) estimate the strength of the relationships between the Rasch analyzed SAS and converging constructs related to stroke outcomes.Methods: Data was from a clinical trial of a community-based intervention targeting participation. A total of 857 SAS questionnaires were completed by 238 people with stroke from up to 5 time points. SAS has 14 items, rated on a 4-point scale with higher values indicating more apathy. Psychometric properties were tested using Rasch partial-credit model, correlation, and regression. Items were rescored so higher scores are interpreted as lower apathy levels.Results: Rasch analysis indicated that the response options were disordered for 8/14 items, pointing to unreliability in the interpretation of the response options; they were consequently reduced from 4 to 3. Only 9/14 items fit the Rasch model and therefore suitable for creating a total score. The new rSAS was deemed unidimensional (residual correlations: < 0.3), reasonably reliable (person separation index: 0.74), with item-locations uniform across time, age, sex, and education. However, 30% of scores were > 2 SD above the standardized mean but only 2/9 items covered this range (construct mistargeting). Apathy (rSAS/SAS) was correlated weakly with anxiety/depression and uncorrelated with physical capacity. Regression showed that the effect of apathy on participation and health perception was similar for rSAS/SAS versions: R2 participation measures ranged from 0.11 to 0.29; R2 for health perception was ∼0.25. When placed on the same scale (0–42), rSAS value was 6.5 units lower than SAS value with minimal floor/ceiling effects. Estimated change over time was identical (0.12 units/month) which was not substantial (1.44 units/year) but greater than expected assuming no change (t: 3.6 and 2.4).Conclusion: The retained items of the rSAS targeted domains of behaviors more than beliefs and results support the rSAS as a robust measure of apathy in people with chronic stroke.

Psychometric Validation of the Spanish Gaming Disorder Test (GDT): Item Response Theory and Measurement Invariance Analysis

Gaming Disorder (GD) has been recently added to the eleventh revision of the International Classification of Diseases (ICD-11) by the World Health Organization (WHO), as such, psychometrically sound psychological measures are required to assess this disorder. The objective of this study was to investigate the psychometric properties of the Spanish Gaming Disorder Test (GDT) by assessing its dimensionality, reliability, convergent validity, and associations with other variables of importance through polytomous Item Response Theory (IRT) and Measurement Invariance (MI) analysis across genders. To achieve this, a sample of 538 gamers (42.94% female, meanage = 23.29 years, SD = 7.24) was recruited. The results obtained supported a one-factor structure for the Spanish GDT with adequate reliability and convergent validity. Furthermore, satisfactory goodness of fit in the partial credit model (PCM) with more precise scores at high trait levels to assess GD was found, and strict invariance across genders was supported. These findings attest to the suitability of the Spanish GDT for clinical assessment and research on disordered gaming beyond community samples.

Performance of Polytomous IRT Models With Rating Scale Data: An Investigation Over Sample Size, Instrument Length, and Missing Data

The implementation of polytomous item response theory (IRT) models such as the graded response model (GRM) and the generalized partial credit model (GPCM) to inform instrument design and validation has been increasing across social and educational contexts where rating scales are usually used. The performance of such models has not been fully investigated and compared across conditions with common survey-specific characteristics such as short test length, small sample size, and data missingness. The purpose of the current simulation study is to inform the literature and guide the implementation of GRM and GPCM under these conditions. For item parameter estimations, results suggest a sample size of at least 300 and/or an instrument length of at least five items for both models. The performance of GPCM is stable across instrument lengths while that of GRM improves notably as the instrument length increases. For person parameters, GRM reveals more accurate estimates when the proportion of missing data is small, whereas GPCM is favored in the presence of a large amount of missingness. Further, it is not recommended to compare GRM and GPCM based on test information. Relative model fit indices (AIC, BIC, LL) might not be powerful when the sample size is less than 300 and the length is less than 5. Synthesis of the patterns of the results, as well as recommendations for the implementation of polytomous IRT models, are presented and discussed.

The Norwegian PROMIS-29: psychometric validation in the general population for Norway

Background The Patient Reported Outcome Measurement Information System profile instruments include “high information” items drawn from large item banks following the application of modern psychometric criteria. The shortest adult profile, PROMIS-29, looks set to replace existing short-form instruments in research and clinical practice. The objective of this study was to undertake the first psychometric evaluation of the Norwegian PROMIS-29, following a postal survey of a random sample of 12,790 Norwegians identified through the National Registry of the Norwegian Tax Administration. Confirmatory factor analysis was used to assess structural validity. Fit to the Rasch partial credit model and differential item functioning (DIF) were assessed in relation to age, gender, and education. PROMIS-29 scores were compared to those for the EQ-5D-5L and the Self-assessed Comorbidity Questionnaire (SCQ), for purposes of assessing validity based on a priori hypotheses. Results There were 3200 (25.9%) respondents with a mean age (SD) of 51 (20.7, range 18 to 97 years) and 55% were female. The PROMIS-29 showed satisfactory structural validity and acceptable fit to Rasch model including unidimensionality, and measurement invariance across age and education levels. One pain interference item had uniform DIF for gender but splitting gave satisfactory fit. Domain reliability estimates ranged from 0.85 to 0.95. Correlations between PROMIS-29 domain, SCQ and EQ-5D scores were largely as expected, the largest being for scores assessing very similar aspects of health. Conclusions The Norwegian version of the PROMIS-29 is a reliable and valid generic self-reported measure of health in the Norwegian general population. The instrument is recommended for further application, but the analysis should be replicated and responsiveness to change assessed in future studies before it can be recommended for clinical and health services evaluation in Norway.

Assessment Instrument Development for a Chronological Thinking Skills with the Rasch Model

Chronological thinking skills are one of the most critical goals in learning history, distinguishing it from other sciences. This skill is not a natural skill that already exists in students but is a skill that needs to be developed in students, especially in studying history. This study aims to develop an instrument for assessing chronological thinking skills with the Rasch model. The development model adopted the Gall Borg development model by adjusting to the research objectives and needs. The adaptation of the Borg Gall model resulted in four stages in this study: (1) needs analysis and preliminary investigation, (2) planning and preparation of product development, (3) expert validation, and (4) instrument implementation. This research was conducted in Senior High School (SMA), involving 120 students from three schools. Sampling was carried out employing the proportional sampling technique. The data was collected using validation sheets, tests, and assessment sheets (scoring rubric). Data analysis was then performed with the Quest Program utilizing the Partial Credit Model (PCM). The results showed that the chronological thinking skills assessment instrument with the Rasch model was valid and reliable. The validity value of items that fit the model ranged from 0.77 to 1.30, and the instrument reliability value of 0.76 was in the high category. Therefore, it could be concluded that the assessment instrument developed can measure students' chronological thinking skills.