Structural validity of the Arabic version of the disabilities of the Arm, Shoulder and Hand (DASH) using Rasch measurement model

Background The disabilities of the arm, shoulder and hand (DASH) is a commonly used region-specific patient-reported outcome measure (PROM) that quantify upper extremity function (activity limitation) and symptoms. Current evidence suggests that measurement properties of the adapted versions of the DASH are not sufficiently examined. The Arabic DASH has evidence supporting its internal consistency, test–retest reliability, construct validity and responsiveness. On the other hand, the validity of the assumed unidimensionality of the Arabic DASH has not been examined previously. The aim of this study was to examine the structural validity of the Arabic DASH in patients with upper extremity musculoskeletal disorders using Rasch measurement model. Methods Patients with upper extremity musculoskeletal disorders were recruited and were asked to complete the Arabic DASH at their initial visit to physical therapy departments. The overall fit of the Arabic DASH to the requirement of the Rasch measurement model was examined using chi-square statistics for item-trait interaction, mean item and person fit residuals. The fit of individual items, thresholds ordering, local dependency, differential item functioning (DIF), and unidimensionality using the t-test approach were also examined. Results The Arabic DASH did not fit the Rasch measurement model initially (χ2 = 179.04, p < 0.001) with major breach of local item independence and a pattern of high residual correlations among the activity-related items and among the impairment-related items. Combining items into activity-limitation and impairment testlets accommodated the local dependency and led to satisfactory fit of the Arabic DASH to the requirement of the Rasch measurement model (χ2 = 3.99, p = 0.41). Conclusions Rasch measurement model supports the structural validity of the Arabic DASH as a unidimensional measure after the accommodation of local dependency.

Rasch analysis for development and reduction of Symptom Questionnaire for Visual Dysfunctions (SQVD)

To develop the Symptom Questionnaire for Visual Dysfunctions (SQVD) and to perform a psychometric analysis using Rasch method to obtain an instrument which allows to detect the presence and frequency of visual symptoms related to any visual dysfunction. A pilot version of 33 items was carried out on a sample of 125 patients from an optometric clinic. Rasch model (using Andrich Rating Scale Model) was applied to investigate the category probability curves and Andrich thresholds, infit and outfit mean square, local dependency using Yen’s Q3 statistic, Differential item functioning (DIF) for gender and presbyopia, person and item reliability, unidimensionality, targeting and ordinal to interval conversion table. Category probability curves suggested to collapse a response category. Rasch analysis reduced the questionnaire from 33 to 14 items. The final SQVD showed that 14 items fit to the model without local dependency and no significant DIF for gender and presbyopia. Person reliability was satisfactory (0.81). The first contrast of the residual was 1.908 eigenvalue, showing unidimensionality and targeting was − 1.59 logits. In general, the SQVD is a well-structured tool which shows that data adequately fit the Rasch model, with adequate psychometric properties, making it a reliable and valid instrument to measure visual symptoms.

Developing and validating a school-based screening tool of Fundamental Movement Skills (FUNMOVES) using Rasch analysis

Background A large proportion of children are not able to perform age-appropriate fundamental movement skills (FMS). Thus, it is important to assess FMS so that children needing additional support can be identified in a timely fashion. There is great potential for universal screening of FMS in schools, but research has established that current assessment tools are not fit for purpose. Objective To develop and validate the psychometric properties of a FMS assessment tool designed specifically to meet the demands of universal screening in schools. Methods A working group consisting of academics from developmental psychology, public health and behavioural epidemiology developed an assessment tool (FUNMOVES) based on theory and prior evidence. Over three studies, 814 children aged 4 to 11 years were assessed in school using FUNMOVES. Rasch analysis was used to evaluate structural validity and modifications were then made to FUNMOVES activities after each study based on Rasch results and implementation fidelity. Results The initial Rasch analysis found numerous psychometric problems including multidimensionality, disordered thresholds, local dependency, and misfitting items. Study 2 showed a unidimensional measure, with acceptable internal consistency and no local dependency, but that did not fit the Rasch model. Performance on a jumping task was misfitting, and there were issues with disordered thresholds (for jumping, hopping and balance tasks). Study 3 revealed a unidimensional assessment tool with good fit to the Rasch model, and no further issues, once jumping and hopping scoring were modified. Implications The finalised version of FUNMOVES (after three iterations) meets standards for accurate measurement, is free and able to assess a whole class in under an hour using resources available in schools. Thus FUNMOVES has the potential to allow schools to efficiently screen FMS to ensure that targeted support can be provided and disability barriers removed.

Taking MT Evaluation Metrics to Extremes: Beyond Correlation with Human Judgments

Automatic Machine Translation (MT) evaluation is an active field of research, with a handful of new metrics devised every year. Evaluation metrics are generally benchmarked against manual assessment of translation quality, with performance measured in terms of overall correlation with human scores. Much work has been dedicated to the improvement of evaluation metrics to achieve a higher correlation with human judgments. However, little insight has been provided regarding the weaknesses and strengths of existing approaches and their behavior in different settings. In this work we conduct a broad meta-evaluation study of the performance of a wide range of evaluation metrics focusing on three major aspects. First, we analyze the performance of the metrics when faced with different levels of translation quality, proposing a local dependency measure as an alternative to the standard, global correlation coefficient. We show that metric performance varies significantly across different levels of MT quality: Metrics perform poorly when faced with low-quality translations and are not able to capture nuanced quality distinctions. Interestingly, we show that evaluating low-quality translations is also more challenging for humans. Second, we show that metrics are more reliable when evaluating neural MT than the traditional statistical MT systems. Finally, we show that the difference in the evaluation accuracy for different metrics is maintained even if the gold standard scores are based on different criteria.