The promise of computer adaptive testing in collection of orthopaedic outcomes: an evaluation of PROMIS utilization

Background A crucial component to improving patient care is better clinician understanding of patients’ health-related quality of life (HRQoL). In orthopaedic surgery, HRQoL assessment instruments such as the NIH developed Patient Reported Outcomes Measurement Information System (PROMIS), provide surgeons with a framework to assess how a treatment or medical condition is affecting each patient’s HRQoL. PROMIS has been demonstrated as a valuable instrument in many diseases; however, the extent to which orthopaedic surgery subspecialties have used and validated PROMIS measures in peer-reviewed research is unclear. Methods Systematic scoping methodology was used to investigate the characteristics of studies using PROMIS to assess HRQoL measures as orthopaedic surgical outcomes as well as studies validating computerized adaptive test (CAT) PROMIS physical health (PH) domains including: Physical Function (PF), Upper Extremity (UE), Lower Extremity (LE). Results A systematic search of PubMed identified 391 publications utilizing PROMIS in orthopaedics; 153 (39%) were PROMIS PH CAT validation publications. One-hundred publications were in Hand and Upper Extremity, 69 in Spine, 44 in Adult Reconstruction, 43 in Foot and Ankle, 43 in Sports, 37 in Trauma, 31 in General orthopaedics, and 24 in Tumor. From 2011 through 2020 there was an upward trend in orthopaedic PROMIS publications each year (range, 1–153) and an increase in studies investigating or utilizing PROMIS PH CAT domains (range, 1–105). Eighty-five percent (n = 130) of orthopaedic surgery PROMIS PH CAT validation publications (n = 153) analyzed PF; 30% (n = 46) analyzed UE; 3% (n = 4) analyzed LE. Conclusions PROMIS utilization within orthopaedics as a whole has significantly increased within the past decade, particularly within PROMIS CAT domains. The existing literature reviewed in this scoping study demonstrates that PROMIS PH CAT domains (PF, UE, and LE) are reliable, responsive, and interpretable in most contexts of patient care throughout all orthopaedic surgery subspecialties. The expanded use of PROMIS CATs in orthopaedic surgery highlights the potential for improved quality of patient care. While challenges of integrating PROMIS into electronic medical records exist, expanded use of PROMIS CAT measurement instruments throughout orthopaedic surgery should be performed. Plain english summary In orthopaedic surgery, health-related quality of life tools such as the NIH developed Patient Reported Outcomes Measurement Information System (PROMIS), offer patients an opportunity to better understand their medical condition and be involved in their own care. Additionally, PROMIS provides surgeons with a framework to assess how a treatment or medical condition is affecting each patient’s functional status and quality of life. The efficacy of PROMIS has been demonstrated in many diseases; however, its application throughout orthopaedic care has yet to be depicted. This study sought to identify the extent to which all orthopaedic surgery subspecialties have used and validated PROMIS measures in peer-reviewed research in order to identify its potential as an applicable and valuable tool across specialties. We determined that PROMIS utilization has significantly increased within the past decade. The existing literature reviewed in this scoping study demonstrates that the PROMIS computerized adaptive test domains evaluating physical function status are reliable, responsive, and interpretable in most contexts of patient care throughout all orthopaedic surgery subspecialties. Based on these results, this study recommends the expanded and more uniform use of PROMIS computerized adaptive test measurement instruments in the clinical care of orthopaedic patients.

Item Pool Development for the School-Aged Life Impact Burn Recovery Evaluation (SA- LIBRE5-12) Profile Computerized Adaptive Test (CAT): An Observer-Reported Outcome Assessment Measuring the Impact of Burn Injuries in School-Aged Children

Introduction The transition from early childhood to teen years (5-12) is a critical time of development, which can be made particularly challenging by a burn injury. Assessing post-burn recovery during these years is important for improving pediatric survivors’ development and health outcomes. Few validated burn-specific measures exist for this age group. The purpose of this study was to generate item pools that will be used to create a future computerized adaptive test (CAT) assessing post-burn recovery in school-aged children. Methods Item pool development was guided by the previously developed School-Aged Life Impact Burn Recovery Evaluation (SA-LIBRE5-12) Conceptual Framework. The item pool development process involved a systematic literature review, extraction of candidate items from existing legacy measures, iterative item review during expert consensus meetings, and parent cognitive interviews. Results The iterative item review with experts consisted of six rounds. A total of 10 parent cognitive interviews were conducted. The three broad themes of concern were items that needed 1) clarification, needed context or were vague, 2) age dependence and relevance, and 3) word choice. The cognitive interviews indicated that survey instructions, recall period, item stem, and response choices were interpretable by respondents. Final item pool based on parental feedback consist of 57, 81, and 60 items in Physical, Psychological, and Family and Social Functioning respectively. Conclusion Developed item pools (n=198) in three domains are consistent with the existing conceptual framework. The next step involves field-testing the item pool and calibration using item response theory to develop and validate the SA-LIBRE5-12 CAT Profile.

Item Selection on The Moodle-Based Computerized Adaptive Test

In this industrial era, all areas of life have been entered. There are five central issues that support performance, namely numerical physical devices, production tools, programs, interfaces, and networks. The Modular Object-Oriented Dynamic Learning Environment (Moodle) in order to support Industrial technology has been equipped with adaptive test facilities. Adaptive Moodle can be used to organize a communicative and interactive test process because of the communication features (chat, messaging, or forum). In addition, Adaptive Moodle can be used to administer online tests. Adaptive tests are tests whose quiz presentations will adjust to the user’s abilities. The results of the research that has been carried out regarding the selection of items on the Moodle-based computerized adaptive test (CAT) were obtained: (1) The Moodle adaptive test worked successfully in accordance with the research objectives, (2) Based on user responses, the selection of items on the Moodle adaptive test items had worked well for the exam, (3) The Moodle adaptive test can run according to its function, namely adaptive to the user’s ability.

Applicability and Efficiency of a Computerized Adaptive Test for the Washington Assessment of the Risks and Needs of Students

The Washington Assessment of the Risks and Needs of Students (WARNS) is a computer-based assessment created to help courts, schools, and youth service providers determine an adolescent’s risks and needs that may lead to truancy, drop out, or delinquency from school. Users are advised to consider the WARNS total score to work with youth. A total score estimate based on fewer items than the full item set may result in less respondent burden, administration time, and fatigue, while not hindering accurate decisions. This simulation study examined the applicability and efficiency of a computerized adaptive test (CAT) to estimate a WARNS total score under a unidimensional item response theory model. The results demonstrate that the CAT provides an accurate estimate of students’ risks and needs and reduces the number of items administered for each examinee compared with the existing version. Future directions and limitations of CAT development with the WARNS are discussed.

Developing and Piloting a Computerized Adaptive Test for a Culturally Appropriate Measure of Adaptive Behavior

Culturally bounded in nature, adaptive behavior is the degree to which a person meets the requirements of personal independence and social responsibilities. This study aimed to develop a computerized adaptive test (CAT) of a culturally appropriate adaptive behavior measure (i.e., the Activities and Participation Rating Scale [APRS]) in the Singapore context for children aged 7 to 12 years, that is, the CAT-APRS-Primary. In Study 1, an item bank consisting of 310 adaptive skills was developed and piloted among 56 caregivers. In Study 2, the refined 310 items were calibrated using Rasch analysis among 352 caregivers. After the removal of five poor items based on the Rasch analysis, the finalized 305 items were used for the CAT development, with these items showing good Rasch person and item fit statistics. Preliminary utility of the CAT-APRS-Primary was established, with participants completing 24 items on average. Limitations and implications for future research were discussed.

Using a Response Time–Based Expected A Posteriori Estimator to Control for Differential Speededness in Computerized Adaptive Test

This study investigates using response times (RTs) with item responses in a computerized adaptive test (CAT) setting to enhance item selection and ability estimation and control for differential speededness. Using van der Linden’s hierarchical framework, an extended procedure for joint estimation of ability and speed parameters for use in CAT is developed following van der Linden; this is called the joint expected a posteriori estimator (J-EAP). It is shown that the J-EAP estimate of ability and speededness outperforms the standard maximum likelihood estimator (MLE) of ability and speededness in terms of correlation, root mean square error, and bias. It is further shown that under the maximum information per time unit item selection method (MICT)—a method which uses estimates for ability and speededness directly—using the J-EAP further reduces average examinee time spent and variability in test times between examinees above the resulting gains of this selection algorithm with the MLE while maintaining estimation efficiency. Simulated test results are further corroborated with test parameters derived from a real data example.

Development of a Computerized Adaptive Test for Quantifying Chinese Medicine Syndrome of Myasthenia Gravis on Basis of Multidimensional Item Response Theory

Background. Making comprehensive management of myasthenia gravis (MG) is a challenge in clinical practice due to heterogeneity and multiple comorbidities among patients. Aim. To develop an end-to-end instrument for individualized assessment of MG in the perspective of Chinese medicine (TCM) with the application of multidisciplinary quantification approaches. Methods. A self-administrated questionnaire was developed integrating typical symptoms of MG and spleen-kidney deficiency syndrome on basis of the conceptual framework of TCM. With data collected in a multicenter cross-sectional study, confirmatory factor analysis together with multidimensional item response theory (MIRT) was used for evaluating the psychometric property of the questionnaire. A computerized adaptive test was developed based on the MIRT model, and scores of syndrome factors were calculated in simulation. A logistics regression model was also estimated for evaluating the consistency between the quantitative result and the clinical diagnosis of syndrome from clinical practitioners. Result. With 337 patients enrolled and assessed, the 14-item questionnaire was evaluated to be with adequate validity and reliability (Cronbach’s alpha indices = 0.87, AIC = 195.827, BIC = 348.631, CFI = 0.921, RMR = 0.006, GFI = 0.954, RMSEA = 0.048, and χ2/df = 1.782). With adequate factor loadings of symptoms on related syndrome factor, the instrument was evaluated with preliminary interpretation and was suitable for evaluating patients with moderate severity of the spleen and kidney deficiency syndrome. Conclusion. Setting typical symptoms of MG together with systemic discomforts in a computerized adaptive test on the basis of MIRT, this study proposed an innovative research paradigm for quantifying individual condition in the perspective of TCM with application of interdisciplinary approaches.