scholarly journals The Optimal Item Pool Design in Multistage Computerized Adaptive Tests With the p-Optimality Method

2020 ◽  
Vol 80 (5) ◽  
pp. 955-974
Author(s):  
Lihong Yang ◽  
Mark D. Reckase
Keyword(s):  
Exposure Control ◽  
Item Pool ◽  
Adaptive Tests ◽  
Computerized Adaptive Test ◽  
Computerized Adaptive Tests ◽  
Statistical Measures ◽  
Scoring Accuracy ◽  
Item Pools ◽  
Item Pool Design ◽  
The Rasch Model

The present study extended the p-optimality method to the multistage computerized adaptive test (MST) context in developing optimal item pools to support different MST panel designs under different test configurations. Using the Rasch model, simulated optimal item pools were generated with and without practical constraints of exposure control. A total number of 72 simulated optimal item pools were generated and evaluated by an overall sample and conditional sample using various statistical measures. Results showed that the optimal item pools built with the p-optimality method provide sufficient measurement accuracy under all simulated MST panel designs. Exposure control affected the item pool size, but not the item distributions and item pool characteristics. This study demonstrated that the p-optimality method can adapt to MST item pool design, facilitate the MST assembly process, and improve its scoring accuracy.

scholarly journals Quality of Item Pool (QIP) Index: A Novel Approach to Evaluating CAT Item Pool Adequacy

2019 ◽  
Vol 79 (6) ◽  
pp. 1133-1155
Author(s):  
Emre Gönülateş
Keyword(s):  
Simulation Studies ◽  
Item Pool ◽  
Adaptive Tests ◽  
Computerized Adaptive Test ◽  
Computerized Adaptive Tests ◽  
Adaptive Test ◽  
Test Specifications ◽  
Novel Approach ◽  
Item Pools

This article introduces the Quality of Item Pool (QIP) Index, a novel approach to quantifying the adequacy of an item pool of a computerized adaptive test for a given set of test specifications and examinee population. This index ranges from 0 to 1, with values close to 1 indicating the item pool presents optimum items to examinees throughout the test. This index can be used to compare different item pools or diagnose the deficiencies of a given item pool by quantifying the amount of deviation from a perfect item pool. Simulation studies were conducted to evaluate the capacity of this index for detecting the inadequacies of two simulated item pools. The value of this index was compared with the existing methods of evaluating the quality of computerized adaptive tests (CAT). Results of the study showed that the QIP Index can detect even slight deviations between a proposed item pool and an optimal item pool. It can also uncover shortcomings of an item pool that other outcomes of CAT cannot detect. CAT developers can use the QIP Index to diagnose the weaknesses of the item pool and as a guide for improving item pools.

Item Pool Development for the School-Aged Life Impact Burn Recovery Evaluation (SA- LIBRE5-12) Profile Computerized Adaptive Test (CAT): An Observer-Reported Outcome Assessment Measuring the Impact of Burn Injuries in School-Aged Children

2021 ◽  
Author(s):  
Silvanys L Rodríguez-Mercedes ◽  
Khushbu F Patel ◽  
Camerin A Rencken ◽  
Gabrielle G Grant ◽  
Kate Surette ◽  
...  
Keyword(s):  
Conceptual Framework ◽  
Critical Time ◽  
Cognitive Interviews ◽  
Item Pool ◽  
Computerized Adaptive Test ◽  
School Aged Children ◽  
Adaptive Test ◽  
Life Impact ◽  
The Impact ◽  
Item Pools

Abstract Introduction The transition from early childhood to teen years (5-12) is a critical time of development, which can be made particularly challenging by a burn injury. Assessing post-burn recovery during these years is important for improving pediatric survivors’ development and health outcomes. Few validated burn-specific measures exist for this age group. The purpose of this study was to generate item pools that will be used to create a future computerized adaptive test (CAT) assessing post-burn recovery in school-aged children. Methods Item pool development was guided by the previously developed School-Aged Life Impact Burn Recovery Evaluation (SA-LIBRE5-12) Conceptual Framework. The item pool development process involved a systematic literature review, extraction of candidate items from existing legacy measures, iterative item review during expert consensus meetings, and parent cognitive interviews. Results The iterative item review with experts consisted of six rounds. A total of 10 parent cognitive interviews were conducted. The three broad themes of concern were items that needed 1) clarification, needed context or were vague, 2) age dependence and relevance, and 3) word choice. The cognitive interviews indicated that survey instructions, recall period, item stem, and response choices were interpretable by respondents. Final item pool based on parental feedback consist of 57, 81, and 60 items in Physical, Psychological, and Family and Social Functioning respectively. Conclusion Developed item pools (n=198) in three domains are consistent with the existing conceptual framework. The next step involves field-testing the item pool and calibration using item response theory to develop and validate the SA-LIBRE5-12 CAT Profile.

Computerized Adaptive Testing: Tracking Candidate Response Patterns

1995 ◽  
Vol 13 (2) ◽  
pp. 151-162 ◽  
Author(s):  
Mary E. Lunz ◽  
Betty Bergstrom
Keyword(s):  
Computerized Adaptive Testing ◽  
Adaptive Testing ◽  
Response Patterns ◽  
Certification Examination ◽  
Adaptive Tests ◽  
Computerized Adaptive Test ◽  
Computerized Adaptive Tests ◽  
Ability Estimates ◽  
Adaptive Test ◽  
Candidate Performance

Computerized adaptive testing (CAT) uses a computer algorithm to construct and score the best possible individualized or tailored tests for each candidate. The computer also provides an absolute record of all responses and changes to responses, as well as their effects on candidate performance. The detail of the data from computerized adaptive tests makes it possible to track initial responses and response alterations, and their effect on candidate estimated ability measures, as well as the statistical performance of the examination. The purpose of this study was to track the effect of candidate response patterns on a computerized adaptive test. A ninety-item certification examination was divided into nine units of ten items each to track the pattern of initial responses and response alterations on ability estimates and test precision across the nine test units. The precision of the test was affected most by response alterations during early segments of the test. While generally, candidates benefit from altering responses, individual candidates showed different patterns of response alterations across test segments. Test precision is minimally affected, suggesting that the tailoring of CAT is minimally affected by response alterations.

scholarly journals Improving Measurement Quality and Efficiency with Adaptive Testing

1982 ◽  
Vol 6 (4) ◽  
pp. 473-492 ◽  
Author(s):  
David J. Weiss
Keyword(s):  
Item Response Theory ◽  
Item Response ◽  
Adaptive Testing ◽  
Simulation Studies ◽  
Item Pool ◽  
Adaptive Tests ◽  
Measurement Quality ◽  
Conventional Tests ◽  
Measurement Efficiency ◽  
Item Pools

Approaches to adaptive (tailored) testing based on item response theory are described and research results summarized. Through appropriate combinations of item pool design and use of different test termination criteria, adaptive tests can be designed (1) to improve both measurement quality and measurement efficiency, resulting in measurements of equal precision at all trait levels; (2) to improve measurement efficiency for test batteries using item pools designed for conventional test administration; and (3) to improve the accuracy and efficiency of testing for classification (e.g., mastery testing). Research results show that tests based on item response theory (IRT) can achieve measurements of equal precision at all trait levels, given an adequately designed item pool; these results contrast with those of conventional tests which require a tradeoff of bandwidth for fidelity/precision of measurements. Data also show reductions in bias, inaccuracy, and root mean square error of ability estimates. Improvements in test fidelity observed in simulation studies are supported by live-testing data, which showed adaptive tests requiring half the number of items as that of conventional tests to achieve equal levels of reliability, and almost one-third the number to achieve equal levels of validity. When used with item pools from conventional tests, both simulation and live-testing results show reductions in test battery length from conventional tests, with no reductions in the quality of measurements. Adaptive tests designed for dichotomous classification also represent improvements over conventional tests designed for the same purpose. Simulation studies show reductions in test length and improvements in classification accuracy for adaptive vs. conventional tests; live-testing studies in which adaptive tests were compared with "optimal" conventional tests support these findings. Thus, the research data show that IRT-based adaptive testing takes advantage of the capabilities of IRT to improve the quality and/or efficiency of measurement for each examinee.

scholarly journals Measurement Precision and Efficiency of Computerized Adaptive Testing for the Activities-specific Balance Confidence Scale in People With Stroke

2021 ◽  
Author(s):  
Bryant A Seamon ◽  
Steven A Kautz ◽  
Craig A Velozo
Keyword(s):  
Rasch Model ◽  
Computerized Adaptive Testing ◽  
Adaptive Testing ◽  
Measurement Precision ◽  
Strongly Correlated ◽  
Computerized Adaptive Test ◽  
Balance Confidence ◽  
Adaptive Test ◽  
The Rasch Model ◽  
Confidence Scale

Abstract Objective Administrative burden often prevents clinical assessment of balance confidence in people with stroke. A computerized adaptive test (CAT) version of the Activities-specific Balance Confidence Scale (ABC CAT) can dramatically reduce this burden. The objective of this study was to test balance confidence measurement precision and efficiency in people with stroke with an ABC CAT. Methods We conducted a retrospective cross-sectional simulation study with data from 406 adults approximately 2-months post-stroke in the Locomotor-Experience Applied Post-Stroke (LEAPS) trial. Item parameters for CAT calibration were estimated with the Rasch model using a random sample of participants (n = 203). Computer simulation was used with response data from remaining 203 participants to evaluate the ABC CAT algorithm under varying stopping criteria. We compared estimated levels of balance confidence from each simulation to actual levels predicted from the Rasch model (Pearson correlations and mean standard error (SE)). Results Results from simulations with number of items as a stopping criterion strongly correlated with actual ABC scores (full item, r = 1, 12-item, r = 0.994; 8-item, r = 0.98; 4-item, r = 0.929). Mean SE increased with decreasing number of items administered (full item, SE = 0.31; 12-item, SE = 0.33; 8-item, SE = 0.38; 4-item, SE = 0.49). A precision-based stopping rule (mean SE = 0.5) also strongly correlated with actual ABC scores (r = .941) and optimized the relationship between number of items administrated with precision (mean number of items 4.37, range [4–9]). Conclusions An ABC CAT can determine accurate and precise measures of balance confidence in people with stroke with as few as 4 items. Individuals with lower balance confidence may require a greater number of items (up to 9) and attributed to the LEAPS trial excluding more functionally impaired persons. Impact Statement Computerized adaptive testing can drastically reduce the ABC’s test administration time while maintaining accuracy and precision. This should greatly enhance clinical utility, facilitating adoption of clinical practice guidelines in stroke rehabilitation. Lay Summary If you have had a stroke, your physical therapist will likely test your balance confidence. A computerized adaptive test version of the ABC scale can accurately identify balance with as few as 4 questions, which takes much less time.

Sign in / Sign up

Export Citation Format

Share Document