Quality of Item Pool (QIP) Index: A Novel Approach to Evaluating CAT Item Pool Adequacy

This article introduces the Quality of Item Pool (QIP) Index, a novel approach to quantifying the adequacy of an item pool of a computerized adaptive test for a given set of test specifications and examinee population. This index ranges from 0 to 1, with values close to 1 indicating the item pool presents optimum items to examinees throughout the test. This index can be used to compare different item pools or diagnose the deficiencies of a given item pool by quantifying the amount of deviation from a perfect item pool. Simulation studies were conducted to evaluate the capacity of this index for detecting the inadequacies of two simulated item pools. The value of this index was compared with the existing methods of evaluating the quality of computerized adaptive tests (CAT). Results of the study showed that the QIP Index can detect even slight deviations between a proposed item pool and an optimal item pool. It can also uncover shortcomings of an item pool that other outcomes of CAT cannot detect. CAT developers can use the QIP Index to diagnose the weaknesses of the item pool and as a guide for improving item pools.

Download Full-text

The Optimal Item Pool Design in Multistage Computerized Adaptive Tests With the p-Optimality Method

Educational and Psychological Measurement ◽

10.1177/0013164419901292 ◽

2020 ◽

Vol 80 (5) ◽

pp. 955-974

Author(s):

Lihong Yang ◽

Mark D. Reckase

Keyword(s):

Exposure Control ◽

Item Pool ◽

Adaptive Tests ◽

Computerized Adaptive Test ◽

Computerized Adaptive Tests ◽

Statistical Measures ◽

Scoring Accuracy ◽

Item Pools ◽

Item Pool Design ◽

The Rasch Model

The present study extended the p-optimality method to the multistage computerized adaptive test (MST) context in developing optimal item pools to support different MST panel designs under different test configurations. Using the Rasch model, simulated optimal item pools were generated with and without practical constraints of exposure control. A total number of 72 simulated optimal item pools were generated and evaluated by an overall sample and conditional sample using various statistical measures. Results showed that the optimal item pools built with the p-optimality method provide sufficient measurement accuracy under all simulated MST panel designs. Exposure control affected the item pool size, but not the item distributions and item pool characteristics. This study demonstrated that the p-optimality method can adapt to MST item pool design, facilitate the MST assembly process, and improve its scoring accuracy.

Download Full-text

Item Pool Development for the School-Aged Life Impact Burn Recovery Evaluation (SA- LIBRE5-12) Profile Computerized Adaptive Test (CAT): An Observer-Reported Outcome Assessment Measuring the Impact of Burn Injuries in School-Aged Children

Journal of Burn Care & Research ◽

10.1093/jbcr/irab247 ◽

2021 ◽

Author(s):

Silvanys L Rodríguez-Mercedes ◽

Khushbu F Patel ◽

Camerin A Rencken ◽

Gabrielle G Grant ◽

Kate Surette ◽

...

Keyword(s):

Conceptual Framework ◽

Critical Time ◽

Cognitive Interviews ◽

Item Pool ◽

Computerized Adaptive Test ◽

School Aged Children ◽

Adaptive Test ◽

Life Impact ◽

The Impact ◽

Item Pools

Abstract Introduction The transition from early childhood to teen years (5-12) is a critical time of development, which can be made particularly challenging by a burn injury. Assessing post-burn recovery during these years is important for improving pediatric survivors’ development and health outcomes. Few validated burn-specific measures exist for this age group. The purpose of this study was to generate item pools that will be used to create a future computerized adaptive test (CAT) assessing post-burn recovery in school-aged children. Methods Item pool development was guided by the previously developed School-Aged Life Impact Burn Recovery Evaluation (SA-LIBRE5-12) Conceptual Framework. The item pool development process involved a systematic literature review, extraction of candidate items from existing legacy measures, iterative item review during expert consensus meetings, and parent cognitive interviews. Results The iterative item review with experts consisted of six rounds. A total of 10 parent cognitive interviews were conducted. The three broad themes of concern were items that needed 1) clarification, needed context or were vague, 2) age dependence and relevance, and 3) word choice. The cognitive interviews indicated that survey instructions, recall period, item stem, and response choices were interpretable by respondents. Final item pool based on parental feedback consist of 57, 81, and 60 items in Physical, Psychological, and Family and Social Functioning respectively. Conclusion Developed item pools (n=198) in three domains are consistent with the existing conceptual framework. The next step involves field-testing the item pool and calibration using item response theory to develop and validate the SA-LIBRE5-12 CAT Profile.

Download Full-text

Computerized Adaptive Testing: Tracking Candidate Response Patterns

Journal of Educational Computing Research ◽

10.2190/gnb7-5hq0-gd8m-p0b2 ◽

1995 ◽

Vol 13 (2) ◽

pp. 151-162 ◽

Cited By ~ 2

Author(s):

Mary E. Lunz ◽

Betty Bergstrom

Keyword(s):

Computerized Adaptive Testing ◽

Adaptive Testing ◽

Response Patterns ◽

Certification Examination ◽

Adaptive Tests ◽

Computerized Adaptive Test ◽

Computerized Adaptive Tests ◽

Ability Estimates ◽

Adaptive Test ◽

Candidate Performance

Computerized adaptive testing (CAT) uses a computer algorithm to construct and score the best possible individualized or tailored tests for each candidate. The computer also provides an absolute record of all responses and changes to responses, as well as their effects on candidate performance. The detail of the data from computerized adaptive tests makes it possible to track initial responses and response alterations, and their effect on candidate estimated ability measures, as well as the statistical performance of the examination. The purpose of this study was to track the effect of candidate response patterns on a computerized adaptive test. A ninety-item certification examination was divided into nine units of ten items each to track the pattern of initial responses and response alterations on ability estimates and test precision across the nine test units. The precision of the test was affected most by response alterations during early segments of the test. While generally, candidates benefit from altering responses, individual candidates showed different patterns of response alterations across test segments. Test precision is minimally affected, suggesting that the tailoring of CAT is minimally affected by response alterations.

Download Full-text

Improving Measurement Quality and Efficiency with Adaptive Testing

Applied Psychological Measurement ◽

10.1177/014662168200600408 ◽

1982 ◽

Vol 6 (4) ◽

pp. 473-492 ◽

Cited By ~ 203

Author(s):

David J. Weiss

Keyword(s):

Item Response Theory ◽

Item Response ◽

Adaptive Testing ◽

Simulation Studies ◽

Item Pool ◽

Adaptive Tests ◽

Measurement Quality ◽

Conventional Tests ◽

Measurement Efficiency ◽

Item Pools

Approaches to adaptive (tailored) testing based on item response theory are described and research results summarized. Through appropriate combinations of item pool design and use of different test termination criteria, adaptive tests can be designed (1) to improve both measurement quality and measurement efficiency, resulting in measurements of equal precision at all trait levels; (2) to improve measurement efficiency for test batteries using item pools designed for conventional test administration; and (3) to improve the accuracy and efficiency of testing for classification (e.g., mastery testing). Research results show that tests based on item response theory (IRT) can achieve measurements of equal precision at all trait levels, given an adequately designed item pool; these results contrast with those of conventional tests which require a tradeoff of bandwidth for fidelity/precision of measurements. Data also show reductions in bias, inaccuracy, and root mean square error of ability estimates. Improvements in test fidelity observed in simulation studies are supported by live-testing data, which showed adaptive tests requiring half the number of items as that of conventional tests to achieve equal levels of reliability, and almost one-third the number to achieve equal levels of validity. When used with item pools from conventional tests, both simulation and live-testing results show reductions in test battery length from conventional tests, with no reductions in the quality of measurements. Adaptive tests designed for dichotomous classification also represent improvements over conventional tests designed for the same purpose. Simulation studies show reductions in test length and improvements in classification accuracy for adaptive vs. conventional tests; live-testing studies in which adaptive tests were compared with "optimal" conventional tests support these findings. Thus, the research data show that IRT-based adaptive testing takes advantage of the capabilities of IRT to improve the quality and/or efficiency of measurement for each examinee.

Download Full-text

EORTC Quality of Life Questionnaire--Computerized Adaptive Test

PsycTESTS Dataset ◽

10.1037/t42356-000 ◽

2013 ◽

Author(s):

Morten Aa. Petersen ◽

Johannes M. Giesinger ◽

Bernhard Holzner ◽

Juan I. Arraras ◽

Thierry Conroy ◽

...

Keyword(s):

Quality Of Life ◽

Life Questionnaire ◽

Computerized Adaptive Test ◽

Quality Of Life Questionnaire ◽

Adaptive Test ◽

Eortc Quality

Download Full-text

Optimizing measurement of vision-related quality of life: a computerized adaptive test for the impact of vision impairment questionnaire (IVI-CAT)

Quality of Life Research ◽

10.1007/s11136-019-02354-y ◽

2019 ◽

Vol 29 (3) ◽

pp. 765-774 ◽

Cited By ~ 1

Author(s):

Eva K. Fenwick ◽

Bao Sheng Loe ◽

Jyoti Khadka ◽

Ryan E. K. Man ◽

Gwyn Rees ◽

...

Keyword(s):

Quality Of Life ◽

Vision Impairment ◽

Computerized Adaptive Test ◽

Adaptive Test ◽

Related Quality ◽

The Impact

Download Full-text

Schizophrenia Quality of Life Questionnaire—Multidimensional Computerized Adaptive Test

PsycTESTS Dataset ◽

10.1037/t72490-000 ◽

2018 ◽

Author(s):

Pierre Michel ◽

Karine Baumstarck ◽

Christophe Lancon ◽

Badih Ghattas ◽

Anderson Loundou ◽

...

Keyword(s):

Quality Of Life ◽

Life Questionnaire ◽

Computerized Adaptive Test ◽

Quality Of Life Questionnaire ◽

Adaptive Test

Download Full-text

Dynamic adaptive audio training tests as a means of monitoring bilingual education

Informatics and Education ◽

10.32517/0234-0453-2019-34-10-46-54 ◽

2019 ◽

pp. 46-54

Author(s):

M. V. Noskov ◽

I. P. Peregudova ◽

P. P. Dyachuk ◽

O. I. Denisenko

Keyword(s):

Foreign Language ◽

Bilingual Education ◽

Language Education ◽

Foreign Language Education ◽

Foreign Languages ◽

Educational Activities ◽

Adaptive Tests ◽

Evaluative Feedback ◽

Adaptive Test

The article discusses the conditions for the use of computerized dynamic adaptive tests to provide developing training and monitoring in the field of foreign language education. Dynamic adaptive test simulators used as a tool for personalizing and monitoring foreign language education are still insufficiently represented in the educational practice of the Russian Federation. The authors of the article offer recommendations on the use of dynamic adaptive audio test simulators for monitoring foreign language education. As indicators of monitoring, the parameters characterizing the learning ability of students are considered: feedback coefficient; the complexity and time pace of learning activities of students of a foreign language.The purpose of the article is to present authors’ recommendations on the implementation of dynamic adaptive audio test simulators based on evaluative feedback in the field of foreign language education, providing for monitoring the quality of bilingual education based on the integration of individualized learning and dynamic adaptive testing of educational activities.The research methodology is an analysis of the use of existing dynamic adaptive test simulators in the field of foreign language education; studying the results of interdisciplinary studies of domestic and foreign scientists on the creation and use of dynamic adaptive tests for the learning process as a whole and the creation of dynamic adaptive tests for foreign education in particular; analysis and generalization of the author’s experience of using dynamic adaptive audio tests simulators as a tool for monitoring educational activities in the field of foreign language education.Results of the reseach are: authors developed computerized dynamic adaptive audio test simulators based on evaluative feedback in the field of foreign language education; indicators of monitoring the learning process in the field of foreign language education are identified, including the total feedback coefficient, actiograms, time pace and laboriousness; testing was carried out and recommendations were given for the use of dynamic adaptive audio test simulators for monitoring educational activities in the field of foreign language education.Analyzing the results of testing dynamic adaptive audio test simulators in a foreign language, the authors conclude that their use in practice implements personalized dynamic monitoring of the process of teaching foreign languages, thereby improving the quality of teaching foreign languages. Dynamic adaptive audio test simulators can provide monitoring of the dynamics of the process of teaching native and foreign languages.

Download Full-text