scholarly journals New Item Selection Method Accommodating Practical Constraints in Cognitive Diagnostic Computerized Adaptive Testing: Maximum Deviation and Maximum Limitation Global Discrimination Indexes

2021 ◽  
Vol 12 ◽  
Author(s):  
Junjie Li ◽  
Lihua Ma ◽  
Pingfei Zeng ◽  
Chunhua Kang
Keyword(s):  
Classification Accuracy ◽  
Computerized Adaptive Testing ◽  
Selection Method ◽  
Discrimination Index ◽  
Item Selection ◽  
Maximum Deviation ◽  
Exposure Control ◽  
Item Exposure ◽  
Item Exposure Control ◽  
Attribute Classification

Maximum deviation global discrimination index (MDGDI) is a new item selection method for cognitive diagnostic computerized adaptive testing that allows for attribute coverage balance. We developed the maximum limitation global discrimination index (MLGDI) from MDGDI, which allows for both attribute coverage balance and item exposure control. MLGDI can realize the attribute coverage balance and exposure control of the item. Our simulation study aimed to evaluate the performance of our new method against maximum global discrimination index (GDI), modified maximum GDI (MMGDI), standardized weighted deviation GDI (SWDGDI), and constraint progressive with SWDGDI (CP_SWDGDI). The results indicated that (1a) under the condition of realizing the attribute coverage balance, MDGDI had the highest attribute classification accuracy; (1b) when the selection strategy accommodated the practical constraints of the attribute coverage balance and item exposure control, MLGDI had the highest attribute classification accuracy; (2) adding the item exposure control mechanism to the item selection method reduces the classification accuracy of the attributes of the item selection method; and (3) compared with GDI, MMGDI, SWDGDI, CP_SWDGDI, and MDGDI, MLGDI can better achieve the attribute-coverage requirement, control item exposure rate, and attribute correct classification rate.

scholarly journals A Dynamic Stratification Method for Improving Trait Estimation in Computerized Adaptive Testing Under Item Exposure Control

2019 ◽  
Vol 44 (3) ◽  
pp. 182-196
Author(s):  
Jyun-Hong Chen ◽  
Hsiu-Yi Chao ◽  
Shu-Ying Chen
Keyword(s):  
Computerized Adaptive Testing ◽  
Item Difficulty ◽  
Adaptive Testing ◽  
Item Selection ◽  
Exposure Control ◽  
Item Exposure ◽  
Stratification Method ◽  
High Discrimination ◽  
Item Exposure Control ◽  
Trait Estimation

When computerized adaptive testing (CAT) is under stringent item exposure control, the precision of trait estimation will substantially decrease. A new item selection method, the dynamic Stratification method based on Dominance Curves (SDC), which is aimed at improving trait estimation, is proposed to mitigate this problem. The objective function of the SDC in item selection is to maximize the sum of test information for all examinees rather than maximizing item information for individual examinees at a single-item administration, as in conventional CAT. To achieve this objective, the SDC uses dominance curves to stratify an item pool into strata with the number being equal to the test length to precisely and accurately increase the quality of the administered items as the test progresses, reducing the likelihood that a high-discrimination item will be administered to an examinee whose ability is not close to the item difficulty. Furthermore, the SDC incorporates a dynamic process for on-the-fly item–stratum adjustment to optimize the use of quality items. Simulation studies were conducted to investigate the performance of the SDC in CAT under item exposure control at different levels of severity. According to the results, the SDC can efficiently improve trait estimation in CAT through greater precision and more accurate trait estimation than those generated by other methods (e.g., the maximum Fisher information method) in most conditions.

scholarly journals Components of the item selection algorithm in computerized adaptive testing

2018 ◽  
Vol 15 ◽  
pp. 7 ◽  
Author(s):  
Kyung (Chris) Tyek Han
Keyword(s):  
Computerized Adaptive Testing ◽  
Selection Criterion ◽  
Information Criterion ◽  
Item Selection ◽  
Exposure Control ◽  
Selection Algorithm ◽  
High Stakes ◽  
Item Exposure ◽  
Content Balancing ◽  
Item Exposure Control

Computerized adaptive testing (CAT) greatly improves measurement efficiency in high-stakes testing operations through the selection and administration of test items with the difficulty level that is most relevant to each individual test taker. This paper explains the 3 components of a conventional CAT item selection algorithm: test content balancing, the item selection criterion, and item exposure control. Several noteworthy methodologies underlie each component. The test script method and constrained CAT method are used for test content balancing. Item selection criteria include the maximized Fisher information criterion, the b-matching method, the astratification method, the weighted likelihood information criterion, the efficiency balanced information criterion, and the KullbackLeibler information criterion. The randomesque method, the Sympson-Hetter method, the unconditional and conditional multinomial methods, and the fade-away method are used for item exposure control. Several holistic approaches to CAT use automated test assembly methods, such as the shadow test approach and the weighted deviation model. Item usage and exposure count vary depending on the item selection criterion and exposure control method. Finally, other important factors to consider when determining an appropriate CAT design are the computer resources requirement, the size of item pools, and the test length. The logic of CAT is now being adopted in the field of adaptive learning, which integrates the learning aspect and the (formative) assessment aspect of education into a continuous, individualized learning experience. Therefore, the algorithms and technologies described in this review may be able to help medical health educators and high-stakes test developers to adopt CAT more actively and efficiently.

scholarly journals Binary Restrictive Threshold Method for Item Exposure Control in Cognitive Diagnostic Computerized Adaptive Testing

2021 ◽  
Vol 12 ◽  
Author(s):  
Xiaojian Sun ◽  
Yizhu Gao ◽  
Tao Xin ◽  
Naiqing Song
Keyword(s):  
Classification Accuracy ◽  
Measurement Accuracy ◽  
Computerized Adaptive Testing ◽  
Critical Issue ◽  
Adaptive Testing ◽  
Exposure Control ◽  
Threshold Method ◽  
Item Exposure ◽  
Item Exposure Control ◽  
Better Than

Although classification accuracy is a critical issue in cognitive diagnostic computerized adaptive testing, attention has increasingly shifted to item exposure control to ensure test security. In this study, we developed the binary restrictive threshold (BRT) method to balance measurement accuracy and item exposure. In addition, a simulation study was conducted to evaluate its performance. The results indicated that the BRT method performed better than the restrictive progressive (RP) and stratified dynamic binary searching (SDBS) approaches but worse than the restrictive threshold (RT) method in terms of classification accuracy. With respect to item exposure control, the BRT method exhibited noticeably stronger performance compared with the RT method, even though its performance was not as high as that of the RP and SDBS methods.

scholarly journals Some Alternatives to Sympson-Hetter Item-Exposure Control in Computerized Adaptive Testing

2003 ◽  
Vol 28 (3) ◽  
pp. 249-265 ◽  
Author(s):  
Wim J. van der Linden
Keyword(s):  
Iterative Process ◽  
Computerized Adaptive Testing ◽  
Adaptive Testing ◽  
Law School ◽  
Exposure Control ◽  
Item Exposure ◽  
Item Exposure Control ◽  
Admission Test ◽  
Formal Properties ◽  
Ability Parameter

The Hetter and Sympson (1997 ; 1985 ) method is a method of probabilistic item-exposure control in computerized adaptive testing. Setting its control parameters to admissible values requires an iterative process of computer simulations that has been found to be time consuming, particularly if the parameters have to be set conditional on a realistic set of values for the examinees’ ability parameter. Formal properties of the method are identified that help us explain why this iterative process can be slow and does not guarantee admissibility. In addition, some alternatives to the SH method are introduced. The behavior of these alternatives was estimated for an adaptive test from an item pool from the Law School Admission Test (LSAT). Two of the alternatives showed attractive behavior and converged smoothly to admissibility for all items in a relatively small number of iteration steps.

scholarly journals Constraining Item Exposure in Computerized Adaptive Testing With Shadow Tests

2004 ◽  
Vol 29 (3) ◽  
pp. 273-291 ◽  
Author(s):  
Wim J van der Linden ◽  
Bernard P Veldkamp
Keyword(s):  
Computerized Adaptive Testing ◽  
Mean Squared Error ◽  
Adaptive Testing ◽  
Law School ◽  
Exposure Control ◽  
Require Time ◽  
Item Exposure ◽  
Squared Error ◽  
Item Exposure Control ◽  
Admission Test

Item-exposure control in computerized adaptive testing is implemented by imposing item-ineligibility constraints on the assembly process of the shadow tests. The method resembles Sympson and Hetter’s (1985) method of item-exposure control in that the decisions to impose the constraints are probabilistic. The method does not, however, require time-consuming simulation studies to set values for control parameters before the operational use of the test. Instead, it can set the probabilities of item ineligibility adaptively during the test using the actual item-exposure rates. An empirical study using an item pool from the Law School Admission Test showed that application of the method yielded perfect control of the item-exposure rates and had negligible impact on the bias and mean-squared error functions of the ability estimator.

Sign in / Sign up

Export Citation Format

Share Document