scholarly journals Making Fixed-Precision Between-Item Multidimensional Computerized Adaptive Tests Even Shorter by Reducing the Asymmetry Between Selection and Stopping Rules

2020 ◽  
Vol 44 (7-8) ◽  
pp. 531-547
Author(s):  
Johan Braeken ◽  
Muirne C. S. Paap
Keyword(s):  
Fisher Information ◽  
Stopping Rule ◽  
Stopping Rules ◽  
Item Selection ◽  
Test Length ◽  
Objective Criterion ◽  
Current Generation ◽  
Adaptive Tests ◽  
Selection Rules ◽  
Computerized Adaptive Tests

Fixed-precision between-item multidimensional computerized adaptive tests (MCATs) are becoming increasingly popular. The current generation of item-selection rules used in these types of MCATs typically optimize a single-valued objective criterion for multivariate precision (e.g., Fisher information volume). In contrast, when all dimensions are of interest, the stopping rule is typically defined in terms of a required fixed marginal precision per dimension. This asymmetry between multivariate precision for selection and marginal precision for stopping, which is not present in unidimensional computerized adaptive tests, has received little attention thus far. In this article, we will discuss this selection-stopping asymmetry and its consequences, and introduce and evaluate three alternative item-selection approaches. These alternatives are computationally inexpensive, easy to communicate and implement, and result in effective fixed-marginal-precision MCATs that are shorter in test length than with the current generation of item-selection approaches.

scholarly journals Measuring language ability of students with compensatory multidimensional CAT: A post-hoc simulation study

2022 ◽  
Author(s):  
Burhanettin Ozdemir ◽  
Selahattin Gelbal
Keyword(s):  
Error Variance ◽  
Stopping Rule ◽  
Language Ability ◽  
Test Reliability ◽  
Item Selection ◽  
Ability Estimation ◽  
Proficiency Tests ◽  
Test Length ◽  
Post Hoc ◽  
Multidimensional Cat

AbstractThe computerized adaptive tests (CAT) apply an adaptive process in which the items are tailored to individuals' ability scores. The multidimensional CAT (MCAT) designs differ in terms of different item selection, ability estimation, and termination methods being used. This study aims at investigating the performance of the MCAT designs used to measure the language ability of students and to compare the results of MCAT designs with the outcomes of corresponding paper–pencil tests. For this purpose, items in the English Proficiency Tests (EPT) were used to create a multi-dimensional item pool that consists of 599 items. The performance of the MCAT designs was evaluated and compared based on the reliability coefficients, root means square error (RMSE), test-length, and root means squared difference (RMSD) statistics, respectively. Therefore, 36 different conditions were investigated in total. The results of the post-hoc simulation designs indicate that the MCAT designs with the A-optimality item selection method outperformed MCAT designs with other item selection methods by decreasing the test length and RMSD values without any sacrifice in test reliability. Additionally, the best error variance stopping rule for each MCAT algorithm with A-optimality item selection could be considered as 0.25 with 27.9 average test length and 30 items for the fixed test-length stopping rule for the Bayesian MAP method. Overall, MCAT designs tend to decrease the test length by 60 to 65 percent and provide ability estimations with higher precision compared to the traditional paper–pencil tests with 65 to 75 items. Therefore, it is suggested to use the A-optimality method for item selection and the Bayesian MAP method for ability estimation for the MCAT designs since the MCAT algorithm with these specifications shows better performance than others.

Outlier Measures and Norming Methods for Computerized Adaptive Tests

2001 ◽  
Vol 26 (1) ◽  
pp. 85-104 ◽  
Author(s):  
Eric T. Bradlow ◽  
Robert E. Weiss
Keyword(s):  
Stopping Rules ◽  
P Value ◽  
Sequential Choice ◽  
Probability Scale ◽  
Adaptive Tests ◽  
Computerized Adaptive Tests ◽  
Paper And Pencil ◽  
Highly Correlated

The problem of identifying outliers has two important aspects: the choice of outlier measures and the method to assess the degree of outlyingness (norming) of those measures. Several classes of measures for identifying outliers in Computerized Adaptive Tests (CATs) are introduced. Some of these measures are new and are constructed to take advantage of CATs’ sequential choice of items; other measures are taken directly from paper and pencil (P&P) tests and are used for baseline comparisons. Assessing the degree of outlyingness of CAT responses, however, can not be applied directly from P&P tests because stopping rules associated with CATs yield examinee responses of varying lengths. Standard outlier measures are highly correlated with the varying lengths which makes comparison across examinees impossible. Therefore, four methods are presented and compared which map outlier statistics to a familiar probability scale (a p value). The application of these methods to CAT data is new. The methods are explored in the context of CAT data from a 1995 Nationally Administered Computerized Examination (NACE).

Sign in / Sign up

Export Citation Format

Share Document