scholarly journals A Heuristic Method for Large-Scale Cognitive-Diagnostic Computerized Adaptive Testing

2017 ◽  
Author(s):  
Jill-Jênn Vie ◽  
Fabrice Popineau ◽  
Françoise Tort ◽  
Benjamin Marteau ◽  
Nathalie Denos
Keyword(s):  
Large Scale ◽  
Computerized Adaptive Testing ◽  
Heuristic Method ◽  
Adaptive Testing

Adaptive Weight Estimation of Latent Ability: Application to Computerized Adaptive Testing With Response Revision

2020 ◽  
pp. 107699862097280
Author(s):  
Shiyu Wang ◽  
Houping Xiao ◽  
Allan Cohen
Keyword(s):  
Large Scale ◽  
Computerized Adaptive Testing ◽  
Estimation Method ◽  
Adaptive Testing ◽  
Ability Estimation ◽  
Weight Estimation ◽  
Data Set ◽  
Adaptive Weight ◽  
Latent Ability ◽  
Estimation Procedures

An adaptive weight estimation approach is proposed to provide robust latent ability estimation in computerized adaptive testing (CAT) with response revision. This approach assigns different weights to each distinct response to the same item when response revision is allowed in CAT. Two types of weight estimation procedures, nonfunctional and functional weight, are proposed to determine the weight adaptively based on the compatibility of each revised response with the assumed statistical model in relation to remaining observations. The application of this estimation approach to a data set collected from a large-scale multistage adaptive testing demonstrates the capability of this method to reveal more information regarding the test taker’s latent ability by using the valid response path compared with only using the very last response. Limited simulation studies were concluded to evaluate the proposed ability estimation method and to compare it with several other estimation procedures in literature. Results indicate that the proposed ability estimation approach is able to provide robust estimation results in two test-taking scenarios.

Controlling Item Exposure Conditional on Ability in Computerized Adaptive Testing

1998 ◽  
Vol 23 (1) ◽  
pp. 57-75 ◽  
Author(s):  
Martha L. Stocking ◽  
Charles Lewis
Keyword(s):  
Large Scale ◽  
Computerized Adaptive Testing ◽  
Simulated Data ◽  
Adaptive Testing ◽  
Testing Environment ◽  
Item Exposure ◽  
Continuous Testing ◽  
Conditional Control ◽  
Periodic Testing ◽  
Testing Algorithm

The interest in the application of large-scale adaptive testing for secure tests has served to focus attention on issues that arise when theoretical advances are made operational. One such issue is that of ensuring item and pool security in the continuous testing environment made possible by the computerized admin-istration of a test, as opposed to the more periodic testing environment typically used for linear paper-and-pencil tests. This article presents a new method of controlling the exposure rate of items conditional on ability level in this continuous testing environment. The properties of such conditional control on the exposure rates of items, when used in conjunction with a particular adaptive testing algorithm, are explored through studies with simulated data.

Evaluating Computerized Adaptive Testing Systems

2009 ◽  
pp. 186-202
Author(s):  
Anastasios A. Economides ◽  
Chrysostomos Roupas
Keyword(s):  
Large Scale ◽  
Computerized Adaptive Testing ◽  
Adaptive Testing ◽  
Evaluation Framework ◽  
Educational Organizations ◽  
Human Errors ◽  
Media Type ◽  
Large Scale Testing ◽  
The Cost

Many educational organizations are trying to reduce the cost of exams, the workload, delay of scoring, and the human errors. Also, organizations try to increase the accuracy and efficiency of the testing. Recently, most examination organizations use Computerized Adaptive Testing (CAT) as the method for large scale testing. This chapter investigates the current state of CAT systems and identifies their strengths and weaknesses. It evaluates 10 CAT systems using an evaluation framework of 15 domains categorized into 3 dimensions: Educational, Technical and Economical. The results show that the majority of the CAT systems give priority to security, reliability, and maintainability. However, they do not offer to the examinee any advanced support and functionalities. Also, the feedback to the examinee is limited and the presentation of the items is poor. Recommendations are made in order to enhance the overall quality of a CAT system. For example, alternative multimedia items should be available so that the examinee would choose his preferred media type. Feedback could be improved by providing more information to the examinee or providing information anytime the examinee wished.

scholarly journals BOBCAT: Bilevel Optimization-Based Computerized Adaptive Testing

2021 ◽  
Author(s):  
Aritra Ghosh ◽  
Andrew Lan
Keyword(s):  
Large Scale ◽  
Computerized Adaptive Testing ◽  
Estimation Error ◽  
Bilevel Optimization ◽  
Adaptive Testing ◽  
Student Response ◽  
Training Data ◽  
Test Length ◽  
Computationally Efficient ◽  
Question Selection

Computerized adaptive testing (CAT) refers to a form of tests that are personalized to every student/test taker. CAT methods adaptively select the next most informative question/item for each student given their responses to previous questions, effectively reducing test length. Existing CAT methods use item response theory (IRT) models to relate student ability to their responses to questions and static question selection algorithms designed to reduce the ability estimation error as quickly as possible; therefore, these algorithms cannot improve by learning from large-scale student response data. In this paper, we propose BOBCAT, a Bilevel Optimization-Based framework for CAT to directly learn a data-driven question selection algorithm from training data. BOBCAT is agnostic to the underlying student response model and is computationally efficient during the adaptive testing process. Through extensive experiments on five real-world student response datasets, we show that BOBCAT outperforms existing CAT methods (sometimes significantly) at reducing test length.

Benefits from Computerized Adaptive Testing as Seen in Simulation Studies

1999 ◽  
Vol 15 (2) ◽  
pp. 91-98 ◽  
Author(s):  
Lutz F. Hornke
Keyword(s):  
Measurement Error ◽  
Computerized Adaptive Testing ◽  
Test Procedure ◽  
Adaptive Testing ◽  
Parameter Estimates ◽  
Simulation Studies ◽  
Computerized Adaptive Test ◽  
Item Banks ◽  
Item Parameters ◽  
General Reliability

Summary: Item parameters for several hundreds of items were estimated based on empirical data from several thousands of subjects. The logistic one-parameter (1PL) and two-parameter (2PL) model estimates were evaluated. However, model fit showed that only a subset of items complied sufficiently, so that the remaining ones were assembled in well-fitting item banks. In several simulation studies 5000 simulated responses were generated in accordance with a computerized adaptive test procedure along with person parameters. A general reliability of .80 or a standard error of measurement of .44 was used as a stopping rule to end CAT testing. We also recorded how often each item was used by all simulees. Person-parameter estimates based on CAT correlated higher than .90 with true values simulated. For all 1PL fitting item banks most simulees used more than 20 items but less than 30 items to reach the pre-set level of measurement error. However, testing based on item banks that complied to the 2PL revealed that, on average, only 10 items were sufficient to end testing at the same measurement error level. Both clearly demonstrate the precision and economy of computerized adaptive testing. Empirical evaluations from everyday uses will show whether these trends will hold up in practice. If so, CAT will become possible and reasonable with some 150 well-calibrated 2PL items.

Methods for Restricting Maximum Exposure Rate in Computerized Adaptative Testing

Methodology ◽  
2007 ◽  
Vol 3 (1) ◽  
pp. 14-23 ◽  
Author(s):  
Juan Ramon Barrada ◽  
Julio Olea ◽  
Vicente Ponsoda
Keyword(s):  
Measurement Accuracy ◽  
Computerized Adaptive Testing ◽  
Computation Time ◽  
Adaptive Testing ◽  
Exposure Rate ◽  
Control Parameters ◽  
The Impact ◽  
Two Alternatives ◽  
Selection Of ◽  
Maximum Exposure

Abstract. The Sympson-Hetter (1985) method provides a means of controlling maximum exposure rate of items in Computerized Adaptive Testing. Through a series of simulations, control parameters are set that mark the probability of administration of an item on being selected. This method presents two main problems: it requires a long computation time for calculating the parameters and the maximum exposure rate is slightly above the fixed limit. Van der Linden (2003) presented two alternatives which appear to solve both of the problems. The impact of these methods in the measurement accuracy has not been tested yet. We show how these methods over-restrict the exposure of some highly discriminating items and, thus, the accuracy is decreased. It also shown that, when the desired maximum exposure rate is near the minimum possible value, these methods offer an empirical maximum exposure rate clearly above the goal. A new method, based on the initial estimation of the probability of administration and the probability of selection of the items with the restricted method ( Revuelta & Ponsoda, 1998 ), is presented in this paper. It can be used with the Sympson-Hetter method and with the two van der Linden's methods. This option, when used with Sympson-Hetter, speeds the convergence of the control parameters without decreasing the accuracy.

Sign in / Sign up

Export Citation Format

Share Document