scholarly journals Simulated Computer Adaptive Testing Administration with Remote Proctoring for Offsite Assessment in Mathematics

2021 ◽  
Author(s):  
Jumoke Oladele ◽  
Mdutshekelwa Ndlovu
Keyword(s):  
Software Package ◽  
Adaptive Testing ◽  
Computer Adaptive Testing ◽  
Free Software ◽  
Site Testing ◽  
Test Security ◽  
Item Exposure ◽  
Test Items ◽  
It Impacts ◽  
Research Questions

<div><p>This study examines remote proctoring as emerging practice for ascertaining the validity of offsite test administration regarding test security. While Computer Adaptive Testing (CAT) has the potentials for greater precision in determining examinees ability level, its gains can be jeopardized with off-site testing if the test is not ensured. This study simulated CAT assessment while focusing on item administration, varying the option of using pre-test items and how it impacts students' ability estimation and item exposure. Monte-Carlo simulation was employed to generate data for answering the research questions raised for the study. The study's findings revealed that CAT administration was more consistent with no pre-test items once tightly controlled at ±2theta level, upon which recommendations were made. This finding is particularly germane, with more institutions moving their assessments online and rapidly becoming a new normal as an aftermath of the Covid-19 pandemic.</p></div><div><br></div>The data for this study were generated from computer simulations using SimulCAT, a free software package designed by Dr. K. C. T. Han of Graduate Management Admission Council.<p></p>

scholarly journals Simulated Computer Adaptive Testing Administration with Remote Proctoring for Offsite Assessment in Mathematics

2021 ◽  
Author(s):  
Jumoke Oladele ◽  
Mdutshekelwa Ndlovu
Keyword(s):  
Software Package ◽  
Adaptive Testing ◽  
Computer Adaptive Testing ◽  
Free Software ◽  
Site Testing ◽  
Test Security ◽  
Item Exposure ◽  
Test Items ◽  
It Impacts ◽  
Research Questions

<div><p>This study examines remote proctoring as emerging practice for ascertaining the validity of offsite test administration regarding test security. While Computer Adaptive Testing (CAT) has the potentials for greater precision in determining examinees ability level, its gains can be jeopardized with off-site testing if the test is not ensured. This study simulated CAT assessment while focusing on item administration, varying the option of using pre-test items and how it impacts students' ability estimation and item exposure. Monte-Carlo simulation was employed to generate data for answering the research questions raised for the study. The study's findings revealed that CAT administration was more consistent with no pre-test items once tightly controlled at ±2theta level, upon which recommendations were made. This finding is particularly germane, with more institutions moving their assessments online and rapidly becoming a new normal as an aftermath of the Covid-19 pandemic.</p></div><div><br></div>The data for this study were generated from computer simulations using SimulCAT, a free software package designed by Dr. K. C. T. Han of Graduate Management Admission Council.<p></p>

scholarly journals Alternative item selection strategies for improving test security in computerized adaptive testing of the algorithm

2020 ◽  
Vol 6 (1) ◽  
Author(s):  
Iwan Suhardi
Keyword(s):  
Computerized Adaptive Testing ◽  
Adaptive Testing ◽  
Item Selection ◽  
Information Function ◽  
Step Size ◽  
Maximum Likeli ◽  
Test Security ◽  
Item Exposure ◽  
Selection Strategies ◽  
Size Model

Salah satu metode estimasi kemampuan  yang banyak diaplikasikan pada algoritma Computerized Adaptive Testing (CAT) adalah Maximum Likeli-hood Estimation (MLE).  Metode MLE mempunyai kekurangan yaitu ketidakmampuan menemukan solusi estimasi kemampuan peserta tes ketika skor peserta masih belum berpola. Bila ada peserta tes yang memperoleh skor 0 atau skor sempurna, maka untuk menentukan estimasi kemampuan peserta tes umumnya menggunakan model step size. Namun, model step-size tersebut mengakibatkan item exposure. Item exposure merupakan fenomena dimana butir-butir soal tertentu akan lebih sering muncul dibandingkan dengan butir-butir soal yang lain. Hal tersebut membuat tes menjadi tidak aman karena butir-butir soal yang sering muncul akan lebih mudah pula untuk dikenali. Kajian ini mencoba memberikan alternatif strategi dengan cara memodifikasi model step-size dan dilanjutkan dengan merandom hasil perhitungan fungsi informasi yang diperoleh. Berdasarkan hasil kajian didapatkan bahwa alternatif strategi pemilihan butir soal ini mampu menghasilkan kemunculan butir soal yang lebih bervariasi sehingga dapat meningkatkan keamanan tes pada CAT.Kata kunci: item exposure, step-size, adaptive testing AbstractOne method of capability estimation that is widely applied to the Computerized Adaptive Testing (CAT) algorithm is Maximum Likeli-hood Estimation (MLE). The MLE method has the disadvantage of being unable to find a solution to the test taker's ability when the participant's score is not patterned. If there are test takers who get a score of 0 or perfect score, then to determine the ability of the test takers to generally use the step size model. However, the step-size model results in exposure items. The exposure item is a phenomenon where certain items will appear more often than other items. This makes the test insecure because items that often appear will be easier to recognize. This study tries to provide an alternative strategy by modifying the step-size model and proceed by randomizing the results of the calculation of the information function obtained. Based on the results of the study, it was found that alternative item selection strategies were able to produce the appearance of more varied items so as to improve the safety of tests on the CAT.

scholarly journals Computer Adaptive Testing Using Upper-Confidence Bound Algorithm for Formative Assessment

2019 ◽  
Vol 9 (20) ◽  
pp. 4303 ◽  
Author(s):  
Jaroslav Melesko ◽  
Vitalij Novickij
Keyword(s):  
Formative Assessment ◽  
Latent Trait ◽  
Corrective Feedback ◽  
Strong Support ◽  
Adaptive Testing ◽  
Computer Adaptive Testing ◽  
Confidence Bound ◽  
Question Item ◽  
E Learning ◽  
Upper Confidence Bound

There is strong support for formative assessment inclusion in learning processes, with the main emphasis on corrective feedback for students. However, traditional testing and Computer Adaptive Testing can be problematic to implement in the classroom. Paper based tests are logistically inconvenient and are hard to personalize, and thus must be longer to accurately assess every student in the classroom. Computer Adaptive Testing can mitigate these problems by making use of Multi-Dimensional Item Response Theory at cost of introducing several new problems, most problematic of which are the greater test creation complexity, because of the necessity of question pool calibration, and the debatable premise that different questions measure one common latent trait. In this paper a new approach of modelling formative assessment as a Multi-Armed bandit problem is proposed and solved using Upper-Confidence Bound algorithm. The method in combination with e-learning paradigm has the potential to mitigate such problems as question item calibration and lengthy tests, while providing accurate formative assessment feedback for students. A number of simulation and empirical data experiments (with 104 students) are carried out to explore and measure the potential of this application with positive results.

Evaluating a Computerized Adaptive Testing Version of a Cognitive Ability Test Using a Simulation Study

2021 ◽  
pp. 073428292110277
Author(s):  
Ioannis Tsaousis ◽  
Georgios D. Sideridis ◽  
Hannan M. AlGhamdi
Keyword(s):  
Cognitive Ability ◽  
Simulation Study ◽  
Computerized Adaptive Testing ◽  
Adaptive Testing ◽  
Estimation Methods ◽  
Item Pool ◽  
Sequential Approach ◽  
Ability Test ◽  
Promising Alternative ◽  
Item Exposure

This study evaluated the psychometric quality of a computerized adaptive testing (CAT) version of the general cognitive ability test (GCAT), using a simulation study protocol put forth by Han, K. T. (2018a). For the needs of the analysis, three different sets of items were generated, providing an item pool of 165 items. Before evaluating the efficiency of the GCAT, all items in the final item pool were linked (equated), following a sequential approach. Data were generated using a standard normal for 10,000 virtual individuals ( M = 0 and SD = 1). Using the measure’s 165-item bank, the ability value (θ) for each participant was estimated. maximum Fisher information (MFI) and maximum likelihood estimation with fences (MLEF) were used as item selection and score estimation methods, respectively. For item exposure control, the fade away method (FAM) was preferred. The termination criterion involved a minimum SE ≤ 0.33. The study revealed that the average number of items administered for 10,000 participants was 15. Moreover, the precision level in estimating the participant’s ability score was very high, as demonstrated by the CBIAS, CMAE, and CRMSE). It is concluded that the CAT version of the test is a promising alternative to administering the corresponding full-length measure since it reduces the number of administered items, prevents high rates of item exposure, and provides accurate scores with minimum measurement error.

A Test Sheet Optimization Approach to Supporting Web-based Learning Diagnosis Using Group Testing Methods

2017 ◽  
Vol 7 (4) ◽  
pp. 1-23
Author(s):  
Chu-Fu Wang ◽  
Chih-Lung Lin ◽  
Gwo-Jen Hwang ◽  
Sheng-Pin Kung ◽  
Shin-Feng Chen
Keyword(s):  
Test Item ◽  
Group Testing ◽  
Diagnostic System ◽  
Point Of View ◽  
Optimization Approach ◽  
Exposure Rate ◽  
Web Based ◽  
Item Exposure ◽  
Test Items ◽  
Doubling Method

Assessment can help teachers to examine the effectiveness of teaching and to diagnose the unfamiliar basic concepts (or attributes) of students within the testing scope. A web-based adaptive testing and diagnostic system can achieve the above objective efficiently and correctly. From a diagnostic point of view, the major concerns are to diagnose whether or not an examinee has learned each basic concept well in the testing scope, while also limiting the number of test items used (the testing length) to as few as possible, which will be directly related to the patience of the examinee. In this paper, we consider a test item selecting optimization diagnostic problem to reveal the mastery profile of an examinee (that is, to diagnose each basic concept's learning status (well learned/unfamiliar) in the testing scope) with a short testing length and a limited test item exposure rate. This paper uses the techniques of Group Testing theory for the design of our test item selecting algorithm. Two test item selecting strategies, the bisecting method and the doubling method, are proposed. The effectiveness of the proposed methods was evaluated by experimental simulations. The results show that both of the proposed algorithms use fewer test items and a limited test item exposure rate compared to the conventional methods.

Sign in / Sign up

Export Citation Format

Share Document