Simulated Computer Adaptive Testing Administration with Remote Proctoring for Offsite Assessment in Mathematics

<div><p>This study examines remote proctoring as emerging practice for ascertaining the validity of offsite test administration regarding test security. While Computer Adaptive Testing (CAT) has the potentials for greater precision in determining examinees ability level, its gains can be jeopardized with off-site testing if the test is not ensured. This study simulated CAT assessment while focusing on item administration, varying the option of using pre-test items and how it impacts students' ability estimation and item exposure. Monte-Carlo simulation was employed to generate data for answering the research questions raised for the study. The study's findings revealed that CAT administration was more consistent with no pre-test items once tightly controlled at ±2theta level, upon which recommendations were made. This finding is particularly germane, with more institutions moving their assessments online and rapidly becoming a new normal as an aftermath of the Covid-19 pandemic.</p></div><div><br></div>The data for this study were generated from computer simulations using SimulCAT, a free software package designed by Dr. K. C. T. Han of Graduate Management Admission Council.<p></p>

Simulated Computer Adaptive Testing Administration with Remote Proctoring for Offsite Assessment in Mathematics

<div><p>This study examines remote proctoring as emerging practice for ascertaining the validity of offsite test administration regarding test security. While Computer Adaptive Testing (CAT) has the potentials for greater precision in determining examinees ability level, its gains can be jeopardized with off-site testing if the test is not ensured. This study simulated CAT assessment while focusing on item administration, varying the option of using pre-test items and how it impacts students' ability estimation and item exposure. Monte-Carlo simulation was employed to generate data for answering the research questions raised for the study. The study's findings revealed that CAT administration was more consistent with no pre-test items once tightly controlled at ±2theta level, upon which recommendations were made. This finding is particularly germane, with more institutions moving their assessments online and rapidly becoming a new normal as an aftermath of the Covid-19 pandemic.</p></div><div><br></div>The data for this study were generated from computer simulations using SimulCAT, a free software package designed by Dr. K. C. T. Han of Graduate Management Admission Council.<p></p>

cdcatR: An R Package for Cognitive Diagnostic Computerized Adaptive Testing

Cognitive diagnosis models (CDMs) are confirmatory latent class models that provide fine-grained information about skills and cognitive processes. These models have gained attention in the last few years because of their usefulness in educational and psychological settings. Recently, numerous developments have been made to allow for the implementation of cognitive diagnosis computerized adaptive testing (CD-CAT). Despite methodological advances, CD-CAT applications are still scarce. To facilitate research and the emergence of empirical applications in this area, we have developed the cdcatR package for R software. The purpose of this document is to illustrate the different functions included in this package. The package includes functionalities for data generation, model selection based on relative fit information, implementation of several item selection rules (including item exposure control), and CD-CAT performance evaluation in terms of classification accuracy, item exposure, and test length. In conclusion, an R package is made available to researchers and practitioners that allows for an easy implementation of CD-CAT in both simulation and applied studies. Ultimately, this is expected to facilitate the development of empirical applications in this area.

Binary Restrictive Threshold Method for Item Exposure Control in Cognitive Diagnostic Computerized Adaptive Testing

Although classification accuracy is a critical issue in cognitive diagnostic computerized adaptive testing, attention has increasingly shifted to item exposure control to ensure test security. In this study, we developed the binary restrictive threshold (BRT) method to balance measurement accuracy and item exposure. In addition, a simulation study was conducted to evaluate its performance. The results indicated that the BRT method performed better than the restrictive progressive (RP) and stratified dynamic binary searching (SDBS) approaches but worse than the restrictive threshold (RT) method in terms of classification accuracy. With respect to item exposure control, the BRT method exhibited noticeably stronger performance compared with the RT method, even though its performance was not as high as that of the RP and SDBS methods.

Sexual jokes at school and students’ life satisfaction: findings from the 2017/18 Swedish Health Behaviour in School-aged Children study

Objective In a previous study we demonstrated that the occurrence of sexual jokes in the class was associated with higher levels of psychological health complaints. Building on and extending these findings, the aim of the current study was to examine if exposure to sexual jokes at the student and at the class level was inversely associated with students’ life satisfaction. Data were derived from the 2017/18 Swedish Health Behaviour in School-aged Children (HBSC) study, with students aged 11, 13 and 15 years (n = 3710 distributed across 209 classes). Exposure to sexual jokes at the student level was captured by one item. Exposure to sexual jokes at the class level was calculated by aggregating this measure. The Cantril ladder was used to operationalise life satisfaction. Two-level logistic regression analyses were performed. Results Students who were exposed to sexual jokes at school were less likely to report high life satisfaction (OR 0.38, 95% CI 0.27–0.53). An inverse association was found between the class proportion of students who were exposed to sexual jokes and students’ likelihood of reporting high life satisfaction, whilst adjusting for exposure to sexual jokes at the student level (OR 0.98, 95% CI 0.97–0.9994). The findings highlight the importance of promoting a school climate without sexual harassment.

Evaluating a Computerized Adaptive Testing Version of a Cognitive Ability Test Using a Simulation Study

This study evaluated the psychometric quality of a computerized adaptive testing (CAT) version of the general cognitive ability test (GCAT), using a simulation study protocol put forth by Han, K. T. (2018a). For the needs of the analysis, three different sets of items were generated, providing an item pool of 165 items. Before evaluating the efficiency of the GCAT, all items in the final item pool were linked (equated), following a sequential approach. Data were generated using a standard normal for 10,000 virtual individuals ( M = 0 and SD = 1). Using the measure’s 165-item bank, the ability value (θ) for each participant was estimated. maximum Fisher information (MFI) and maximum likelihood estimation with fences (MLEF) were used as item selection and score estimation methods, respectively. For item exposure control, the fade away method (FAM) was preferred. The termination criterion involved a minimum SE ≤ 0.33. The study revealed that the average number of items administered for 10,000 participants was 15. Moreover, the precision level in estimating the participant’s ability score was very high, as demonstrated by the CBIAS, CMAE, and CRMSE). It is concluded that the CAT version of the test is a promising alternative to administering the corresponding full-length measure since it reduces the number of administered items, prevents high rates of item exposure, and provides accurate scores with minimum measurement error.

New Item Selection Method Accommodating Practical Constraints in Cognitive Diagnostic Computerized Adaptive Testing: Maximum Deviation and Maximum Limitation Global Discrimination Indexes

Maximum deviation global discrimination index (MDGDI) is a new item selection method for cognitive diagnostic computerized adaptive testing that allows for attribute coverage balance. We developed the maximum limitation global discrimination index (MLGDI) from MDGDI, which allows for both attribute coverage balance and item exposure control. MLGDI can realize the attribute coverage balance and exposure control of the item. Our simulation study aimed to evaluate the performance of our new method against maximum global discrimination index (GDI), modified maximum GDI (MMGDI), standardized weighted deviation GDI (SWDGDI), and constraint progressive with SWDGDI (CP_SWDGDI). The results indicated that (1a) under the condition of realizing the attribute coverage balance, MDGDI had the highest attribute classification accuracy; (1b) when the selection strategy accommodated the practical constraints of the attribute coverage balance and item exposure control, MLGDI had the highest attribute classification accuracy; (2) adding the item exposure control mechanism to the item selection method reduces the classification accuracy of the attributes of the item selection method; and (3) compared with GDI, MMGDI, SWDGDI, CP_SWDGDI, and MDGDI, MLGDI can better achieve the attribute-coverage requirement, control item exposure rate, and attribute correct classification rate.