Monitoring Item Performance With CUSUM Statistics in Continuous Testing

In many educational assessments, items are reused in different administrations throughout the life of the assessments. Ideally, a reused item should perform relatively similarly over time. In reality, an item may become easier with exposure, especially when item preknowledge has occurred. This article presents a novel cumulative sum procedure for detecting item preknowledge in continuous testing where data for each reused item may be obtained from small and varying sample sizes across administrations. Its performance is evaluated with simulations and analytical work. The approach is effective in detecting item preknowledge quickly with group size at least 10 and is easy to implement with varying item parameters. In addition, it is robust to the ability estimation error introduced in the simulations.

A Deterministic Gated Lognormal Response Time Model to Identify Examinees with Item Preknowledge

Response time (RT) information has recently attracted a significant amount of attention in the literature as it may provide meaningful information about item preknowledge. In this study, a Deterministic Gated Lognormal Response Time (DG-LNRT) model is proposed to identify examinees with potential item preknowledge using RT information. The proposed model is applied to a real experimental dataset provided by Toton and Maynes (2019) in which item preknowledge was manipulated, and its performance is demonstrated. Then, the performance of the DG-LNRT model is investigated through a simulation study. The model is estimated using the Bayesian framework via Stan. The results indicate that the proposed model is viable and has the potential to be useful in detecting cheating by using response time differences between compromised and uncompromised items.

The Effect of Item Preknowledge on Response Time: Analysis of an Experimental Dataset Using the Multiple-Group Lognormal Response Time Model with a Gating Mechanism

Response time information has recently attracted a significant amount of attention in the literature as it may provide meaningful information about item preknowledge. The methods that propose the use of response time information in identifying examinees with potential item preknowledge make an implicit assumption that the examinees with item preknowledge differ in their response time patterns compared to other examinees without item preknowledge. In this study, we analyzed the differences in response time of examinees with potential item preknowledge and examinees without item preknowledge based on a real experimental dataset provided by Toton and Maynes (2019). A multiple-group extension of van der Linden’s Lognormal Response Model with a gating mechanism was used to capture the differences in latent speed for control and experimental groups on disclosed and undisclosed items. The model used in the study and estimated parameters from this experimental dataset may inform future simulation studies in this area of research to simulate realistic datasets with item preknowledge behavior.

Detection of Item Preknowledge Using Response Times

Benefiting from item preknowledge is a major type of fraudulent behavior during educational assessments. This article suggests a new statistic that can be used for detecting the examinees who may have benefited from item preknowledge using their response times. The statistic quantifies the difference in speed between the compromised items and the non-compromised items of the examinees. The distribution of the statistic under the null hypothesis of no preknowledge is proved to be the standard normal distribution. A simulation study is used to evaluate the Type I error rate and power of the suggested statistic. A real data example demonstrates the usefulness of the new statistic that is found to provide information that is not provided by statistics based only on item scores.

Detecting Examinees With Item Preknowledge in Large-Scale Testing Using Extreme Gradient Boosting (XGBoost)

Researchers frequently use machine-learning methods in many fields. In the area of detecting fraud in testing, there have been relatively few studies that have used these methods to identify potential testing fraud. In this study, a technical review of a recently developed state-of-the-art algorithm, Extreme Gradient Boosting (XGBoost), is provided and the utility of XGBoost in detecting examinees with potential item preknowledge is investigated using a real data set that includes examinees who engaged in fraudulent testing behavior, such as illegally obtaining live test content before the exam. Four different XGBoost models were trained using different sets of input features based on (a) only dichotomous item responses, (b) only nominal item responses, (c) both dichotomous item responses and response times, and (d) both nominal item responses and response times. The predictive performance of each model was evaluated using the area under the receiving operating characteristic curve and several classification measures such as the false-positive rate, true-positive rate, and precision. For comparison purposes, the results from two person-fit statistics on the same data set were also provided. The results indicated that XGBoost successfully classified the honest test takers and fraudulent test takers with item preknowledge. Particularly, the classification performance of XGBoost was reasonably good when the response time information and item responses were both taken into account.