scholarly journals Detection of Item Preknowledge Using Response Times

2020 ◽  
Vol 44 (5) ◽  
pp. 376-392
Author(s):  
Sandip Sinharay
Keyword(s):  
Type I Error ◽  
Response Times ◽  
Real Data ◽  
Standard Normal Distribution ◽  
Major Type ◽  
Type I ◽  
Standard Normal ◽  
The Difference ◽  
Item Scores ◽  
Item Preknowledge

Benefiting from item preknowledge is a major type of fraudulent behavior during educational assessments. This article suggests a new statistic that can be used for detecting the examinees who may have benefited from item preknowledge using their response times. The statistic quantifies the difference in speed between the compromised items and the non-compromised items of the examinees. The distribution of the statistic under the null hypothesis of no preknowledge is proved to be the standard normal distribution. A simulation study is used to evaluate the Type I error rate and power of the suggested statistic. A real data example demonstrates the usefulness of the new statistic that is found to provide information that is not provided by statistics based only on item scores.

scholarly journals Which Statistic Should Be Used to Detect Item Preknowledge When the Set of Compromised Items Is Known?

2017 ◽  
Vol 41 (6) ◽  
pp. 403-421 ◽  
Author(s):  
Sandip Sinharay
Keyword(s):  
Likelihood Ratio ◽  
Likelihood Ratio Test ◽  
Type I Error ◽  
Null Distribution ◽  
Real Data ◽  
Major Type ◽  
Ratio Test ◽  
Type I ◽  
Posterior Shift ◽  
Item Preknowledge

Benefiting from item preknowledge is a major type of fraudulent behavior during educational assessments. Belov suggested the posterior shift statistic for detection of item preknowledge and showed its performance to be better on average than that of seven other statistics for detection of item preknowledge for a known set of compromised items. Sinharay suggested a statistic based on the likelihood ratio test for detection of item preknowledge; the advantage of the statistic is that its null distribution is known. Results from simulated and real data and adaptive and nonadaptive tests are used to demonstrate that the Type I error rate and power of the statistic based on the likelihood ratio test are very similar to those of the posterior shift statistic. Thus, the statistic based on the likelihood ratio test appears promising in detecting item preknowledge when the set of compromised items is known.

Detecting Fraudulent Erasures at an Aggregate Level

2017 ◽  
Vol 43 (3) ◽  
pp. 286-315 ◽  
Author(s):  
Sandip Sinharay
Keyword(s):  
Type I Error ◽  
Null Distribution ◽  
Real Data ◽  
Error Rates ◽  
Standard Normal Distribution ◽  
Type I ◽  
Group Level ◽  
Nominal Level ◽  
Type I Error Rates ◽  
Standard Normal

Wollack, Cohen, and Eckerly suggested the “erasure detection index” (EDI) to detect fraudulent erasures for individual examinees. Wollack and Eckerly extended the EDI to detect fraudulent erasures at the group level. The EDI at the group level was found to be slightly conservative. This article suggests two modifications of the EDI for the group level. The asymptotic null distribution of the two modified indices is proved to be the standard normal distribution. In a simulation study, the modified indices are shown to have Type I error rates close to the nominal level and larger power than the index of Wollack and Eckerly. A real data example is also included.

scholarly journals Three New Methods for Analysis of Answer Changes

2016 ◽  
Vol 77 (1) ◽  
pp. 54-81 ◽  
Author(s):  
Sandip Sinharay ◽  
Matthew S. Johnson
Keyword(s):  
Normal Distribution ◽  
Null Hypothesis ◽  
Type I Error ◽  
Error Rates ◽  
Standard Normal Distribution ◽  
Type I ◽  
Data Set ◽  
Continuity Correction ◽  
Type I Error Rates ◽  
Standard Normal

In a pioneering research article, Wollack and colleagues suggested the “erasure detection index” (EDI) to detect test tampering. The EDI can be used with or without a continuity correction and is assumed to follow the standard normal distribution under the null hypothesis of no test tampering. When used without a continuity correction, the EDI often has inflated Type I error rates. When used with a continuity correction, the EDI has satisfactory Type I error rates, but smaller power compared with the EDI without a continuity correction. This article suggests three methods for detecting test tampering that do not rely on the assumption of a standard normal distribution under the null hypothesis. It is demonstrated in a detailed simulation study that the performance of each suggested method is slightly better than that of the EDI. The EDI and the suggested methods were applied to a real data set. The suggested methods, although more computation intensive than the EDI, seem to be promising in detecting test tampering.

Detection of Item Preknowledge Using Likelihood Ratio Test and Score Test

2016 ◽  
Vol 42 (1) ◽  
pp. 46-68 ◽  
Author(s):  
Sandip Sinharay
Keyword(s):  
Type I Error ◽  
Score Test ◽  
Error Rates ◽  
Ratio Test ◽  
Type I ◽  
Polytomous Items ◽  
Detailed Simulation ◽  
Standard Normal ◽  
New Statistics ◽  
Item Preknowledge

An increasing concern of producers of educational assessments is fraudulent behavior during the assessment (van der Linden, 2009). Benefiting from item preknowledge (e.g., Eckerly, 2017; McLeod, Lewis, & Thissen, 2003) is one type of fraudulent behavior. This article suggests two new test statistics for detecting individuals who may have benefited from item preknowledge; the statistics can be used for both nonadaptive and adaptive assessments that may include either or both of dichotomous and polytomous items. Each new statistic has an asymptotic standard normal n distribution. It is demonstrated in detailed simulation studies that the Type I error rates of the new statistics are close to the nominal level and the values of power of the new statistics are larger than those of an existing statistic for addressing the same problem.

Cheating on Unproctored Internet Test Applications: An Analysis of a Verification Test in a Real Personnel Selection Context

2018 ◽  
Vol 21 ◽  
Author(s):  
David Aguado ◽  
Alejandro Vidal ◽  
Julio Olea ◽  
Vicente Ponsoda ◽  
Juan Ramón Barrada ◽  
...  
Keyword(s):  
Personnel Selection ◽  
Type I Error ◽  
Response Times ◽  
Item Difficulty ◽  
Real Data ◽  
Type I ◽  
Test Statistic ◽  
Computerized Adaptive Test ◽  
English Grammar ◽  
Verification Test

AbstractThis study analyses the extent to which cheating occurs in a real selection setting. A two-stage, unproctored and proctored, test administration was considered. Test score inconsistencies were concluded by applying a verification test (Guo and Drasgow Z-test). An initial simulation study showed that the Z-test has adequate Type I error and power rates in the specific selection settings explored. A second study applied the Z-test statistic verification procedure to a sample of 954 employment candidates. Additional external evidence based on item time response to the verification items was gathered. The results revealed a good performance of the Z-test statistic and a relatively low, but non-negligible, number of suspected cheaters that showed higher distorted ability estimates. The study with real data provided additional information on the presence of suspected cheating in unproctored applications and the viability of using item response times as an additional evidence of cheating. In the verification test, suspected cheaters spent 5.78 seconds per item more than expected considering the item difficulty and their assumed ability in the unproctored stage. We found that the percentage of suspected cheaters in the empirical study could be estimated at 13.84%. In summary, the study provides evidence of the usefulness of the Z-test in the detection of cheating in a specific setting, in which a computerized adaptive test for assessing English grammar knowledge was used for personnel selection.

The Use of Theory of Linear Mixed-Effects Models to Detect Fraudulent Erasures at an Aggregate Level

2021 ◽  
pp. 001316442199489
Author(s):  
Luyao Peng ◽  
Sandip Sinharay
Keyword(s):  
Type I Error ◽  
Real Data ◽  
Mixed Effects ◽  
Error Rates ◽  
Mixed Effects Models ◽  
Type I ◽  
Aggregate Level ◽  
Linear Mixed Effects Models ◽  
Linear Mixed Effects ◽  
Best Linear Unbiased

Wollack et al. (2015) suggested the erasure detection index (EDI) for detecting fraudulent erasures for individual examinees. Wollack and Eckerly (2017) and Sinharay (2018) extended the index of Wollack et al. (2015) to suggest three EDIs for detecting fraudulent erasures at the aggregate or group level. This article follows up on the research of Wollack and Eckerly (2017) and Sinharay (2018) and suggests a new aggregate-level EDI by incorporating the empirical best linear unbiased predictor from the literature of linear mixed-effects models (e.g., McCulloch et al., 2008). A simulation study shows that the new EDI has larger power than the indices of Wollack and Eckerly (2017) and Sinharay (2018). In addition, the new index has satisfactory Type I error rates. A real data example is also included.

Approximate confidence intervals for the difference in proportions for partially observed binary data

2021 ◽  
pp. 096228022110605
Author(s):  
Ujjwal Das ◽  
Ranojoy Basu
Keyword(s):  
Binary Data ◽  
Type I Error ◽  
Missing At Random ◽  
Matched Pair ◽  
Data Sets ◽  
Type I ◽  
Real World Data ◽  
Partially Observed ◽  
The Difference ◽  
Interval Estimator

We consider partially observed binary matched-pair data. We assume that the incomplete subjects are missing at random. Within this missing framework, we propose an EM-algorithm based approach to construct an interval estimator of the proportion difference incorporating all the subjects. In conjunction with our proposed method, we also present two improvements to the interval estimator through some correction factors. The performances of the three competing methods are then evaluated through extensive simulation. Recommendation for the method is given based on the ability to preserve type-I error for various sample sizes. Finally, the methods are illustrated in two real-world data sets. An R-function is developed to implement the three proposed methods.

scholarly journals Detecting Differential Item Functioning Using Multiple-Group Cognitive Diagnosis Models

2020 ◽  
Vol 45 (1) ◽  
pp. 37-53
Author(s):  
Wenchao Ma ◽  
Ragip Terzi ◽  
Jimmy de la Torre
Keyword(s):  
Differential Item Functioning ◽  
Type I Error ◽  
Search Algorithm ◽  
Real Data ◽  
Error Rates ◽  
Cognitive Diagnosis ◽  
Type I ◽  
Wald Tests ◽  
Multiple Group ◽  
Item Functioning

This study proposes a multiple-group cognitive diagnosis model to account for the fact that students in different groups may use distinct attributes or use the same attributes but in different manners (e.g., conjunctive, disjunctive, and compensatory) to solve problems. Based on the proposed model, this study systematically investigates the performance of the likelihood ratio (LR) test and Wald test in detecting differential item functioning (DIF). A forward anchor item search procedure was also proposed to identify a set of anchor items with invariant item parameters across groups. Results showed that the LR and Wald tests with the forward anchor item search algorithm produced better calibrated Type I error rates than the ordinary LR and Wald tests, especially when items were of low quality. A set of real data were also analyzed to illustrate the use of these DIF detection procedures.

scholarly journals The asymptotic distribution of modularity in weighted signed networks

Biometrika ◽  
2020 ◽  
Author(s):  
Rong Ma ◽  
Ian Barnett
Keyword(s):  
Community Structure ◽  
Asymptotic Distribution ◽  
Statistical Power ◽  
Type I Error ◽  
Real Data ◽  
Asymptotic Distributions ◽  
Edge Weight ◽  
Type I ◽  
Largest Eigenvalue ◽  
The Largest Eigenvalue

Summary Modularity is a popular metric for quantifying the degree of community structure within a network. The distribution of the largest eigenvalue of a network’s edge weight or adjacency matrix is well studied and is frequently used as a substitute for modularity when performing statistical inference. However, we show that the largest eigenvalue and modularity are asymptotically uncorrelated, which suggests the need for inference directly on modularity itself when the network is large. To this end, we derive the asymptotic distribution of modularity in the case where the network’s edge weight matrix belongs to the Gaussian orthogonal ensemble, and study the statistical power of the corresponding test for community structure under some alternative models. We empirically explore universality extensions of the limiting distribution and demonstrate the accuracy of these asymptotic distributions through Type I error simulations. We also compare the empirical powers of the modularity-based tests and some existing methods. Our method is then used to test for the presence of community structure in two real data applications.

Sign in / Sign up

Export Citation Format

Share Document