Application of Change Point Analysis of Response Time Data to Detect Test Speededness

Computer-based and web-based testing have become increasingly popular in recent years. Their popularity has dramatically expanded the availability of response time data. Compared to the conventional item response data that are often dichotomous or polytomous, response time has the advantage of being continuous and can be collected in an unobstrusive manner. It therefore has great potential to improve many measurement activities. In this paper, we propose a change point analysis (CPA) procedure to detect test speededness using response time data. Specifically, two test statistics based on CPA, the likelihood ratio test and Wald test, are proposed to detect test speededness. A simulation study has been conducted to evaluate the performance of the proposed CPA procedure, as well as the use of asymptotic and empirical critical values. Results indicate that the proposed procedure leads to high power in detecting test speededness, while keeping the false positive rate under control, even when simplistic and liberal critical values are used. Accuracy of the estimation of the actual change point, however, is highly dependent on the true change point. A real data example is also provided to illustrate the utility of the proposed procedure and its contrast to the response-only procedure. Implications of the findings are discussed at the end.

Clarifying the Effect of Test Speededness

In the context of high-stakes tests, test takers who do not have enough time to complete a test rush toward the end and may engage in speeded behavior when tests do not penalize guessing. Using mathematical derivations and simulations, previous research showed that random guessing responses should attenuate interitem correlations, and therefore, decrease estimates of reliability. Meanwhile, other researchers showed that random guessing could in fact inflate reliability estimates using real data. We provide analytical derivations on how speededness could affect correlations between two dichotomous items in multiple ways, depending on the manifestation and prevalence of test speededness. Furthermore, we provide two simulation studies that evaluate the magnitude of impact of test speededness on interitem correlations and Cronbach’s alpha. We found that the impact of test speededness can vary between item pairs and that it depends on the manifestation of test speededness and item level characteristics. Furthermore, speeded responses will, in general, attenuate or not affect reliability estimates, depending on the prevalence of such responses and conceptual interpretation of speeded responses. Implications of the findings are discussed.

Clarifying the Effect of Test Speededness

In the context of high-stakes tests, test takers who do not have enough time to complete a test rush towards the end and may engage in speeded behavior when tests do not penalize guessing. Using mathematical derivations and simulations, Attali (2005) showed that random guessing responses should attenuate inter-item correlations, and therefore decrease estimates of reliability. Meanwhile, Wise and DeMars (2009) showed that random guessing could in fact inflate reliability estimates using real data. We provide analytical derivations on how speededness could affect correlations between two dichotomous items in multiple ways, depending on the manifestation and prevalence of test speededness. Furthermore, we provide two simulation studies that evaluate the magnitude of impact of test speededness on inter-item correlations and Cronbach's Alpha. We found that the impact of test speededness can vary between item pairs and that it depends on the manifestation of test speededness and item level characteristics. Furthermore, speeded responses will, in general, attenuate or not affect reliability estimates, depending on the prevalence of such responses and conceptual interpretation of speeded responses. Implications of the findings are discussed.

Evaluating the Accuracy of the Empirical Item Characteristic Curve Preequating Method in the Presence of Test Speededness

This study aimed to assess the accuracy of the empirical item characteristic curve (EICC) preequating method given the presence of test speededness. The simulation design of this study considered the proportion of speededness, speededness point, speededness rate, proportion of missing on speeded items, sample size, and test length. After crossing all of the manipulated factors and then normalizing the evaluation criteria (bias and root mean square difference [ RMSD]) with regard to test length, the results revealed that (1) when test speededness was present, conversions from the EICC preequating method tended to be positively distorted; (2) no practically meaningful moderation effect associated with sample size was found on the relationship between test speededness and the accuracy of EICC preequating; and (3) the location of the speededness point was the driving factor in terms of its impact on the accuracy of EICC preequating. Implications and suggestions were discussed.