scholarly journals Influence of Context on Item Parameters in Forced-Choice Personality Assessments

2016 ◽  
Vol 77 (3) ◽  
pp. 389-414 ◽  
Author(s):  
Yin Lin ◽  
Anna Brown
Keyword(s):  
Small Proportion ◽  
Computerized Adaptive Testing ◽  
Adaptive Testing ◽  
Forced Choice ◽  
Item Parameter ◽  
Parameter Estimates ◽  
Personality Assessments ◽  
Fundamental Assumption ◽  
Item Parameters ◽  
Item Parameter Estimates

A fundamental assumption in computerized adaptive testing is that item parameters are invariant with respect to context—items surrounding the administered item. This assumption, however, may not hold in forced-choice (FC) assessments, where explicit comparisons are made between items included in the same block. We empirically examined the influence of context on item parameters by comparing parameter estimates from two FC instruments. The first instrument was composed of blocks of three items, whereas in the second, the context was manipulated by adding one item to each block, resulting in blocks of four. The item parameter estimates were highly similar. However, a small number of significant deviations were observed, confirming the importance of context when designing adaptive FC assessments. Two patterns of such deviations were identified, and methods to reduce their occurrences in an FC computerized adaptive testing setting were proposed. It was shown that with a small proportion of violations of the parameter invariance assumption, score estimation remained stable.

Benefits from Computerized Adaptive Testing as Seen in Simulation Studies

1999 ◽  
Vol 15 (2) ◽  
pp. 91-98 ◽  
Author(s):  
Lutz F. Hornke
Keyword(s):  
Measurement Error ◽  
Computerized Adaptive Testing ◽  
Test Procedure ◽  
Adaptive Testing ◽  
Parameter Estimates ◽  
Simulation Studies ◽  
Computerized Adaptive Test ◽  
Item Banks ◽  
Item Parameters ◽  
General Reliability

Summary: Item parameters for several hundreds of items were estimated based on empirical data from several thousands of subjects. The logistic one-parameter (1PL) and two-parameter (2PL) model estimates were evaluated. However, model fit showed that only a subset of items complied sufficiently, so that the remaining ones were assembled in well-fitting item banks. In several simulation studies 5000 simulated responses were generated in accordance with a computerized adaptive test procedure along with person parameters. A general reliability of .80 or a standard error of measurement of .44 was used as a stopping rule to end CAT testing. We also recorded how often each item was used by all simulees. Person-parameter estimates based on CAT correlated higher than .90 with true values simulated. For all 1PL fitting item banks most simulees used more than 20 items but less than 30 items to reach the pre-set level of measurement error. However, testing based on item banks that complied to the 2PL revealed that, on average, only 10 items were sufficient to end testing at the same measurement error level. Both clearly demonstrate the precision and economy of computerized adaptive testing. Empirical evaluations from everyday uses will show whether these trends will hold up in practice. If so, CAT will become possible and reasonable with some 150 well-calibrated 2PL items.

Small Sample Estimation in Dichotomous Item Response Models: Effect of Priors Based on Judgmental Information on the Accuracy of Item Parameter Estimates

2003 ◽  
Vol 27 (1) ◽  
pp. 27-51 ◽  
Author(s):  
Hariharan Swaminathan ◽  
Ronald K. Hambleton ◽  
Stephen G. Sireci ◽  
Dehui Xing ◽  
Saba M. Rizavi
Keyword(s):  
Prior Information ◽  
Small Sample ◽  
Item Parameter ◽  
Small Samples ◽  
Future Research ◽  
Parameter Estimates ◽  
Test Items ◽  
Item Parameters ◽  
Item Parameter Estimates ◽  
The One

Large item banks with properly calibrated test items are essential for ensuring the validity of computer-based tests. At the same time, item calibrations with small samples are desirable to minimize the amount of pretesting and limit item exposure. Bayesian estimation procedures show considerable promise with small examinee samples. The purposes of the study were (a) to examine how prior information for Bayesian item parameter estimation can be specified and (b) to investigate the relationship between sample size and the specification of prior information on the accuracy of item parameter estimates. The results of the simulation study were clear: Estimation of IRT model item parameters can be improved considerably. Improvements in the one-parameter model were modest; considerable improvements with the two- and three-parameter models were observed. Both the study of different forms of priors and ways to improve the judgmental data used in forming the priors appear to be promising directions for future research.

Detecting Noneffortful Responses Based on a Residual Method Using an Iterative Purification Process

2021 ◽  
pp. 107699862199436
Author(s):  
Yue Liu ◽  
Hongyun Liu
Keyword(s):  
True Positive Rate ◽  
Item Parameter ◽  
Parameter Estimates ◽  
Detection Accuracy ◽  
Purification Process ◽  
Parameter Recovery ◽  
Residual Method ◽  
Data Contamination ◽  
Item Parameters ◽  
The Impact

The prevalence and serious consequences of noneffortful responses from unmotivated examinees are well-known in educational measurement. In this study, we propose to apply an iterative purification process based on a response time residual method with fixed item parameter estimates to detect noneffortful responses. The proposed method is compared with the traditional residual method and noniterative method with fixed item parameters in two simulation studies in terms of noneffort detection accuracy and parameter recovery. The results show that when severity of noneffort is high, the proposed method leads to a much higher true positive rate with a small increase of false discovery rate. In addition, parameter estimation is significantly improved by the strategies of fixing item parameters and iteratively cleansing. These results suggest that the proposed method is a potential solution to reduce the impact of data contamination due to severe low test-taking effort and to obtain more accurate parameter estimates. An empirical study is also conducted to show the differences in the detection rate and parameter estimates among different approaches.

scholarly journals Evaluating Robust Scale Transformation Methods With Multiple Outlying Common Items Under IRT True Score Equating

2019 ◽  
Vol 44 (4) ◽  
pp. 296-310
Author(s):  
Yong He ◽  
Zhongmin Cui
Keyword(s):  
Item Parameter ◽  
Scale Transformation ◽  
Parameter Estimates ◽  
Test Form ◽  
Multiple Outliers ◽  
Common Item ◽  
Item Parameter Estimates ◽  
True Score Equating ◽  
Common Items ◽  
Transformation Methods

Item parameter estimates of a common item on a new test form may change abnormally due to reasons such as item overexposure or change of curriculum. A common item, whose change does not fit the pattern implied by the normally behaved common items, is defined as an outlier. Although improving equating accuracy, detecting and eliminating of outliers may cause a content imbalance among common items. Robust scale transformation methods have recently been proposed to solve this problem when only one outlier is present in the data, although it is not uncommon to see multiple outliers in practice. In this simulation study, the authors examined the robust scale transformation methods under conditions where there were multiple outlying common items. Results indicated that the robust scale transformation methods could reduce the influences of multiple outliers on scale transformation and equating. The robust methods performed similarly to a traditional outlier detection and elimination method in terms of reducing the influence of outliers while keeping adequate content balance.

scholarly journals The Impact of Item Calibration Error on Variable-Length Cognitive Diagnostic Computerized Adaptive Testing

2020 ◽  
Vol 11 ◽  
Author(s):  
Xiaojian Sun ◽  
Yanlou Liu ◽  
Tao Xin ◽  
Naiqing Song
Keyword(s):  
Adverse Effects ◽  
Measurement Accuracy ◽  
Computerized Adaptive Testing ◽  
Adaptive Testing ◽  
Unified Model ◽  
Variable Length ◽  
Calibration Error ◽  
Test Length ◽  
Test Efficiency ◽  
Item Parameters

Calibration errors are inevitable and should not be ignored during the estimation of item parameters. Items with calibration error can affect the measurement results of tests. One of the purposes of the current study is to investigate the impacts of the calibration errors during the estimation of item parameters on the measurement accuracy, average test length, and test efficiency for variable-length cognitive diagnostic computerized adaptive testing. The other purpose is to examine the methods for reducing the adverse effects of calibration errors. Simulation results show that (1) calibration error has negative effect on the measurement accuracy for the deterministic input, noisy “and” gate (DINA) model, and the reduced reparameterized unified model; (2) the average test lengths is shorter, and the test efficiency is overestimated for items with calibration errors; (3) the compensatory reparameterized unified model (CRUM) is less affected by the calibration errors, and the classification accuracy, average test length, and test efficiency are slightly stable in the CRUM framework; (4) methods such as improving the quality of items, using large calibration sample to calibrate the parameters of items, as well as using cross-validation method can reduce the adverse effects of calibration errors on CD-CAT.

scholarly journals Conditional Maximum Likelihood Estimation in Probability-Branched Multistage Designs

2021 ◽  
Author(s):  
Jan Steinfeld ◽  
Alexander Robitzsch
Keyword(s):  
Parameter Estimation ◽  
Maximum Likelihood ◽  
Likelihood Estimation ◽  
Parameter Estimate ◽  
Item Parameter ◽  
Parameter Estimates ◽  
Conditional Maximum Likelihood ◽  
Item Parameter Estimates ◽  
Item Parameter Estimation ◽  
Conditional Maximum

This article describes the conditional maximum likelihood-based item parameter estimation in probabilistic multistage designs. In probabilistic multistage designs, the routing is not solely based on a raw score j and a cut score c as well as a rule for routing into a module such as j < c or j ≤ c but is based on a probability p(j) for each raw score j. It can be shown that the use of a conventional conditional maximum likelihood parameter estimate in multistage designs leads to severely biased item parameter estimates. Zwitser and Maris (2013) were able to show that with deterministic routing, the integration of the design into the item parameter estimation leads to unbiased estimates. This article extends this approach to probabilistic routing and, at the same time, represents a generalization. In a simulation study, it is shown that the item parameter estimation in probabilistic designs leads to unbiased item parameter estimates.

Sign in / Sign up

Export Citation Format

Share Document