Measuring language ability of students with compensatory multidimensional CAT: A post-hoc simulation study

The computerized adaptive tests (CAT) apply an adaptive process in which the items are tailored to individuals' ability scores. The multidimensional CAT (MCAT) designs differ in terms of different item selection, ability estimation, and termination methods being used. This study aims at investigating the performance of the MCAT designs used to measure the language ability of students and to compare the results of MCAT designs with the outcomes of corresponding paper–pencil tests. For this purpose, items in the English Proficiency Tests (EPT) were used to create a multi-dimensional item pool that consists of 599 items. The performance of the MCAT designs was evaluated and compared based on the reliability coefficients, root means square error (RMSE), test-length, and root means squared difference (RMSD) statistics, respectively. Therefore, 36 different conditions were investigated in total. The results of the post-hoc simulation designs indicate that the MCAT designs with the A-optimality item selection method outperformed MCAT designs with other item selection methods by decreasing the test length and RMSD values without any sacrifice in test reliability. Additionally, the best error variance stopping rule for each MCAT algorithm with A-optimality item selection could be considered as 0.25 with 27.9 average test length and 30 items for the fixed test-length stopping rule for the Bayesian MAP method. Overall, MCAT designs tend to decrease the test length by 60 to 65 percent and provide ability estimations with higher precision compared to the traditional paper–pencil tests with 65 to 75 items. Therefore, it is suggested to use the A-optimality method for item selection and the Bayesian MAP method for ability estimation for the MCAT designs since the MCAT algorithm with these specifications shows better performance than others.

Data Driven Monitoring in Community Based Management of SAM children using Psychometric Techniques: An Operational Framework

Background- The success of the Community Based Management of Severe Malnutrition (CSAM)programme, largely depends on the knowledge and skills of Front-Line Workers (FLWs).A robust supportive supervision system in CSAM should be tailored to individualistic learning needs by distinguishing the FLWs as per their ability and simultaneously identifying the task domains to be emphasized more in supervisory visits.This paper details the ability assessment strategy developed and employed in the selected geographical locations in state of Madhya Pradesh (Central India) among the 197 Anganwadi workers (FLWs involved in CSAM implementation) Methodology-. A 25 items tool was developed based on an analytical construct for ability estimation through Rasch Analysis (RA). RA models the probability of right/wrong answer as a function of person(participants) and item (questions) parameters and calculates the item difficulty in relation with person ability on same unidimensional linear scale. The fitting of the data to Rasch model (Rasch diagnostic) was tested by both numeric (Anderson LR and Wald test) and graphical method. Suitable visualization like Item Characteristic Curve (ICC) and Person Item Map (PIM) were plotted in RA. Further a quadratic allocation of all AWWs into 4 quadrants were done as per the ability estimation (Rasch score) and adjusted numbers of SAM/MAM children in her center. Results-. . The item easiness parameter (β) value related to Diarrhoeal assessment was lowest (-2.32, -2.91 to -1.73) and related to peer assessment consequential action (2.009, 1.669- 2.348)) was highest (most difficult). Anderson LR test (LR=31.32, df=24, p=0.079) showed the absence of global outliers. Quadrant analysis using the permutations of ability score and adjusted burden of malnutrition further mapped 41/197 (20.8%) FLWs to low ability -high burden quadrant and 44/197(25%) as low ability low burden quadrant. Conclusion- RASCH assessment may address the innate challenges to maintain homogeneity, discrimination capacity and linearity in a raw score-based measurement construct. The monitoring strategy developed on this thus may offer a judicious, pragmatic and thematic approach to supportive supervision in CSAM program. Keywords: Severe Acute Malnutrition, RASCH assessment, data driven monitoring

Monitoring Item Performance With CUSUM Statistics in Continuous Testing

In many educational assessments, items are reused in different administrations throughout the life of the assessments. Ideally, a reused item should perform relatively similarly over time. In reality, an item may become easier with exposure, especially when item preknowledge has occurred. This article presents a novel cumulative sum procedure for detecting item preknowledge in continuous testing where data for each reused item may be obtained from small and varying sample sizes across administrations. Its performance is evaluated with simulations and analytical work. The approach is effective in detecting item preknowledge quickly with group size at least 10 and is easy to implement with varying item parameters. In addition, it is robust to the ability estimation error introduced in the simulations.

Adaptive Weight Estimation of Latent Ability: Application to Computerized Adaptive Testing With Response Revision

An adaptive weight estimation approach is proposed to provide robust latent ability estimation in computerized adaptive testing (CAT) with response revision. This approach assigns different weights to each distinct response to the same item when response revision is allowed in CAT. Two types of weight estimation procedures, nonfunctional and functional weight, are proposed to determine the weight adaptively based on the compatibility of each revised response with the assumed statistical model in relation to remaining observations. The application of this estimation approach to a data set collected from a large-scale multistage adaptive testing demonstrates the capability of this method to reveal more information regarding the test taker’s latent ability by using the valid response path compared with only using the very last response. Limited simulation studies were concluded to evaluate the proposed ability estimation method and to compare it with several other estimation procedures in literature. Results indicate that the proposed ability estimation approach is able to provide robust estimation results in two test-taking scenarios.