The Reliability of the Posterior Probability of Skill Attainment in Diagnostic Classification Models

One common score reported from diagnostic classification assessments is the vector of posterior means of the skill mastery indicators. As with any assessment, it is important to derive and report estimates of the reliability of the reported scores. After reviewing a reliability measure suggested by Templin and Bradshaw, this article suggests three new measures of reliability of the posterior means of skill mastery indicators and methods for estimating the measures when the number of items on the assessment and the number of skills being assessed render exact calculation computationally burdensome. The utility of the new measures is demonstrated using simulated and real data examples. Two of the suggested measures are recommended for future use.

Download Full-text

A Variational Bayesian Inference Posterior Approximation Algorithm for Hidden Markov Diagnostic Classification Models

10.31234/osf.io/28jkf ◽

2021 ◽

Author(s):

Kazuhiro Yamaguchi ◽

Alfonso J. Martinez

Keyword(s):

Hidden Markov ◽

Cognitive Learning ◽

Real Data ◽

Diagnostic Classification ◽

Mcmc Methods ◽

Parameter Estimates ◽

Classification Models ◽

Diagnostic Classification Models ◽

Variational Bayesian ◽

Posterior Approximation

General diagnostic classification models (DCMs) can be used to capture individual students’ cognitive learning status. Moreover, DCMs for longitudinal data are appropriate to track students transition of cognitive elements. This study developed an effective Bayesian posterior approximation method called variational Bayesian (VB) inference method for hidden Markov type longitudinal general DCMs. Simulation study indicated the proposed algorithm could satisfactorily recover true parameters. Comparative study of the VB and previously developed Markov chain Monte Carlo (MCMC) methods was conducted in real data example. The result revealed that the VB method provided similar parameter estimates to the MCMC with faster estimation time.

Download Full-text

Test-level and Item-level Model Fit Comparison of General vs. Specific Diagnostic Classification Models: A Case of True DCM

Language Testing in Asia ◽

10.1186/s40468-021-00148-z ◽

2021 ◽

Vol 11 (1) ◽

Author(s):

Mahdieh Shafipoor ◽

Hamdollah Ravand ◽

Parviz Maftoon

Keyword(s):

Model Fit ◽

Diagnostic Classification ◽

Fit Indices ◽

Classification Models ◽

Diagnostic Classification Models ◽

Item Level ◽

Model Fit Indices ◽

Skill Mastery ◽

Level Model ◽

Test Level

AbstractThe current study compared the model fit indices, skill mastery probabilities, and classification accuracy of six Diagnostic Classification Models (DCMs): a general model (G-DINA) against five specific models (LLM, RRUM, ACDM, DINA, and DINO). To do so, the response data to the grammar and vocabulary sections of a General English Achievement Test, designed specifically for cognitive diagnostic purposes from scratch, was analyzed. The results of the test-level-model fit values obtained strong evidence in supporting the G-DINA and LLM models possessing the best model fit. In addition, the ACDM and RRUM were almost very identical to that of the G-DINA. The value indices of the DINO and DINA models were very close to each other but larger than those of the G-DINA and LLM. The model fit was also investigated at the item level, and the results revealed that model selection should be performed at the item level rather than the test level, and most of the specific models might perform well for the test. The findings of this study suggested that the relationships among the attributes of grammar and vocabulary are not ‘either-or’ compensatory or non-compensatory but a combination of both.

Download Full-text

The accuracy and consistency of mastery for each content domain using the Rasch and deterministic inputs, noisy “and” gate diagnostic classification models: a simulation study and a real-world analysis using data from the Korean Medical Licensing Examination

Journal of Educational Evaluation for Health Professions ◽

10.3352/jeehp.2021.18.15 ◽

2021 ◽

Vol 18 ◽

pp. 15

Author(s):

Dong Gi Seo ◽

Jae Kum Kim

Keyword(s):

Simulation Study ◽

Real Data ◽

Diagnostic Classification ◽

Rasch Models ◽

Classification Models ◽

Diagnostic Classification Models ◽

Dina Model ◽

Medical Licensing ◽

Licensing Examination ◽

High Level

Purpose: Diagnostic classification models (DCMs) were developed to identify the mastery or non-mastery of the attributes required for solving test items, but their application has been limited to very low-level attributes, and the accuracy and consistency of high-level attributes using DCMs have rarely been reported compared with classical test theory (CTT) and item response theory models. This paper compared the accuracy of high-level attribute mastery between deterministic inputs, noisy “and” gate (DINA) and Rasch models, along with sub-scores based on CTT.Methods: First, a simulation study explored the effects of attribute length (number of items per attribute) and the correlations among attributes with respect to the accuracy of mastery. Second, a real-data study examined model and item fit and investigated the consistency of mastery for each attribute among the 3 models using the 2017 Korean Medical Licensing Examination with 360 items.Results: Accuracy of mastery increased with a higher number of items measuring each attribute across all conditions. The DINA model was more accurate than the CTT and Rasch models for attributes with high correlations (>0.5) and few items. In the real-data analysis, the DINA and Rasch models generally showed better item fits and appropriate model fit. The consistency of mastery between the Rasch and DINA models ranged from 0.541 to 0.633 and the correlations of person attribute scores between the Rasch and DINA models ranged from 0.579 to 0.786.Conclusion: Although all 3 models provide a mastery decision for each examinee, the individual mastery profile using the DINA model provides more accurate decisions for attributes with high correlations than the CTT and Rasch models. The DINA model can also be directly applied to tests with complex structures, unlike the CTT and Rasch models, and it provides different diagnostic information from the CTT and Rasch models.

Download Full-text