A Diagnostic Classification Analysis of Problem-Solving Competence using Process Data
Process data refers to data recorded by computer-based assessments (CBA) that reflect respondents’ problem-solving processes and provide greater insight into how students solve problems, instead of merely how well they solve them. Using the rich information contained in process data, this study proposed an item-specific psychometric method for analyzing process data in order to comprehensively understand respondents’ problem-solving competence. By incorporating diagnostic classification into process data analysis, the proposed method cannot only estimate respondents’ problem-solving ability along a continuum, but can also classify respondents according to their problem-solving strategies. To illustrate the application and advantages of the proposed method, a Programme for International Student Assessment (PISA) problem-solving task was used. The results indicated that (a) the estimated latent classes provided more detailed diagnoses of respondents’ problem-solving strategies than the observed score classes; (b) although only one item was used, estimated higher-order latent ability reflected the respondents’ problem-solving ability more accurately than the estimated unidimensional latent ability taken from the outcome data; and (c) the interactions between problem-solving skills may follow the conjunctive condensation rule, which assumes that only when a respondent has mastered all the required problem-solving skills can the specific action sequence appear. Overall, the main conclusion drawn from this study was that using diagnostic classification is a feasible and promising method for analyzing process data.