Validating a Reading Assessment Within the Cognitive Diagnostic Assessment Framework: Q-Matrix Construction and Model Comparisons for Different Primary Grades

Cognitive diagnostic assessment (CDA) has been developed rapidly to provide fine-grained diagnostic feedback on students’ subskills and to provide insights on remedial instructions in specific domains. To date, most cognitive diagnostic studies on reading tests have focused on retrofitting a single booklet from a large-scale assessment (e.g., PISA and PIRLS). Critical issues in CDA involve the scarcity of research to develop diagnostic tests and the lack of reliability and validity evidence. This study explored the development and validation of the Diagnostic Chinese Reading Comprehension Assessment (DCRCA) for primary students under the CDA framework. Reading attributes were synthesized based on a literature review, the national curriculum criteria, the results of expert panel judgments, and student think-aloud protocols. Then, the tentative attributes were used to construct three booklets of reading comprehension items for 2–6 graders at three key stages. The assessment was administered to a large population of students (N = 21,466) in grades 2–6 from 20 schools in a district of Changchun City, China. Q-matrices were compared and refined using the model-data fit and an empirical validation procedure, and five representative cognitive diagnostic models (CDMs) were compared for optimal performance. The fit indices suggested that a six-attribute structure and the G-DINA model were best fitted for the reading comprehension assessment. In addition, diagnostic reliability, construct, internal and external validity results were provided, supporting CDM classifications as reliable, accurate, and useful. Such diagnostic information could be utilized by students, teachers, and administrators of reading programs and instructions.

Download Full-text

Developing individualized feedback for listening assessment: Combining standard setting and cognitive diagnostic assessment approaches

Language Testing ◽

10.1177/0265532221995475 ◽

2021 ◽

pp. 026553222199547

Author(s):

Shangchao Min ◽

Lianzhen He

Keyword(s):

Large Scale ◽

Standard Setting ◽

Diagnostic Assessment ◽

Efl Learners ◽

Cognitive Diagnostic Assessment ◽

Item Level ◽

Language Assessments ◽

Individualized Feedback ◽

Proficiency Classifications ◽

Acceptable Reliability

In this study, we present the development of individualized feedback for a large-scale listening assessment by combining standard setting and cognitive diagnostic assessment (CDA) approaches. We used the performance data from 3358 students’ item-level responses to a field test of a national EFL test primarily intended for tertiary-level EFL learners. The results showed that proficiency classifications and subskill mastery classifications were generally of acceptable reliability, and the two kinds of classifications were in alignment with each other at individual and group levels. The outcome of the study is a set of descriptors that describe each test taker’s ability to understand certain level of oral texts and his or her cognitive performance. The current study, by illustrating the feasibility of combining standard setting and CDA approaches to produce individualized feedback, contributes to the enhancement of score reporting and addresses the long-standing criticism that large-scale language assessments fail to provide individualized feedback to link assessment with instruction.

Download Full-text

Introduction to Reliability and Validity of International Large-Scale Assessment

IEA Research for Education - Reliability and Validity of International Large-Scale Assessment ◽

10.1007/978-3-030-53081-5_1 ◽

2020 ◽

pp. 1-5

Author(s):

Hans Wagemaker

Keyword(s):

Large Scale ◽

Reliability And Validity ◽

Large Scale Assessment ◽

Scale Assessment

Download Full-text

Reliability and Validity of International Large-Scale Assessment

10.1007/978-3-030-53081-5 ◽

2020 ◽

Keyword(s):

Large Scale ◽

Reliability And Validity ◽

Large Scale Assessment ◽

Scale Assessment

Download Full-text

ALIGNMENT OF 6TH GRADE LARGE-SCALE ASSESSMENT CONSTRUCTS WITH THE REVISED CURRICULUM FRAMEWORK

SOCIETY INTEGRATION EDUCATION Proceedings of the International Scientific Conference ◽

10.17770/sie2019vol2.3811 ◽

2019 ◽

Vol 2 ◽

pp. 387

Author(s):

Pāvels Pestovs ◽

Dace Namsone ◽

Līga Čakāne ◽

Ilze Saleniece

Keyword(s):

Cognitive Skills ◽

Large Scale ◽

National Curriculum ◽

National Development ◽

Theoretical Framework ◽

Test Theory ◽

Test Items ◽

Large Scale Assessment ◽

International Student Assessment ◽

Scale Assessment

One of the goals of the National Development Plan 2014-2020 is to reduce the proportion of students with low cognitive skills, and at the same time increase the proportion of students with higher level cognitive skills. In line with those goals, the National Centre for Education is implementing the project “Competency-based approach to curriculum”, funded by the European Social Fund. The purpose of the research described in this article is to find out to what extent the current large-scale national assessments for 6th Grade are coherent with the new curriculum and what improvements are needed for aligning the national assessments with the national curriculum. The theoretical framework of the research is developed by analysing the frameworks of the programme for international student assessment (PISA), trends in international mathematics and science study (TIMSS), progress in international reading literacy study (PIRLS), as well as the framework of the revised national curriculum in Latvia. National 6th Grade assessments of the year 2018 are analysed by using Classical test theory and Rasch model. The indicators of the test items are mapped according to the developed theoretical framework. Authors conclude that the national 6th Grade tests assess the elements of literacy, numeracy and scientific literacy. Students have a high level of performance in test items with low cognitive depths, but there is an insufficient number of test items with high cognitive depths, allowing pupils to demonstrate skills in new contexts, which is an essential goal of the new national curriculum. Further research is required on the use of data from the large-scale assessment in supporting and guiding student instruction and learning.

Download Full-text

Reading Comprehension in Problems Solving in Mathematics Test of Prova Brasil

Signum Estudos da Linguagem ◽

10.5433/2237-4876.2019v22n1p88 ◽

2019 ◽

Vol 22 (1) ◽

pp. 88

Author(s):

Alessandra Pereira Gomes Machado ◽

Andréa Maria dos Santos Matos

Keyword(s):

Reading Comprehension ◽

Problem Solving ◽

Large Scale ◽

Standardized Test ◽

Mathematical Problem ◽

Large Scale Assessment ◽

Reading Abilities ◽

Mathematics Test ◽

Comprehension Skills ◽

Mathematical Problems

Difficulty in reading comprehension and problem solving presented in the results of large- scale assessment may be in basic cognitive processes of reading. Researches suggest the relation between reading comprehension skills and mathematical problem solving. The aim of this article is to present the articulation between reading and problem solving, considering the cognitive dimension, and to accentuate the importance of automaticity and reading comprehension for solving problems in the Mathematics test of Prova Brasil. We analysed the descriptors of three topics of the reference matrix of Portuguese Language that indicate the reading abilities in items of Prova Brasil of a standardized test, elaborated in order to emphasize the need for the student to have developed the processes of reading comprehension to be successful in solving mathematical problems.

Download Full-text

Examining the L2 reading comprehension ability of adult ELLs: Developing a diagnostic test within the cognitive diagnostic assessment framework

Language Testing ◽

10.1177/0265532220941470 ◽

2020 ◽

pp. 026553222094147

Author(s):

Tugba Elif Toprak ◽

Abdulvahit Cakir

Keyword(s):

Reading Comprehension ◽

Cognitive Model ◽

Diagnostic Assessment ◽

Classification Model ◽

L2 Reading ◽

Fine Grained ◽

Cognitive Diagnostic Assessment ◽

Comprehension Ability ◽

Log Linear ◽

L2 Reading Comprehension

Cognitive diagnostic assessment (CDA) has been applied to language assessment in a number of studies in which a diagnostic classification model (DCM) was retrofitted to the results of a non-diagnostic assessment. However, the need to apply CDA through utilization of an inductive rather than a retrofitted approach has been a recurrent theme in these studies. Thus, this study aimed to develop a diagnostic L2 reading comprehension test in English to investigate adult examinees’ reading performances in an EFL (English as a Foreign Language) academic setting. The test was based on a cognitive model of L2 reading comprehension and was administered to a sample of 1058 examinees across Turkey. The results were analyzed using log-linear cognitive diagnosis modeling (LCDM), which is one of the general DCM families subsuming other core DCMs. The findings of the study indicated that obtaining fine-grained diagnostic information about examinees’ performances in a given domain would be possible by coupling an adequate understanding of the construct with a CDA framework.

Download Full-text