Nonparametric CAT for CD in Educational Settings With Small Samples

Cognitive diagnostic computerized adaptive testing (CD-CAT) has been suggested by researchers as a diagnostic tool for assessment and evaluation. Although model-based CD-CAT is relatively well researched in the context of large-scale assessment systems, this type of system has not received the same degree of research and development in small-scale settings, such as at the course-based level, where this system would be the most useful. The main obstacle is that the statistical estimation techniques that are successfully applied within the context of a large-scale assessment require large samples to guarantee reliable calibration of the item parameters and an accurate estimation of the examinees’ proficiency class membership. Such samples are simply not obtainable in course-based settings. Therefore, the nonparametric item selection (NPS) method that does not require any parameter calibration, and thus, can be used in small educational programs is proposed in the study. The proposed nonparametric CD-CAT uses the nonparametric classification (NPC) method to estimate an examinee’s attribute profile and based on the examinee’s item responses, the item that can best discriminate the estimated attribute profile and the other attribute profiles is then selected. The simulation results show that the NPS method outperformed the compared parametric CD-CAT algorithms and the differences were substantial when the calibration samples were small.

Download Full-text

On the Treatment of Missing Item Responses in Educational Large-scale Assessment Data: The Case of PISA 2018 Mathematics

10.20944/preprints202110.0107.v1 ◽

2021 ◽

Author(s):

Alexander Robitzsch

Keyword(s):

Missing Data ◽

Large Scale ◽

Model Fit ◽

Large Scale Assessment ◽

Missing Data Treatments ◽

Scale Assessment ◽

Scaling Models ◽

Item Responses ◽

Response Propensity ◽

Missing Item

Missing item responses are prevalent in educational large-scale assessment studies like the programme for international student assessment (PISA). The current operational practice scores missing item responses as wrong, but several psychometricians advocated a model-based treatment based on latent ignorability assumption. In this approach, item responses and response indicators are jointly modeled conditional on a latent ability and a latent response propensity variable. Alternatively, imputation-based approaches can be used. The latent ignorability assumption is weakened in the Mislevy-Wu model that characterizes a nonignorable missingness mechanism and allows the missingness of an item to depend on the item itself. The scoring of missing item responses as wrong and the latent ignorable model are submodels of the Mislevy-Wu model. This article uses the PISA 2018 mathematics dataset to investigate the consequences of different missing data treatments on country means. Obtained country means can substantially differ for the different scaling models. In contrast to previous statements in the literature, the scoring of missing item responses as incorrect provided a better model fit than a latent ignorable model for most countries. Furthermore, the dependence of the missingness of an item from the item itself after conditioning on the latent response propensity was much more pronounced for constructed-response items than for multiple-choice items. As a consequence, scaling models that presuppose latent ignorability should be refused from two perspectives. First, the Mislevy-Wu model is preferred over the latent ignorable model for reasons of model fit. Second, we argue that model fit should only play a minor role in choosing psychometric models in large-scale assessment studies because validity aspects are most relevant. Missing data treatments that countries can simply manipulate (and, hence, their students) result in unfair country comparisons.

Download Full-text

Intasc Standard Cores: Raising Students’ English Modality Competence

Lingua Cultura ◽

10.21512/lc.v10i2.929 ◽

2016 ◽

Vol 10 (2) ◽

pp. 111

Author(s):

Muliani Muliani

Keyword(s):

Content Knowledge ◽

Large Scale ◽

Small Scale ◽

Teaching Model ◽

Teaching English ◽

Tense And Aspect ◽

Large Scale Assessment ◽

Scale Assessment ◽

Assessment Standard ◽

Validated Instrument

This research aimed at raising the students’ modality competence with the implementation of a teaching model which was called as Interstate New Teachers Assessment and Support Consortium (INTASC) model that covering ten standards. It was expected that this research could give numerous contribution in teaching English, particularly in teaching English Modality where the problem found was that the students got difficulty in using modal verbs regarding both tense and aspect in which consequently would affect the communicative competence of the students. In the form of Research and Development, this research was carried by means of implementing validated instrument and 10 modules in the small and large scale assessments that involving 50 students in the small scale assessment and 80 students in the large-scale assessment. Standard 1-2 dealt with the students’ need and diversity of learning while standard 3-7 dealt with various instructions teaching the content knowledge regarding the use of English modality. Furthermore, standard 8-10 dealt with summative assessment, reflection, and professional development. Eventually, it is found that the level of learning of the students raise supported by the data that 94% of the level of learning can be achieved by the students while it was only 6% of the modality expressions cannot be used properly. It can be noted that this teaching model can assist the students in achieving the modality competence by having a very well-sequenced procedures of teaching in which this teaching model starts from considering the prior knowledge, the need, and the students’ diversity before creating further instructions regarding the content knowledge where the modality competence is the main goal to achieve.

Download Full-text

On the Treatment of Missing Item Responses in Educational Large-Scale Assessment Data: An Illustrative Simulation Study and a Case Study Using PISA 2018 Mathematics Data

European Journal of Investigation in Health, Psychology and Education ◽

10.3390/ejihpe11040117 ◽

2021 ◽

Vol 11 (4) ◽

pp. 1653-1687

Author(s):

Alexander Robitzsch

Keyword(s):

Large Scale ◽

Model Fit ◽

Large Scale Assessment ◽

Missing Data Treatments ◽

Scale Assessment ◽

Scaling Models ◽

Item Responses ◽

Response Propensity ◽

Missing Item

Missing item responses are prevalent in educational large-scale assessment studies such as the programme for international student assessment (PISA). The current operational practice scores missing item responses as wrong, but several psychometricians have advocated for a model-based treatment based on latent ignorability assumption. In this approach, item responses and response indicators are jointly modeled conditional on a latent ability and a latent response propensity variable. Alternatively, imputation-based approaches can be used. The latent ignorability assumption is weakened in the Mislevy-Wu model that characterizes a nonignorable missingness mechanism and allows the missingness of an item to depend on the item itself. The scoring of missing item responses as wrong and the latent ignorable model are submodels of the Mislevy-Wu model. In an illustrative simulation study, it is shown that the Mislevy-Wu model provides unbiased model parameters. Moreover, the simulation replicates the finding from various simulation studies from the literature that scoring missing item responses as wrong provides biased estimates if the latent ignorability assumption holds in the data-generating model. However, if missing item responses are generated such that they can only be generated from incorrect item responses, applying an item response model that relies on latent ignorability results in biased estimates. The Mislevy-Wu model guarantees unbiased parameter estimates if the more general Mislevy-Wu model holds in the data-generating model. In addition, this article uses the PISA 2018 mathematics dataset as a case study to investigate the consequences of different missing data treatments on country means and country standard deviations. Obtained country means and country standard deviations can substantially differ for the different scaling models. In contrast to previous statements in the literature, the scoring of missing item responses as incorrect provided a better model fit than a latent ignorable model for most countries. Furthermore, the dependence of the missingness of an item from the item itself after conditioning on the latent response propensity was much more pronounced for constructed-response items than for multiple-choice items. As a consequence, scaling models that presuppose latent ignorability should be refused from two perspectives. First, the Mislevy-Wu model is preferred over the latent ignorable model for reasons of model fit. Second, in the discussion section, we argue that model fit should only play a minor role in choosing psychometric models in large-scale assessment studies because validity aspects are most relevant. Missing data treatments that countries can simply manipulate (and, hence, their students) result in unfair country comparisons.

Download Full-text

Alignment, Inclusion, and Technical Adequacy in Large-Scale Assessment and Accountability.

PsycCRITIQUES ◽

10.1037/004824 ◽

2004 ◽

Vol 49 (5) ◽

Author(s):

William D. Schafer

Keyword(s):

Large Scale ◽

Technical Adequacy ◽

Large Scale Assessment ◽

Scale Assessment

Download Full-text

Using Web-based Testing for Large-Scale Assessment

PsycEXTRA Dataset ◽

10.1037/e421172005-001 ◽

2013 ◽

Author(s):

Laura S. Hamilton ◽

Stephen P. Klein ◽

William Lorie

Keyword(s):

Large Scale ◽

Web Based ◽

Large Scale Assessment ◽

Scale Assessment

Download Full-text

Predicting Shifts in Land Suitability for Maize Cultivation Worldwide Due to Climate Change: A Modeling Approach

Land ◽

10.3390/land10030295 ◽

2021 ◽

Vol 10 (3) ◽

pp. 295

Author(s):

Yuan Gao ◽

Anyu Zhang ◽

Yaojie Yue ◽

Jing’ai Wang ◽

Peng Su

Keyword(s):

Climate Change ◽

Crop Production ◽

Large Scale ◽

Land Suitability ◽

Large Scale Assessment ◽

Scale Assessment ◽

Maize Cultivation ◽

Large Scale Screening ◽

Selection Of

Suitable land is an important prerequisite for crop cultivation and, given the prospect of climate change, it is essential to assess such suitability to minimize crop production risks and to ensure food security. Although a variety of methods to assess the suitability are available, a comprehensive, objective, and large-scale screening of environmental variables that influence the results—and therefore their accuracy—of these methods has rarely been explored. An approach to the selection of such variables is proposed and the criteria established for large-scale assessment of land, based on big data, for its suitability to maize (Zea mays L.) cultivation as a case study. The predicted suitability matched the past distribution of maize with an overall accuracy of 79% and a Kappa coefficient of 0.72. The land suitability for maize is likely to decrease markedly at low latitudes and even at mid latitudes. The total area suitable for maize globally and in most major maize-producing countries will decrease, the decrease being particularly steep in those regions optimally suited for maize at present. Compared with earlier research, the method proposed in the present paper is simple yet objective, comprehensive, and reliable for large-scale assessment. The findings of the study highlight the necessity of adopting relevant strategies to cope with the adverse impacts of climate change.

Download Full-text