Are multiple-choice items unfair? And if so, for whom?

Due to their test economy and objective evaluability, multiple-choice items are used much more frequently to test knowledge than constructed-response questions. However, studies point out that dependencies may exist between the individual test result and the test format (multiple-choice or constructed-response). Studies testing economic knowledge (one dimension of economic competence) are using mainly multiple-choice items and indicate gender-specific performance in the corresponding tests in favour of male test-takers. As an explanation for these “gender differences” gender-specific affinities and differences in cognitive abilities are mentioned. Moreover, the test format itself is mentioned but has hardly been investigated in detail to date. In order to answer the question to what extent students test performance depends on the item format, we test economic knowledge using two test formats (constructed-response and multiple-choice), but with the same content. Results from 201 business and business education students show that the usage of constructed-response items can compensate for existing gender differences in 53% of all cases. This underlines that no general, gender-specific advantage or disadvantage can be assumed in relation to the item format. However, the mixed use of constructed-response and multiple-choice items seem promising to compensate for potential gender differences.

Optimal Response Formats for Online Surveys: Branch, Grid, or Single Item?

This article reports the results of an experiment comparing branch, grid, and single-item question formats in an internet survey with a nationally representative probability sample. We compare the three formats in terms of administration time, item nonresponse, survey breakoff rates, response distribution, and criterion validity. On average, the grid format obtained the fastest answers, the single-item format was intermediate, and the branch format took the longest. Item nonresponse rates were lowest for the single-item format, intermediate for the grid, and highest for branching, but these results were not statistically significant when modeling the full experimental design. Survey breakoff rates among the formats are not statistically distinguishable. Criterion validity was weakest in the branching format, and there was no significant difference between the grid and single-item formats. This evidence indicates that the branching format is not well suited to internet data collection and that both single-item and short, well-constructed grids are better question formats.

Evaluation of the Effect of Items’ Format and Type on Psychometric Properties of Sixth Year Pharmacy Students Clinical Clerkship Assessment Tools

Background Examination is the traditional assessment tool, and are used to guide the improvement of academic programs. Accreditation committees’ emphasis on the implementation of standardized examinations. The aims of the current study are to evaluate the quality of assessment tools of sixth year PharmD students as a function of assessment item format and type/structure and to assess the effect of the number of response choices on the characteristics of MCQs as assessment items. Methods. A total of 173 assessment items used in the examinations of sixth year rotations of PharmD program were included. Items were classified as case based or non-case based and as MCQs or open-ended. The psychometric characteristics of examinations were studied as a function of the level of the Bloom’s levels addressed by an item, item format, and number of choices in MCQs. Results. Items addressing intellectual and analysis skills were more difficult, while items associated with multiple cognitive levels were more discriminative. No differences were found between case based and noncase based items in terms of their difficulty, with a slightly better discrimination in the letter. Open-ended items were easier, yet more discriminative. MCQs with higher number of options were easier and more discriminative. Open-ended questions were significantly easier and more discriminative in comparison to MCQs as case based items while they were more difficult and more discriminative as non-case based items. Conclusion. Item formats, structure, and number of options in MCQs, affected students’ performance and overall examination quality. The discrimination of items associated with multiple Bloom’s levels was significantly higher than those associated with a single level. Noncase based items and open-ended items were easier and more discriminative than case based items and MCQs, respectively.

Assessing Equivalency of PSC-17 Ratings: Does It Matter if Mixed or Grouped Item Format Is Used?

This study investigated the effect of item position on descriptive statistics, psychometric information, and factor structure of the Pediatric Symptoms Checklist 17-item social-emotional screening instrument (PSC-17). The goal was to determine whether item position, either grouped by factor or mixed across constructs, produced similar results. Descriptive statistics, reliability estimates, and model-data fit were similar across the two versions of the screener. Factor invariance tests supported strict invariance across the two versions, and very small differences between latent means for the three factors measured by the PSC-17. Both forms are equivalent for use with screening activities.

Random Item Response Model Approaches to Evaluating Item Format Effects

The PISA 2006 science assessment is composed of open response, multiple-choice, and constructed multiple choice items. The current study introduced the random item response models to investigate the item format effects on item difficulties, and these models include the linear logistic test model with raThe PISA 2006 science assessment is composed of open response, multiple-choice, and constructed multiple choice items. The current study introduced the random item response models to investigate the item format effects on item difficulties, and these models include the linear logistic test model with random item effects (i.e., the LLTM-R) and the hierarchical item response model (i.e., the hierarchical IRM). In this study these models were applied to the PISA 2006 science data set to explore the relationship between items' format and their difficulties. The empirical analysis results in the PISA 2006 science assessment first find that the LLTM-R and the hierachical IRM provides equivalent item difficulty estimates compared with those from the Rasch model and the LLTM, and also clearly show that the item difficulties are substantially affected by item formats. This result implies that item difficulties may be different to each other depending on the item format although they deal with the same content.ndom item effects (i.e., the LLTM-R) and the hierarchical item response model (i.e., the hierarchical IRM). In this study these models were applied to the PISA 2006 science data set to explore the relationship between items' format and their difficulties. The empirical analysis results in the PISA 2006 science assessment first find that the LLTM-R and the hierachical IRM provides equivalent item difficulty estimates compared with those from the Rasch model and the LLTM, and also clearly show that the item difficulties are substantially affected by item formats. This result implies that item difficulties may be different to each other depending on the item format although they deal with the same content.

The Relationship Between Test Item Format and Gender Achievement Gaps on Math and ELA Tests in Fourth and Eighth Grades

Prior research suggests that males outperform females, on average, on multiple-choice items compared to their relative performance on constructed-response items. This paper characterizes the extent to which gender achievement gaps on state accountability tests across the United States are associated with those tests’ item formats. Using roughly 8 million fourth- and eighth-grade students’ scores on state assessments, we estimate state- and district-level math and reading male-female achievement gaps. We find that the estimated gaps are strongly associated with the proportions of the test scores based on multiple-choice and constructed-response questions on state accountability tests, even when controlling for gender achievement gaps as measured by the National Assessment of Educational Progress (NAEP) or Northwest Evaluation Association (NWEA) Measures of Academic Progress (MAP) assessments, which have the same item format across states. We find that test item format explains approximately 25% of the variation in gender achievement gaps among states.