Setting a standard for low reading proficiency: A comparison of the bookmark procedure and constrained mixture Rasch model

In order to draw pertinent conclusions about persons with low reading skills, it is essential to use validated standard-setting procedures by which they can be assigned to their appropriate level of proficiency. Since there is no standard-setting procedure without weaknesses, external validity studies are essential. Traditionally, studies have assessed validity by comparing different judgement-based standard-setting procedures. Only a few studies have used model-based approaches for validating judgement-based procedures. The present study addressed this shortcoming and compared agreement of the cut score placement between a judgement-based approach (i.e., Bookmark procedure) and a model-based one (i.e., constrained mixture Rasch model). This was performed by differentiating between individuals with low reading proficiency and those with a functional level of reading proficiency in three independent samples of the German National Educational Panel Study that included students from the ninth grade (N = 13,897) as well as adults (Ns = 5,335 and 3,145). The analyses showed quite similar mean cut scores for the two standard-setting procedures in two of the samples, whereas the third sample showed more pronounced differences. Importantly, these findings demonstrate that model-based approaches provide a valid and resource-efficient alternative for external validation, although they can be sensitive to the ability distribution within a sample.

Cut-Score Operating Function Extensions: Penalty-Based Errors and Uncertainty in Standard Settings

We model pass/fail examinations aiming to provide a systematic tool to minimize classification errors. We use the method of cut-score operating functions to generate specific cut-scores on the basis of minimizing several important misclassification measures. The goal of this research is to examine the combined effects of a known distribution of examinee abilities and uncertainty in the standard setting on the optimal choice of the cut-score. In addition, we describe an online application that allows others to utilize the cut-score operating function for their own standard settings.

Validity Evidence for the Medical Student Core Physical Examination

Background The Core Physical Exam (CPE) has been proposed as a set of key physical exam (PE) items for teaching and assessing PE skills in medical students, and as the basis of a Core + Cluster curriculum. Beyond the initial development of the CPE and proposal of the CPE and the Core + Cluster curriculum, no additional validity evidence has been presented for use of the CPE to teach or assess PE skills of medical students. As a result, a modified version of the CPE was developed by faculty at the University of Colorado School of Medicine (UCSOM) and implemented in the school’s clinical skills course in the context of an evolving Core + Cluster curriculum. Methods Validity evidence for the 25-item University of Colorado School of Medicine (UCSOM) CPE was analyzed using longitudinal assessment data from 366 medical students (Classes of 2019 and 2020), obtained from September 2015 through December 2019. Using Messick's unified validity framework, validity evidence specific to content, response process, internal structure, relationship to other variables, and consequences was gathered. Results Content and response process validity evidence included expert content review and rater training. For internal structure, a generalizability study phi coefficient of 0.258 suggests low reliability for a single assessment due to variability in learner performance by occasion and CPE items. Correlations of performance on the UCSOM CPE with other PE assessments were low, ranging from .00-.34. Consequences were explored through determination of a pass-fail cut score. Following a modified Angoff process, clinical skills course directors selected a consensus pass-fail cut score of 80% as a defensible and practical threshold for entry into precepted clinical experiences. Conclusions Validity evidence supports the use of the UCSOM CPE as an instructional strategy for teaching PE skills and as a formative assessment of readiness for precepted clinical experiences. The low generalizability coefficient suggests that inferences about PE skills based on the UCSOM CPE alone should be made with caution, and that the UCSOM CPE in isolation should be used primarily as a formative assessment.

Comparing the cut score for the borderline group method and borderline regression method with norm-referenced standard setting in an objective structured clinical examination in medical school in Korea

Purpose: Setting standards is critical in health professions. However, appropriate standard setting methods do not always apply to the set cut score in performance assessment. The aim of this study was to compare the cut score when the standard setting is changed from the norm-referenced method to the borderline group method (BGM) and borderline regression method (BRM) in an objective structured clinical examination (OSCE) in medical school.Methods: This was an explorative study to model of the BGM and BRM. A total of 107 fourth-year medical students attended the OSCE at seven stations with encountering standardized patients (SPs) and one station with performing skills on a manikin on 15 July 2021. Thirty-two physician examiners evaluated the performance by completing a checklist and global rating scales.Results: The cut score of the norm-referenced method was lower than that of the BGM (p<0.01) and BRM (p<0.02). There was no significant difference in the cut score between the BGM and BRM (p=0.40). The station with the highest standard deviation and the highest proportion of the borderline group showed the largest cut score difference in standard setting methods.Conclusion: Prefixed cut scores by the norm-referenced method without considering station contents or examinee performance can vary due to station difficulty and content, affecting the appropriateness of standard setting decisions. If there is an adequate consensus on the criteria for the borderline group, standard setting with the BRM could be applied as a practical and defensible method to determine the cut score for OSCE.

A-39 New Cut-Score to Improve Performance of the Memory Validity Profile (MVP) in Heterogenous Clinical Populations

Objective The Memory Validity Profile (MVP) is a standalone performance validity test developed specifically for use with children. Prior research has demonstrated the MVP’s strength in its ease of administration to children with a wide range of intellectual abilities. However, it has been found to lack sensitivity in detecting noncredible performance in select clinical populations using published cut-offs. The current study examines the MVP’s performance in a diagnostically heterogeneous clinical sample and proposes a new cut-off for optimization of sensitivity and specificity. Method Archival clinical data was examined from 96 participants referred for a comprehensive neuropsychological evaluation (ages 6–18). Receiver operating characteristic analysis was used to assess the discriminative ability of MVP in detecting cases of noncredible performance defined as failures on both the Test of Memory Malingering and Reliable Digit Span. Results Using published cut-offs, the MVP demonstrated perfect specificity (100%) but suboptimal sensitivity (33.3%). Receiver operating characteristic analysis revealed strong discrimination using MVP Total score (AUC = 0.891 (p &lt; 0.001) and a MVP Total cut-score of ≤30 resulted in optimal sensitivity (89%) and specificity (63%). Conclusions Our findings provide additional evidence that published MVP cut-offs may be too lenient to adequately capture instances of noncredible performance and indicate an MVP Total score cut-off of ≤30 may be more appropriate for use with heterogeneous clinical populations.

The Response Vector for Mastery Method of Standard Setting

Proposed is a new method of standard setting referred to as response vector for mastery (RVM) method. Under the RVM method, the task of panelists that participate in the standard setting process does not involve conceptualization of a borderline examinee and probability judgments as it is the case with the Angoff and bookmark methods. Also, the RVM-based computation of a cut-score is not based on a single item (e.g., marked in an ordered item booklet) but, instead, on a response vector (1/0 scores) on items and their parameters calibrated in item response theory or under the recently developed D-scoring method. Illustrations with hypothetical and real-data scenarios of standard setting are provided and methodological aspects of the RVM method are discussed.

Effects of double-dose algebra on college persistence and degree attainment

In 2003, Chicago Public Schools introduced double-dose algebra, requiring two periods of math—one period of algebra and one of algebra support—for incoming ninth graders with eighth-grade math scores below the national median. Using a regression discontinuity design, earlier studies showed promising results from the program: For median-skill students, double-dose algebra improved algebra test scores, pass rates, high school graduation rates, and college enrollment. This study follows the same students 12 y later. Our findings show that, for median-skill students in the 2003 cohort, double-dose significantly increased semesters of college attended and college degree attainment. These results were not replicated for the 2004 cohort. Importantly, the impact of the policy on median-skill students depended largely on how classes were organized. In 2003, the impacts on college persistence and degree attainment were large in schools that strongly adhered to the cut-score-based course assignment, but without grouping median-skill students with lower-skill peers. Few schools implemented the policy in such a way in 2004.

Predictors of trajectories of obsessive-compulsive symptoms during the COVID-19 pandemic in the general population in Germany

The COVID-19 pandemic has been associated with an increase in obsessive-compulsive disorder/symptoms (OCD/OCS). However, knowledge is limited regarding the trajectories of OCS during the pandemic, as well as their predictors and mechanisms (e.g., experiential avoidance, EA). The aim of this study was to describe the trajectories of OCS and the identification of associated factors. We assessed 1207 participants of the general population in March 2020 (t1) and June 2020 (t2). Pre-pandemic data was available from March 2014 for a subsample (n = 519). To define trajectories, we determined OCS status (OCS+/−). We performed a hierarchical multinomial logistic regression to investigate predictors of trajectories. Between t1 and t2, 66% of participants had an asymptomatic trajectory (OCS−/OCS−); 18% had a continuously symptomatic trajectory (OCS+/OCS+). Ten percent had a delayed-onset trajectory (OCS−/OCS+), and the recovery trajectory group (OCS+/OCS−) was the smallest group (6%). Higher education reduced the odds of an OCS+/OCS− trajectory. OCS in 2014 was associated with increased odds of showing an OCS+/OCS+ or OCS−/OCS+ trajectory. When EA at t1 and change in EA from t1 to t2 were added to the model, higher EA at t1 was associated with increased odds of scoring above the cut score on one or more of the assessments. A higher decrease in EA from t1 to t2 reduced the probability of showing an OCS+/OCS+ and an OCS−/OCS+ trajectory. While the current data supports a slight increase in OCS during the pandemic, trajectories differed, and EA seems to represent an important predictor for an unfavorable development.

Using Rasch analysis for determining the cut score of a computer science placement exam

A key strategy for broadening computer science participation in the Chicago Public Schools has been the enactment of a yearlong computer science course as a high school graduation requirement. The Exploring Computer Science (ECS) curriculum and professional development program serves as a core foundation for supporting policy enactment. However, students with prior background in computer science might find the course repetitive. This paper reports on district efforts to develop a placement exam for students to take an advanced computer science course in lieu of the introductory computer science course. The placement exam tasks were modeled after the ECS exam tasks but with higher difficulty. We used Rasch modeling to equate the placement exam tasks to the ECS exams and to establish a cut score for passing the placement exam.