Setting a standard for low reading proficiency: A comparison of the bookmark procedure and constrained mixture Rasch model

In order to draw pertinent conclusions about persons with low reading skills, it is essential to use validated standard-setting procedures by which they can be assigned to their appropriate level of proficiency. Since there is no standard-setting procedure without weaknesses, external validity studies are essential. Traditionally, studies have assessed validity by comparing different judgement-based standard-setting procedures. Only a few studies have used model-based approaches for validating judgement-based procedures. The present study addressed this shortcoming and compared agreement of the cut score placement between a judgement-based approach (i.e., Bookmark procedure) and a model-based one (i.e., constrained mixture Rasch model). This was performed by differentiating between individuals with low reading proficiency and those with a functional level of reading proficiency in three independent samples of the German National Educational Panel Study that included students from the ninth grade (N = 13,897) as well as adults (Ns = 5,335 and 3,145). The analyses showed quite similar mean cut scores for the two standard-setting procedures in two of the samples, whereas the third sample showed more pronounced differences. Importantly, these findings demonstrate that model-based approaches provide a valid and resource-efficient alternative for external validation, although they can be sensitive to the ability distribution within a sample.

Screening Depression and Anxiety via Brief Measures of Psychological Inflexibility

We evaluated the usefulness of scores from two transdiagnostic scales—the 8-item version of the Avoidance and Fusion Questionnaire for Youth and the second edition of the Avoidance and Action Questionnaire—for estimating symptom severity on two measures of depression and anxiety. Responses from 797 college students, who mostly identified as White and female, to both measures of psychological inflexibility were analyzed to determine how well scores estimated anxiety and depression above or below a given severity level and at specific categories of symptom severity. Findings indicated that scores from both measures were acceptable to excellent screeners of concurrent ratings of anxiety and depression. Results varied somewhat depending on the measure used, level of severity targeted, and scope of screening. By investigating the screening accuracy of these transdiagnostic measures and potential cut scores to ease in interpreting results, we hope these measures might prove useful for addressing barriers in public health screening endeavors.

Improving Detection of Amnestic Mild Cognitive Impairment with Sex-Specific Cognitive Norms

Background Despite a female advantage in verbal memory, normative data for verbal memory tests used to diagnose Alzheimer’s disease (AD) dementia and amnestic mild cognitive impairment (aMCI) often are not sex-adjusted. Objective To determine whether sex-adjusted norms improve aMCI diagnostic accuracy when accuracy was evaluated by progression to AD dementia over time. Methods Non-sex-specific and sex-specific verbal memory test norms were incorporated into Jak/Bondi aMCI criteria and applied to older (age 65–90) non-demented women (N = 1,036) and men (N = 355) from the Rush Memory and Aging Project. Using sex-specific aMCI diagnosis as the “true” condition versus non-sex-specific aMCI diagnosis as the “predicted” condition, we identified True Positives, False Positives, True Negatives, and False Negatives and compared AD dementia risk over 10 years among groups. Results Rates of aMCI were higher in men versus women (χ 2 = 15.39, p < 0.001) when determined based on typical diagnostic criteria, but this difference reversed when using sex-specific diagnostic criteria (χ 2 = 8.38, p = 0.004). We identified 8%of women as False Negatives and 12%of men as False Positives. Risk of incident AD dementia in False Positive men was significantly lower than in True Positive men (HR = 0.26, 95%CI = 0.12–0.58, p = 0.001). Risk of incident AD dementia in False Negative women was substantially higher than in True Negative women (HR = 3.11, 95%CI = 2.09–4.63, p < 0.001). Conclusion Results suggest that previous reports of higher aMCI rates in men versus women may be an artifact of non-sex-adjusted norms/cut-scores. Incorporation of sex-specific norms/cut-scores for verbal memory impairment into aMCI diagnostic criteria may improve diagnostic accuracy and avoid diagnostic errors in approximately 20%.

Cross-validating the Dot Counting Test Among an Adult ADHD Clinical Sample and Analyzing the Effect of ADHD Subtype and Comorbid Psychopathology

This study cross-validated the dot counting test (DCT) as a performance validity test (PVT) in an adult attention-deficit/hyperactivity disorder (ADHD) clinical population and examined the effect of ADHD subtype and psychiatric comorbidity on accuracy for detecting invalidity. DCT performance was assessed among 210 consecutive adult ADHD referrals who underwent neuropsychological evaluation and were classified into valid ( n = 175) or invalid ( n = 35) groups based on seven independent criterion PVTs. The invalid group had significantly worse DCT performance than the valid group using both the standard and unrounded scoring procedure ([Formula: see text]). Classification accuracy was excellent, with 54.3% sensitivity/92% specificity at optimal cut-scores of ≥14 (rounded) and ≥13.38 (unrounded). Nonsignificant DCT performance differences emerged based on ADHD subtype or the presence/absence of comorbid psychopathology. The DCT functions well as a nonmemory-based PVT in an ethnoracially diverse ADHD population, supporting its clinical utility for detecting invalid neurocognitive performance during ADHD evaluations.

Cut-Score Operating Function Extensions: Penalty-Based Errors and Uncertainty in Standard Settings

We model pass/fail examinations aiming to provide a systematic tool to minimize classification errors. We use the method of cut-score operating functions to generate specific cut-scores on the basis of minimizing several important misclassification measures. The goal of this research is to examine the combined effects of a known distribution of examinee abilities and uncertainty in the standard setting on the optimal choice of the cut-score. In addition, we describe an online application that allows others to utilize the cut-score operating function for their own standard settings.

Comparing the cut score for the borderline group method and borderline regression method with norm-referenced standard setting in an objective structured clinical examination in medical school in Korea

Purpose: Setting standards is critical in health professions. However, appropriate standard setting methods do not always apply to the set cut score in performance assessment. The aim of this study was to compare the cut score when the standard setting is changed from the norm-referenced method to the borderline group method (BGM) and borderline regression method (BRM) in an objective structured clinical examination (OSCE) in medical school.Methods: This was an explorative study to model of the BGM and BRM. A total of 107 fourth-year medical students attended the OSCE at seven stations with encountering standardized patients (SPs) and one station with performing skills on a manikin on 15 July 2021. Thirty-two physician examiners evaluated the performance by completing a checklist and global rating scales.Results: The cut score of the norm-referenced method was lower than that of the BGM (p<0.01) and BRM (p<0.02). There was no significant difference in the cut score between the BGM and BRM (p=0.40). The station with the highest standard deviation and the highest proportion of the borderline group showed the largest cut score difference in standard setting methods.Conclusion: Prefixed cut scores by the norm-referenced method without considering station contents or examinee performance can vary due to station difficulty and content, affecting the appropriateness of standard setting decisions. If there is an adequate consensus on the criteria for the borderline group, standard setting with the BRM could be applied as a practical and defensible method to determine the cut score for OSCE.

A-102 Performance Validity Testing in Telehealth Research: A VA Traumatic Brain Injury Model Systems Study

Objective Studies on Performance Validity Tests (PVTs) for tele-neuropsychology (TeleNP) are sparse. Verbal PVTs appear to better translate to TeleNP, so the primary objective of this study was to provide initial data on two well-established, verbal PVTs administered via TeleNP for research participants with traumatic brain injury (TBI). Methods This secondary analysis of the Veterans Affairs TBI Model Systems data included 53 participants enrolled in a PVT module study (3/01/2020–09/20/2020) with documented moderate-to-severe TBI per Glasgow Coma Score (M = 6.5, SD = 4.4), posttraumatic amnesia duration (M = 42.7 days, SD = 47.1), and/or time to follow commands (M = 10.5 days, SD = 16.3). Participants completed two PVTs—Reliable Digit Span (RDS) and the 21-Item Test (21-IT)—alongside telephone-based cognitive assessment 1–7 years after TBI. Descriptive analyses were performed to compare PVT performances to previously established cut scores. Chi square analyses were employed to examine 21-IT and RDS as dichotomous outcomes (pass/fail) at selected cutoffs. Results RDS ranged from 5 to 16 (M = 10.5, SD = 2.4). 21-IT ranged from 7 to 21 (M = 16.4, SD = 3.1). For RDS, 9.8% were invalid with a cutscore of ≤7 and 19.6% using a cutscore of ≤8. For the 21-IT, 7.8% were in invalid using a cutscore of ≤11, and 13.7% using a cutscore of ≤12. Conclusion(s) Using previously established cut scores, telephone-administered RDS and 21-IT resulted in relatively low rates of invalid performance among individuals with moderate-to-severe TBI. These findings provide preliminary support for the RDS and 21-IT in TeleNP.

Screening Depression and Anxiety via Brief Measures of Psychological Inflexibility

We evaluated the usefulness of scores from two transdiagnostic scales—the 8-item version of the Avoidance and Fusion Questionnaire for Youth (AFQ-Y8) and the second edition of the Avoidance and Action Questionnaire (AAQ-II)—for estimating symptom severity on two measures of depression and anxiety. Responses to both measures of psychological inflexibility were analyzed to determine how well scores estimated anxiety and depression above or below a given severity level and at specific categories of symptom severity. Findings indicated that scores from both measures were acceptable to excellent screeners of concurrent ratings of anxiety and depression. Results varied somewhat depending on the measure used, level of severity targeted, and scope of screening. By investigating the screening accuracy of these transdiagnostic measures and potential cut scores to ease in interpreting results, we hope these measures might prove useful for addressing barriers in public health screening endeavors.

The International Cognitive Ability Resource: A Free Cognitive Measure With Utility for Postsecondary Giftedness Research

Research on high-ability postsecondary students has increased in recent years; yet identifying such students can be challenging. The International Cognitive Ability Resource (ICAR) is an online, open-access tool designed to facilitate measurement of cognitive abilities in research. We evaluated whether the ICAR is appropriate to identify high-ability postsecondary students for research; high ability was classified by a General Ability Index score of 120 or higher on the WAIS-IV. In a sample of 97 students from a U.S. university (Mean age 22.47 years, Mean General Ability Index score 115.13) the 60-item ICAR demonstrated adequate diagnostic accuracy to identify high ability with three appropriate cut scores (33, 34, or 35 items correct out of 60). The 16-item ICAR had no appropriate cut scores but demonstrated validity as a brief cognitive ability measure that could be used to examine relations between intelligence and other variables. Findings suggest that the ICAR could be a useful open-source tool for research with high-ability college students