Development and Application of Novel Performance Validity Metrics for Computerized Neurocognitive Batteries

Objective: Data from neurocognitive assessments may not be accurate in the context of factors impacting validity, such as disengagement, unmotivated responding, or intentional underperformance. Performance validity tests (PVTs) were developed to address these phenomena and assess underperformance on neurocognitive tests. However, PVTs can be burdensome, rely on cutoff scores that reduce information, do not examine potential variations in task engagement across a battery, and are typically not well-suited to acquisition of large cognitive datasets. Here we describe the development of novel performance validity measures that could address some of these limitations by leveraging psychometric modeling from data embedded within the Penn Computerized Neurocognitive Battery (PennCNB). Method: We first developed these validity measures using simulations of invalid response patterns with parameters drawn from real data. Next, we examined their application in two large, independent samples: 1) children and adolescents from the Philadelphia Neurodevelopmental Cohort (n=9,498); and 2) adult servicemembers from the Marine Resiliency Study-II (n=1,444). Results: Our performance validity metrics detected patterns of invalid responding in simulated data, even at subtle levels. Furthermore, a combination of these metrics significantly predicted previously established validity rules for these tests in both developmental and adult datasets. Moreover, most clinical diagnostic groups did not show reduced validity estimates. Conclusion: These results provide proof-of-concept evidence for multivariate, data-driven performance validity metrics. These metrics offer a novel method for determining the performance validity for individual neurocognitive tests that is scalable, applicable across different tests, less burdensome, and dimensional. However, more research is needed into their application.

A-207 Applying the Hartford Consistency Index to the Brief Visuospatial Memory Test—Revised for Assessing Performance Validity in an Outpatient Sample: A Pilot Study

Objective Prior studies have examined indices within the Brief Visuospatial Memory Test—Revised (BVMT-R) as potential embedded performance validity tests (PVT). Findings from these studies, however, are limited and with mixed results. Therefore, the purpose of the current study was to compare the classification accuracy of the Hartford Consistency Index (HCI) with published BVMT-R performance validity measures in an outpatient sample. Method A total of 115 archival files met study inclusion criteria: a) ≥ 18 years-old; b) administered > 2 PVTs (Reliable Digit Span, Dot Counting Test, and Test of Memory Malingering); and c) no diagnoses of intellectual disability or dementia. Utilizing standard cutoffs, participants were classified as ‘Valid’ (n = 94) or ‘Invalid’ (n = 21). ‘Valid’ profiles passed all PVTs and were free of known external incentives while ‘Invalid’ profiles failed ≥2 PVTs. Results An HCI cutoff of < 1 yielded 90% specificity, 48% sensitivity, and the area under the curve (AUC = .70) was adequate. Applying published cutoffs for Recognition Hits (≤4) and Percent Retention (≤58%) to our sample produced > 90% specificity, but sensitivity rates were < 40% and AUCs were consistently < .70. Similarly, the Recognition Discrimination (≤4) cutoff revealed inadequate specificity (84%), but acceptable sensitivity (63%), and AUC (.73). Conclussions Results from our study support the use of the HCI as an embedded PVT within the BVMT-R for non-demented outpatient samples. Furthermore, the HCI outperformed other embedded PVTs examined. Limitations of our study and future directions are discussed.

A-228 Are Those Who Fail Performance Validity Measures High Utilizers of Healthcare?

Objective One study reported that suboptimal effort on performance validity tests (PVTs) is associated with higher healthcare utilization within a VA setting, defined as the number of Emergency Department visits and inpatient hospitalizations. The current study sought to expand on this by examining whether PVT failure is associated with higher number of outpatient visits in a military sample with a history of mild traumatic brain injury (mTBI). Method The medical records of 43 participants, 13 of whom failed the PVT Green’s Word Memory Test (WMT), were reviewed for the number of encounters since the mTBI and the reason for the encounter. The two groups (passed vs. failed) did not differ significantly on demographic variables (39 males, mean age 39, 65% Caucasian). Results The overall number of medical encounters was not significant between the two groups after controlling for years since the mTBI (F(1, 40) = 2.67, p = .11); however, once three participants with (>2 years) missing records were excluded (final n = 40), the PVT failure group was seen significantly more often, (F(1, 37) = 8.23, p = .01). The PVT failures had a higher number of encounters with physical therapy (t(38) = −2.79, p = .01) and orthopedics (t(38) = −2.10, p = .04). Conclusions Preliminary results suggest that suboptimal effort is not associated with higher healthcare utilization; however, when participants with more than two years of missing records were excluded, those who failed PVTs were seen more frequently by physical therapy and orthopedic specialties. Limitations for future investigations are highlighted.

Test of Premorbid Functioning: You’re Doing It Wrong, but Does It Matter?

Objective Clarify procedures to correctly score Test of Premorbid Functioning (TOPF) and assess the accuracy of TOPF scores in the estimation of premorbid intellectual functioning. Method In this cross-sectional study, post-9/11 veterans (N = 233, 84.12% male) completed the TOPF, the Wechsler Adult Intelligence Scale-Fourth Edition (WAIS-IV), and performance validity measures. Results All TOPF scores were significantly correlated with WAIS-IV FSIQ scores (range r = 0.56–.73). The degree of discrepancy between TOPF scores and FSIQ varied with 10%–17% of TOPF scores deviating from FSIQ scores by one SD or more. Conclusions Proper TOPF scoring procedures are presented. The TOPF Actual and Predicted scores were related to FSIQ. However, a higher percentage of Actual and Predicted scores were discrepant from FSIQ compared with the other three TOPF estimates, arguing against their use as independent premorbid estimates. Use of the TOPF as was designed is recommended.

Gender Differences in Veterans Referred for Neuropsychological Evaluation in an Outpatient Neuropsychology Consultation Service

Objective The number of women veterans seeking Veterans Health Administration services has substantially increased over the past decade. Neuropsychology remains an understudied area in the examination of gender differences. The present study sought to delineate similarities and differences in men and women veterans presenting for neuropsychological evaluation in terms of demographics, referral, medical conditions, effort, and outcome diagnosis. Method A database collected from an outpatient VA neuropsychology clinic from 2013 to 2019 was analyzed (n = 232 women, 2642 men). Additional analyses examined younger (n = 836 men, 155 women) and older (n = 1805 men, 77 women) age cohorts. Results Women veterans were younger and more educated than men, whereas men had higher prevalence of vascular risk factors. Both groups were most often referred from mental health clinics and memory was the most common referral question. Although men performed worse on performance validity measures, clinicians rated women as evidencing poorer effort on a cumulative rating based on formal and embedded performance validity measures, behavioral observations, and inconsistent test patterns. Older women reported more depressive symptoms than older men and were more commonly diagnosed with depression. Conclusions This exploratory study fills a gap in the understanding of gender differences in veterans presenting for neuropsychological evaluations. Findings emphasize consideration for the intersection of gender with demographics, medical factors, effort, and psychological symptoms by VA neuropsychologists. A better understanding of relationships between gender and these factors may inform neuropsychologists’ test selection, interpretation of behavioral observations, and diagnostic considerations to best treat women veterans.

PGR - 2 Neuropsychology’s Role in Identifying Functional Neurological Disorder

Objective To outline the usefulness of neuropsychological evaluation in identifying functional neurological disorder. Functional neurological disorder accounts for an estimated 16% of neurology referrals and is a “crisis of neurology” (Edwards & Bhatia, 2012). Adolescents with a history of neurologic compromise, including autoimmune disorders, are at increased risk for comorbid functional neurological disorder (Reuber, Mitchell, Howlett, Crimlisk, & Grünewald, 2005). Method 16-year-old female with a history of Hoshimoto’s encephalopathy referred by her neurologist. Following diagnosis and treatment, she developed a constellation of symptoms, including wide set gait, nystagmus, incontinence, and dystonic episodes. She also reported lapses in memory that lasted a few minutes at a time. During these episodes, she forgot who her mother was and forgot details about her home, which resulted in panic. Results Two neuropsychological evaluations were completed over two years. Psychometric intelligence declined by two standard deviations compared to previous testing one year before. Her performance on memory tasks declined dramatically as well. Despite these declines, there was no decline in activities of daily living. She failed embedded and stand-alone performance validity measures (RDS = 4; TOMM = 29, 30, 28). These findings were described to her neurologist and psychiatrist and we discussed the importance of identifying functional symptoms in the context of her medical history. Conclusions Our evaluation identified significant discrepancies between neuropsychological performance and daily cognitive functioning. The memory lapses she described were inconsistent with any known etiology or illness. Neuropsychological evaluation identified symptoms of a functional nature and assisted in appropriate treatment planning.