A-176 Undercover Performance Validity Testing: Proposed Embedded Validity Indicators for Pediatric Neuropsychological Assessment

2021 ◽  
Vol 36 (6) ◽  
pp. 1231-1231
Author(s):  
Naomi R Kaswan ◽  
Ryan C Thompson ◽  
Yelena Markiv ◽  
Aubrey Deenen ◽  
Haig V Pilavjian ◽  
...  
Keyword(s):  
Sport Participation ◽  
Word Reading ◽  
Digit Span ◽  
Characteristic Curve ◽  
Test Word ◽  
Healthy Children ◽  
Performance Validity ◽  
Validity Indicators ◽  
Neuropsychological Evaluations ◽  
Embedded Validity Indicators

Abstract Objective Literature supports the use of the Delis-Kaplan Executive Function System Trail Making Test Conditions 4/2 ratio (TMT 4/2) and Stroop Color Word Test Word Reading (WR) as embedded validity indicators (EVIs) with adults (Erdodi et al., 2018; Guise et al., 2012) and the Wechsler Abbreviated Scale of Intelligence, 2nd Edition Matrix Reasoning (MR) as an EVI with children (Sussman et al., 2017). This study assessed the utility of these measures as EVIs in healthy children, compared to the Test of Memory Malingering Trial 1 (TOMM1 < 45; Perna & Loughan, 2013) and Reliable Digit Span (RDS). Method Participants (n = 99, 68.7% male, Mage = 11.9) completed baseline neuropsychological evaluations for sport participation, including the aforementioned measures. Receiver operator characteristic curve analysis was used to determine whether TMT 4/2, MR, and WR accurately categorized valid performance. Results TMT 4/2 yielded adequate sensitivity (0.83–1.00) but poor specificity (0.07–0.09) when predicting TOMM1 and RDS pass/fail performances. MR yielded adequate sensitivity (1.00) and specificity (0.92) when predicting RDS pass/fail performance and adequate specificity (0.92) and poor sensitivity (0.18) when predicting TOMM1 pass/fail performance. The only EVI that produced better than chance accuracy was MR when predicting RDS pass/fail performance (area under the curve [AUC] = 0.98). All participants failed the WR cutoff, suggesting poor specificity. Conclusion Results suggest that MR was the only EVI that achieved minimally acceptable specificity (≥0.90) in children. MR performed adequately when detecting valid performances but variably when detecting invalid performances; therefore, MR may be used alongside well-established performance validity tests with children but not independently.

A-187 Lack of Agreement between Reliable Digit Span and Rey-15 Item Test as Validity Indicators for Test of Memory Malingering Performance in Youth Athletes

2021 ◽  
Vol 36 (6) ◽  
pp. 1242-1242
Author(s):  
Rachel M Murley ◽  
Daniel Baldini ◽  
Julius Flowers ◽  
Aiden Hall ◽  
Jamie P Kiefer ◽  
...  
Keyword(s):  
Clinical Judgment ◽  
Test Performance ◽  
Sport Participation ◽  
Digit Span ◽  
Healthy Population ◽  
Healthy Children ◽  
Performance Validity ◽  
Test Of Memory Malingering ◽  
Roc Curve Analysis ◽  
Youth Athletes

Abstract Objective In neuropsychological assessments, it is critical to evaluate examinee performance validity. Reliable Digit Span (RDS) and the Rey-15 Item Test (FIT) are performance validity tests (PVTs) used to assess the accuracy of results during neuropsychological assessment (NA; Lichtenstein et al., 2017). While many studies have examined adult populations, few examined children to determine the base rate level of PVT failure even in a healthy population. The present study assessed RDS and FIT as PVTs in youth athletes, with valid performance operationalized by Test of Memory Malingering (TOMM) performance. Method Youth athletes (n = 109, 79% male, Mage = 11.9) completed a NA including RDS (cutoff <7), TOMM Trial 2 (cutoff <50), and FIT (cutoff <26) during baseline evaluation for sport participation. A stringent TOMM cutoff was used to improve sensitivity (Schroeder et al., 2012). Receiver operator characteristic (ROC) curve analysis determined whether RDS or FIT performance accurately categorized participants’ validity test performance, based upon TOMM Pass/Fail. Results Both RDS and FIT produced minimal accuracy in categorizing participants performance on TOMM; areas under the curve ranged from 0.41 to 0.60 and did not significantly differ from chance (0.50). Additionally, there was a lack of agreement across PVTs, as no participant failed all three PVTs. Conclusion RDS and FIT did not adequately predict TOMM performance in healthy youth. These findings illustrate that even in a brief NA, healthy children demonstrated variable performance across validity measures, with low consistency across three commonly used PVTs. Findings highlight the importance of clinical judgment in evaluating discrepant PVT performance in children.

Digit Span Forward as a Performance Validity Test in Dementia Evaluations: Specificity in Mild Cognitive Impairment, Mild Dementia, and Moderate Dementia

2019 ◽  
Vol 34 (6) ◽  
pp. 837-837
Author(s):  
H Clark ◽  
P Martin ◽  
R Schroeder
Keyword(s):  
Cognitive Impairment ◽  
Mild Cognitive Impairment ◽  
Digit Span ◽  
Future Research ◽  
Performance Validity ◽  
Mild Dementia ◽  
Dementia Patients ◽  
Moderate Dementia ◽  
Neuropsychological Evaluations ◽  
Validity Tests

Abstract Objective Traditional performance validity tests (PVTs) often yield high false positive rates in dementia evaluations. The current study examined the frequency of extremely low scores (≤ 2 percentile) on WAIS-IV Digit Span Forward (DSF) in older adults with Mild Cognitive Impairment (MCI) or dementia to evaluate its possible utility as a PVT in these populations. Method Archival data from outpatient neuropsychological evaluations were analyzed. Individuals who were not diagnosed with a neurocognitive disorder, had missing data, or were believed to be invalidly performing were excluded. Participants (n = 195; mean age = 72.8; mean education = 13.2 years) were classified according to their evaluation diagnosis of MCI (n = 72; mean RBANS Total Score = 86.8) or dementia. Dementia patients were further divided by MoCA score into groups of mild dementia (n = 90; MoCA≥15; mean RBANS Total Score = 71.0) or moderate dementia (n = 33; MoCA < 15; mean RBANS Total Score = 55.9). Frequencies of scaled scores were analyzed to calculate specificity values for each group. Results A WAIS-IV DSF scaled score of ≤4 (≤ 2 percentile) resulted in specificity values of 0.99 and 0.94 in MCI and mild dementia, respectively. Conversely, in moderate dementia, ≥0.90 specificity was achieved only when using a more conservative cutoff of ≤2. Conclusions Low DSF scaled scores occurred infrequently in MCI and mild dementia, indicating strong specificity and potential utility as a PVT in these populations. However, in moderate dementia, low DSF scores were more common, requiring use of a more stringent cutoff. Future research should examine DSF sensitivity to invalid performance, as well as DSF specificity according to specific etiologies of MCI and dementia.

Digit Span embedded validity indicators in neurologically-intact veterans

2019 ◽  
Vol 34 (5) ◽  
pp. 1025-1037 ◽  
Author(s):  
Robert D. Shura ◽  
Sarah L. Martindale ◽  
Katherine H. Taber ◽  
Alana M. Higgins ◽  
Jared A. Rowland
Keyword(s):  
Digit Span ◽  
Validity Indicators ◽  
Neurologically Intact ◽  
Embedded Validity Indicators

scholarly journals A-219 Cross-validation of a Delis-Kaplan Executive Function System Color Word Interference Test embedded performance validity test

2020 ◽  
Vol 35 (6) ◽  
pp. 1014-1014
Author(s):  
Schroeder R ◽  
Clark H ◽  
Martin P
Keyword(s):  
Executive Function ◽  
Operating Characteristic ◽  
Color Word ◽  
Characteristic Curve ◽  
Area Under The Curve ◽  
Function System ◽  
Performance Validity ◽  
Validity Test ◽  
Interference Test ◽  
Neuropsychological Evaluations

Abstract Objective Eglit and colleagues (2019) found that the sum of the four age-corrected scaled scores from the Color Word Interference Test (CWIT) of the Delis-Kaplan Executive Function System (D-KEFS) could act as a useful embedded performance validity test (PVT). As such, we attempted to cross-validate their findings. Method Patients included 312 individuals who completed neuropsychological evaluations. Individuals were excluded if they were not administered CWIT, were not administered at least 4 criterion PVTs, had diagnoses of dementia or intellectual disability, or had indeterminate validity results (i.e., failure of one PVT). Valid performers (n = 231) were those who passed all criterion PVTs while invalid performers (n = 81) failed two or more criterion PVTs. A receiver operating characteristic curve was conducted for the CWIT embedded PVT. Results Area under the curve (AUC) was .804. At a cutoff of &lt; 26, specificity was 90% and sensitivity was 48%. At a cutoff of &lt; 20 (the first cutoff at which 90% specificity was found by Eglit et al.), specificity was 95% and sensitivity was 32%. At an even more conservative cutoff of &lt; 18 (the cutoff recommended by Eglit et al.), specificity was 96% and sensitivity was 22%. Conclusions These results cross-validate Eglit et al.’s findings, indicating that the sum of age-corrected scaled scores across the four CWIT trials can effectively serve as an embedded PVT. A more liberal cutoff was able to be applied in our sample but, even at conservative cutoffs documented in Eglit et al., sensitivity rates were adequate enough to warrant use of the index as an embedded PVT.

scholarly journals A-205 Digit Span Raw, A Newly Derived Embedded Performance Validity Test

2020 ◽  
Vol 35 (6) ◽  
pp. 1000-1000
Author(s):  
Schroeder R ◽  
Soden D ◽  
Clark H ◽  
Martin P
Keyword(s):  
Digit Span ◽  
Area Under The Curve ◽  
Performance Validity ◽  
Adult Intelligence Scale ◽  
Wechsler Adult Intelligence Scale ◽  
Sensitivity Rate ◽  
Neuropsychological Evaluations ◽  
Validity Tests ◽  
Wechsler Adult Intelligence ◽  
Performance Validity Tests

Abstract Objective Outside of Reliable Digit Span (RDS), there has been minimal research examining the utility of Digit Span (DS) score combinations from the Wechsler Adult Intelligence Scale—4th Edition (WAIS-IV) as possible performance validity tests (PVTs). We sought to determine if other DS scores/score combinations might work more effectively than RDS as a PVT. Method Patients included 318 individuals who completed neuropsychological evaluations. Individuals were excluded if they were not administered DS; were not administered at least 4 criterion PVTs; had diagnoses of dementia, intellectual disability, or left hemisphere cerebrovascular accident; or had indeterminate validity results (i.e., failure of one PVT). Valid performers (n = 248) were those who passed all criterion PVTs while invalid performers (n = 70) failed two or more criterion PVTs. Receiver operating characteristic curves were conducted for multiple DS indices. Results Area under the curve (AUC) was highest for the DS index that combined raw scores from all three trials (Digit Span Raw; AUC = .821). Likewise, when examining cutoffs that maintained 90% specificity for each DS index, a Digit Span Raw cutoff of &lt; 20 produced the highest sensitivity rate (52%) of all indices. For comparison, RDS, RDS with sequencing, and DS scaled score had AUC values of .758, .802, and .811, respectively. When maintaining specificity at 90%, sensitivity rates were 28%, 43%, and 43%, respectively. Conclusions Results suggest that the most effective embedded DS index might be a new one, which we term Digit Span Raw. Cross-validation of these findings could provide support for using this index instead of the more commonly examined RDS.

A Systematic Review and Meta-Analysis of the Utility of the Test of Memory Malingering in Pediatric Examinees

2020 ◽  
Vol 35 (8) ◽  
pp. 1312-1322
Author(s):  
Hilary A Clark ◽  
Phillip K Martin ◽  
Hayrettin Okut ◽  
Ryan W Schroeder
Keyword(s):  
Systematic Review ◽  
Meta Analysis ◽  
Healthy Children ◽  
Performance Validity ◽  
Test Of Memory Malingering ◽  
Eligibility Criteria ◽  
Random Effects Models ◽  
Neuropsychological Evaluations ◽  
Meta Analyses ◽  
The Impact

Abstract Objective This is the first systematic review and meta-analysis of the Test of Memory Malingering (TOMM) in pediatric examinees. It adheres to Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines. Method A systematic literature search was conducted using PsycINFO and PubMed, reviewing articles from January 1997 to July 2019. Books providing data on pediatric validity testing were also reviewed for references to relevant articles. Eligibility criteria included publication in a peer-reviewed journal, utilizing a pediatric sample, providing sufficient data to calculate specificity and/or sensitivity, and providing a means for evaluating validity status external to the TOMM. After selection criteria were applied, 9 articles remained for meta-analysis. Samples included clinical patients and healthy children recruited for research purposes; ages ranged from 5 to 18. Fixed and random effects models were used to calculate classification accuracy statistics. Results Traditional adult-derived cutoffs for Trial 2 and Retention were highly specific (0.96–0.99) in pediatric examinees for both clinical and research samples. Sensitivity was relatively strong (0.68–0.70), although only two studies reported sensitivity rates. A supplemental review of the literature corroborated these findings, revealing that traditional adult-based TOMM cutoffs are supported in most pediatric settings. However, limited research exists on the impact of very young age, extremely low cognitive functioning, and varying clinical diagnoses. Conclusions The TOMM, at traditional adult cutoffs, has strong specificity as a performance validity test in pediatric neuropsychological evaluations. This meta-analysis found that specificity values in children are comparable to those of adults. Areas for further research are discussed.

scholarly journals A-222 Efficacy of D–KEFS Trail Making Test Conditions 1–5 Using Age-Corrected Scaled Score Cutoffs as Embedded Validity Indicators in a Sample of Youth Athletes

2020 ◽  
Vol 35 (6) ◽  
pp. 1017-1017
Author(s):  
Thompson R ◽  
Arastu S ◽  
Markuson S ◽  
Deneen A ◽  
Hirst R
Keyword(s):  
Research Evaluation ◽  
Area Under The Curve ◽  
Trail Making Test ◽  
Performance Validity ◽  
Neuropsychological Battery ◽  
Youth Athletes ◽  
Test Conditions ◽  
Validity Indicators ◽  
Trail Making ◽  
Embedded Validity Indicators

Abstract Objective Embedded validity indicators (EVI) assist in the evaluation of performance validity across a neuropsychological battery without increasing the time or cognitive demand on patients during testing. Erdodi et al. (2018) evaluated Delis–Kaplan Executive Function System (D–KEFS) Trail Making Test Conditions 1–5 (TMT1–5) as EVIs in adults using age-corrected scaled score (ACSS) cutoffs, demonstrating adequate specificity but inadequate sensitivity. This study assessed the TMT1–5 as EVIs in children using a specificity threshold of &gt; .89 (Boone, 2013). Method Youth athletes (n = 134, M age = 12.2) completed a two-hour neuropsychological battery that included TMT1–5 and TOMM during a larger sport-related concussion clinical research evaluation. Specificity, sensitivity, and area under the curve (AUC) were calculated for TMT1–5 using ACSS cutoffs to predict TOMM Trial 2 (TOMM2) pass/fail performance. Results When predicting TOMM2 (cutoff.89) and inadequate sensitivity (.00), with TMT3 and TMT4 having the highest AUC (.81 and .74, respectively). The following ACSS cutoffs met the specificity threshold (&gt;.89): TMT1 ACSS &lt; 7, TMT2 &lt; 9, TMT3 &lt; 6, TMT4 &lt; 7, and TMT5.79, we increased sensitivity for TMT1–3 (.33). Conclusion D–KEFS TMT1–5 achieved excellent specificity (&gt;.89) when predicting TOMM2 performance but demonstrated inadequate sensitivity. Lowering the specificity threshold slightly still yielded low sensitivity. These data suggest that D–KEFS TMT3 and TMT4 were most effective at detecting adequate effort, but should be supported with other validity measures when assessing performance validity across a neuropsychological battery.

Verbal fluency and digit span variables as performance validity indicators in experimentally induced malingering and real world patients with TBI

2020 ◽  
Vol 9 (4) ◽  
pp. 337-354 ◽  
Author(s):  
Jessica Hurtubise ◽  
Tabarak Baher ◽  
Isabelle Messa ◽  
Laura Cutler ◽  
Ayman Shahein ◽  
...  
Keyword(s):  
Real World ◽  
Verbal Fluency ◽  
Digit Span ◽  
Performance Validity ◽  
Validity Indicators ◽  
Experimentally Induced

scholarly journals Performance validity testing via telehealth and failure rate in veterans with moderate-to-severe traumatic brain injury: A veterans affairs TBI model systems study

2021 ◽  
pp. 1-9
Author(s):  
Robert Kanser ◽  
Justin O’Rourke ◽  
Marc A. Silva
Keyword(s):  
Traumatic Brain Injury ◽  
Brain Injury ◽  
Severe Traumatic Brain Injury ◽  
Digit Span ◽  
Model Systems ◽  
Performance Validity ◽  
Face To Face ◽  
Neuropsychological Evaluations ◽  
Validity Tests ◽  
Positive Classification

BACKGROUND: The COVID-19 pandemic has led to increased utilization of teleneuropsychology (TeleNP) services. Unfortunately, investigations of performance validity tests (PVT) delivered via TeleNP are sparse. OBJECTIVE: The purpose of this study was to examine the specificity of the Reliable Digit Span (RDS) and 21-item test administered via telephone METHOD: Participants were 51 veterans with moderate-to-severe traumatic brain injury (TBI). All participants completed the RDS and 21-item test in the context of a larger TeleNP battery. Specificity rates were examined across multiple cutoffs for both PVTs. RESULTS: Consistent with research employing traditional face-to-face neuropsychological evaluations, both PVTs maintained adequate specificity (i.e., >  90%) across previously established cutoffs. Specifically, defining performance invalidity as RDS <  7 or 21-item test forced choice total correct <  11 led to <  10%false positive classification errors. CONCLUSIONS: Findings add to the limited body of research examining and provide preliminary support for the use of the RDS and 21-item test in TeleNP via telephone. Both measures maintained adequate specificity in veterans with moderate-to-severe TBI. Future investigations including clinical or experimental “feigners” in a counter-balanced cross-over design (i.e., face-to-face vs. TeleNP) are recommended.

Sign in / Sign up

Export Citation Format

Share Document