scholarly journals Development and Application of Novel Performance Validity Metrics for Computerized Neurocognitive Batteries

2021 ◽  
Author(s):  
J. Cobb Scott ◽  
Tyler M. Moore ◽  
David R Roalf ◽  
Theodore D. Satterthwaite ◽  
Daniel H. Wolf ◽  
...  
Keyword(s):  
Simulated Data ◽  
Real Data ◽  
Response Patterns ◽  
Performance Validity ◽  
Neurocognitive Tests ◽  
Validity Measures ◽  
Novel Method ◽  
Performance Validity Measures ◽  
Validity Tests ◽  
Performance Validity Tests

Objective: Data from neurocognitive assessments may not be accurate in the context of factors impacting validity, such as disengagement, unmotivated responding, or intentional underperformance. Performance validity tests (PVTs) were developed to address these phenomena and assess underperformance on neurocognitive tests. However, PVTs can be burdensome, rely on cutoff scores that reduce information, do not examine potential variations in task engagement across a battery, and are typically not well-suited to acquisition of large cognitive datasets. Here we describe the development of novel performance validity measures that could address some of these limitations by leveraging psychometric modeling from data embedded within the Penn Computerized Neurocognitive Battery (PennCNB). Method: We first developed these validity measures using simulations of invalid response patterns with parameters drawn from real data. Next, we examined their application in two large, independent samples: 1) children and adolescents from the Philadelphia Neurodevelopmental Cohort (n=9,498); and 2) adult servicemembers from the Marine Resiliency Study-II (n=1,444). Results: Our performance validity metrics detected patterns of invalid responding in simulated data, even at subtle levels. Furthermore, a combination of these metrics significantly predicted previously established validity rules for these tests in both developmental and adult datasets. Moreover, most clinical diagnostic groups did not show reduced validity estimates. Conclusion: These results provide proof-of-concept evidence for multivariate, data-driven performance validity metrics. These metrics offer a novel method for determining the performance validity for individual neurocognitive tests that is scalable, applicable across different tests, less burdensome, and dimensional. However, more research is needed into their application.

The Rey Dot Counting Test as a Tool for Detecting Suboptimal Performance in Athlete Baseline Testing

2020 ◽  
Author(s):  
Andrew DaCosta ◽  
Frank Webbe ◽  
Anthony LoGalbo
Keyword(s):  
Cognitive Testing ◽  
Performance Validity ◽  
Baseline Assessment ◽  
Baseline Testing ◽  
Validity Measures ◽  
Significant Disagreement ◽  
Two Measures ◽  
Validity Tests ◽  
Performance Validity Tests ◽  
Suboptimal Effort

Abstract Objective The limitations of Immediate Post-Concussion Assessment and Cognitive Testing (ImPACT)’s embedded validity measures (EVMs) are well-documented, as estimates suggest up to 35% of invalid baseline performances go undetected. Few studies have examined standalone performance validity tests (PVT) as a supplement to ImPACT’s EVMs. Method College athletes (n = 1,213) were administered a preseason baseline assessment that included ImPACT and the Rey Dot Counting Test (DCT), a standalone PVT, among other measures. Results Sixty-nine athletes (5.69%) met criteria for suboptimal effort on either ImPACT or the DCT. The DCT detected more cases of suboptimal effort (n = 50) than ImPACT (n = 21). A χ2 test of independence detected significant disagreement between the two measures, as only two individuals produced suboptimal effort on both (χ2(2) = 1.568, p = .210). Despite this disagreement, there were significant differences between the suboptimal effort DCT group and the adequate effort DCT group across all four ImPACT neurocognitive domains (U = 19,225.000, p < .001; U = 17,859.000, p < .001; U = 13,854.000, p < .001; U = 17,850.500, p < .001). Conclusions The DCT appears to detect suboptimal effort otherwise undetected by ImPACT’s EVMs.

scholarly journals A-225 The Rey Dot Counting Test as a Tool for Detecting Suboptimal Performance in Athlete Baseline Testing

2020 ◽  
Vol 35 (6) ◽  
pp. 1020-1020
Author(s):  
Dacosta A ◽  
Roccaforte A ◽  
Sohoni R ◽  
Crane A ◽  
Webbe F ◽  
...  
Keyword(s):  
Cognitive Testing ◽  
Performance Validity ◽  
Baseline Assessment ◽  
Baseline Testing ◽  
Validity Measures ◽  
Significant Disagreement ◽  
Two Measures ◽  
Validity Tests ◽  
Performance Validity Tests ◽  
Suboptimal Effort

Abstract Objective The limitations of Immediate Post-Concussion Assessment and Cognitive Testing (ImPACT)‘s embedded validity measures (EVMs) are well-documented, as estimates suggest up to 35% of invalid baseline performances go undetected (Gaudet & Weyandt, 2017). Few studies have examined standalone performance validity tests (PVT) as a supplement to ImPACT’s EVMs (Gaudet & Weyandt, 2017). Method College athletes (n = 1,213) were administered a preseason baseline assessment that included ImPACT and the Rey Dot Counting Test (DCT), a standalone PVT, among other measures. Results Sixty-nine athletes (5.69%) met criteria for suboptimal effort on either ImPACT or the DCT. The DCT detected more cases of suboptimal effort (n = 50) than ImPACT (n = 21). A χ2 test of independence detected significant disagreement between the two measures, as only 2 individuals produced suboptimal effort on both (χ2(2) = 1.568, p = .210). Despite this disagreement, there were significant differences between the suboptimal effort DCT group and the adequate effort DCT group across all four ImPACT neurocognitive domains (U = 19225.000, p < .001; U = 17859.000, p < .001; U = 13854.000, p < .001; U = 17850.500, p < .001). Conclusions The DCT appears to detect suboptimal effort otherwise undetected by ImPACT’s EVMs.

scholarly journals A-207 Applying the Hartford Consistency Index to the Brief Visuospatial Memory Test—Revised for Assessing Performance Validity in an Outpatient Sample: A Pilot Study

2020 ◽  
Vol 35 (6) ◽  
pp. 1002-1002
Author(s):  
Sheikh K ◽  
Peck C
Keyword(s):  
Digit Span ◽  
Area Under The Curve ◽  
Memory Test ◽  
Consistency Index ◽  
Visuospatial Memory ◽  
Performance Validity ◽  
Performance Validity Measures ◽  
Validity Tests ◽  
External Incentives ◽  
Performance Validity Tests

Abstract Objective Prior studies have examined indices within the Brief Visuospatial Memory Test—Revised (BVMT-R) as potential embedded performance validity tests (PVT). Findings from these studies, however, are limited and with mixed results. Therefore, the purpose of the current study was to compare the classification accuracy of the Hartford Consistency Index (HCI) with published BVMT-R performance validity measures in an outpatient sample. Method A total of 115 archival files met study inclusion criteria: a) ≥ 18 years-old; b) administered > 2 PVTs (Reliable Digit Span, Dot Counting Test, and Test of Memory Malingering); and c) no diagnoses of intellectual disability or dementia. Utilizing standard cutoffs, participants were classified as ‘Valid’ (n = 94) or ‘Invalid’ (n = 21). ‘Valid’ profiles passed all PVTs and were free of known external incentives while ‘Invalid’ profiles failed ≥2 PVTs. Results An HCI cutoff of < 1 yielded 90% specificity, 48% sensitivity, and the area under the curve (AUC = .70) was adequate. Applying published cutoffs for Recognition Hits (≤4) and Percent Retention (≤58%) to our sample produced > 90% specificity, but sensitivity rates were < 40% and AUCs were consistently < .70. Similarly, the Recognition Discrimination (≤4) cutoff revealed inadequate specificity (84%), but acceptable sensitivity (63%), and AUC (.73). Conclussions Results from our study support the use of the HCI as an embedded PVT within the BVMT-R for non-demented outpatient samples. Furthermore, the HCI outperformed other embedded PVTs examined. Limitations of our study and future directions are discussed.

Development and Initial Validation of the Perceptual Assessment of Memory (PASSOM): A Simulator Study

2021 ◽  
Author(s):  
Justin C Koenitzer ◽  
Janice E Herron ◽  
Jesse W Whitlow ◽  
Catherine M Barbuscak ◽  
Nitin R Patel ◽  
...  
Keyword(s):  
Undergraduate Students ◽  
Performance Validity ◽  
Cut Scores ◽  
Response Latencies ◽  
Initial Validation ◽  
Neurocognitive Tests ◽  
Linear Regression Modeling ◽  
Validity Tests ◽  
Perceptual Assessment ◽  
Performance Validity Tests

Abstract Objective Performance validity tests (PVTs) are an integral component of neuropsychological assessment. There is a need for the development of more PVTs, especially those employing covert determinations. The aim of the present study was to provide initial validation of a new computerized PVT, the Perceptual Assessment of Memory (PASSOM). Method Participants were 58 undergraduate students randomly assigned to a simulator (SIM) or control (CON) group. All participants were provided written instructions for their role prior to testing and were administered the PASSOM as part of a brief battery of neurocognitive tests. Indices of interest included response accuracy for Trials 1 and 2, and total errors across Trials, as well as response time (RT) for Trials 1 and 2, and total RT for both Trials. Results The SIM group produced significantly more errors than the CON group for Trials 1 and 2, and committed more total errors across trials. Significantly longer response latencies were found for the SIM group compared to the CON group for all RT indices examined. Linear regression modeling indicated excellent group classification for all indices studied, with areas under the curve ranging from 0.92 to 0.95. Sensitivity and specificity rates were good for several cut scores across all of the accuracy and RT indices, and sensitivity improved greatly by combining RT cut scores with the more traditional accuracy cut scores. Conclusion Findings demonstrate the ability of the PASSOM to distinguish individuals instructed to feign cognitive impairment from those told to perform to the best of their ability.

Inconsistent responding on the MMPI-2-RF and uncooperative attitude: Evidence from cognitive performance validity measures.

2018 ◽  
Vol 30 (3) ◽  
pp. 410-415 ◽  
Author(s):  
Roger O. Gervais ◽  
Anthony M. Tarescavage ◽  
Manfred F. Greiffenstein ◽  
Dustin B. Wygant ◽  
Cheryl Deslauriers ◽  
...  
Keyword(s):  
Cognitive Performance ◽  
Performance Validity ◽  
Validity Measures ◽  
Performance Validity Measures ◽  
Inconsistent Responding

A Bayesian Approach to Mixed Group Validation of Performance Validity Tests

2014 ◽  
Author(s):  
Douglas Mossman ◽  
William Miller ◽  
Elliot Lee ◽  
Roger Gervais ◽  
Kathleen Hart ◽  
...  
Keyword(s):  
Bayesian Approach ◽  
Performance Validity ◽  
Mixed Group ◽  
Validity Tests ◽  
Performance Validity Tests

scholarly journals Calibration of Camera and Flash LiDAR System with a Triangular Pyramid Target

2021 ◽  
Vol 11 (2) ◽  
pp. 582
Author(s):  
Zean Bu ◽  
Changku Sun ◽  
Peng Wang ◽  
Hang Dong
Keyword(s):  
Simulated Data ◽  
Real Data ◽  
Calibration Method ◽  
Multiple Sensors ◽  
Triangular Pyramid ◽  
World Coordinate System ◽  
Flash Lidar ◽  
Novel Method ◽  
3D Information ◽  
Incremental Validation

Calibration between multiple sensors is a fundamental procedure for data fusion. To address the problems of large errors and tedious operation, we present a novel method to conduct the calibration between light detection and ranging (LiDAR) and camera. We invent a calibration target, which is an arbitrary triangular pyramid with three chessboard patterns on its three planes. The target contains both 3D information and 2D information, which can be utilized to obtain intrinsic parameters of the camera and extrinsic parameters of the system. In the proposed method, the world coordinate system is established through the triangular pyramid. We extract the equations of triangular pyramid planes to find the relative transformation between two sensors. One capture of camera and LiDAR is sufficient for calibration, and errors are reduced by minimizing the distance between points and planes. Furthermore, the accuracy can be increased by more captures. We carried out experiments on simulated data with varying degrees of noise and numbers of frames. Finally, the calibration results were verified by real data through incremental validation and analyzing the root mean square error (RMSE), demonstrating that our calibration method is robust and provides state-of-the-art performance.

Sign in / Sign up

Export Citation Format

Share Document