Development of a new virtual reality test of cognition: assessing the test-retest reliability, convergent and ecological validity of CONVIRT

Abstract Background: The Box and Block Test (BBT) has been widely used to assess gross upper extremity (UE) motor function. We designed a haptic-combined virtual reality (VR) system, named VBBT, to improve BBT with kinematic parameters for more specific assessments.Methods: According to the block-transfer task in BBT, we designed a VBBT system with a haptic device to provide a sense of the grasping force and block's gravity during task performance for UE function assessment. Besides the number of transferred blocks (N-TB), some kinematic parameters were also collected in VBBT for more specific assessment, including the number of zero-crossings of moving acceleration (NZC-ACC), the number of zero-crossings of derivative of releasing force (NZC-DRF), the ratio of path length and linear length (PLR) and the distance between barrier partition and drop position (DDP). 16 post-stroke patients and 113 healthy subjects were recruited to examine quantitative performances, concurrent validity, test-retest reliability and subjective preference between the BBT and VBBT.Results: Based on the measurements of 95% healthy subjects, the normative ranges of quantitative performances were established for BBT and VBBT. Deficiencies in patients’ UE function could be identified when their measurements fell outsides. A moderate correlation was found in the N-TBs between the VBBT and BBT (r = 0.42). The measurements in VBBT presented a stronger age-related correlation than that in BBT (R2 = 0.57 and R2 = 0.16). The N-TBs in both BBT and VBBT were strongly correlated to the Action Research Arm Test (ARAT) (|r| = 0.84 and 0.83), and the NZC-ACC and NZC-DRF in VBBT also showed significant correlations. (|r| = 0.76 and 0.79). The N-TB, NZC-ACC and NZC-DRF in VBBT showed a good test-retest reliability (ICC = 0.75, 0.78 and 0.80) while a moderate reliability was found in BBT (ICC = 0.62). For the patient preference, VBBT was given a higher score than the BBT (p < 0.05) for its enjoyment and completion effort performing.Conclusion: The VBBT improved a gross manual assessment in BBT, which can provide clinically validated, reliable and motivative assessment with kinematic parameters for specific UE motor functions of post-stroke patients.

Download Full-text

Measuring Correlates of Mental Workload During Simulated Driving Using cEEGrid Electrodes: A Test–Retest Reliability Analysis

Frontiers in Neuroergonomics ◽

10.3389/fnrgo.2021.729197 ◽

2021 ◽

Vol 2 ◽

Author(s):

Stephan Getzmann ◽

Julian E. Reiser ◽

Melanie Karthaus ◽

Georg Rudinger ◽

Edmund Wascher

Keyword(s):

Driving Simulator ◽

Mental Workload ◽

Ecological Validity ◽

Internal Validity ◽

Steering Wheel ◽

Mental Processes ◽

Retest Reliability ◽

Frequency Bands ◽

Theta Frequency ◽

Test Retest Reliability

The EEG reflects mental processes, especially modulations in the alpha and theta frequency bands are associated with attention and the allocation of mental resources. EEG has also been used to study mental processes while driving, both in real environments and in virtual reality. However, conventional EEG methods are of limited use outside of controlled laboratory settings. While modern EEG technologies offer hardly any restrictions for the user, they often still have limitations in measurement reliability. We recently showed that low-density EEG methods using film-based round the ear electrodes (cEEGrids) are well-suited to map mental processes while driving a car in a driving simulator. In the present follow-up study, we explored aspects of ecological and internal validity of the cEEGrid measurements. We analyzed longitudinal data of 127 adults, who drove the same driving course in a virtual environment twice at intervals of 12–15 months while the EEG was recorded. Modulations in the alpha and theta frequency bands as well as within behavioral parameters (driving speed and steering wheel angular velocity) which were highly consistent over the two measurement time points were found to reflect the complexity of the driving task. At the intraindividual level, small to moderate (albeit significant) correlations were observed in about 2/3 of the participants, while other participants showed significant deviations between the two measurements. Thus, the test-retest reliability at the intra-individual level was rather low and challenges the value of the application for diagnostic purposes. However, across all participants the reliability and ecological validity of cEEGrid electrodes were satisfactory in the context of driving-related parameters.

Download Full-text

Multimodal immersive trail making – virtual reality paradigm to study cognitive-motor interactions

10.1101/2020.05.27.118760 ◽

2020 ◽

Author(s):

Meir Plotnik ◽

Oran Ben-Gal ◽

Glen M. Doniger ◽

Amihai Gottlieb ◽

Yotam Bahat ◽

...

Keyword(s):

Virtual Reality ◽

Executive Function ◽

Construct Validity ◽

Repeated Measures ◽

Discriminant Validity ◽

Middle Aged ◽

Retest Reliability ◽

Pencil And Paper ◽

Test Retest Reliability ◽

Completion Times

ABSTRACTBackgroundNeuropsychological tests of executive function have limited real-world predictive and functional relevance. An emerging solution for this limitation is to adapt the tests for implementation in virtual reality (VR). We thus developed two VR-based versions of the classic Color-Trails Test (CTT), a well-validated pencil-and-paper executive function test assessing sustained (Trails A) and divided (Trails B) attention - one for a large-scale VR system (DOME-CTT) and the other for a portable head-mount display VR system (HMD-CTT). We then evaluated construct validity, test-retest reliability, and age-related discriminant validity of the VR-based versions and explored effects on motor function.MethodsHealthy adults (n=147) in three age groups (young: n=50; middle-aged: n=80; older: n=17) participated. All participants were administered the original CTT, some completing the DOME-CTT (14 young, 29 middle-aged) and the rest completing the HMD-CTT. Primary outcomes were Trails A and B completion times (tA, tB). Spatiotemporal characteristics of upper-limb reaching movements during VR test performance were reconstructed from motion capture data. Statistics included correlations and repeated measures analysis of variance.ResultsConstruct validity was substantiated by moderate correlations between the ‘gold standard’ pencil-and-paper CTT and the VR adaptations (DOME-CTT: tA 0.58, tB 0.71; HMD-CTT: tA 0.62, tB 0.69). VR versions showed relatively high test-retest reliability (intraclass correlation; VR: tA 0.60-0.75, tB 0.59-0.89; original: tA 0.75-0.85, tB 0.77-0.80) and discriminant validity (area under the curve; VR: tA 0.70-0.92, tB 0.71-0.92; original: tA 0.73-0.95, tB 0.77-0.95). VR completion times were longer than for the original pencil-and-paper test; completion times were longer with advanced age. Compared with Trails A, Trails B target-to-target VR hand trajectories were characterized by delayed, more erratic acceleration and deceleration, consistent with the greater executive function demands of divided vs. sustained attention; acceleration onset later for older participants.ConclusionsThe present study demonstrates the feasibility and validity of converting a neuropsychological test from two-dimensional pencil-and-paper to three-dimensional VR-based format while preserving core neuropsychological task features. Findings on the spatiotemporal morphology of motor planning/execution during the cognitive tasks may lead to multimodal analysis methods that enrich the ecological validity of VR-based neuropsychological testing, representing a novel paradigm for studying cognitive-motor interactions.

Download Full-text

Test-retest reliability of the virtual reality sickness evaluation using electroencephalography (EEG)

Neuroscience Letters ◽

10.1016/j.neulet.2020.135589 ◽

2021 ◽

Vol 743 ◽

pp. 135589

Author(s):

Hyun Kyoon Lim ◽

Kyungha Ji ◽

Ye Shin Woo ◽

Dong-uk Han ◽

Dong-Hyun Lee ◽

...

Keyword(s):

Virtual Reality ◽

Retest Reliability ◽

Test Retest Reliability

Download Full-text

Stability of Attachment across Time and Context in a Dutch Sample

Psychological Reports ◽

10.2466/pr0.1986.58.1.23 ◽

1986 ◽

Vol 58 (1) ◽

pp. 23-32 ◽

Cited By ~ 12

Author(s):

F. A. Goossens ◽

M. H. Van Ijzendoorn ◽

L. W. C. Tavecchio ◽

P. M. Kroonenberg

Keyword(s):

Ecological Validity ◽

Strange Situation ◽

Attachment Relationship ◽

Retest Reliability ◽

The Stability ◽

Dutch Sample ◽

Test Retest Reliability ◽

Insight Into ◽

Ainsworth's Strange Situation ◽

At Home

Children were observed twice in Ainsworth's Strange Situation to measure the stability of their attachment relationship with the caregiver across 1 mo. To get some insight into the ecological validity of the procedure, the children were randomly divided over four sets of conditions: the home-home, home-lab, lab-home, and lab-lab conditions. The test-retest reliability of the interactive scores was high as was the stability of the classifications. The results of the Strange Situation at home and in the laboratory remain non-comparable.

Download Full-text

Test-retest reliability of functional connectivity networks during naturalistic fMRI paradigms

10.1101/087197 ◽

2016 ◽

Cited By ~ 2

Author(s):

Jiahui Wang ◽

Yudan Ren ◽

Xintao Hu ◽

Vinh Thai Nguyen ◽

Lei Guo ◽

...

Keyword(s):

Functional Connectivity ◽

Resting State ◽

Visual Processing ◽

Ecological Validity ◽

Brain Networks ◽

Viewing Condition ◽

Retest Reliability ◽

Fmri Session ◽

Test Retest Reliability ◽

Resting State Connectivity

AbstractFunctional connectivity analysis has become a powerful tool for probing the human brain function and its breakdown in neuropsychiatry disorders. So far, most studies adopted resting state paradigm to examine functional connectivity networks in the brain, thanks to its low demand and high tolerance that are essential for clinical studies. However, the test-retest reliability of resting state connectivity measures is moderate, potentially due to its low behavioral constraint. On the other hand, naturalistic neuroimaging paradigms, an emerging approach for cognitive neuroscience with high ecological validity, could potentially improve the reliability of functional connectivity measures. To test this hypothesis, we characterized the test-retest reliability of functional connectivity measures during a natural viewing condition, and benchmarked it against resting state connectivity measures acquired within the same functional magnetic resonance imaging (fMRI) session. We found that the reliability of connectivity and graph theoretical measures of brain networks is significantly improved during natural viewing conditions over resting state conditions, with an average increase of almost 50% across various connectivity measures. Not only sensory networks for audio-visual processing become more reliable, higher order brain networks, such as default mode and attention networks, also appear to show higher reliability during natural viewing. Our results support the use of natural viewing paradigms in estimating functional connectivity of brain networks, and have important implications for clinical application of fMRI.

Download Full-text

The Comparison of a Virtual Reality Headset and Force plate to Investigate Human Postural Control (Preprint)

10.2196/preprints.24950 ◽

2020 ◽

Author(s):

Brian Sylcott ◽

Chia-Cheng Lin ◽

Keith Williams ◽

Mark Hinderaker

Keyword(s):

Virtual Reality ◽

Postural Control ◽

Postural Sway ◽

Center Of Pressure ◽

Low Cost ◽

Intraclass Correlation ◽

Force Plate ◽

Retest Reliability ◽

Position Data ◽

Test Retest Reliability

BACKGROUND Accurately measuring postural sway is an important part of balance assessment and rehabilitation. While force plates give accurate measurements, their cost and space requirements make their use impractical in many situations. OBJECTIVE The work presented here is aimed at addressing this issue by validating a virtual reality (VR) headset as a relatively low-cost alternative for postural sway measurement. The HTC Vive (VR) headset has built-in sensors that allow for position and orientation tracking making it a potentially eﬀective tool for balance assessments. METHODS Participants in this study were asked to stand upright on a force plate (Neurocom platform) while wearing the HTC ViveTM. Position data was collected from the headset and force plate simultaneously as participants experienced a custom-built VR environment that covered their entire field of view. The Intraclass Correlation Coefficient (ICC) was used to examined the test-retest reliability on the postural control variables, including normalized path length (NPL), root-mean-square (RMS), and peak-to-peak (P2P), computed from the VR position output and the center of pressure (COP) data from the force plate. Liner regression were used to investigate the correlation between the VR and force plate measurements. RESULTS Results showed that the test-retest reliability of VR headset was similar to COP in RMS and P2P, ranging from .285 to .636 in VR and .228 to .759. Linear regression between VR and COP measures showed significant correlation in RMS and P2P. CONCLUSIONS Based on our results, the VR headset has potential for use in postural control measurements. However, further development of software and testing protocols for balance assessments is needed.

Download Full-text

Development and Evaluation of an Ecological Task to Assess Executive Functioning Post Childhood TBI: The Children's Cooking Task

Brain Impairment ◽

10.1375/brim.11.2.125 ◽

2010 ◽

Vol 11 (2) ◽

pp. 125-143 ◽

Cited By ~ 28

Author(s):

Mathilde P. Chevignard ◽

Cathy Catroppa ◽

Jane Galvin ◽

Vicki Anderson

Keyword(s):

Executive Function ◽

Executive Functioning ◽

Executive Functions ◽

Concurrent Validity ◽

Discriminant Validity ◽

Ecological Validity ◽

Retest Reliability ◽

Behavior Rating Inventory ◽

Severe Tbi ◽

Test Retest Reliability

AbstractPurpose:Traumatic brain injury (TBI) often leads to executive functions deficits, which may be responsible for severe and longstanding disabilities in everyday activities. Sensitivity and ecological validity of neuropsychological tests of executive functions have been questioned. The aims of this study were to pilot an ecological open-ended assessment of executive functions in children, the ‘Children's Cooking Task’ (CCT), specifically to report its reliability, discriminant validity and concurrent validity.Methods:Twenty-five children with mild (n= 10) or moderate-to-severe TBI (n= 15), and 21 matched controls (aged 8 to 20 years) participated in the study. An open-ended cooking task was designed to test multi-tasking abilities. It required the preparation of two simple recipes using specific instructions. Outcome measures included the number of errors and an overall qualitative analysis of the task. Validating measures of executive functions included the Delis Kaplan Executive Function System, the Six-Part Test and two questionnaires completed by the child's primary care-giver: the Behavior Rating Inventory of Executive Function and the Dysexecutive Questionnaire for Children.Results:Internal consistency of the Children's Cooking Task was high (Cronbach's alpha = .86), as was test–retest reliability (ICC = .89). Children with moderate-to-severe TBI, as well as children with mild TBI made significantly more errors in the Children's Cooking Task in comparison to controls (p< .001). The CCT was correlated with several tests and one questionnaire of executive functioning (Trails, verbal fluency, sorting, 20 questions, Dysexecutive Questionnaire).Discussion and Conclusion:The Children's Cooking Task has good interrater and test–retest reliability, as well as good discriminant and concurrent validity.

Download Full-text