Multimodal biomarkers based on behavioral, neurophysiological, and cognitive measurements have recently increased in popularity for the detection of cognitive stress and neurologically based disorders. Such conditions significantly and adversely affect human performance and quality of life in a large fraction of the world’s population. Example modalities used in detection of these conditions include speech, facial expression, physiology, eye tracking, gait, and electroencephalography (EEG). Toward the goal of finding simple, noninvasive means to detect, predict, and monitor cognitive stress and neurological conditions, MIT Lincoln Laboratory is developing biomarkers that satisfy three criteria. First, we seek biomarkers that reflect core components of cognitive status, such as working memory capacity, processing speed, attention, and arousal. Second, and as importantly, we seek biomarkers that reflect timing and coordination relations both within components of each modality and across different modalities. This is based on the hypothesis that neural coordination across different parts of the brain is essential in cognition. An example of timing and coordination within a modality is the set of finely timed and synchronized physiological components of speech production, whereas an example of coordination across modalities is the timing and synchrony that occur between speech and facial expression during speaking. Third, we seek multimodal biomarkers that contribute in a complementary fashion under various channel and background conditions. In this chapter, as an illustration of the biomarker approach, we focus on cognitive stress and the particular case of detecting different cognitive load levels. We also briefly show how similar feature-extraction principles can be applied to a neurological condition through the example of major depressive disorder (MDD). MDD is one of several neuropsychiatric disorders where multimodal biomarkers based on principles of timing and coordination are important for detection (Cummins et al., 2015; Helfer et al., 2014; Quatieri & Malyska, 2012; Trevino, Quatieri, & Malyska, 2011; Williamson, Quatieri, Helfer, Ciccarelli, & Mehta, 2014; Williamson et al., 2013, 2015; Yu, Quatieri, Williamson, & Mundt, 2014).
Faces and fitness: attractive evolutionary relationship or ugly hypothesis?
