A kinematic-acoustic analysis of gesture-speech coupling in persons with aphasia

Aphasia is a profound speech pathology hampering speech production and/or comprehension. People with Aphasia (PWA) use more manual gestures than neurotypicals. This intuitively invokes the idea of some kind of compensatory mechanism, but there is variable evidence of this gesture-boosting effect on speech processes. The status quo in gesture research with PWA is an emphasis on categorical analysis of gesture types, how often they are recruited, and whether they aid communication or speaking. However, there are increasingly louder calls for the investigation of gesture and speech as continuous entangled modes of expression. In neurotypical adults, expressive moments of gesture and speech are synchronized. It is unknown how this multimodal prosody is instantiated in PWA. In the current study we perform the first acoustic-kinematic gesture-speech analysis in persons of aphasia (Wernicke, Broca, Anomic) relative to age-matched controls, where we apply several multimodal signal processing methods (e.g., motion tracking). Specifically, we related the speech peaks (smoothed amplitude envelope change) with that of the nearest peaks in the gesture acceleration profile. We obtained that the magnitude of gesture vs. speech peaks are positively related across the groups, though more variably for PWA, and such coupling was related to less severe Aphasia-related symptoms in PWA. When comparing the temporal ordering of speech envelope versus acceleration peaks, no statistically reliable differences between controls and PWA were observed. We did however obtain when gesture kinematic peaks tended to anticipate speech peaks more, this was related to less severe aphasia-related symptoms. Finally, we show that both gesture and speech have slower quasi-rhythmic structure, indicating that next to speech, gesture is slowed down too. We conclude that future aphasia research will benefit from the study of gesture-speech interconnections.