Neurophysiological evidence for rapid processing of verbal and gestural information in understanding communicative actions

Abstract During everyday social interaction, gestures are a fundamental part of human communication. The communicative pragmatic role of hand gestures and their interaction with spoken language has been documented at the earliest stage of language development, in which two types of indexical gestures are most prominent: the pointing gesture for directing attention to objects and the give-me gesture for making requests. Here we study, in adult human participants, the neurophysiological signatures of gestural-linguistic acts of communicating the pragmatic intentions of naming and requesting by simultaneously presenting written words and gestures. Already at ~150 ms, brain responses diverged between naming and request actions expressed by word-gesture combination, whereas the same gestures presented in isolation elicited their earliest neurophysiological dissociations significantly later (at ~210 ms). There was an early enhancement of request-evoked brain activity as compared with naming, which was due to sources in the frontocentral cortex, consistent with access to action knowledge in request understanding. In addition, an enhanced N400-like response indicated late semantic integration of gesture-language interaction. The present study demonstrates that word-gesture combinations used to express communicative pragmatic intentions speed up the brain correlates of comprehension processes – compared with gesture-only understanding – thereby calling into question current serial linguistic models viewing pragmatic function decoding at the end of a language comprehension cascade. Instead, information about the social-interactive role of communicative acts is processed instantaneously.

Download Full-text

Content-based Dissociation of Hippocampal Involvement in Prediction

Journal of Cognitive Neuroscience ◽

10.1162/jocn_a_01509 ◽

2020 ◽

Vol 32 (3) ◽

pp. 527-545 ◽

Cited By ~ 4

Author(s):

Peter Kok ◽

Lindsay I. Rait ◽

Nicholas B. Turk-Browne

Keyword(s):

Brain Activity ◽

Predictive Processing ◽

Auditory Cues ◽

Domain Specific ◽

Visual Objects ◽

Complex Shapes ◽

Different Types ◽

Human Participants ◽

Abstract Shapes

Recent work suggests that a key function of the hippocampus is to predict the future. This is thought to depend on its ability to bind inputs over time and space and to retrieve upcoming or missing inputs based on partial cues. In line with this, previous research has revealed prediction-related signals in the hippocampus for complex visual objects, such as fractals and abstract shapes. Implicit in such accounts is that these computations in the hippocampus reflect domain-general processes that apply across different types and modalities of stimuli. An alternative is that the hippocampus plays a more domain-specific role in predictive processing, with the type of stimuli being predicted determining its involvement. To investigate this, we compared hippocampal responses to auditory cues predicting abstract shapes (Experiment 1) versus oriented gratings (Experiment 2). We measured brain activity in male and female human participants using high-resolution fMRI, in combination with inverted encoding models to reconstruct shape and orientation information. Our results revealed that expectations about shape and orientation evoked distinct representations in the hippocampus. For complex shapes, the hippocampus represented which shape was expected, potentially serving as a source of top–down predictions. In contrast, for simple gratings, the hippocampus represented only unexpected orientations, more reminiscent of a prediction error. We discuss several potential explanations for this content-based dissociation in hippocampal function, concluding that the computational role of the hippocampus in predictive processing may depend on the nature and complexity of stimuli.

Download Full-text

Eye-movement reinstatement and neural reactivation during mental imagery

10.1101/107953 ◽

2017 ◽

Cited By ~ 1

Author(s):

Michael B. Bone ◽

Marie St-Laurent ◽

Christa Dang ◽

Douglas A. McQuiggan ◽

Jennifer D. Ryan ◽

...

Keyword(s):

Eye Movement ◽

Mental Imagery ◽

Brain Activity ◽

Healthy Human ◽

Supportive Role ◽

Constructive Process ◽

Human Participants ◽

Multivariate Pattern ◽

With Memory

AbstractHalf a century ago, Donald Hebb posited that mental imagery is a constructive process that emulates perception. Specifically, Hebb claimed that visual imagery results from the reactivation of neural activity associated with viewing images. He also argued that neural reactivation and imagery benefit from the re-enactment of eye movement patterns that first occurred at viewing (fixation reinstatement). To investigate these claims, we applied multivariate pattern analyses to functional MRI (fMRI) and eye-tracking data collected while healthy human participants repeatedly viewed and visualized complex images. We observed that the specificity of neural reactivation correlated positively with vivid imagery and with memory for stimulus image details. Moreover, neural reactivation correlated positively with fixation reinstatement, meaning that image-specific eye movements accompanied image-specific patterns of brain activity during visualization. These findings support the conception of mental imagery as a simulation of perception, and provide evidence of the supportive role of eye-movement in neural reactivation.

Download Full-text

The role of trait empathy in the processing of observed actions in a false-belief task

Social Cognitive and Affective Neuroscience ◽

10.1093/scan/nsaa009 ◽

2020 ◽

Vol 15 (1) ◽

pp. 53-61

Author(s):

Christian Bellebaum ◽

Marta Ghio ◽

Marie Wollmer ◽

Benjamin Weismüller ◽

Patrizia Thoma

Keyword(s):

Brain Activity ◽

Event Related Potential ◽

False Belief ◽

Reactivity Index ◽

False Belief Task ◽

Linear Mixed Effects ◽

Brain Responses ◽

Trait Empathy ◽

Empathy Quotient

Abstract Empathic brain responses are characterized by overlapping activations between active experience and observation of an emotion in another person, with the pattern for observation being modulated by trait empathy. Also for self-performed and observed errors, similar brain activity has been described, but findings concerning the role of empathy are mixed. We hypothesized that trait empathy modulates the processing of observed responses if expectations concerning the response are based on the beliefs of the observed person. In the present study, we utilized a false-belief task in which observed person’s and observer’s task-related knowledge were dissociated and errors and correct responses could be expected or unexpected. While theta power was generally modulated by the expectancy of the observed response, a negative mediofrontal event-related potential (ERP) component was more pronounced for unexpected observed actions only in participants with higher trait empathy (assessed by the Empathy Quotient), as revealed by linear mixed effects analyses. Cognitive and affective empathy, assessed by the Interpersonal Reactivity Index, were not significantly related to the ERP component. The results suggest that trait empathy can facilitate the generation of predictions and thereby modulate specific aspects of the processing of observed actions, while the contributions of specific empathy components remain unclear.

Download Full-text

The left angular gyrus is causally involved in information buffering and context formation: evidence from a narrative reading task

10.1101/841478 ◽

2019 ◽

Author(s):

Francesca M. Branzi ◽

Gorana Pobric ◽

JeYoung Jung ◽

Matthew A. Lambon Ralph

Keyword(s):

Semantic Processing ◽

Contextual Information ◽

Memory Task ◽

Semantic Integration ◽

Angular Gyrus ◽

Reading Task ◽

Context Dependent ◽

Human Participants ◽

Left Angular Gyrus

AbstractThe role of the left angular gyrus (Ag) in semantic processing remains unclear. In this study, we used transcranial magnetic stimulation (TMS) to test the hypothesis that the left Ag supports the information buffering processes necessary for context-dependent semantic integration (narrative reading task). We applied online TMS to the left Ag to disrupt the buffering processes while female and male human participants integrated information between two paragraphs of text presented sequentially (i.e., context and target conditions). We assessed the effect of TMS on the left Ag by recording reading times for target conditions during the reading task and by asking participants to retrieve contextual information given the target condition as cue in a successive memory task. TMS applied over the left Ag during the reading task impaired the retrieval of contextual information in the memory task without affecting reading times. These results suggest that the Ag supports information buffering and context formation. This study provides the first evidence of a causal role of the left Ag in language tasks that require context-dependent semantic integration.Significance statementThe present study offers novel insights into the processes that are associated with the left angular gyrus (Ag) activation during context-dependent naturalistic language processing. We used TMS while human participants read written narratives to interfere with online language integration. We measured how TMS over the left Ag was affecting reading times and retrieval of integrated narrative representations. We were able to show that the left Ag is causally involved in information buffering and context formation.

Download Full-text

Content-based dissociation of hippocampal involvement in prediction

10.1101/568303 ◽

2019 ◽

Author(s):

Peter Kok ◽

Lindsay I. Rait ◽

Nicholas B. Turk-Browne

Keyword(s):

Prior Knowledge ◽

Brain Activity ◽

Memory Systems ◽

General Purpose ◽

Predictive Processing ◽

Complex Object ◽

Complex Objects ◽

Human Participants ◽

Simple Features

AbstractIt has recently become clear that one of the key functions of the hippocampus is to predict future inputs. In line with this, previous research has revealed prediction-related signals in the hippocampus for complex visual objects, such as fractals and abstract shapes. Based on this, it has been suggested that the hippocampus may generate perceptual expectations, especially when relying on rapidly learned predictive associations between arbitrary stimuli. However, it is currently unknown whether the hippocampus implements general-purpose computations that subserve all associative predictions, regardless of stimulus properties, or whether the involvement of the hippocampus is stimulus-dependent. To investigate this, we exposed male and female human participants to complex auditory cues predicting either the shape of a complex object (Experiment 1) or the orientation of a simple line grating (Experiment 2). We measured brain activity using high-resolution functional magnetic resonance imaging (fMRI), in combination with inverted encoding models to reconstruct shape and orientation representations in visual cortex and the hippocampus. Our results revealed that expectations about shape and orientation evoked distinct representations in the hippocampus. For complex shapes, the hippocampus represented which shape was expected, potentially serving as a source of top-down predictions. In contrast, for simple gratings, the hippocampus represented only unexpected orientations, more reminiscent of a prediction error. We discuss several potential explanations for this dissociation, concluding that the computational role of the hippocampus in predictive processing depends upon the nature and complexity of stimuli.Significance StatementTo deal with the noisy and ambiguous sensory signals received by our brain, it is crucial to use prior knowledge of the world to guide perception. Previous research suggests that the hippocampus is involved in predicting upcoming visual stimuli based on prior knowledge. In the current study, we show that hippocampal prediction is specific to expectations of complex objects, whereas for simple features the hippocampus generates an opposite prediction error signal instead. These findings demonstrate that the computational role of the hippocampus can be content-sensitive and refine our understanding of the involvement of memory systems in predictive processing.

Download Full-text

Behavior in the Brain

Journal of Psychophysiology ◽

10.1027/0269-8803/a000016 ◽

2010 ◽

Vol 24 (2) ◽

pp. 76-82 ◽

Cited By ~ 15

Author(s):

Martin M. Monti ◽

Adrian M. Owen

Keyword(s):

Motor Behavior ◽

Functional Neuroimaging ◽

Brain Activity ◽

Motor Output ◽

The Unconscious ◽

Ethical Implications ◽

Brain Responses ◽

Minimally Conscious ◽

Brain Insults ◽

Brain Behavior

Recent evidence has suggested that functional neuroimaging may play a crucial role in assessing residual cognition and awareness in brain injury survivors. In particular, brain insults that compromise the patient’s ability to produce motor output may render standard clinical testing ineffective. Indeed, if patients were aware but unable to signal so via motor behavior, they would be impossible to distinguish, at the bedside, from vegetative patients. Considering the alarming rate with which minimally conscious patients are misdiagnosed as vegetative, and the severe medical, legal, and ethical implications of such decisions, novel tools are urgently required to complement current clinical-assessment protocols. Functional neuroimaging may be particularly suited to this aim by providing a window on brain function without requiring patients to produce any motor output. Specifically, the possibility of detecting signs of willful behavior by directly observing brain activity (i.e., “brain behavior”), rather than motoric output, allows this approach to reach beyond what is observable at the bedside with standard clinical assessments. In addition, several neuroimaging studies have already highlighted neuroimaging protocols that can distinguish automatic brain responses from willful brain activity, making it possible to employ willful brain activations as an index of awareness. Certainly, neuroimaging in patient populations faces some theoretical and experimental difficulties, but willful, task-dependent, brain activation may be the only way to discriminate the conscious, but immobile, patient from the unconscious one.

Download Full-text

Modality and folk psychology

Studies in history and philosophy of science and technology ◽

10.15421/271903 ◽

2019 ◽

Vol 28 (1) ◽

pp. 19-27

Author(s):

Ja. O. Petik

Keyword(s):

Modal Logic ◽

Brain Activity ◽

Formal System ◽

Philosophy Of Logic ◽

Psychological States ◽

Formal Systems ◽

Modern Psychology ◽

Philosophical Interpretation ◽

Important Direction

The connection of the modern psychology and formal systems remains an important direction of research. This paper is centered on philosophical problems surrounding relations between mental and logic. Main attention is given to philosophy of logic but certain ideas are introduced that can be incorporated into the practical philosophical logic. The definition and properties of basic modal logic and descending ones which are used in study of mental activity are in view. The defining role of philosophical interpretation of modality for the particular formal system used for research in the field of psychological states of agents is postulated. Different semantics of modal logic are studied. The hypothesis about the connection of research in cognitive psychology (semantics of brain activity) and formal systems connected to research of psychological states is stated.

Download Full-text

Music and Gesture

10.1093/oso/9780190653637.003.0004 ◽

2017 ◽

Author(s):

Lawrence M. Zbikowski

Keyword(s):

Dynamic Processes ◽

Thought Processes ◽

Human Communication ◽

Charlie Chaplin ◽

Swing Time ◽

The Relationship

This chapter explores the relationship between music and physical gesture, drawing on recent research on the spontaneous gestures that accompany speech. Such gestures appear to be motivated by thought processes that are independent from speech and that in many cases offer analogs for dynamic processes. The chapter outlines the infrastructure for human communication that supports language and gesture as well as music. This outline provides a framework for exploring how music and gesture are similar and for how they are different. These comparisons are made through analyses of the movements Fred Astaire makes while accompanying himself at the piano in the 1936 film Swing Time and those Charlie Chaplin makes to Brahms’s Hungarian Dance No. 5 in the 1941 film The Great Dictator. These analyses further explicate the role of syntactic processes and syntactic layers in musical grammar and introduce referential frameworks, which serve as perceptual anchors for syntactic processes.

Download Full-text

Delta band activity contributes to the identification of command following in disorder of consciousness

Scientific Reports ◽

10.1038/s41598-021-95818-6 ◽

2021 ◽

Vol 11 (1) ◽

Author(s):

Gonzalo Rivera-Lillo ◽

Emmanuel A. Stamatakis ◽

Tristan A. Bekinschtein ◽

David K. Menon ◽

Srivas Chennu

Keyword(s):

Functional Neuroimaging ◽

Brain Activity ◽

Disorder Of Consciousness ◽

Fine Grained ◽

Electrical Brain Activity ◽

Delta Band ◽

Delta Modulation ◽

Command Following ◽

Band Activity

AbstractThe overt or covert ability to follow commands in patients with disorders of consciousness is considered a sign of awareness and has recently been defined as cortically mediated behaviour. Despite its clinical relevance, the brain signatures of the perceptual processing supporting command following have been elusive. This multimodal study investigates the temporal spectral pattern of electrical brain activity to identify features that differentiated healthy controls from patients both able and unable to follow commands. We combined evidence from behavioural assessment, functional neuroimaging during mental imagery and high-density electroencephalography collected during auditory prediction, from 21 patients and 10 controls. We used a penalised regression model to identify command following using features from electroencephalography. We identified seven well-defined spatiotemporal signatures in the delta, theta and alpha bands that together contribute to identify DoC subjects with and without the ability to follow command, and further distinguished these groups of patients from controls. A fine-grained analysis of these seven signatures enabled us to determine that increased delta modulation at the frontal sensors was the main feature in command following patients. In contrast, higher frequency theta and alpha modulations differentiated controls from both groups of patients. Our findings highlight a key role of spatiotemporally specific delta modulation in supporting cortically mediated behaviour including the ability to follow command. However, patients able to follow commands nevertheless have marked differences in brain activity in comparison with healthy volunteers.

Download Full-text

An Analysis of the External Validity of EEG Spectral Power in an Uncontrolled Outdoor Environment during Default and Complex Neurocognitive States

Brain Sciences ◽

10.3390/brainsci11030330 ◽

2021 ◽

Vol 11 (3) ◽

pp. 330

Author(s):

Dalton J. Edwards ◽

Logan T. Trujillo

Keyword(s):

External Validity ◽

Spectral Power ◽

Brain Activity ◽

Outdoor Environment ◽

Oscillatory Activity ◽

Brain State ◽

Beta Band ◽

Eeg Spectral Power ◽

Eeg Power ◽

Human Participants

Traditionally, quantitative electroencephalography (QEEG) studies collect data within controlled laboratory environments that limit the external validity of scientific conclusions. To probe these validity limits, we used a mobile EEG system to record electrophysiological signals from human participants while they were located within a controlled laboratory environment and an uncontrolled outdoor environment exhibiting several moderate background influences. Participants performed two tasks during these recordings, one engaging brain activity related to several complex cognitive functions (number sense, attention, memory, executive function) and the other engaging two default brain states. We computed EEG spectral power over three frequency bands (theta: 4–7 Hz, alpha: 8–13 Hz, low beta: 14–20 Hz) where EEG oscillatory activity is known to correlate with the neurocognitive states engaged by these tasks. Null hypothesis significance testing yielded significant EEG power effects typical of the neurocognitive states engaged by each task, but only a beta-band power difference between the two background recording environments during the default brain state. Bayesian analysis showed that the remaining environment null effects were unlikely to reflect measurement insensitivities. This overall pattern of results supports the external validity of laboratory EEG power findings for complex and default neurocognitive states engaged within moderately uncontrolled environments.

Download Full-text