Supplemental Material for The Role of Audiovisual Integration in the Perception of Attractiveness

We assessed the role of audiovisual integration in selective attention by testing selective attention to sound. Participants were asked to focus on one audio speech stream out of two audio streams presented simultaneously at different pitch. We measured recall of words from the cued or the uncued sentence using a 2AFC at the end of each trial. A video-clip of the mouth of a speaker was presented in the middle of the display, matching one of the two simultaneous auditory streams (50% of the time it matched the cued sentence and the rest the uncued one). In Experiment 1 the cue was 75% valid. Recall in the valid trials was better than in the invalid ones. The critical result was, however, that only in the valid condition we did find differences between audio–visual matching and audio-visually mismatching sentences. On the invalid condition these differences were not found. In Experiment 2 the cue to the relevant sentence was 100% valid, and we included a control condition where the lips didn’t match either of the sentences. When the lips matched the cued sentence performance was better than when they matched the uncued sentence or none of them, suggesting a benefit of audiovisual matching rather than a cost of mismatch. Our results indicate that attention to acoustic frequency (pitch) plays an important role in what sounds benefit from multisensory integration.

Download Full-text

The role of audiovisual integration in the perception of attractiveness.

Evolutionary Behavioral Sciences ◽

10.1037/ebs0000128 ◽

2019 ◽

Vol 13 (1) ◽

pp. 1-15

Author(s):

Alexis T. Mook ◽

Aaron D. Mitchel

Keyword(s):

Audiovisual Integration

Download Full-text

Assessing the role of attention in the audiovisual integration of speech

Information Fusion ◽

10.1016/j.inffus.2009.04.001 ◽

2010 ◽

Vol 11 (1) ◽

pp. 4-11 ◽

Cited By ~ 39

Author(s):

Jordi Navarra ◽

Agnès Alsius ◽

Salvador Soto-Faraco ◽

Charles Spence

Keyword(s):

Audiovisual Integration

Download Full-text

The role of emotion in dynamic audiovisual integration of faces and voices

Social Cognitive and Affective Neuroscience ◽

10.1093/scan/nsu105 ◽

2014 ◽

Vol 10 (5) ◽

pp. 713-720 ◽

Cited By ~ 32

Author(s):

Jenny Kokinous ◽

Sonja A. Kotz ◽

Alessandro Tavano ◽

Erich Schröger

Keyword(s):

Audiovisual Integration

Download Full-text

Semantic Congruency Modulates the Effect of Attentional Load on the Audiovisual Integration of Animate Images and Sounds

i-Perception ◽

10.1177/2041669520981096 ◽

2020 ◽

Vol 11 (6) ◽

pp. 204166952098109

Author(s):

Qingqing Li ◽

Qiong Wu ◽

Yiyang Yu ◽

Fengxia Wu ◽

Satoshi Takahashi ◽

...

Keyword(s):

Critical Role ◽

Semantic Content ◽

Visual Presentation ◽

Audiovisual Integration ◽

Attentional Processes ◽

Semantic Congruency ◽

Attentional Load ◽

Animal Identification ◽

Animal Images

Attentional processes play a complex and multifaceted role in the integration of input from different sensory modalities. However, whether increased attentional load disrupts the audiovisual (AV) integration of common objects that involve semantic content remains unclear. Furthermore, knowledge regarding how semantic congruency interacts with attentional load to influence the AV integration of common objects is limited. We investigated these questions by examining AV integration under various attentional-load conditions. AV integration was assessed by adopting an animal identification task using unisensory (animal images and sounds) and AV stimuli (semantically congruent AV objects and semantically incongruent AV objects), while attentional load was manipulated by using a rapid serial visual presentation task. Our results indicate that attentional load did not attenuate the integration of semantically congruent AV objects. However, semantically incongruent animal sounds and images were not integrated (as there was no multisensory facilitation), and the interference effect produced by the semantically incongruent AV objects was reduced by increased attentional-load manipulations. These findings highlight the critical role of semantic congruency in modulating the effect of attentional load on the AV integration of common objects.

Download Full-text

Effect of bilateral opercular syndrome on speech perception

Neurobiology of Language ◽

10.1162/nol_a_00037 ◽

2021 ◽

pp. 1-45

Author(s):

Grant M. Walker ◽

Patrick Rollo ◽

Nitin Tandon ◽

Gregory Hickok

Keyword(s):

Speech Perception ◽

Language Processing ◽

Short Term Memory ◽

Audiovisual Integration ◽

Mcgurk Effect ◽

Control Center ◽

Cortical Motor ◽

Opercular Syndrome ◽

Motor Representations

Abstract Speech perception ability and structural neuroimaging were investigated in two cases of bilateral opercular syndrome. Due to bilateral ablation of the motor control center for the lower face and surrounds, these rare cases provide an opportunity to evaluate the necessity of cortical motor representations for speech perception, a cornerstone of some neurocomputational theories of language processing. Speech perception, including audiovisual integration (i.e., the McGurk effect), was mostly unaffected in these cases, although verbal short-term memory impairment hindered performance on several tasks that are traditionally used to evaluate speech perception. The results suggest that the role of the cortical motor system in speech perception is context dependent and supplementary, not inherent or necessary.

Download Full-text

Komunikacja niewerbalna a tłumaczenie symultaniczne: aspekt teoretyczny

Acta Neophilologica ◽

10.31648/an.4746 ◽

2019 ◽

Vol 2 (XXI) ◽

pp. 65-76

Author(s):

Konrad Rachut

Keyword(s):

Human Brain ◽

Nonverbal Communication ◽

Theoretical Basis ◽

Audiovisual Integration ◽

Simultaneous Interpreting ◽

Quality Of Work ◽

Cognitive Resource

This paper draws attention to the role of nonverbal communication in the process of simultaneous interpreting. The theoretical basis is provided by the phe-nomenon of audiovisual integration: the ability of the human brain to incorporate both verbal and nonverbal signals into comprehension. Referring to previous works by F. Poyatos, S. Viaggio and K. Seeber, the author attempts to distinguish core functions of nonverbal signals in simultaneous interpreting and to analyse the model of cognitive resource footprint by K. Seeber. Additionally, theoretical and practical ramifications of taking nonverbal signals into consideration for the psychology and quality of work of simultaneous interpreters are pinpointed.

Download Full-text