scholarly journals Auditory-Visual Speech Behaviors are Resilient to Left pSTS Damage

2020 ◽  
Author(s):  
David Brang ◽  
John Plass ◽  
Sofia Kakaizada ◽  
Shawn L. Hervey-Jumper
Keyword(s):  
Speech Perception ◽  
Emotional Health ◽  
Brain Regions ◽  
Mcgurk Effect ◽  
Visual Speech ◽  
Critical Areas ◽  
Lesion Mapping ◽  
Left Posterior ◽  
Speech Behaviors ◽  
Speech Information

ABSTRACTThe ability to understand spoken language is essential for social, vocational, and emotional health, but can be disrupted by environmental noise, injury, or hearing loss. These auditory deficits can be ameliorated by visual speech signals that convey redundant or supplemental speech information, but the brain regions critically responsible for these audiovisual (AV) interactions remain poorly understood. Previous TMS and lesion-mapping studies suggest that the left posterior superior temporal sulcus (pSTS) is causally implicated in producing the McGurk effect, an AV illusion in which auditory and visual speech are perceptually “fused.” However, previous research suggests that the McGurk effect is neurally and behaviorally dissociable from other visual effects on speech perception and, therefore, may not provide a generalizable index of AV interactions in speech perception more broadly. To examine whether the left pSTS is critically responsible for AV speech integration more broadly, we measured the strength of the McGurk effect, AV facilitation effects, and AV conflict effects longitudinally over 2 years in patients undergoing surgery for intrinsic tumors in the left pSTS (n = 2) or frontal lobes (control; n = 14). Results demonstrated that left pSTS lesions impaired experience of the McGurk effect, but did not uniformly reduce visual influences on speech perception. Additionally, when multisensory behaviors were affected by a lesion, AV speech perception abilities could recover over time. Our results suggest a causal dissociation between perceptual benefits produced by congruent AV speech and perceptual modulations produced by incongruent AV speech (the McGurk effect).These data are consistent with models proposing that that the pSTS is only one of multiple critical areas necessary for AV speech interactions.

scholarly journals Auditory-Visual Speech Perception in Bipolar Disorder: A Preliminary Study

2018 ◽  
Author(s):  
Arzu Yordamlı ◽  
Doğu Erdener
Keyword(s):  
Bipolar Disorder ◽  
Speech Perception ◽  
Mcgurk Effect ◽  
Visual Speech ◽  
Physiological Data ◽  
Control Group ◽  
Visual Speech Perception ◽  
Visual Speech Information ◽  
Speech Information ◽  
Depressive Episodes

The focus of this study was to investigate how individuals with bipolar disorder integrate auditory and visual speech information compared to non-disordered individuals and whether there were any differences in auditory and visual speech integration in the manic and depressive episodes in bipolar disorder patients. It was hypothesized that bipolar groups’ auditory-visual speech integration would be less robust than the control group. Further, it was predicted that those in the manic phase of bipolar disorder would integrate visual speech information more than their depressive phase counterparts. To examine these, the McGurk effect paradigm was used with typical auditory-visual speech (AV) as well as auditory-only (AO) speech perception on visual-only (VO) stimuli. Results. Results showed that the disordered and non-disordered groups did not differ on auditory-visual speech (AV) integration and auditory-only (AO) speech perception but on visual-only (VO) stimuli. The results are interpreted to pave the way for further research whereby both behavioural and physiological data are collected simultaneously. This will allow us understand the full dynamics of how, actually, the auditory and visual (relatively impoverished in bipolar disorder) speech information are integrated in people with bipolar disorder.

Audiovisual Speech Perception and the McGurk Effect

2019 ◽  
Author(s):  
Lawrence D. Rosenblum
Keyword(s):  
Speech Perception ◽  
Mcgurk Effect ◽  
Visual Speech ◽  
Multisensory Perception ◽  
Future Research ◽  
Audiovisual Speech ◽  
Audiovisual Speech Perception ◽  
Common Information ◽  
The Senses ◽  
Speech Information

Research on visual and audiovisual speech information has profoundly influenced the fields of psycholinguistics, perception psychology, and cognitive neuroscience. Visual speech findings have provided some of most the important human demonstrations of our new conception of the perceptual brain as being supremely multimodal. This “multisensory revolution” has seen a tremendous growth in research on how the senses integrate, cross-facilitate, and share their experience with one another. The ubiquity and apparent automaticity of multisensory speech has led many theorists to propose that the speech brain is agnostic with regard to sense modality: it might not know or care from which modality speech information comes. Instead, the speech function may act to extract supramodal informational patterns that are common in form across energy streams. Alternatively, other theorists have argued that any common information existent across the modalities is minimal and rudimentary, so that multisensory perception largely depends on the observer’s associative experience between the streams. From this perspective, the auditory stream is typically considered primary for the speech brain, with visual speech simply appended to its processing. If the utility of multisensory speech is a consequence of a supramodal informational coherence, then cross-sensory “integration” may be primarily a consequence of the informational input itself. If true, then one would expect to see evidence for integration occurring early in the perceptual process, as well in a largely complete and automatic/impenetrable manner. Alternatively, if multisensory speech perception is based on associative experience between the modal streams, then no constraints on how completely or automatically the senses integrate are dictated. There is behavioral and neurophysiological research supporting both perspectives. Much of this research is based on testing the well-known McGurk effect, in which audiovisual speech information is thought to integrate to the extent that visual information can affect what listeners report hearing. However, there is now good reason to believe that the McGurk effect is not a valid test of multisensory integration. For example, there are clear cases in which responses indicate that the effect fails, while other measures suggest that integration is actually occurring. By mistakenly conflating the McGurk effect with speech integration itself, interpretations of the completeness and automaticity of multisensory may be incorrect. Future research should use more sensitive behavioral and neurophysiological measures of cross-modal influence to examine these issues.

Second Language Instruction

2020 ◽  
pp. 105-123
Author(s):  
Doğu Erdener
Keyword(s):  
Speech Perception ◽  
Applied Field ◽  
General Framework ◽  
Language Instruction ◽  
Visual Speech ◽  
Second Language Instruction ◽  
Perception Process ◽  
L2 Instruction ◽  
Visual Speech Information ◽  
Speech Information

Speech perception has long been taken for granted as an auditory-only process. However, it is now firmly established that speech perception is an auditory-visual process in which visual speech information in the form of lip and mouth movements are taken into account in the speech perception process. Traditionally, foreign language (L2) instructional methods and materials are auditory-based. This chapter presents a general framework of evidence that visual speech information will facilitate L2 instruction. The author claims that this knowledge will form a bridge to cover the gap between psycholinguistics and L2 instruction as an applied field. The chapter also describes how orthography can be used in L2 instruction. While learners from a transparent L1 orthographic background can decipher phonology of orthographically transparent L2s –overriding the visual speech information – that is not the case for those from orthographically opaque L1s.

scholarly journals Auditory–Visual Speech Integration in Bipolar Disorder: A Preliminary Study

Languages ◽  
2018 ◽  
Vol 3 (4) ◽  
pp. 38 ◽  
Author(s):  
Arzu Yordamlı ◽  
Doğu Erdener
Keyword(s):  
Bipolar Disorder ◽  
Mcgurk Effect ◽  
Visual Speech ◽  
Physiological Data ◽  
Striking Difference ◽  
Control Group ◽  
Speech Stimuli ◽  
Visual Speech Information ◽  
Speech Information ◽  
And Control

This study aimed to investigate how individuals with bipolar disorder integrate auditory and visual speech information compared to healthy individuals. Furthermore, we wanted to see whether there were any differences between manic and depressive episode bipolar disorder patients with respect to auditory and visual speech integration. It was hypothesized that the bipolar group’s auditory–visual speech integration would be weaker than that of the control group. Further, it was predicted that those in the manic phase of bipolar disorder would integrate visual speech information more robustly than their depressive phase counterparts. To examine these predictions, a McGurk effect paradigm with an identification task was used with typical auditory–visual (AV) speech stimuli. Additionally, auditory-only (AO) and visual-only (VO, lip-reading) speech perceptions were also tested. The dependent variable for the AV stimuli was the amount of visual speech influence. The dependent variables for AO and VO stimuli were accurate modality-based responses. Results showed that the disordered and control groups did not differ in AV speech integration and AO speech perception. However, there was a striking difference in favour of the healthy group with respect to the VO stimuli. The results suggest the need for further research whereby both behavioural and physiological data are collected simultaneously. This will help us understand the full dynamics of how auditory and visual speech information are integrated in people with bipolar disorder.

Sound Location Can Influence Audiovisual Speech Perception When Spatial Attention Is Manipulated

2011 ◽  
Vol 24 (1) ◽  
pp. 67-90 ◽  
Author(s):  
Riikka Möttönen ◽  
Kaisa Tiippana ◽  
Mikko Sams ◽  
Hanna Puharinen
Keyword(s):  
Speech Perception ◽  
Spatial Attention ◽  
Reaction Times ◽  
Mcgurk Effect ◽  
Visual Speech ◽  
Audiovisual Speech ◽  
Sound Location ◽  
Audiovisual Speech Perception ◽  
The Right ◽  
Talking Face

AbstractAudiovisual speech perception has been considered to operate independent of sound location, since the McGurk effect (altered auditory speech perception caused by conflicting visual speech) has been shown to be unaffected by whether speech sounds are presented in the same or different location as a talking face. Here we show that sound location effects arise with manipulation of spatial attention. Sounds were presented from loudspeakers in five locations: the centre (location of the talking face) and 45°/90° to the left/right. Auditory spatial attention was focused on a location by presenting the majority (90%) of sounds from this location. In Experiment 1, the majority of sounds emanated from the centre, and the McGurk effect was enhanced there. In Experiment 2, the major location was 90° to the left, causing the McGurk effect to be stronger on the left and centre than on the right. Under control conditions, when sounds were presented with equal probability from all locations, the McGurk effect tended to be stronger for sounds emanating from the centre, but this tendency was not reliable. Additionally, reaction times were the shortest for a congruent audiovisual stimulus, and this was the case independent of location. Our main finding is that sound location can modulate audiovisual speech perception, and that spatial attention plays a role in this modulation.

Own-race faces promote integrated audiovisual speech information

2021 ◽  
pp. 174702182110444
Author(s):  
Yuta Ujiie ◽  
Kohske Takahashi
Keyword(s):  
Speech Perception ◽  
Mcgurk Effect ◽  
The Other ◽  
Emotional Expressions ◽  
Audiovisual Speech ◽  
Audiovisual Speech Perception ◽  
Facial Identity ◽  
Race Effect ◽  
Speech Information ◽  
Effect Experiment

The other-race effect indicates a perceptual advantage when processing own-race faces. This effect has been demonstrated in individuals’ recognition of facial identity and emotional expressions. However, it remains unclear whether the other-race effect also exists in multisensory domains. We conducted two experiments to provide evidence for the other-race effect in facial speech recognition, using the McGurk effect. Experiment 1 tested this issue among East Asian adults, examining the magnitude of the McGurk effect during stimuli using speakers from two different races (own-race vs. other-race). We found that own-race faces induced a stronger McGurk effect than other-race faces. Experiment 2 indicated that the other-race effect was not simply due to different levels of attention being paid to the mouths of own- and other-race speakers. Our findings demonstrated that own-race faces enhance the weight of visual input during audiovisual speech perception, and they provide evidence of the own-race effect in the audiovisual interaction for speech perception in adults.

scholarly journals Hearing Lips and Seeing Voices: the Origins and Development of the ‘McGurk Effect’ and Reflections on Audio–Visual Speech Perception Over the Last 40 Years

2018 ◽  
Vol 31 (1-2) ◽  
pp. 7-18 ◽  
Author(s):  
John MacDonald
Keyword(s):  
Speech Perception ◽  
Visual Illusion ◽  
Simultaneous Presentation ◽  
Mcgurk Effect ◽  
Visual Speech ◽  
Audiovisual Speech ◽  
Audiovisual Speech Perception ◽  
Profound Impact ◽  
Visual Speech Perception

In 1976 Harry McGurk and I published a paper in Nature, entitled ‘Hearing Lips and Seeing Voices’. The paper described a new audio–visual illusion we had discovered that showed the perception of auditorily presented speech could be influenced by the simultaneous presentation of incongruent visual speech. This hitherto unknown effect has since had a profound impact on audiovisual speech perception research. The phenomenon has come to be known as the ‘McGurk effect’, and the original paper has been cited in excess of 4800 times. In this paper I describe the background to the discovery of the effect, the rationale for the generation of the initial stimuli, the construction of the exemplars used and the serendipitous nature of the finding. The paper will also cover the reaction (and non-reaction) to the Nature publication, the growth of research on, and utilizing the ‘McGurk effect’ and end with some reflections on the significance of the finding.

scholarly journals Speech Perception as a Multimodal Phenomenon

2008 ◽  
Vol 17 (6) ◽  
pp. 405-409 ◽  
Author(s):  
Lawrence D. Rosenblum
Keyword(s):  
Speech Perception ◽  
Semantic Context ◽  
Visual Speech ◽  
Audiovisual Speech ◽  
Lexical Status ◽  
Auditory Speech ◽  
Lip Reading ◽  
Imaging Research ◽  
Speech Information ◽  
The Brain

Speech perception is inherently multimodal. Visual speech (lip-reading) information is used by all perceivers and readily integrates with auditory speech. Imaging research suggests that the brain treats auditory and visual speech similarly. These findings have led some researchers to consider that speech perception works by extracting amodal information that takes the same form across modalities. From this perspective, speech integration is a property of the input information itself. Amodal speech information could explain the reported automaticity, immediacy, and completeness of audiovisual speech integration. However, recent findings suggest that speech integration can be influenced by higher cognitive properties such as lexical status and semantic context. Proponents of amodal accounts will need to explain these results.

Sign in / Sign up

Export Citation Format

Share Document