scholarly journals Substituting facial movements in singers changes the sounds of musical intervals

2021 ◽  
Vol 11 (1) ◽  
Author(s):  
Bruno Laeng ◽  
Sarjo Kuyateh ◽  
Tejaswinee Kelkar
Keyword(s):  
Mcgurk Effect ◽  
Natural Speech ◽  
Musical Interval ◽  
Facial Movements ◽  
Auditory Signals ◽  
Compression Effects ◽  
Stretching Effect

AbstractCross-modal integration is ubiquitous within perception and, in humans, the McGurk effect demonstrates that seeing a person articulating speech can change what we hear into a new auditory percept. It remains unclear whether cross-modal integration of sight and sound generalizes to other visible vocal articulations like those made by singers. We surmise that perceptual integrative effects should involve music deeply, since there is ample indeterminacy and variability in its auditory signals. We show that switching videos of sung musical intervals changes systematically the estimated distance between two notes of a musical interval so that pairing the video of a smaller sung interval to a relatively larger auditory led to compression effects on rated intervals, whereas the reverse led to a stretching effect. In addition, after seeing a visually switched video of an equally-tempered sung interval and then hearing the same interval played on the piano, the two intervals were judged often different though they differed only in instrument. These findings reveal spontaneous, cross-modal, integration of vocal sounds and clearly indicate that strong integration of sound and sight can occur beyond the articulations of natural speech.

scholarly journals Animated virtual characters to explore audio-visual speech in controlled and naturalistic environments

2020 ◽  
Vol 10 (1) ◽  
Author(s):  
Raphaël Thézé ◽  
Mehdi Ali Gadiri ◽  
Louis Albert ◽  
Antoine Provost ◽  
Anne-Lise Giraud ◽  
...  
Keyword(s):  
Speech Processing ◽  
Background Noise ◽  
Mcgurk Effect ◽  
Visual Speech ◽  
Natural Speech ◽  
Virtual Characters ◽  
Speech Stimuli ◽  
Stimulus Timing ◽  
Phonetic Features ◽  
Set Up

Abstract Natural speech is processed in the brain as a mixture of auditory and visual features. An example of the importance of visual speech is the McGurk effect and related perceptual illusions that result from mismatching auditory and visual syllables. Although the McGurk effect has widely been applied to the exploration of audio-visual speech processing, it relies on isolated syllables, which severely limits the conclusions that can be drawn from the paradigm. In addition, the extreme variability and the quality of the stimuli usually employed prevents comparability across studies. To overcome these limitations, we present an innovative methodology using 3D virtual characters with realistic lip movements synchronized on computer-synthesized speech. We used commercially accessible and affordable tools to facilitate reproducibility and comparability, and the set-up was validated on 24 participants performing a perception task. Within complete and meaningful French sentences, we paired a labiodental fricative viseme (i.e. /v/) with a bilabial occlusive phoneme (i.e. /b/). This audiovisual mismatch is known to induce the illusion of hearing /v/ in a proportion of trials. We tested the rate of the illusion while varying the magnitude of background noise and audiovisual lag. Overall, the effect was observed in 40% of trials. The proportion rose to about 50% with added background noise and up to 66% when controlling for phonetic features. Our results conclusively demonstrate that computer-generated speech stimuli are judicious, and that they can supplement natural speech with higher control over stimulus timing and content.

scholarly journals Speech and non-speech measures of audiovisual integration are not correlated

2021 ◽  
Author(s):  
Jonathan Wilbiks ◽  
Julia Feld Strand ◽  
Violet Aurora Brown
Keyword(s):  
Individual Differences ◽  
Convergent Validity ◽  
Visual Stimuli ◽  
Audiovisual Integration ◽  
Mcgurk Effect ◽  
Large Individual ◽  
Speech Stimuli ◽  
Multiple Tasks ◽  
Auditory Signals ◽  
Audiovisual Stimuli

Many natural events generate both visual and auditory signals, and humans are remarkably adept at integrating information from those sources. However, individuals appear to differ markedly in their ability or propensity to combine what they hear with what they see. Individual differences in audiovisual integration have been established using a range of materials including speech stimuli (seeing and hearing a talker) and simpler audiovisual stimuli (seeing flashes of light combined with tones). Although there are multiple tasks in the literature that are referred to as “measures of audiovisual integration,” the tasks themselves differ widely with respect to both the type of stimuli used (speech versus non-speech) and the nature of the tasks themselves (e.g., some tasks use conflicting auditory and visual stimuli whereas others use congruent stimuli). It is not clear whether these varied tasks are actually measuring the same underlying construct: audiovisual integration. This study tested the convergent validity of four commonly-used measures of audiovisual integration, two of which use speech stimuli (susceptibility to the McGurk effect and a measure of audiovisual benefit), and two of which use non-speech stimuli (the sound-induced flash illusion and audiovisual integration capacity). We replicated previous work showing large individual differences in each measure, but found no significant correlations between any of the measures. These results suggest that tasks that are commonly referred to as measures of audiovisual integration may not be tapping into the same underlying construct.

The McGurk Effect Requires Conscious Visual Perception

2008 ◽  
Author(s):  
Terry D. Palmer ◽  
Ashley Ramsey ◽  
Maria Grosch ◽  
Dustin KermanKerman ◽  
Mike Urlakis
Keyword(s):  
Visual Perception ◽  
Mcgurk Effect

Influence of imaged pictures and sounds on detection of visual and auditory signals.

1970 ◽  
Vol 83 (3, Pt.1) ◽  
pp. 458-464 ◽  
Author(s):  
Sydney J. Segal ◽  
Vincent Fusella
Keyword(s):  
Auditory Signals

The Role of Learning in the Production and Comprehension of Auditory Signals by Pinnipeds

2006 ◽  
Vol 32 (4) ◽  
pp. 483-490 ◽  
Author(s):  
Kristy L. Lindemann ◽  
Colleen Reichmuth Kastak ◽  
Ronald J. Schusterman
Keyword(s):  
Auditory Signals

A Study on Polymer Particle Flow in a Disk Zone of a Screw-Disk Extruder

2014 ◽  
Vol 35 (1) ◽  
pp. 121-135 ◽  
Author(s):  
Tomasz Rydzkowski ◽  
Iwona Michalska-Pożoga
Keyword(s):  
Mechanical Properties ◽  
Computer Simulations ◽  
Particle Motion ◽  
Polymer Melt ◽  
Experimental Tests ◽  
Polymer Particle ◽  
Particle Flow ◽  
Material Particle ◽  
Motion Trajectories ◽  
Stretching Effect

Abstract The paper presents the summary of research on polymer melt particle motion trajectories in a disc zone of a screw-disk extruder. We analysed two models of its structure, different in levels of taken simplifications. The analysis includes computer simulations of material particle flow and results of experimental tests to determine the properties of the resultant extrudate. Analysis of the results shows that the motion of melt in the disk zone of a screw-disk extruder is a superposition of pressure and dragged streams. The observed trajectories of polymer particles and relations of mechanical properties and elongation of the molecular chain proved the presence of a stretching effect on polymer molecular chains.

A Meta-Analysis of the Facial Feedback Literature: Effects of Facial Feedback on Emotional Experience Are Small and Variable

2017 ◽  
Author(s):  
Nicholas Alvaro Coles ◽  
Jeff T. Larsen ◽  
Heather Lench
Keyword(s):  
Publication Bias ◽  
Emotional Experience ◽  
Meta Analysis ◽  
Effect Sizes ◽  
Detection Methods ◽  
Feedback Effects ◽  
Facial Feedback ◽  
Facial Movements ◽  
Facial Feedback Hypothesis ◽  
Variance Estimates

The facial feedback hypothesis suggests that an individual’s experience of emotion is influenced by feedback from their facial movements. To evaluate the cumulative evidence for this hypothesis, we conducted a meta-analysis on 286 effect sizes derived from 138 studies that manipulated facial feedback and collected emotion self-reports. Using random effects meta-regression with robust variance estimates, we found that the overall effect of facial feedback was significant, but small. Results also indicated that feedback effects are stronger in some circumstances than others. We examined 12 potential moderators, and three were associated with differences in effect sizes. 1. Type of emotional outcome: Facial feedback influenced emotional experience (e.g., reported amusement) and, to a greater degree, affective judgments of a stimulus (e.g., the objective funniness of a cartoon). Three publication bias detection methods did not reveal evidence of publication bias in studies examining the effects of facial feedback on emotional experience, but all three methods revealed evidence of publication bias in studies examining affective judgments. 2. Presence of emotional stimuli: Facial feedback effects on emotional experience were larger in the absence of emotionally evocative stimuli (e.g., cartoons). 3. Type of stimuli: When participants were presented with emotionally evocative stimuli, facial feedback effects were larger in the presence of some types of stimuli (e.g., emotional sentences) than others (e.g., pictures). The available evidence supports the facial feedback hypothesis’ central claim that facial feedback influences emotional experience, although these effects tend to be small and heterogeneous.

Complex Auditory Signals

1994 ◽  
Author(s):  
David M. Green
Keyword(s):  
Auditory Signals

scholarly journals Design of Desktop Audiovisual Entertainment System with Deep Learning and Haptic Sensations

Symmetry ◽  
2020 ◽  
Vol 12 (10) ◽  
pp. 1718
Author(s):  
Chien-Hsing Chou ◽  
Yu-Sheng Su ◽  
Che-Ju Hsu ◽  
Kong-Chang Lee ◽  
Ping-Hsuan Han
Keyword(s):  
Deep Learning ◽  
Object Detection ◽  
User Experience ◽  
Recognition System ◽  
Scene Recognition ◽  
Single Shot ◽  
Auditory Signals ◽  
Hot Weather ◽  
Viewing Experience ◽  
At Home

In this study, we designed a four-dimensional (4D) audiovisual entertainment system called Sense. This system comprises a scene recognition system and hardware modules that provide haptic sensations for users when they watch movies and animations at home. In the scene recognition system, we used Google Cloud Vision to detect common scene elements in a video, such as fire, explosions, wind, and rain, and further determine whether the scene depicts hot weather, rain, or snow. Additionally, for animated videos, we applied deep learning with a single shot multibox detector to detect whether the animated video contained scenes of fire-related objects. The hardware module was designed to provide six types of haptic sensations set as line-symmetry to provide a better user experience. After the system considers the results of object detection via the scene recognition system, the system generates corresponding haptic sensations. The system integrates deep learning, auditory signals, and haptic sensations to provide an enhanced viewing experience.

The Difference in Facial Movement Between the Medial and the Lateral Midface: A 3D Skin Surface Vector Analysis

2021 ◽  
Author(s):  
David L Freytag ◽  
Michael G Alfertshofer ◽  
Konstantin Frank ◽  
Dmitry V Melnikov ◽  
Nicholas Moellhoff ◽  
...  
Keyword(s):  
3D Imaging ◽  
Functional Anatomy ◽  
Skin Surface ◽  
Facial Movements ◽  
Anatomic Location ◽  
Objective Evidence ◽  
The Face ◽  
Functional Boundary ◽  
The Mean ◽  
The Difference

Abstract Background Our understanding of the functional anatomy of the face is constantly improving. To date, it is unclear whether the anatomic location of the line of ligaments has any functional importance during normal facial movements such as smiling. Objectives It is the objective of the present study to identify differences in facial movements between the medial and lateral midface by means of skin vector displacement analyses derived from 3D imaging and to further ascertain whether the line of ligaments has both a structural and functional significance in these movements. Methods The study sample consisted of 21 healthy volunteers (9 females & 12 males) of Caucasian ethnic background with a mean age of 30.6 (8.3) years and a mean BMI of 22.57 (2.5) kg/m 2. 3D images of the volunteers’ faces in repose and during smiling (Duchenne type) were taken. 3D imaging-based skin vector displacement analyses were conducted. Results The mean horizontal skin displacement was 0.08 (2.0) mm in the medial midface (lateral movement) and was -0.08 (1.96) mm in the lateral midface (medial movement) (p = 0.711). The mean vertical skin displacement (cranial movement of skin toward the forehead/temple) was 6.68 (2.4) mm in the medial midface whereas it was 5.20 (2.07) mm in the lateral midface (p = 0.003). Conclusions The results of this study provide objective evidence for an antagonistic skin movement between the medial and the lateral midface. The functional boundary identified by 3D imaging corresponds to the anatomic location of the line of ligaments.

Sign in / Sign up

Export Citation Format

Share Document