Gesture-vocal coupling in music performance: A kinematic-acoustic analysis of a south Indian vocal music corpus

In many musical styles worldwide, vocalists manually gesture while they sing. Coupling between gesture kinematics and vocalisation has been examined in speech contexts, but it is an open question how these couple in music making. We examine this in a corpus of South Indian vocal music in the Karnatak style, containing audio and motion-capture recordings. Through linear mixed regression and GAM modeling, we assessed whether peaks in vertical velocity, speed or acceleration were more strongly temporally predictive for changes in vocal acoustics, namely pitch and amplitude. Kinematic coupling was stronger for pitch change versus amplitude. An acceleration-based model was the most predictive for change in pitch and also had the most reliable magnitude coupling with vocal acoustics, showing a 1/3 power relation. That acceleration rather than other kinematics is maximally predictive for vocalization is interesting because accelerations entail that forces are produced onto the body. Thus force-transfer may prove to be salient for gesture-vocal synchrony in this context. As a theoretical contribution, we argue that gesturing in musical contexts should be understood in relation to the body’s tensegrity structure and also vocalists’ performance goals. We therefore propose that gesture-vocal coupling should be viewed as a neuro-bodily distributed aesthetic entanglement.

Facial and vocal markers of schizophrenia measured using remote smartphone assessments (Preprint)

BACKGROUND Machine learning-based facial and vocal measurements have demonstrated relationships with schizophrenia diagnosis and severity. Here, we determine their accuracy when acquired through automated assessments conducted remotely through smartphones. Demonstrating utility and validity of remote and automated assessments conducted outside of controlled experimental settings can facilitate scaling such measurement tools to aid in risk assessment and tracking of treatment response in difficult to engage populations. OBJECTIVE We aim to assess the accuracy of these facial and vocal markers through remote assessments and compare them with traditional clinical assessments of schizophrenia severity. METHODS Measurements of facial and vocal characteristics including facial expressivity, vocal acoustics, and speech prevalence were assessed in 20 schizophrenia patients over the course of 2 weeks in response to two classes of prompts previously utilized in experimental laboratory assessments: evoked prompts, where subjects are guided to produce specific facial expressions and phonations, and spontaneous prompts, where subjects are presented stimuli in the form of emotionally evocative imagery and asked to freely respond. Facial and vocal measurements were assessed in relation to schizophrenia symptom severity using the Positive and Negative Syndrome Scale. RESULTS Vocal markers including speech prevalence, vocal jitter, fundamental frequency, and vocal intensity demonstrated specificity as markers of negative symptom severity while measurement of facial expressivity demonstrated itself as a robust marker of overall schizophrenia severity. CONCLUSIONS Established facial and vocal measurements, collected remotely in schizophrenia patients via smartphones in response to automated task prompts, demonstrated accuracy as markers of schizophrenia severity. Clinical implications are discussed.

Facial and vocal markers of schizophrenia measured using remote smartphone assessments

BackgroundMachine learning-based facial and vocal measurements have demonstrated relationships with schizophrenia diagnosis and severity. Here, we determine their accuracy of when acquired through automated assessments conducted remotely through smartphones. Demonstrating utility and validity of remote and automated assessments conducted outside of controlled experimental settings can facilitate scaling such measurement tools to aid in risk assessment and tracking of treatment response in difficult to engage populations.MethodsMeasurements of facial and vocal characteristics including facial expressivity, vocal acoustics, and speech prevalence were assessed in 20 schizophrenia patients over the course of 2 weeks in response to two classes of prompts previously utilized in experimental laboratory assessments: evoked prompts, where subjects are guided to produce specific facial expressions and phonations, and spontaneous prompts, where subjects are presented stimuli in the form of emotionally evocative imagery and asked to freely respond. Facial and vocal measurements were assessed in relation to schizophrenia symptom severity using the Positive and Negative Syndrome Scale.ResultsVocal markers including speech prevalence, vocal jitter, fundamental frequency, and vocal intensity demonstrated specificity as markers of negative symptom severity while measurement of facial expressivity demonstrated itself as a robust marker of overall schizophrenia severity.ConclusionEstablished facial and vocal measurements, collected remotely in schizophrenia patients via smartphones in response to automated task prompts, demonstrated accuracy as markers of schizophrenia severity. Clinical implications are discussed.

Internal states and extrinsic factors both determine monkey vocal production

A key question for understanding speech evolution is whether or not the vocalizations of our closest living relatives—nonhuman primates—represent the precursors to speech. Some believe that primate vocalizations are not volitional but are instead inextricably linked to internal states like arousal and thus bear little resemblance to human speech. Others disagree and believe that since many primates can use their vocalizations strategically, this demonstrates a degree of voluntary vocal control. In the current study, we present a behavioral paradigm that reliably elicits different types of affiliative vocalizations from marmoset monkeys while measuring their heart rate fluctuations using noninvasive electromyography. By modulating both the physical distance between marmosets and the sensory information available to them, we find that arousal levels are linked, but not inextricably, to vocal production. Different arousal levels are, generally, associated with changes in vocal acoustics and the drive to produce different call types. However, in contexts where marmosets are interacting, the production of these different call types is also affected by extrinsic factors such as the timing of a conspecific’s vocalization. These findings suggest that variability in vocal output as a function of context might reflect trade-offs between the drive to perpetuate vocal contact and conserving energy.

The Effects of Pitch Shifts on Delay-induced Changes in Vocal Sequencing in a Songbird

Like human speech, vocal behavior in songbirds depends critically on auditory feedback. In both humans and songbirds, vocal skills are acquired by a process of imitation whereby current vocal production is compared to an acoustic target. Similarly, performance in adulthood relies strongly on auditory feedback, and online manipulations of auditory signals can dramatically alter acoustic production even after vocalizations have been well learned. Artificially delaying auditory feedback can disrupt both speech and birdsong, and internal delays in auditory feedback have been hypothesized as a cause of vocal dysfluency in persons who stutter. Furthermore, in both song and speech online shifts of the pitch (fundamental frequency) of auditory feedback lead to compensatory changes in vocal pitch for small perturbations, but larger pitch shifts produce smaller changes in vocal output. Intriguingly, large pitch shifts can partially restore normal speech in some dysfluent speakers, suggesting that the effects of auditory feedback delays might be ameliorated by online pitch manipulations. While birdsong provides a promising model system for understanding speech production, the interaction between sensory feedback delays and pitch shifts have not yet been assessed in songbirds. To investigate this, we asked whether the addition of a pitch shift modulates delay-induced changes in Bengalese finch song, hypothesizing that pitch shifts would reduce the effects of feedback delays. Compared the effects of delays alone, combined delays and pitch shifts resulted in a significant reduction in behavioral changes in one type of sequencing (branch points) but not another (distribution of repeated syllables).Significance StatementVocal behavior depends critically on an organism’s ability to monitor the sound of its own voice (“auditory feedback”). Studies of both humans and songbirds have demonstrated that successful vocal performance depends critically on the quality and timing of such feedback, however the interaction between vocal acoustics and the timing of auditory feedback is unclear. Here we used songbirds to examine this interaction by measuring vocal performance during delays and distortions (pitch shifts) of auditory feedback.

Early development of turn-taking with parents shapes vocal acoustics in infant marmoset monkeys

In humans, vocal turn-taking is a ubiquitous form of social interaction. It is a communication system that exhibits the properties of a dynamical system: two individuals become coupled to each other via acoustic exchanges and mutually affect each other. Human turn-taking develops during the first year of life. We investigated the development of vocal turn-taking in infant marmoset monkeys, a New World species whose adult vocal behaviour exhibits the same universal features of human turn-taking. We find that marmoset infants undergo the same trajectory of change for vocal turn-taking as humans, and do so during the same life-history stage. Our data show that turn-taking by marmoset infants depends on the development of self-monitoring, and that contingent parental calls elicit more mature-sounding calls from infants. As in humans, there was no evidence that parental feedback affects the rate of turn-taking maturation. We conclude that vocal turn-taking by marmoset monkeys and humans is an instance of convergent evolution, possibly as a result of pressures on both species to adopt a cooperative breeding strategy and increase volubility.