Gesture, Prosody and Information Structure Synchronisation in Turkish

<p>This thesis investigates the synchronisation of gesture with prosody and information structure in Turkish. Speech and gesture have a close relationship in human communication, and they are tightly coordinated in production. Research has shown that gestural units are synchronised with prosodic units on a prominence-related micro level (i.e., pitch accents and gesture apexes), however these studies have largely been on a small number of languages of a similar prosodic type, not including Turkish, which has prominence-less prosodic words. It is known that both gesture and speech, through prosody, are hierarchically structured with nested phrasal constituents, but little is known about gesture-prosody synchronisation at this macro level. Even less is known about the timing relationships of gesture with information structure, which is also closely related to prosody. This thesis links gesture to information structure as a part of a three-way synchronisation relationship of gesture, prosody, and information structure. Four participants were filmed in a narrative task, resulting in three hours of Turkish natural speech and gesture data. Selected sections were annotated for prosody using an adapted scheme for Turkish in the Autosegmental-Metrical framework, for information structure and for gesture. In total, there were over 20,000 annotations. The synchronisation of gesture and speech units was systematically investigated at (1) the micro level, and (2) the macro level. At the micro level, this thesis asked which tones apexes are synchronised with, and whether this synchronisation depends on other prosodic and gestural features. It was found that gesture apexes were synchronised with pitch accents if there were pitch accents in the relevant prosodic phrases; if not, they were synchronised with low tones that marked the onsets of prosodic words. This synchronisation pattern was largely consistent across different prosodic and gestural contexts, although it was tighter in the nuclear area. These findings confirm prominence as a constraint on synchronisation with evidence of pitch accent-apex synchronisation. The findings also extend our knowledge of the typology of micro-level synchronisation to cases where prominence is locally absent showing that micro-level synchronisation also obeys the prosodic hierarchy. At the macro level, the aim was to find the prosodic anchor for single gesture phrases while testing for the possible effects of prosodic, gestural and information structural contexts. The findings showed that there was no one-to-one synchronisation of single gesture phrases with single intermediate or intonational phrases. However, it was found that gesture phrases often spanned over multiple consecutive intermediate phrases, and the synchronisation of gesture phrase boundaries was with the boundaries of these intermediate phrase groupings. In addition, these groupings tended to be combinations of pre-nuclear and nuclear intermediate phrases constituting the default focus position in Turkish. This synchronisation behaviour over the focal domain implied that there might be another speech element governing the speech-gesture synchronisation which also informs prosody, i.e. information structure. Based on this finding and a few other associations in the earlier studies, it was hypothesised that gesture is also informed by and synchronised with information structure. In order to test this hypothesis, it was investigated whether gesture phrases were synchronised with information structural units, i.e., topics, foci and background. The findings showed that gesture phrases tended to accompany discursively prominent foci over topics and background. However, gesture phrases did not show perfect synchronisation with any of these information structure units, although there was a systematic overlap in which foci and topics were contained within the duration of complete gesture phrases. Further investigations revealed that gesture phrase parts that bear apex related meaning provided a much better anchor for the synchronisation of information structure units. The preference for accompanying and synchronisation with the parts of gesture bearing gesturally prominent apical meaning also highlighted that prominence is a driving factor of synchronisation at the macro level as well as at the micro level. This thesis has revealed pivotal links between gesture, prosody and information structure through a systematic investigation of synchronisation of these structures. The implications of these links have also been discussed within the thesis, and a model of speech and gesture production integrating synchronisation has been proposed. Overall, the thesis contributes to a deeper understanding of speech and gesture production, explaining how these interact during natural speech.</p>

Prosodic Characteristics of Sentences with Communicatively Marked and Communicatively Unmarked Alignment

The paper presents the results from a study of the interaction between intonation and information structure in SVO and OVS sentences with communicatively (un)marked alignment of information structure elements. We analyzed the prosodic features of pre-nuclear and nuclear pitch accents. The information structure elements were characterized using Steedman’s (2000) model which classifies sentence constituents as belonging to one of the following categories: theme-background, theme-focus, rheme-background and rheme-focus. Our study found that unmarked and marked alignment has no effect on the pitch range of the rheme-focus. In cases of communicatively unmarked alignment, the pitch range of the theme-background (and rheme-background) group in OVS sentences is wider than in SVO sentences. Word order has no effect on the duration of the accented syllable. Topicalized constituents belonging to the theme-background in OVS sentences with unmarked alignment form separate intermediate phrases. In cases of marked alignment, the rheme-focus ends with a phrase accent and sometimes a pause. The rheme-background and rheme-focus always take a pitch accent, whereas the theme-background is marked by a pitch accent only in cases of communicatively unmarked alignment. The theme-background is deaccented when the sentence is communicatively marked.

Gesture, Prosody and Information Structure Synchronisation in Turkish

<p>This thesis investigates the synchronisation of gesture with prosody and information structure in Turkish. Speech and gesture have a close relationship in human communication, and they are tightly coordinated in production. Research has shown that gestural units are synchronised with prosodic units on a prominence-related micro level (i.e., pitch accents and gesture apexes), however these studies have largely been on a small number of languages of a similar prosodic type, not including Turkish, which has prominence-less prosodic words. It is known that both gesture and speech, through prosody, are hierarchically structured with nested phrasal constituents, but little is known about gesture-prosody synchronisation at this macro level. Even less is known about the timing relationships of gesture with information structure, which is also closely related to prosody. This thesis links gesture to information structure as a part of a three-way synchronisation relationship of gesture, prosody, and information structure. Four participants were filmed in a narrative task, resulting in three hours of Turkish natural speech and gesture data. Selected sections were annotated for prosody using an adapted scheme for Turkish in the Autosegmental-Metrical framework, for information structure and for gesture. In total, there were over 20,000 annotations. The synchronisation of gesture and speech units was systematically investigated at (1) the micro level, and (2) the macro level. At the micro level, this thesis asked which tones apexes are synchronised with, and whether this synchronisation depends on other prosodic and gestural features. It was found that gesture apexes were synchronised with pitch accents if there were pitch accents in the relevant prosodic phrases; if not, they were synchronised with low tones that marked the onsets of prosodic words. This synchronisation pattern was largely consistent across different prosodic and gestural contexts, although it was tighter in the nuclear area. These findings confirm prominence as a constraint on synchronisation with evidence of pitch accent-apex synchronisation. The findings also extend our knowledge of the typology of micro-level synchronisation to cases where prominence is locally absent showing that micro-level synchronisation also obeys the prosodic hierarchy. At the macro level, the aim was to find the prosodic anchor for single gesture phrases while testing for the possible effects of prosodic, gestural and information structural contexts. The findings showed that there was no one-to-one synchronisation of single gesture phrases with single intermediate or intonational phrases. However, it was found that gesture phrases often spanned over multiple consecutive intermediate phrases, and the synchronisation of gesture phrase boundaries was with the boundaries of these intermediate phrase groupings. In addition, these groupings tended to be combinations of pre-nuclear and nuclear intermediate phrases constituting the default focus position in Turkish. This synchronisation behaviour over the focal domain implied that there might be another speech element governing the speech-gesture synchronisation which also informs prosody, i.e. information structure. Based on this finding and a few other associations in the earlier studies, it was hypothesised that gesture is also informed by and synchronised with information structure. In order to test this hypothesis, it was investigated whether gesture phrases were synchronised with information structural units, i.e., topics, foci and background. The findings showed that gesture phrases tended to accompany discursively prominent foci over topics and background. However, gesture phrases did not show perfect synchronisation with any of these information structure units, although there was a systematic overlap in which foci and topics were contained within the duration of complete gesture phrases. Further investigations revealed that gesture phrase parts that bear apex related meaning provided a much better anchor for the synchronisation of information structure units. The preference for accompanying and synchronisation with the parts of gesture bearing gesturally prominent apical meaning also highlighted that prominence is a driving factor of synchronisation at the macro level as well as at the micro level. This thesis has revealed pivotal links between gesture, prosody and information structure through a systematic investigation of synchronisation of these structures. The implications of these links have also been discussed within the thesis, and a model of speech and gesture production integrating synchronisation has been proposed. Overall, the thesis contributes to a deeper understanding of speech and gesture production, explaining how these interact during natural speech.</p>

Expanding the gestural model of lexical tone: evidence from two dialects of Serbian

Featural and gestural models of tone differ on the degree to which they include timing information in the representation. However, both assume some kind of simultaneity between tones and their tone-bearing units, where featural models emphasize the role of acoustic relationships and gestural models instead emphasize articulatory coordination. We present the results of two acoustic production studies on two dialects of Serbian, a lexical pitch accent language. In the Belgrade dialect, pitch accents are aligned relatively late in the tone-bearing unit, while in the Valjevo dialect, pitch accents are phonetically retracted, sometimes into the preceding syllable. We varied the syllable onsets of tone-bearing units in falling (experiment 1) and rising (experiment 2) pitch accents, and measured the effects on F0 contours. Despite these differences in phonetic alignment, the phonological system is the same in both dialects. We argue that this apparent mismatch between the phonology and phonetics can be expressed straightforwardly in the Articulatory Phonology framework by allowing tone gestures to coordinate with other gestures in all the ways that segmental gestures can, rather than restricting tone to c-center coordination.

The prosodic properties of narrow information focus in Central Mexican Spanish: Pitch accents, de-emphasis and phrasing

This paper investigates the different prosodic strategies used for the marking of information focus in Central Mexican Spanish. For this purpose, we carried out a study of the prosodic properties of information focus both in clause final position and in situ. Our results show important differences when compared to other varieties of Spanish. Specifically, we observe that the most frequent accent signaling information focus is a monotonal pitch accent (L* or !H*) and not L+H*. Furthermore, in many cases we observe that the pitch accent is not the only mechanism used to signal the focus: this is because we observe the presence of prosodic edges to the left of the focus, presumably functioning as an additional prosodic cue to identify it. Additionally, while we do not observe deaccenting of post-focal material, we do observe a sequence of non-rising forms (a flat pattern or “de-emphasis”) following the pitch accent that signals an in situ information focus forced by the test. With respect to phonological phrasing, our results confirm the analysis in Prieto (2006), where it is proposed that syntactic constituency is not the primary factor that regulates phrasing in Spanish.

The prosody of focus projection: word stress and F0 patterns in Georgian

Based on experimental evidence, this paper shows that focus projection/percolation – the phenomenon by way of which prosodic prominence on a sub-constituent signals focus on the whole constituent – has a consistent prosodic realization in Georgian. The novelty of these findings lies in two properties of Georgian that have not been explored from the perspective of focus projection: it is a language with a dedicated focus position (linearly immediately preverbal) and one that does not rely on pitch accents in the expression of phrasal prosody (Skopeteas & Féry 2010; 2016). According to focus projection accounts (Selkirk 1984; Cinque 1993; Ladd 1996; Zubizarreta 1998, a. o.), utterances with narrow focus on the direct object are realized in the same way as broad focus utterances, since in all three cases prosodic prominence is realized on the direct object. In contrast, in utterances with narrow focus on the subject, the subject is the most prosodically prominent element, which means that the whole utterance has a different prosodic realization from that of broad focus contexts. This paper shows that the distribution of prosodic prominence in object- and subject-focus contexts in Georgian fits with this generalization. Specifically, the realization of utterances with narrowly focused objects does not differ from broad focus contexts in their F0 patterns and prominence of the stressed syllable, while narrowly focused subjects differ from subjects in broad focus utterances in both of these parameters.

How focus particles and accents affect attachment

This project shows that focus and information structure, as indicated by the focus particle “only” and pitch accents, influence syntactic attachment, in contrast to the well-known effects of prosodic boundaries on attachment. One written questionnaire, one completion study, and several auditory questionnaires show that the position of “only” strongly affects attachment preferences in ambiguous sentences, while contrastive pitch accents have smaller effects. The two types of focus marking do not interact but independently impact attachment. These results support a modified version of the Focus Attraction Hypothesis, with ambiguous material drawn to attach to the most important information in a sentence. This research shows that information structure can affect sentence structure as well as discourse coherence.

Focus Attracts Attachment

There is growing evidence that pitch accents as well as prosodic boundaries can affect syntactic attachment. But is this an effect of their perceptual salience (the Salience Hypothesis), or is it because accents mark the position of focus (the Focus Attraction Hypothesis)? A pair of auditory comprehension experiments shows that focus position, as indicated by preceding wh-questions instead of by pitch accents, affects attachment by drawing the ambiguous phrase to the focus. This supports the Focus Attraction Hypothesis (or a pragmatic version of salience) for both these results and previous results of accents on attachment. These experiments show that information structure, as indicated with prosody or other means, influences sentence interpretation, and suggests a view on which modifiers are drawn to the most important information in a sentence.

Perception in Black and White: Effects of Intonational Variables and Filtering Conditions on Sociolinguistic Judgments With Implications for ASR

This study tests the effects of intonational contours and filtering conditions on listener judgments of ethnicity to arrive at a more comprehensive understanding on how prosody influences these judgments, with implications for austomatic speech recognition systems as well as speech synthesis. In a perceptual experiment, 40 American English listeners heard phrase-long clips which were controlled for pitch accent type and focus marking. Each clip contained either two H* (high) or two L+H* (low high) pitch accents and a L-L% (falling) boundary tone, and had also previously been labelled for broad or narrow focus. Listeners rated clips in two tasks, one with unmodified stimuli and one with stimuli lowpass filtered at 400 Hz, and were asked to judge whether the speaker was “Black” or “White”. In the filtered condition, tokens with the L+H* pitch accent were more likely to be rated as “Black”, with an interaction such that broad focus enhanced this pattern, supporting earlier findings that listeners may perceive African American Language as having more variation in possible pitch accent meanings. In the unfiltered condition, tokens with the L+H* pitch accent were less likely to be rated as Black, with no effect of focus, likely due to the fact that listeners relied more heavily on available segmental information in this condition. These results enhance our understanding of cues listeners rely on in making social judgments about speakers, especially in ethnic identification and linguistic profiling, by highlighting perceptual differences due to listening environment as well as predicted meaning of specific intonational contours. They also contribute to our understanding of the role of how human listeners interpret meaning within a holistic context, which has implications for the construction of computational systems designed to replicate the properties of natural language. In particular, they have important applicability to speech synthesis and speech recognition programs, which are often limited in their capacities due to the fact that they do not make such holistic sociolinguistic considerations of the meanings of input or output speech.