Gestures as scaffolding for L2 narrative recall: The role of gesture type, task complexity, and working memory

This study investigated the extent to which different pedagogical gestures contribute to learners’ foreign or second language (L2) narrative recall, and further discussed how task complexity and task difficulty (i.e. working memory capacity or WMC) influence recall performance. Sixty-four adolescent learners, assigned to four different gesture viewing conditions (iconic gestures, deictic gestures, beat gestures, or no gesture), were required to listen to an instructor telling two stories (one complex and one simple) and then retell both stories twice: once immediately after listening (immediate recall) and a second time two weeks later (delayed recall). Recall performance was evaluated by the number of relevant pieces of event and motion information produced in the participants’ retelling. The results show that L2 learners who were exposed to deictic and iconic gesture conditions outperformed the other gesture groups, particularly in delayed narrative recall, but only in complex tasks where cognitive demands were increased. It was also found that event and motion information was retained for a longer period of time in the deictic and iconic conditions respectively. Although both high and low WMC groups benefitted from viewing gestures, this finding further indicates that the beneficial effect of gestures on learners could possibly compensate for low WMC by providing scaffolding that reduces cognitive burden in narrative recall.

EXPRESS: Lending a Hand to Storytelling: Gesture’s Effects on Narrative Comprehension Moderated by Task Difficulty and Cognitive Ability

Compelling evidence suggests observing iconic gestures benefits learning. While emerging evidence suggests typical iconic gestures benefit comprehension to a greater extent than atypical iconic gestures, it is unclear precisely when and for whom these gestures will be most helpful. The current study investigated factors that may moderate when and for whom gesture benefits narrative comprehension most, including the type of gesture, task difficulty, and individual differences in cognitive ability. Participants were shown a video narrative in which they observed either typical gestures (commonly produced gestures, highly semantically related to accompanying speech), atypical gestures (gestures that are seldom produced), or no gestures. The video narrative was either viewed with interference (background noise to increase task difficulty) or no interference (no background noise). To determine whether the effects of gesture observation and externally imposed task difficulty on narrative comprehension further depend on an individual’s cognitive abilities, participants completed four measures of cognitive abilities (immediate and delayed non-verbal memory, attention, and intellectual ability). Observing typical gestures significantly benefited narrative comprehension compared to atypical and no gestures combined, which did not differ significantly. Participants with below average and average levels of delayed non-verbal memory benefited more from typical gestures than atypical or no gestures compared to those with an above average level of delayed non-verbal memory. However, this interaction was only significant when the task was difficult (i.e., with interference) but not when the task was simple (i.e., no interference). This finding suggests that the type of iconic gesture observed may impact gesture’s beneficial effect on narrative comprehension, and that such gestures may be more beneficial in difficult tasks, but only for certain individuals.

Task-Specific Iconic Gesturing During Spoken Discourse in Aphasia

Purpose In persons living with aphasia, we will explore the relationship between iconic gesture production during spontaneous speech and discourse task, spoken language, and demographic information. Method Employing the AphasiaBank database, we coded iconic gestures in 75 speakers with aphasia during two spoken discourse tasks: a procedural narrative, which involved participants telling the experimenter how to make a sandwich (“Sandwich”), and a picture sequence narrative, which had participants describe the picture sequence to the experimenter (“Window”). Forty-three produced a gesture during both tasks, and we further evaluate data from this subgroup as a more direct comparison between tasks. Results More iconic gestures, at a higher rate, were produced during the procedural narrative. For both tasks, there was a relationship between iconic gesture rate, modeled as iconic gestures per word, and metrics of language dysfluency extracted from the discourse task as well as a metric of fluency extracted from a standardized battery. Iconic gesture production was correlated with aphasia duration, which was driven by performance during only a single task (Window), but not with other demographic metrics, such as aphasia severity or age. We also provide preliminary evidence for task differences shown through the lens of two types of iconic gestures. Conclusions While speech-language pathologists have utilized gesture in therapy for poststroke aphasia, due to its possible facilitatory role in spoken language, there has been considerably less work in understanding how gesture differs across naturalistic tasks and how we can best utilize this information to better assess gesture in aphasia and improve multimodal treatment for aphasia. Furthermore, our results contribute to gesture theory, particularly, about the role of gesture across naturalistic tasks and its relationship with spoken language. Supplemental Material https://doi.org/10.23641/asha.14614941

The Role of Iconic Gestures in Speech Comprehension: An Overview of Various Methodologies

Iconic gesture-speech integration is a relatively recent field of investigation with numerous researchers studying its various aspects. The results obtained are just as diverse. The definition of iconic gestures is often overlooked in the interpretations of results. Furthermore, while most behavioral studies have demonstrated an advantage of bimodal presentation, brain activity studies show a diversity of results regarding the brain regions involved in the processing of this integration. Clinical studies also yield mixed results, some suggesting parallel processing channels, others a unique and integrated channel. This review aims to draw attention to the methodological variations in research on iconic gesture-speech integration and how they impact conclusions regarding the underlying phenomena. It will also attempt to draw together the findings from other relevant research and suggest potential areas for further investigation in order to better understand processes at play during speech integration process.

Gesture Meaning of Tortor Used in Simalungun Wedding Ceremony

This study discusses about the gesture meaning of tortor in Simalungun Wedding Ceremony. The theory used in this study is McNeill's theory. The data collection was conducted by real work methods it were interviews, documentation and literature study. The methods is used in this study is qualitative method and descriptive approach. The result of this study show that the meaning of totor in simalungun wedding ceremony are showing happiness, respect, compassion, delighted. Each gesture of tortor also serves show a close sense of relationship between two families. In human life, God creates men and women who will have marital relationships and are called husband and wife. With the merging of two families with a marriage, the happiness of both families it show by tortor. Based on McNeill's theory gesture, tortor of Simalungun wedding ceremony included in the types of iconic gesture. This is because the gesture contains a formal relationship.

Using Gesture to Facilitate L2 Phoneme Acquisition: The Importance of Gesture and Phoneme Complexity

Most language learners have difficulties acquiring the phonemes of a second language (L2). Unfortunately, they are often judged on their L2 pronunciation, and segmental inaccuracies contribute to miscommunication. Therefore, we aim to determine how to facilitate phoneme acquisition. Given the close relationship between speech and co-speech gesture, previous work unsurprisingly reports that gestures can benefit language acquisition, e.g., in (L2) word learning. However, gesture studies on L2 phoneme acquisition present contradictory results, implying that both specific properties of gestures and phonemes used in training, and their combination, may be relevant. We investigated the effect of phoneme and gesture complexity on L2 phoneme acquisition. In a production study, Dutch natives received instruction on the pronunciation of two Spanish phonemes, /u/ and /θ/. Both are typically difficult to produce for Dutch natives because their orthographic representation differs between both languages. Moreover, /θ/ is considered more complex than /u/, since the Dutch phoneme inventory contains /u/ but not /θ/. The instruction participants received contained Spanish examples presented either via audio-only, audio-visually without gesture, audio-visually with a simple, pointing gesture, or audio-visually with a more complex, iconic gesture representing the relevant speech articulator(s). Preceding and following training, participants read aloud Spanish sentences containing the target phonemes. In a perception study, Spanish natives rated the target words from the production study on accentedness and comprehensibility. Our results show that combining gesture and speech in L2 phoneme training can lead to significant improvement in L2 phoneme production, but both gesture and phoneme complexity affect successful learning: Significant learning only occurred for the less complex phoneme /u/ after seeing the more complex iconic gesture, whereas for the more complex phoneme /θ/, seeing the more complex gesture actually hindered acquisition. The perception results confirm the production findings and show that items containing /θ/ produced after receiving training with a less complex pointing gesture are considered less foreign-accented and more easily comprehensible as compared to the same items after audio-only training. This shows that gesture can facilitate task performance in L2 phonology acquisition, yet complexity affects whether certain gestures work better for certain phonemes than others.

Introducing the NEMO-Lowlands iconic gesture dataset, collected through a gameful human–robot interaction

This paper describes a novel dataset of iconic gestures, together with a publicly available robot-based elicitation method to record these gestures, which consists of playing a game of charades with a humanoid robot. The game was deployed at a science museum (NEMO) and a large popular music festival (Lowlands) in the Netherlands. This resulted in recordings of 428 participants, both adults and children, performing 3715 silent iconic gestures for 35 different objects in a naturalistic setting. Our dataset adds to existing collections of iconic gesture recordings in two important ways. First, participants were free to choose how they represented the broad concepts using gestures, and they were asked to perform a second attempt if the robot did not recognize their gesture the first time. This provides insight into potential repair strategies that might be used. Second, by making the interactive game available we enable other researchers to collect additional recordings, for different concepts, and in diverse cultures or contexts. This can be done in a consistent manner because a robot is used as a confederate in the elicitation procedure, which ensures that every data collection session plays out in the same way. The current dataset can be used for research into human gesturing behavior, and as input for the gesture recognition and production capabilities of robots and virtual agents.

Task-specific iconic gesturing during spoken discourse in aphasia

Purpose: Identify main effect of task, as well as roles of spoken language and demographic details, on iconic gesture production during spontaneous speech in persons living with aphasia.Method: Employing the AphasiaBank database, we coded iconic gestures in N=75 speakers with aphasia during two spoken discourse tasks: a procedural narrative, which involved participants telling the experimenter how to make a sandwich (“Sandwich”), and a picture sequence narrative, which had participants describe the picture sequence to the experimenter (“Window”). N=43 produced a gesture during both tasks, and this subgroup serves as a more direct comparison between tasks.Results: More iconic gestures, at a higher rate, were produced during the procedural narrative. For both tasks, there was a relationship between iconic gesture rate, modeled as iconic gestures per word, and metrics of language dysfluency extracted from the discourse task, and a metric of fluency extracted from a standardized battery. Iconic gesture production was correlated with aphasia duration, which was driven by performance during only a single task (Window), but not with other demographic metrics, such as aphasia severity or age. Conclusions: Whilst speech-language pathologists have utilized gesture in therapy for post-stroke aphasia, due to its possible facilitatory role in spoken language, there has been considerably less work in understanding how gesture differs across naturalistic tasks, and how we can best utilize this information to better assess and treat gesture in aphasia. Further, our results contribute to gesture theory, in particular about the role of gesture across naturalistic tasks, and its relationship with spoken language.