Multisensory Integration: The Case of a Time Window of Gesture–Speech Integration

2015 ◽  
Vol 27 (2) ◽  
pp. 292-307 ◽  
Author(s):  
Christian Obermeier ◽  
Thomas C. Gunter

This experiment investigates the integration of gesture and speech from a multisensory perspective. In a disambiguation paradigm, participants were presented with short videos of an actress uttering sentences like “She was impressed by the BALL, because the GAME/DANCE….” The ambiguous noun (BALL) was accompanied by an iconic gesture fragment containing information to disambiguate the noun toward its dominant or subordinate meaning. We used four different temporal alignments between noun and gesture fragment: the identification point (IP) of the noun was either prior to (+120 msec), synchronous with (0 msec), or lagging behind the end of the gesture fragment (−200 and −600 msec). ERPs triggered to the IP of the noun showed significant differences for the integration of dominant and subordinate gesture fragments in the −200, 0, and +120 msec conditions. The outcome of this integration was revealed at the target words. These data suggest a time window for direct semantic gesture–speech integration ranging from at least −200 up to +120 msec. Although the −600 msec condition did not show any signs of direct integration at the homonym, significant disambiguation was found at the target word. An explorative analysis suggested that gesture information was directly integrated at the verb, indicating that there are multiple positions in a sentence where direct gesture–speech integration takes place. Ultimately, this would implicate that in natural communication, where a gesture lasts for some time, several aspects of that gesture will have their specific and possibly distinct impact on different positions in an utterance.

2007 ◽  
Vol 19 (7) ◽  
pp. 1175-1192 ◽  
Author(s):  
Henning Holle ◽  
Thomas C. Gunter

The present series of experiments explored the extent to which iconic gestures convey information not found in speech. Electroencephalogram (EEG) was recorded as participants watched videos of a person gesturing and speaking simultaneously. The experimental sentences contained an unbalanced homonym in the initial part of the sentence (e.g., She controlled the ball …) and were disambiguated at a target word in the subsequent clause (which during the game … vs. which during the dance …). Coincident with the initial part of the sentence, the speaker produced an iconic gesture which supported either the dominant or the subordinate meaning. Event-related potentials were time-locked to the onset of the target word. In Experiment 1, participants were explicitly asked to judge the congruency between the initial homonym-gesture combination and the subsequent target word. The N400 at target words was found to be smaller after a congruent gesture and larger after an incongruent gesture, suggesting that listeners can use gestural information to disambiguate speech. Experiment 2 replicated the results using a less explicit task, indicating that the disambiguating effect of gesture is somewhat task-independent. Unrelated grooming movements were added to the paradigm in Experiment 3. The N400 at subordinate targets was found to be smaller after subordinate gestures and larger after dominant gestures as well as grooming, indicating that an iconic gesture can facilitate the processing of a lesser frequent word meaning. The N400 at dominant targets no longer varied as a function of the preceding gesture in Experiment 3, suggesting that the addition of meaningless movements weakened the impact of gesture. Thus, the integration of gesture and speech in comprehension does not appear to be an obligatory process but is modulated by situational factors such as the amount of observed meaningful hand movements.


2020 ◽  
Vol 228 (4) ◽  
pp. 278-290 ◽  
Author(s):  
Eylul Tekin ◽  
Henry L. Roediger

Abstract. Recent studies have shown that judgments of learning (JOLs) are reactive measures in paired-associate learning paradigms. However, evidence is scarce concerning whether JOLs are reactive in other paradigms. In old/new recognition experiments, we investigated the reactivity effects of JOLs in a levels-of-processing (LOP) paradigm. In Experiments 1 and 2, for each word, subjects saw a yes/no orienting question followed by the target word and a response. Then, they either did or did not make a JOL. The yes/no questions were about target words’ appearances, rhyming properties, or category memberships. In Experiment 3, for each word, subjects gave a pleasantness rating or counted the letter “e ”. Our results revealed that JOLs enhanced recognition across all orienting tasks in Experiments 1 and 2, and for the e-counting task in Experiment 3. This reactive effect was salient for shallow tasks, attenuating – but not eliminating – the LOP effect after making JOLs. We conclude that JOLs are reactive in LOP paradigms and subjects encode words more effectively when providing JOLs.


Phonetica ◽  
2021 ◽  
Vol 78 (1) ◽  
pp. 65-94
Author(s):  
Katsura Aoyama ◽  
Barbara L. Davis

Abstract The aim of the present study was to investigate relationships between characteristics of children’s target words and their actual productions during the single-word period in American English. Word productions in spontaneous and functional speech from 18 children acquiring American English were analyzed. Consonant sequences in 3,328 consonant-vowel-consonant (C1VC2) target words were analyzed in terms of global place of articulation (labials, coronals, and dorsals). Children’s actual productions of place sequences were compared between target words containing repeated place sequences (e.g., mom, map, dad, not) and target words containing variegated place sequences (e.g., mat, dog, cat, nap). Overall, when the target word contained two consonants at the same global place of articulation (e.g., labial-labial, map; coronal-coronal, not), approximately 50% of children’s actual productions matched consonant place characteristics. Conversely, when the target word consisted of variegated place sequences (e.g., mat, dog, cat, nap), only about 20% of the productions matched the target consonant sequences. These results suggest that children’s actual productions are influenced by their own production abilities as well as by the phonetic forms of target words.


2013 ◽  
Vol 2 (1) ◽  
Author(s):  
Radina Mohamad Deli

Barcroft (2007) found opportunities for word retrieval to be advantageous during second language vocabulary learning. This study extended such a finding and investigated the effect of increased time in target-word retrieval for learning new vocabulary in the L2, as well as the effect of presentation orders of different time conditions on word retrieval. The data were obtained from 17 native Arab speakers who attempted to learn 24 new English words by viewing 24 word-picture pairs. Each picture and its corresponding word were viewed with different time lags of 0, 6 and 12 seconds between them in different presentation orders. The results showed that, although the increased time does not positively affect word retrieval, the overall findings correspond to Barcroft’s (2007) view, at least in the case of 6 seconds lag. The results also showed that the production of target words in both the control and retrieval-oriented conditions depend on and vary according to the order of presentation, particularly in the case of 6 seconds lag in which word gain is found to be highest when the lag is presented first and second.


1987 ◽  
Vol 8 (1) ◽  
pp. 55-66 ◽  
Author(s):  
Nancy W. Streim ◽  
Robin S. Chapman

ABSTRACTThis study asked whether lexical availability affects the length, complexity, order of mention, and fluency of children's utterances. Lexical availability was manipulated through discourse support (present or absent) and word frequency (high or low) for 40 target nouns. Length was indexed by mean number of words per communication unit. Complexity was indexed by mean number of verbs per communication unit. Earlier mention was measured by mean number of words preceding the target word in each communication unit. Thirty-six subjects, aged 4, 6, and 8, described 40 illustrations containing a high or low frequency target noun referent. In the discourse support condition, provided for one half of the target words, subjects named the target word prior to the description task. Results showed that the number of responses containing the target word varied with age, word frequency and discourse support condition; length of responses varied with age and its interaction with discourse support; earlier mention varied with age and discourse support condition; and fluency varied with discourse support condition. The results are discussed from the viewpoint of Bock's process model of sentence production.


2021 ◽  
Author(s):  
◽  
TJ Boutorwick

<p>This thesis compares two approaches to extensive reading to determine the extent that they facilitate vocabulary development. The first approach is a traditional reading-only approach, and the second approach is a task-based approach which supplements reading with post-reading meaning-focused discussions. These two approaches are compared using a battery of tests, most notably a measure for productive knowledge of word associations.  For years, scholars have believed that word associations have potential to reveal important information about a person’s language proficiency. One reason word associations are intriguing is that a large amount of a person’s lexicon can be assessed (Meara, 2009). This is possible because a large amount of data from the learner can be gathered in a short period of time. Another intriguing aspect of word association data is that it is one aspect of vocabulary knowledge that is not based on correct performance. This raises the question of an appropriate means of assigning value to the associations, a question which still hinders research to this day. Recent research has made progress in this area with a multi-level taxonomy (i.e., Fitzpatrick, 2007), creating a picture of the types of associations which exist in a learner’s lexicon. However, this taxonomy does not address the strength of the association. Wilks and Meara (2007) have attempted to tackle association strength through the use of self-report measures, whereby a test-taker reports strength of association on a four-point scale from weak to strong. This has left them with "...problems which we have not yet solved, notably a tendency for some test takers to claim that most associations are strong, while others appear to be very reluctant to identify strong associations..." (Meara, 2009, p. 80). In other words, the question of how to appropriately determine association strength is still unanswered.  In the current study lexical development, in the form of word association knowledge, was measured using a multi-response word association test. Participants were assessed on their knowledge of 60 target words which occurred in five graded readers that they read over the course of the study. The learners first self-reported their knowledge of the 60 target words in terms of no knowledge, form knowledge, or meaning knowledge. The students provided up to five associations for each word that they reported at either the form or meaning levels. They did this once before reading the five graded readers, and again after finishing the graded readers.  The associations provided by the students were analyzed using Latent Semantic Analysis, a method for computing semantic similarity between words (Landauer & Dumais, 1997). The associations a learner provided for each target word were assigned a similarity value representing how similar they were to the target word to which they were provided. The hypothesis was that the students who engaged in the post-reading discussion activities would show greater increases in associational knowledge of the target words than those students who did not participate in the discussions.  The major finding from this thesis was that the students who struggled with a word during the post-reading discussion and were provided an opportunity to discuss the word with their group developed associational knowledge to a significantly greater degree than those students who did not encounter the words during the discussions. This emphasizes the facilitative role that meaning-focused output activities have on vocabulary development. In addition, the associational knowledge developed at the initial stages of word learning (i.e., from no knowledge to form knowledge), continued to develop from form knowledge of a word to meaning knowledge of the word, and was also developing even when words did not change in reported knowledge. This suggests a continual restructuring of the learners’ lexicon, exemplifying past research (e.g., Henriksen, 1999). Overall, the findings suggest that an extensive reading approach which includes opportunities for meaning-focused interaction has greater benefits for lexical development when compared to a traditional reading-only approach to extensive reading.</p>


2007 ◽  
Vol 60 (8) ◽  
pp. 1155-1167 ◽  
Author(s):  
Tania Malouf ◽  
Sachiko Kinoshita

Two experiments investigated whether priming due to a match in just the onset between a masked prime and target is found with high-frequency target words. Forster and Davis (1991, Exp. 5) reported that the masked onset priming effect was absent for high-frequency words and used the finding to argue that the effect has its locus in the grapheme–phoneme mapping process that operates serially within the nonlexical route. Experiment 1 used primes that were unrelated to targets and found a masked onset priming effect of equal size for high-frequency and low-frequency target words. Experiment 2 used form-related primes as used by Forster and Davis, and again found that the effect of onset mismatch was not dependent on target word frequency. These results are interpreted in terms of an alternative view that the masked onset priming effect has its origin in the process of preparing a speech response.


2011 ◽  
Vol 32 (4) ◽  
pp. 739-759 ◽  
Author(s):  
HEATHER WINSKEL

ABSTRACTFour eye movement experiments investigated whether readers use parafoveal input to gain information about the phonological or orthographic forms of consonants, vowels, and tones in word recognition when reading Thai silently. Target words were presented in sentences preceded by parafoveal previews in which consonant, vowel, or tone information was manipulated. Previews of homophonous consonants (Experiment 1) and concordant vowels (Experiment 2) did not substantially facilitate processing of the target word, whereas the identical previews did. Hence, orthography appears to be playing the prominent role in early word recognition for consonants and vowels. Incorrect tone marker previews (Experiment 3) substantially retarded the subsequent processing of the target word, indicating that lexical tone plays an important role in early word recognition. Vowels in VOP (Experiment 4) did not facilitate processing, which points to vowel position being a significant factor. Primarily, orthographic codes of consonants and vowels (HOP) in conjunction with tone information are assembled from parafoveal input and used for early lexical access.


1986 ◽  
Vol 25 ◽  
pp. 118-128 ◽  
Author(s):  
Pauline Stip ◽  
Jan H. Hulstijn

Two investigations were carried out to estimate the usefulness of the context method for the explanation of word meanings. This method provides the L2 learner with a sample sentence for each target word, without translation into LI. A sample of 100 sentences was selected from a monolingual vocabu-lary learning course book for adult learners of Dutch as a second language (Hart & Polter 1983). In the first study, these sample sentences were presented to 40 educated Dutch native speakers. They were asked to judge the appropriateness of the sentences (i.e. the extent to which the contexts constrained the meaning of the target words). In the second study, the same sentences, from which the target words had been deleted, were presented as a "fill-in-the-blank" test to 20 11/12-year old and 20 17/18-year old Dutch native speakers. In the discussion, it is argued that word meanings can seldom be successfully induced from the context. In general, the context method seems to be an ineffective way for initial explanation of word meanings, frequently causing noncomprehension or miscompre-hension on the part of the learner. However, the context method seems to offer an excellent means for further illustrating word meanings after they have been initially explained by translation.


Sign in / Sign up

Export Citation Format

Share Document