scholarly journals Children flexibly seek visual information during signed and spoken language comprehension

2018 ◽  
Author(s):  
Kyle Earl MacDonald ◽  
Virginia Marchman ◽  
Anne Fernald ◽  
Michael C. Frank
Keyword(s):  
Eye Movements ◽  
American Sign Language ◽  
Real Time ◽  
Language Processing ◽  
Language Comprehension ◽  
Visual Information ◽  
Social Partner ◽  
Gaze Shifts ◽  
Spoken English ◽  
English Speaking

During grounded language comprehension, listeners must link the incoming linguistic signal to the visual world despite noise in the input. Information gathered through visual fixations can facilitate understanding. But do listeners flexibly seek supportive visual information? Here, we propose that even young children can adapt their gaze and actively gather information that supports their language understanding. We present two case studies of eye movements during real-time language processing where the value of fixating on a social partner varies across different contexts. First, compared to children learning spoken English (n=80), young American Sign Language (ASL) learners (n=30) delayed gaze shifts away from a language source and produced a higher proportion of language-consistent eye movements. This result suggests that ASL learners adapt to dividing attention between language and referents, which both compete for processing via the same channel: vision. Second, English-speaking preschoolers (n=39) and adults (n=31) delayed the timing of gaze shifts away from a speaker’s face while processing language in a noisy auditory environment. This delay resulted in a higher proportion of language-consistent gaze shifts. These results suggest that young listeners can adapt their gaze to seek supportive visual information from social partners during real-time language comprehension.

scholarly journals Real-time lexical comprehension in young children learning American Sign Language

2017 ◽  
Author(s):  
Kyle Earl MacDonald ◽  
Todd LaMarr ◽  
David Corina ◽  
Virginia Marchman ◽  
Anne Fernald
Keyword(s):  
Eye Movements ◽  
Visual Attention ◽  
American Sign Language ◽  
Real Time ◽  
Sign Language ◽  
Language Comprehension ◽  
Auditory Information ◽  
American Sign ◽  
Visual Channel ◽  
Visual World

When children interpret spoken language in real time, linguistic information drives rapidshifts in visual attention to objects in the visual world. This language-vision interaction canprovide insights into children's developing efficiency in language comprehension. But how doeslanguage influence visual attention when the linguistic signal and the visual world are bothprocessed via the visual channel? Here, we measured eye movements during real-timecomprehension of a visual-manual language, American Sign Language (ASL), by 29 nativeASL-learning children (16-53 mos, 16 deaf, 13 hearing) and 16 fluent deaf adult signers. Allsigners showed evidence of rapid, incremental language comprehension, tending to initiate aneye movement before sign offset. Deaf and hearing ASL-learners showed similar gaze patterns,suggesting that the in-the-moment dynamics of eye movements during ASL processing areshaped by the constraints of processing a visual language in real time and not by differentialaccess to auditory information in day-to-day life. Finally, variation in children’s ASL processingwas positively correlated with age and vocabulary size. Thus, despite competition for attentionwithin a single modality, the timing and accuracy of visual fixations during ASL comprehensionreflect information processing skills that are fundamental for language acquisition regardless oflanguage modality.

Spoken language comprehension: insights from eye movements

2007 ◽  
pp. 308-326 ◽  
Author(s):  
Michael K. Tanenhaus
Keyword(s):  
Eye Movements ◽  
Language Processing ◽  
Language Comprehension ◽  
Language Production ◽  
World Literature ◽  
Spoken Language ◽  
Response Measure ◽  
Visual World ◽  
Spoken Language Processing ◽  
Spoken Language Comprehension

Recently, eye movements have become a widely used response measure for studying spoken language processing in both adults and children, in situations where participants comprehend and generate utterances about a circumscribed “Visual World” while fixation is monitored, typically using a free-view eye-tracker. Psycholinguists now use the Visual World eye-movement method to study both language production and language comprehension, in studies that run the gamut of current topics in language processing. Eye movements are a response measure of choice for addressing many classic questions about spoken language processing in psycholinguistics. This article reviews the burgeoning Visual World literature on language comprehension, highlighting some of the seminal studies and examining how the Visual World approach has contributed new insights to our understanding of spoken word recognition, parsing, reference resolution, and interactive conversation. It considers some of the methodological issues that come to the fore when psycholinguists use eye movements to examine spoken language comprehension.

Eye movements as a window into real-time spoken language comprehension in natural contexts

1995 ◽  
Vol 24 (6) ◽  
pp. 409-436 ◽  
Author(s):  
Kathleen M. Eberhard ◽  
Michael J. Spivey-Knowlton ◽  
Julie C. Sedivy ◽  
Michael K. Tanenhaus
Keyword(s):  
Eye Movements ◽  
Real Time ◽  
Language Comprehension ◽  
Spoken Language ◽  
Spoken Language Comprehension

scholarly journals Electrophysiological modulations of lexical prediction by conversation requirements

2019 ◽  
Author(s):  
Cristina Baus ◽  
Anne-Sophie Dubarry ◽  
F.-Xavier Alario
Keyword(s):  
Social Context ◽  
Language Processing ◽  
Language Comprehension ◽  
Social Life ◽  
Language Production ◽  
Social Partner ◽  
Social Relevance ◽  
The Social ◽  
Priming Paradigm ◽  
Interactive Nature

Language mediates most of our social life and yet, despite such social relevance and ubiquity, little is known about language processing during social interactions. To explore this issue, two experiments were designed to isolate two basic components of a conversation: 1) the interplay between language production and comprehension systems, and 2) the participation of a social partner. We explored how prediction processes in language comprehension are modulated by two basic components of a conversation. Participants were asked to perform a cross-modal priming paradigm in two blocks, one involving only comprehension trials and another in which trials requiring production and comprehension were intermixed. In the first experiment, participants were alone during the task and in the second experiment, participants believed they were performing the task jointly with an interactive partner. A critical electrophysiological signature of lexical prediction was observed, the N400 component, allowing to assess its modulation across conditions and experiments. when production was involved in the task, the effect of lexical predictability was enhanced at the early stages of language comprehension (anticipatory phase), irrespective of the social context. In contrast, language production reduced the effect of lexical predictability at later stages (integration phase), only when participants performed the task alone but not in the social context. These results support production based-models and reveal the importance of exploring language considering its interactive nature.

Regarding Scenes

2007 ◽  
Vol 16 (4) ◽  
pp. 219-222 ◽  
Author(s):  
John M. Henderson
Keyword(s):  
Eye Movements ◽  
Real Time ◽  
Real World ◽  
Visual Information ◽  
Scene Perception ◽  
Saccadic Eye Movements ◽  
Gaze Direction ◽  
Behavioral Activity ◽  
Gaze Control ◽  
Gaze Stability

When we view the visual world, our eyes flit from one location to another about three times each second. These frequent changes in gaze direction result from very fast saccadic eye movements. Useful visual information is acquired only during fixations, periods of relative gaze stability. Gaze control is defined as the process of directing fixation through a scene in real time in the service of ongoing perceptual, cognitive, and behavioral activity. This article discusses current approaches and new empirical findings that are allowing investigators to unravel how human gaze control operates during active real-world scene perception.

Google translate in healthcare: preliminary evaluation of transcription, translation and speech synthesis accuracy

2021 ◽  
pp. bmjinnov-2019-000347
Author(s):  
Jack Birkenbeuel ◽  
Helen Joyce ◽  
Ronald Sahyouni ◽  
Dillon Cheung ◽  
Marlon Maducdoc ◽  
...  
Keyword(s):  
Real Time ◽  
Limited English Proficiency ◽  
Speech Synthesis ◽  
Oral Communication ◽  
Preliminary Evaluation ◽  
Medical Terminology ◽  
Spoken English ◽  
Time Translation ◽  
English Speaking ◽  
Translation Errors

ObjectiveTo assess the ability of Google Translate (GT) to accurately interpret single sentences and series of sentences commonly used in healthcare encounters from English to Spanish.DesignEnglish-speaking volunteers used GT to interpret a list of 83 commonly used sentences and series of sentences of different lengths containing both medical and non-medical terminology. A certified medical interpreter evaluated whether the meaning of these sentences was preserved.ParticipantsEighteen English-speaking subjects (nine males and nine females), with a mean age of 36 years, volunteered for this study to read sentences.Main outcome measuresThe accuracy of GTs (1) real-time voice recognition (ie, transcription) of English sentences, (2) real-time translation of these transcribed English sentences to Spanish, and (3) GTs speech synthesis ability to preserve the meaning of spoken English sentences after translation to Spanish.ResultsSpeech synthesis accuracy, with preservation of the original English-spoken sentence(s), was 89.4% for single sentences with ≤8 words; 90.6% for single sentences with >8 words; 52.2% for two sentences and 26.6% for three sentences. Furthermore, the number of transcription and translation errors per sentence(s) significantly increased with the number of sentences (p<0.05).ConclusionsDespite the fact that GTs accuracy was widely variable and dependent on the length of the spoken sentence(s), GT is readily accessible, has no associated monetary costs, and offers nearly immediate interpretation services. As such, it has the potential to routinely facilitate effective one-way oral communication between English-speaking physicians and Spanish-speaking patients with limited English proficiency.

scholarly journals Discovering prominence and its role in language processing: An individual (differences) approach

2015 ◽  
Vol 1 (1) ◽  
Author(s):  
Phillip M. Alday ◽  
Matthias Schlesewsky ◽  
Ina Bornkessel-Schlesewsky
Keyword(s):  
Individual Differences ◽  
Real Time ◽  
Language Processing ◽  
Language Comprehension ◽  
Processing System ◽  
Partial Ordering ◽  
Human Language ◽  
Qualitative Work ◽  
State Of Affairs ◽  
The Relationship

AbstractIt has been suggested that, during real time language comprehension, the human language processing system attempts to identify the argument primarily responsible for the state of affairs (the “actor”) as quickly and unambiguously as possible. However, previous work on a prominence (e.g. animacy, definiteness, case marking) based heuristic for actor identification has suffered from underspecification of the relationship between different cue hierarchies. Qualitative work has yielded a partial ordering of many features (e.g.: OpenSesame experiment and Python support scripts, sample stimuli, R scripts for analysis

scholarly journals Integration of gaze information during online language comprehension and learning

2019 ◽  
Author(s):  
Kyle Earl MacDonald ◽  
Elizabeth Swanson ◽  
Michael C. Frank
Keyword(s):  
Visual Attention ◽  
Eye Tracking ◽  
Language Processing ◽  
Language Comprehension ◽  
Visual Information ◽  
Eye Gaze ◽  
The Other ◽  
Face To Face ◽  
Social Partners ◽  
Processing Mechanisms

Face-to-face communication provides access to visual information that can support language processing. But do listeners automatically seek social information without regard to the language processing task? Here, we present two eye-tracking studies that ask whether listeners’ knowledge of word-object links changes how they actively gather a social cue to reference (eye gaze) during real-time language processing. First, when processing familiar words, children and adults did not delay their gaze shifts to seek a disambiguating gaze cue. When processing novel words, however, children and adults fixated longer on a speaker who provided a gaze cue, which led to an increase in looking to the named object and less looking to the other object in the scene. These results suggest that listeners use their knowledge of object labels when deciding how to allocate visual attention to social partners, which in turn changes the visual input to language processing mechanisms.

scholarly journals Real-time processing of event descriptions for partially- and fully-completed events: Evidence from the visual world paradigm

2021 ◽  
Vol 6 (1) ◽  
pp. 118
Author(s):  
Max J. Kaplan ◽  
Amulya Raju ◽  
Sudha Arunachalam
Keyword(s):  
Real Time ◽  
Language Processing ◽  
Visual World ◽  
Visual World Paradigm ◽  
Real Time Processing ◽  
Time Processing ◽  
Semantic Account ◽  
Verb Forms ◽  
Partial Completion ◽  
English Speaking

The current study investigated non-culminating accomplishments through an experimental lens. We used a well-established paradigm for studying real-time language processing using eye-tracking, the visual world paradigm. Our study was modeled after Altmann and Kamide’s (2007) investigation of processing of aspectual information contained in a perfect verb form (e.g., has eaten). We compared English-speaking adults’ interpretations of sentences like ‘The girl has eaten a cookie’ and ‘The girl was eating a cookie’ in the context of one of two visual scenes. In the Full Completion condition, the scene depicted two referents that were compatible with the predicate: one was compatible with the expected end state of the event (e.g., an empty plate), the other with an unrealized version of the event (e.g., an uneaten cookie). In the Partial Completion condition, the scene depicted a referent that was compatible with a partially-completed version of the event (e.g., part of a cookie on a plate) and an unrealized interpretation (e.g., an uneaten cookie). For verb forms in the perfect (e.g., has eaten) but not in the progressive, we found a difference between conditions; listeners preferred to look at the fully-affected referent in the Full Completion condition as compared to the partially-affected referent in the Partial Completion condition. We take the results as suggestive in favor of a pragmatic rather than semantic account of non-culmination interpretations in English.

scholarly journals Visual context constrains language-mediated anticipatory eye movements

2019 ◽  
Vol 73 (3) ◽  
pp. 458-467
Author(s):  
Florian Hintz ◽  
Antje S Meyer ◽  
Falk Huettig
Keyword(s):  
Eye Movements ◽  
Language Processing ◽  
Visual Information ◽  
Eye Gaze ◽  
Target Object ◽  
Predictive Processing ◽  
Selectional Restrictions ◽  
Anticipatory Eye Movements ◽  
Multiple Levels ◽  
Visual Shape

Contemporary accounts of anticipatory language processing assume that individuals predict upcoming information at multiple levels of representation. Research investigating language-mediated anticipatory eye gaze typically assumes that linguistic input restricts the domain of subsequent reference (visual target objects). Here, we explored the converse case: Can visual input restrict the dynamics of anticipatory language processing? To this end, we recorded participants’ eye movements as they listened to sentences in which an object was predictable based on the verb’s selectional restrictions (“The man peels a banana”). While listening, participants looked at different types of displays: the target object (banana) was either present or it was absent. On target-absent trials, the displays featured objects that had a similar visual shape as the target object (canoe) or objects that were semantically related to the concepts invoked by the target (monkey). Each trial was presented in a long preview version, where participants saw the displays for approximately 1.78 s before the verb was heard (pre-verb condition), and a short preview version, where participants saw the display approximately 1 s after the verb had been heard (post-verb condition), 750 ms prior to the spoken target onset. Participants anticipated the target objects in both conditions. Importantly, robust evidence for predictive looks to objects related to the (absent) target objects in visual shape and semantics was found in the post-verb but not in the pre-verb condition. These results suggest that visual information can restrict language-mediated anticipatory gaze and delineate theoretical accounts of predictive processing in the visual world.

Sign in / Sign up

Export Citation Format

Share Document