scholarly journals Prediction as a Basis for Skilled Reading: Insights from Natural Language Engineering Models

2021 ◽  
Author(s):  
Benedetta Cevoli ◽  
Chris Watkins ◽  
Kathleen Rastle
Keyword(s):  
Natural Language ◽  
Eye Movement ◽  
Internal State ◽  
Movement Behaviour ◽  
Language Engineering ◽  
Fine Grained ◽  
Modern Natural ◽  
Language Research ◽  
Engineering Models ◽  
English Speaking

Reading is not an inborn human capability, and yet, English-speaking adults read with impressive speed. This study considered how predictions of upcoming words impact on this skilled behaviour. We used a powerful computer model from natural language engineering (GPT-2) to derive predictions of upcoming words in text passages. These predictions were highly accurate, and showed a tight relationship to fine-grained aspects of eye-movement behaviour when adults read those same passages, including whether to skip the next word and how long to spend on it. Strong predictions that did not materialise resulted in a prediction error cost on fixation durations. Our findings suggest that predictions for upcoming words can be made based on relatively superficial statistical information in reading, and that these predictions guide how our eyes interrogate text. This study is the first to demonstrate a relationship between the internal state of a modern natural language engineering model and eye-movement behaviour in reading, opening substantial new opportunities for language research and application.

Fast Neural Network Engine for Natural Science Language Processing: A Drug-Search Case.

2020 ◽  
Author(s):  
Vadim V. Korolev ◽  
Artem Mitrofanov ◽  
Kirill Karpov ◽  
Valery Tkachenko
Keyword(s):  
Neural Network ◽  
Natural Language Processing ◽  
Natural Language ◽  
Language Processing ◽  
Natural Science ◽  
Therapeutic Agent ◽  
Semantic Relations ◽  
Chemical Data ◽  
Processing Methods ◽  
Modern Natural

The main advantage of modern natural language processing methods is a possibility to turn an amorphous human-readable task into a strict mathematic form. That allows to extract chemical data and insights from articles and to find new semantic relations. We propose a universal engine for processing chemical and biological texts. We successfully tested it on various use-cases and applied to a case of searching a therapeutic agent for a COVID-19 disease by analyzing PubMed archive.

scholarly journals Datives with psych nouns and adjectives in Basque

2020 ◽  
Vol 54 (3) ◽  
pp. 647-696
Author(s):  
Beatriz Fernández ◽  
Fernando Zúñiga ◽  
Ane Berro
Keyword(s):  
Natural Language ◽  
Linguistic Theory ◽  
Psychological State ◽  
Formal Expression ◽  
Fine Grained ◽  
Psych Verbs ◽  
Other Regarding ◽  
Psychological Verbs

Abstract This paper explores the formal expression of two Basque dative argument types in combination with psych nouns and adjectives, in intransitive and transitive clauses: (i) those that express the experiencer, and (ii) those that express the stimulus of the psychological state denoted by the psych noun and adjective. In the intransitive structure involving a dative experiencer (DatExpIS), the stimulus is in the absolutive case, and the intransitive copula izan ‘be’ shows both dative and absolutive agreement. This construction basically corresponds to those built upon the piacere type of psychological verbs typified in (Belletti, Adriana & Luigi Rizzi. 1988. Psych-verbs and θ-theory. Natural Language and Linguistic Theory 6. 291–352) three-way classification of Italian psych verbs. In the intransitive structure involving a dative stimulus (DatStimIS), the experiencer is marked by absolutive case, and the same intransitive copula shows both absolutive and dative agreement (with the latter corresponding to the dative stimulus and not to the experiencer). We show that the behavior of the dative argument in the two constructions is just the opposite of each other regarding a number of morphosyntactic tests, including agreement, constituency, hierarchy and selection. Additionally, we explore two parallel transitive constructions that involve either a dative experiencer and an ergative stimulus (DatExpTS) or a dative stimulus and an ergative experiencer (DatStimTS), which employ the transitive copula *edun ‘have’. Considering these configurations, we propose an extended and more fine-grained typology of psych predicates.

scholarly journals How Efficiency Shapes Human Language, TICS 2019

2019 ◽  
Author(s):  
Edward Gibson ◽  
Richard Futrell ◽  
Steven T. Piantadosi ◽  
Isabelle Dautriche ◽  
Kyle Mahowald ◽  
...  
Keyword(s):  
Information Theory ◽  
Cognitive Psychology ◽  
Natural Language ◽  
Cognitive Science ◽  
Human Language ◽  
Language Research ◽  
The Way

Cognitive science applies diverse tools and perspectives to study human language. Recently, an exciting body of work has examined linguistic phenomena through the lens of efficiency in usage: what otherwise puzzling features of language find explanation in formal accounts of how language might be optimized for communication and learning? Here, we review studies that deploy formal tools from probability and information theory to understand how and why language works the way that it does, focusing on phenomena ranging from the lexicon through syntax. These studies show how apervasive pressure for efficiency guides the forms of natural language and indicate that a rich future for language research lies in connecting linguistics to cognitive psychology and mathematical theories of communication and inference.

scholarly journals Emerging trends: A gentle introduction to fine-tuning

2021 ◽  
Vol 27 (6) ◽  
pp. 763-778
Author(s):  
Kenneth Ward Church ◽  
Zeyu Chen ◽  
Yanjun Ma
Keyword(s):  
Natural Language ◽  
Language Processing ◽  
Question Answering ◽  
General Purpose ◽  
Fine Tuning ◽  
Language Engineering ◽  
Training Models ◽  
Emerging Trends ◽  
Foundation Model ◽  
Programming Skills

AbstractThe previous Emerging Trends article (Church et al., 2021. Natural Language Engineering27(5), 631–645.) introduced deep nets to poets. Poets is an imperfect metaphor, intended as a gesture toward inclusion. The future for deep nets will benefit by reaching out to a broad audience of potential users, including people with little or no programming skills, and little interest in training models. That paper focused on inference, the use of pre-trained models, as is, without fine-tuning. The goal of this paper is to make fine-tuning more accessible to a broader audience. Since fine-tuning is more challenging than inference, the examples in this paper will require modest programming skills, as well as access to a GPU. Fine-tuning starts with a general purpose base (foundation) model and uses a small training set of labeled data to produce a model for a specific downstream application. There are many examples of fine-tuning in natural language processing (question answering (SQuAD) and GLUE benchmark), as well as vision and speech.

scholarly journals Densely Supervised Hierarchical Policy-Value Network for Image Paragraph Generation

2019 ◽  
Author(s):  
Siying Wu ◽  
Zheng-Jun Zha ◽  
Zilei Wang ◽  
Houqiang Li ◽  
Feng Wu
Keyword(s):  
Natural Language ◽  
State Of The Art ◽  
Cross Entropy ◽  
Image Captioning ◽  
Value Network ◽  
Entropy Loss ◽  
Fine Grained ◽  
Performance Improvements ◽  
Single Sentence ◽  
Multiple State

Image paragraph generation aims to describe an image with a paragraph in natural language. Compared to image captioning with a single sentence, paragraph generation provides more expressive and fine-grained description for storytelling. Existing approaches mainly optimize paragraph generator towards minimizing word-wise cross entropy loss, which neglects linguistic hierarchy of paragraph and results in ``sparse" supervision for generator learning. In this paper, we propose a novel Densely Supervised Hierarchical Policy-Value (DHPV) network for effective paragraph generation. We design new hierarchical supervisions consisting of hierarchical rewards and values at both sentence and word levels. The joint exploration of hierarchical rewards and values provides dense supervision cues for learning effective paragraph generator. We propose a new hierarchical policy-value architecture which exploits compositionality at token-to-token and sentence-to-sentence levels simultaneously and can preserve the semantic and syntactic constituent integrity. Extensive experiments on the Stanford image-paragraph benchmark have demonstrated the effectiveness of the proposed DHPV approach with performance improvements over multiple state-of-the-art methods.

scholarly journals Incremental comprehension of pitch relationships in written music: Evidence from eye movements

2018 ◽  
Vol 71 (1) ◽  
pp. 211-219 ◽  
Author(s):  
Lauren V Hadley ◽  
Patrick Sturt ◽  
Tuomas Eerola ◽  
Martin J Pickering
Keyword(s):  
Eye Movements ◽  
Eye Movement ◽  
Sight Reading ◽  
Harmonic Structure ◽  
Movement Behaviour ◽  
Prior Context ◽  
Concurrent Performance ◽  
Text Reading ◽  
Processing Difficulty ◽  
Structural Processing

To investigate how proficient pianists comprehend pitch relationships in written music when they first encounter it, we conducted two experiments in which proficient pianists’ eyes were tracked while they read and played single-line melodies. In Experiment 1, participants played at their own speed; in Experiment 2, they played with an external metronome. The melodies were either congruent or anomalous, with the anomaly involving one bar being shifted in pitch to alter the implied harmonic structure (e.g. non-resolution of a dominant). In both experiments, anomaly led to rapid disruption in participants’ eye movements in terms of regressions from the target bar, indicating that pianists process written pitch relationships online. This is particularly striking because in musical sight-reading, eye movement behaviour is constrained by the concurrent performance. Both experiments also showed that anomaly induced pupil dilation. Together, these results indicate that proficient pianists rapidly integrate the music that they read into the prior context and that anomalies in terms of pitch relationships lead to processing difficulty. These findings parallel those of text reading, suggesting that structural processing involves similar constraints across domains.

Sign in / Sign up

Export Citation Format

Share Document