scholarly journals How Furiously Can Colorless Green Ideas Sleep? Sentence Acceptability in Context

2020 ◽  
Vol 8 ◽  
pp. 296-310
Author(s):  
Jey Han Lau ◽  
Carlos Armendariz ◽  
Shalom Lappin ◽  
Matthew Purver ◽  
Chang Shu
Keyword(s):  
Computational Modeling ◽  
Cognitive Load ◽  
Sentence Processing ◽  
State Of The Art ◽  
Language Models ◽  
Discourse Coherence ◽  
Relevant Context ◽  
Green Ideas ◽  
Coherence Effect

We study the influence of context on sentence acceptability. First we compare the acceptability ratings of sentences judged in isolation, with a relevant context, and with an irrelevant context. Our results show that context induces a cognitive load for humans, which compresses the distribution of ratings. Moreover, in relevant contexts we observe a discourse coherence effect that uniformly raises acceptability. Next, we test unidirectional and bidirectional language models in their ability to predict acceptability ratings. The bidirectional models show very promising results, with the best model achieving a new state-of-the-art for unsupervised acceptability prediction. The two sets of experiments provide insights into the cognitive aspects of sentence processing and central issues in the computational modeling of text and discourse.

Towards corpus and model: Hierarchical structured-attention-based features for Indonesian named entity recognition

2021 ◽  
pp. 1-12
Author(s):  
Yingwen Fu ◽  
Nankai Lin ◽  
Xiaotian Lin ◽  
Shengyi Jiang
Keyword(s):  
Language Processing ◽  
State Of The Art ◽  
Named Entity Recognition ◽  
Entity Recognition ◽  
Language Models ◽  
Neural Models ◽  
Performance Models ◽  
Named Entity ◽  
High Resource ◽  
Benchmark Datasets

Named entity recognition (NER) is fundamental to natural language processing (NLP). Most state-of-the-art researches on NER are based on pre-trained language models (PLMs) or classic neural models. However, these researches are mainly oriented to high-resource languages such as English. While for Indonesian, related resources (both in dataset and technology) are not yet well-developed. Besides, affix is an important word composition for Indonesian language, indicating the essentiality of character and token features for token-wise Indonesian NLP tasks. However, features extracted by currently top-performance models are insufficient. Aiming at Indonesian NER task, in this paper, we build an Indonesian NER dataset (IDNER) comprising over 50 thousand sentences (over 670 thousand tokens) to alleviate the shortage of labeled resources in Indonesian. Furthermore, we construct a hierarchical structured-attention-based model (HSA) for Indonesian NER to extract sequence features from different perspectives. Specifically, we use an enhanced convolutional structure as well as an enhanced attention structure to extract deeper features from characters and tokens. Experimental results show that HSA establishes competitive performance on IDNER and three benchmark datasets.

scholarly journals Attending to Entities for Better Text Understanding

2020 ◽  
Vol 34 (05) ◽  
pp. 7554-7561
Author(s):  
Pengxiang Cheng ◽  
Katrin Erk
Keyword(s):  
Large Scale ◽  
Human Performance ◽  
State Of The Art ◽  
Syntactic Structure ◽  
Semantic Knowledge ◽  
Training Data ◽  
Language Models ◽  
Long Distance ◽  
Future Directions ◽  
Text Understanding

Recent progress in NLP witnessed the development of large-scale pre-trained language models (GPT, BERT, XLNet, etc.) based on Transformer (Vaswani et al. 2017), and in a range of end tasks, such models have achieved state-of-the-art results, approaching human performance. This clearly demonstrates the power of the stacked self-attention architecture when paired with a sufficient number of layers and a large amount of pre-training data. However, on tasks that require complex and long-distance reasoning where surface-level cues are not enough, there is still a large gap between the pre-trained models and human performance. Strubell et al. (2018) recently showed that it is possible to inject knowledge of syntactic structure into a model through supervised self-attention. We conjecture that a similar injection of semantic knowledge, in particular, coreference information, into an existing model would improve performance on such complex problems. On the LAMBADA (Paperno et al. 2016) task, we show that a model trained from scratch with coreference as auxiliary supervision for self-attention outperforms the largest GPT-2 model, setting the new state-of-the-art, while only containing a tiny fraction of parameters compared to GPT-2. We also conduct a thorough analysis of different variants of model architectures and supervision configurations, suggesting future directions on applying similar techniques to other problems.

scholarly journals MSA Transformer

2021 ◽  
Author(s):  
Roshan Rao ◽  
Jason Liu ◽  
Robert Verkuil ◽  
Joshua Meier ◽  
John F. Canny ◽  
...  
Keyword(s):  
Structure Learning ◽  
State Of The Art ◽  
Language Model ◽  
Language Modeling ◽  
Language Models ◽  
Multiple Sequence ◽  
Wide Margin ◽  
Current State ◽  
Individual Sequences ◽  
And Function

AbstractUnsupervised protein language models trained across millions of diverse sequences learn structure and function of proteins. Protein language models studied to date have been trained to perform inference from individual sequences. The longstanding approach in computational biology has been to make inferences from a family of evolutionarily related sequences by fitting a model to each family independently. In this work we combine the two paradigms. We introduce a protein language model which takes as input a set of sequences in the form of a multiple sequence alignment. The model interleaves row and column attention across the input sequences and is trained with a variant of the masked language modeling objective across many protein families. The performance of the model surpasses current state-of-the-art unsupervised structure learning methods by a wide margin, with far greater parameter efficiency than prior state-of-the-art protein language models.

scholarly journals Modeling Topical Relevance for Multi-Turn Dialogue Generation

2020 ◽  
Author(s):  
Hainan Zhang ◽  
Yanyan Lan ◽  
Liang Pang ◽  
Hongshen Chen ◽  
Zhuoye Ding ◽  
...  
Keyword(s):  
Topic Model ◽  
State Of The Art ◽  
Training Dataset ◽  
Common Phenomenon ◽  
New Model ◽  
Sentence Level ◽  
Topic Distribution ◽  
Topic Drift ◽  
Relevant Context ◽  
Customer Services

Topic drift is a common phenomenon in multi-turn dialogue. Therefore, an ideal dialogue generation models should be able to capture the topic information of each context, detect the relevant context, and produce appropriate responses accordingly. However, existing models usually use word or sentence level similarities to detect the relevant contexts, which fail to well capture the topical level relevance. In this paper, we propose a new model, named STAR-BTM, to tackle this problem. Firstly, the Biterm Topic Model is pre-trained on the whole training dataset. Then, the topic level attention weights are computed based on the topic representation of each context. Finally, the attention weights and the topic distribution are utilized in the decoding process to generate the corresponding responses. Experimental results on both Chinese customer services data and English Ubuntu dialogue data show that STAR-BTM significantly outperforms several state-of-the-art methods, in terms of both metric-based and human evaluations.

Ascent of Pre-trained State-of-the-Art Language Models

2020 ◽  
pp. 269-280
Author(s):  
Keval Nagda ◽  
Anirudh Mukherjee ◽  
Milind Shah ◽  
Pratik Mulchandani ◽  
Lakshmi Kurup
Keyword(s):  
State Of The Art ◽  
Language Models

scholarly journals Fast, Small and Exact: Infinite-order Language Modelling with Compressed Suffix Trees

2016 ◽  
Vol 4 ◽  
pp. 477-490 ◽  
Author(s):  
Ehsan Shareghi ◽  
Matthias Petri ◽  
Gholamreza Haffari ◽  
Trevor Cohn
Keyword(s):  
Infinite Order ◽  
State Of The Art ◽  
Language Model ◽  
Language Models ◽  
Memory Usage ◽  
Suffix Trees ◽  
Construction Time ◽  
Modest Increase ◽  
Order Language ◽  
Language Modelling

Efficient methods for storing and querying are critical for scaling high-order m-gram language models to large corpora. We propose a language model based on compressed suffix trees, a representation that is highly compact and can be easily held in memory, while supporting queries needed in computing language model probabilities on-the-fly. We present several optimisations which improve query runtimes up to 2500×, despite only incurring a modest increase in construction time and memory usage. For large corpora and high Markov orders, our method is highly competitive with the state-of-the-art KenLM package. It imposes much lower memory requirements, often by orders of magnitude, and has runtimes that are either similar (for training) or comparable (for querying).

scholarly journals Theory and practice of modeling van der Waals interactions in electronic-structure calculations

2019 ◽  
Vol 48 (15) ◽  
pp. 4118-4154 ◽  
Author(s):  
Martin Stöhr ◽  
Troy Van Voorhis ◽  
Alexandre Tkatchenko
Keyword(s):  
Electronic Structure ◽  
Computational Modeling ◽  
State Of The Art ◽  
Van Der Waals ◽  
Electronic Structure Calculations ◽  
Black Box ◽  
Theory And Practice ◽  
Van Der Waals Interactions ◽  
Dispersion Interactions ◽  
Structure Calculations

Opening the black box of van der Waals-inclusive electronic structure calculations: a tutorial-style introduction to van der Waals dispersion interactions, state-of-the-art methods in computational modeling and complementary experimental techniques.

scholarly journals QASC: A Dataset for Question Answering via Sentence Composition

2020 ◽  
Vol 34 (05) ◽  
pp. 8082-8090
Author(s):  
Tushar Khot ◽  
Peter Clark ◽  
Michal Guerquin ◽  
Peter Jansen ◽  
Ashish Sabharwal
Keyword(s):  
Common Sense ◽  
Human Performance ◽  
Question Answering ◽  
State Of The Art ◽  
Multiple Choice ◽  
Training Data ◽  
Language Models ◽  
Current State ◽  
New Concepts ◽  
Large Corpus

Composing knowledge from multiple pieces of texts is a key challenge in multi-hop question answering. We present a multi-hop reasoning dataset, Question Answering via Sentence Composition (QASC), that requires retrieving facts from a large corpus and composing them to answer a multiple-choice question. QASC is the first dataset to offer two desirable properties: (a) the facts to be composed are annotated in a large corpus, and (b) the decomposition into these facts is not evident from the question itself. The latter makes retrieval challenging as the system must introduce new concepts or relations in order to discover potential decompositions. Further, the reasoning model must then learn to identify valid compositions of these retrieved facts using common-sense reasoning. To help address these challenges, we provide annotation for supporting facts as well as their composition. Guided by these annotations, we present a two-step approach to mitigate the retrieval challenges. We use other multiple-choice datasets as additional training data to strengthen the reasoning model. Our proposed approach improves over current state-of-the-art language models by 11% (absolute). The reasoning and retrieval problems, however, remain unsolved as this model still lags by 20% behind human performance.

Improving word prediction for augmentative communication by using idiolects and sociolects

2014 ◽  
Vol 3 (2) ◽  
pp. 137-154 ◽  
Author(s):  
Wessel Stoop ◽  
Antal van den Bosch
Keyword(s):  
Data Collection ◽  
State Of The Art ◽  
Training Data ◽  
Language Models ◽  
Prediction System ◽  
Word Prediction ◽  
The People ◽  
The Social ◽  
Statistical Language Models ◽  
Assistive Communication

Word prediction, or predictive editing, has a long history as a tool for augmentative and assistive communication. Improvements in the state-of-the-art can still be achieved, for instance by training personalized statistical language models. We developed the word prediction system Soothsayer. The main innovation of Soothsayer is that it not only uses idiolects, the language of one individual person, as training data, but also sociolects, the language of the social circle around that person. We use Twitter for data collection and experimentation. The idiolect models are based on individual Twitter feeds, the sociolect models are based on the tweets of a particular person and the tweets of the people he often communicates with. The sociolect approach achieved the best results. For a number of users, more than 50% of the keystrokes could have been saved if they had used Soothsayer.

scholarly journals Towards computational models of multilingual sentence processing

2019 ◽  
Author(s):  
Stefan L. Frank
Keyword(s):  
Computational Linguistics ◽  
Sentence Processing ◽  
Computational Models ◽  
State Of The Art ◽  
Cognitive Models ◽  
The State ◽  
Cognitive Modelling ◽  
Current Review ◽  
Multiple Languages ◽  
Human Sentence Processing

Although computational models can simulate aspects of human sentence processing, research on this topic has remained almost exclusively limited to the single language case. The current review presents an overview of the state of the art in computational cognitive models of sentence processing, and discusses how recent sentence-processing models can be used to study bi- and multilingualism. Recent results from cognitive modelling and computational linguistics suggest that phenomena specific to bilingualism can emerge from systems that have no dedicated components for handling multiple languages. Hence, accounting for human bi-/multilingualism may not require models that are much more sophisticated than those for the monolingual case.

Sign in / Sign up

Export Citation Format

Share Document