Automated Paraphrase Quality Assessment Using Recurrent Neural Networks and Language Models

The geolocation task of textual data shared on social networks like Twitter attracts a progressive attention. Since those data are supported by advanced geographic information systems for multipurpose spatial analysis, new trends to extend the paradigm of geolocated data become more emergent. Differently from statistical language models that are widely adopted in prior works, the authors propose a new approach that is adopted to the geolocation of both tweets and users through the application of embedding models. The authors boost the geolocation strategy with a sequential modelling using recurrent neural networks to delimit the importance of words in tweets with respect to contextual information. They evaluate the power of this strategy in order to determine locations of unstructured texts that reflect unlimited user's writing styles. Especially, the authors demonstrate that semantic proprieties and word forms can be effective to geolocate texts without specifying local words or topics' descriptions per region.

Download Full-text

Representation of Linguistic Form and Function in Recurrent Neural Networks

Computational Linguistics ◽

10.1162/coli_a_00300 ◽

2017 ◽

Vol 43 (4) ◽

pp. 761-780 ◽

Cited By ~ 18

Author(s):

Ákos Kádár ◽

Grzegorz Chrupała ◽

Afra Alishahi

Keyword(s):

Neural Networks ◽

Recurrent Neural Networks ◽

Network Architecture ◽

Language Model ◽

Visual Pathway ◽

Recurrent Network ◽

Point Of View ◽

Language Models ◽

Grammatical Function ◽

Form And Function

We present novel methods for analyzing the activation patterns of recurrent neural networks from a linguistic point of view and explore the types of linguistic structure they learn. As a case study, we use a standard standalone language model, and a multi-task gated recurrent network architecture consisting of two parallel pathways with shared word embeddings: The Visual pathway is trained on predicting the representations of the visual scene corresponding to an input sentence, and the Textual pathway is trained to predict the next word in the same sentence. We propose a method for estimating the amount of contribution of individual tokens in the input to the final prediction of the networks. Using this method, we show that the Visual pathway pays selective attention to lexical categories and grammatical functions that carry semantic information, and learns to treat word types differently depending on their grammatical function and their position in the sequential structure of the sentence. In contrast, the language models are comparatively more sensitive to words with a syntactic function. Further analysis of the most informative n-gram contexts for each model shows that in comparison with the Visual pathway, the language models react more strongly to abstract contexts that represent syntactic constructions.

Download Full-text

Temporal hierarchies in multilayer gated recurrent neural networks for language models

2017 International Joint Conference on Neural Networks (IJCNN) ◽

10.1109/ijcnn.2017.7966115 ◽

2017 ◽

Cited By ~ 1

Author(s):

Moirangthem Dennis Singh ◽

Minho Lee

Keyword(s):

Neural Networks ◽

Recurrent Neural Networks ◽

Language Models

Download Full-text

Tabula Nearly Rasa: Probing the Linguistic Knowledge of Character-level Neural Language Models Trained on Unsegmented Text

Transactions of the Association for Computational Linguistics ◽

10.1162/tacl_a_00283 ◽

2019 ◽

Vol 7 ◽

pp. 467-484 ◽

Cited By ~ 1

Author(s):

Michael Hahn ◽

Marco Baroni

Keyword(s):

Neural Networks ◽

Language Learning ◽

Language Processing ◽

Recurrent Neural Networks ◽

Input Data ◽

Language Models ◽

Linguistic Knowledge ◽

Word Level ◽

Linguistic Unit ◽

Word Boundaries

Recurrent neural networks (RNNs) have reached striking performance in many natural language processing tasks. This has renewed interest in whether these generic sequence processing devices are inducing genuine linguistic knowledge. Nearly all current analytical studies, however, initialize the RNNs with a vocabulary of known words, and feed them tokenized input during training. We present a multi-lingual study of the linguistic knowledge encoded in RNNs trained as character-level language models, on input data with word boundaries removed. These networks face a tougher and more cognitively realistic task, having to discover any useful linguistic unit from scratch based on input statistics. The results show that our “near tabula rasa” RNNs are mostly able to solve morphological, syntactic and semantic tasks that intuitively presuppose word-level knowledge, and indeed they learned, to some extent, to track word boundaries. Our study opens the door to speculations about the necessity of an explicit, rigid word lexicon in language learning and usage.

Download Full-text