scholarly journals Tabula Nearly Rasa: Probing the Linguistic Knowledge of Character-level Neural Language Models Trained on Unsegmented Text

2019 ◽  
Vol 7 ◽  
pp. 467-484 ◽  
Author(s):  
Michael Hahn ◽  
Marco Baroni
Keyword(s):  
Neural Networks ◽  
Language Learning ◽  
Language Processing ◽  
Recurrent Neural Networks ◽  
Input Data ◽  
Language Models ◽  
Linguistic Knowledge ◽  
Word Level ◽  
Linguistic Unit ◽  
Word Boundaries

Recurrent neural networks (RNNs) have reached striking performance in many natural language processing tasks. This has renewed interest in whether these generic sequence processing devices are inducing genuine linguistic knowledge. Nearly all current analytical studies, however, initialize the RNNs with a vocabulary of known words, and feed them tokenized input during training. We present a multi-lingual study of the linguistic knowledge encoded in RNNs trained as character-level language models, on input data with word boundaries removed. These networks face a tougher and more cognitively realistic task, having to discover any useful linguistic unit from scratch based on input statistics. The results show that our “near tabula rasa” RNNs are mostly able to solve morphological, syntactic and semantic tasks that intuitively presuppose word-level knowledge, and indeed they learned, to some extent, to track word boundaries. Our study opens the door to speculations about the necessity of an explicit, rigid word lexicon in language learning and usage.

scholarly journals Learning, visualizing and exploring 16S rRNA structure using an attention-based deep neural network

2021 ◽  
Vol 17 (9) ◽  
pp. e1009345
Author(s):  
Zhengqiao Zhao ◽  
Stephen Woloszynek ◽  
Felix Agbavor ◽  
Joshua Chang Mell ◽  
Bahrad A. Sokhansanj ◽  
...  
Keyword(s):  
Neural Networks ◽  
Deep Learning ◽  
Microbial Community ◽  
Language Processing ◽  
Recurrent Neural Networks ◽  
Network Architecture ◽  
Sequence Data ◽  
Community Sample ◽  
Marker Genes ◽  
Link Type

Recurrent neural networks with memory and attention mechanisms are widely used in natural language processing because they can capture short and long term sequential information for diverse tasks. We propose an integrated deep learning model for microbial DNA sequence data, which exploits convolutional neural networks, recurrent neural networks, and attention mechanisms to predict taxonomic classifications and sample-associated attributes, such as the relationship between the microbiome and host phenotype, on the read/sequence level. In this paper, we develop this novel deep learning approach and evaluate its application to amplicon sequences. We apply our approach to short DNA reads and full sequences of 16S ribosomal RNA (rRNA) marker genes, which identify the heterogeneity of a microbial community sample. We demonstrate that our implementation of a novel attention-based deep network architecture, Read2Pheno, achieves read-level phenotypic prediction. Training Read2Pheno models will encode sequences (reads) into dense, meaningful representations: learned embedded vectors output from the intermediate layer of the network model, which can provide biological insight when visualized. The attention layer of Read2Pheno models can also automatically identify nucleotide regions in reads/sequences which are particularly informative for classification. As such, this novel approach can avoid pre/post-processing and manual interpretation required with conventional approaches to microbiome sequence classification. We further show, as proof-of-concept, that aggregating read-level information can robustly predict microbial community properties, host phenotype, and taxonomic classification, with performance at least comparable to conventional approaches. An implementation of the attention-based deep learning network is available at https://github.com/EESI/sequence_attention (a python package) and https://github.com/EESI/seq2att (a command line tool).

scholarly journals Relation extraction for coal mine safety information using recurrent neural networks with bidirectional minimal gated unit

2021 ◽  
Vol 2021 (1) ◽  
Author(s):  
Xiulei Liu ◽  
Shoulu Hou ◽  
Zhihui Qin ◽  
Sihan Liu ◽  
Jian Zhang
Keyword(s):  
Neural Networks ◽  
Coal Mine ◽  
Recurrent Neural Networks ◽  
Practical Importance ◽  
Safety Information ◽  
Relation Extraction ◽  
Mine Safety ◽  
Linguistic Knowledge ◽  
Coal Mine Safety ◽  
Time Accuracy

AbstractThe data of coal mine safety field are massive, multi-source and heterogeneous. It is of practical importance to extract information from big data to achieve disaster precaution and emergency response. Existing approaches need to build more features and rely heavily on the linguistic knowledge of researchers, leading to inefficiency, poor portability, and slow update speed. This paper proposes a new relation extraction approach using recurrent neural networks with bidirectional minimal gated unit (MGU) model. This is achieved by adding a back-to-front MGU layer based on original MGU model. It does not require to construct complex text features and can capture the global context information by combining the forward and backward features. Evident from extensive experiments, the proposed approach outperforms the existing initiatives in terms of training time, accuracy, recall and F value.

scholarly journals Learning molecular dynamics with simple language model built upon long short-term memory neural network

2020 ◽  
Vol 11 (1) ◽  
Author(s):  
Sun-Ting Tsai ◽  
En-Jui Kuo ◽  
Pratyush Tiwary
Keyword(s):  
Neural Networks ◽  
Language Processing ◽  
Recurrent Neural Networks ◽  
Short Term Memory ◽  
Language Model ◽  
Short Term ◽  
Term Memory ◽  
Molecular Systems ◽  
Simple Language ◽  
Long Short Term Memory

Abstract Recurrent neural networks have led to breakthroughs in natural language processing and speech recognition. Here we show that recurrent networks, specifically long short-term memory networks can also capture the temporal evolution of chemical/biophysical trajectories. Our character-level language model learns a probabilistic model of 1-dimensional stochastic trajectories generated from higher-dimensional dynamics. The model captures Boltzmann statistics and also reproduces kinetics across a spectrum of timescales. We demonstrate how training the long short-term memory network is equivalent to learning a path entropy, and that its embedding layer, instead of representing contextual meaning of characters, here exhibits a nontrivial connectivity between different metastable states in the underlying physical system. We demonstrate our model’s reliability through different benchmark systems and a force spectroscopy trajectory for multi-state riboswitch. We anticipate that our work represents a stepping stone in the understanding and use of recurrent neural networks for understanding the dynamics of complex stochastic molecular systems.

scholarly journals Sentence embeddings in NLI with iterative refinement encoders

2019 ◽  
Vol 25 (4) ◽  
pp. 467-482 ◽  
Author(s):  
Aarne Talman ◽  
Anssi Yli-Jyrä ◽  
Jörg Tiedemann
Keyword(s):  
Neural Networks ◽  
Natural Language Processing ◽  
Natural Language ◽  
Language Processing ◽  
Recurrent Neural Networks ◽  
State Of The Art ◽  
Iterative Refinement ◽  
Learning Tasks ◽  
Sentence Level ◽  
Refinement Strategy

AbstractSentence-level representations are necessary for various natural language processing tasks. Recurrent neural networks have proven to be very effective in learning distributed representations and can be trained efficiently on natural language inference tasks. We build on top of one such model and propose a hierarchy of bidirectional LSTM and max pooling layers that implements an iterative refinement strategy and yields state of the art results on the SciTail dataset as well as strong results for Stanford Natural Language Inference and Multi-Genre Natural Language Inference. We can show that the sentence embeddings learned in this way can be utilized in a wide variety of transfer learning tasks, outperforming InferSent on 7 out of 10 and SkipThought on 8 out of 9 SentEval sentence embedding evaluation tasks. Furthermore, our model beats the InferSent model in 8 out of 10 recently published SentEval probing tasks designed to evaluate sentence embeddings’ ability to capture some of the important linguistic properties of sentences.

scholarly journals Recurrent Kernel Machines: Computing with Infinite Echo State Networks

2012 ◽  
Vol 24 (1) ◽  
pp. 104-133 ◽  
Author(s):  
Michiel Hermans ◽  
Benjamin Schrauwen
Keyword(s):  
Neural Networks ◽  
Recurrent Neural Networks ◽  
Input Data ◽  
Dimensional Space ◽  
Theoretical Research ◽  
High Dimensional ◽  
Support Vector ◽  
Practical Applications ◽  
Echo State Networks ◽  
Vector Machines

Echo state networks (ESNs) are large, random recurrent neural networks with a single trained linear readout layer. Despite the untrained nature of the recurrent weights, they are capable of performing universal computations on temporal input data, which makes them interesting for both theoretical research and practical applications. The key to their success lies in the fact that the network computes a broad set of nonlinear, spatiotemporal mappings of the input data, on which linear regression or classification can easily be performed. One could consider the reservoir as a spatiotemporal kernel, in which the mapping to a high-dimensional space is computed explicitly. In this letter, we build on this idea and extend the concept of ESNs to infinite-sized recurrent neural networks, which can be considered recursive kernels that subsequently can be used to create recursive support vector machines. We present the theoretical framework, provide several practical examples of recursive kernels, and apply them to typical temporal tasks.

scholarly journals Natural language processing for cognitive therapy: Extracting schemas from thought records

PLoS ONE ◽  
2021 ◽  
Vol 16 (10) ◽  
pp. e0257832
Author(s):  
Franziska Burger ◽  
Mark A. Neerincx ◽  
Willem-Paul Brinkman
Keyword(s):  
Mental Health ◽  
Neural Networks ◽  
Natural Language Processing ◽  
Natural Language ◽  
Language Processing ◽  
Recurrent Neural Networks ◽  
Support Vector ◽  
Free Text ◽  
Cognitive Approach ◽  
Text Input

The cognitive approach to psychotherapy aims to change patients’ maladaptive schemas, that is, overly negative views on themselves, the world, or the future. To obtain awareness of these views, they record their thought processes in situations that caused pathogenic emotional responses. The schemas underlying such thought records have, thus far, been largely manually identified. Using recent advances in natural language processing, we take this one step further by automatically extracting schemas from thought records. To this end, we asked 320 healthy participants on Amazon Mechanical Turk to each complete five thought records consisting of several utterances reflecting cognitive processes. Agreement between two raters on manually scoring the utterances with respect to how much they reflect each schema was substantial (Cohen’s κ = 0.79). Natural language processing software pretrained on all English Wikipedia articles from 2014 (GLoVE embeddings) was used to represent words and utterances, which were then mapped to schemas using k-nearest neighbors algorithms, support vector machines, and recurrent neural networks. For the more frequently occurring schemas, all algorithms were able to leverage linguistic patterns. For example, the scores assigned to the Competence schema by the algorithms correlated with the manually assigned scores with Spearman correlations ranging between 0.64 and 0.76. For six of the nine schemas, a set of recurrent neural networks trained separately for each of the schemas outperformed the other algorithms. We present our results here as a benchmark solution, since we conducted this research to explore the possibility of automatically processing qualitative mental health data and did not aim to achieve optimal performance with any of the explored models. The dataset of 1600 thought records comprising 5747 utterances is published together with this article for researchers and machine learning enthusiasts to improve upon our outcomes. Based on our promising results, we see further opportunities for using free-text input and subsequent natural language processing in other common therapeutic tools, such as ecological momentary assessments, automated case conceptualizations, and, more generally, as an alternative to mental health scales.

scholarly journals Fast and efficient: Statistical learning predicts online reading times in a second language

2020 ◽  
Author(s):  
Luca Onnis ◽  
Hongoak Yun
Keyword(s):  
Second Language ◽  
Language Learning ◽  
Statistical Learning ◽  
Language Processing ◽  
Common Mechanism ◽  
Online Reading ◽  
Learning Skills ◽  
Sequential Structure ◽  
Word Level ◽  
Statistical Regularities

Two long-standing and interrelated questions have been central to language and cognition. The first question is whether shared or separate mechanisms subserve language learning versus language processing. The second question is whether shared or separate mechanisms underlie processing in a first versus second language. Using an individual differences paradigm, we sought evidence for common mechanisms by probing implicit statistical learning (SL) skills and online reading in a second language. We found that individuals with better statistical learning skills more efficiently incorporate word-level statistical regularities while reading in their second language, making them more efficient L2 readers. In addition, sensitivity to forward and backward lexical predictability reduced reading times. Thus, sensitivity to statistical sequential structure may be a common mechanism of implicit learning as well as processing of language.

Sign in / Sign up

Export Citation Format

Share Document