scholarly journals Evaluating Computational Language Models with Scaling Properties of Natural Language

2019 ◽  
Vol 45 (3) ◽  
pp. 481-513
Author(s):  
Shuntaro Takahashi ◽  
Kumiko Tanaka-Ishii
Keyword(s):  
Neural Networks ◽  
Natural Language ◽  
Recurrent Neural Networks ◽  
Long Memory ◽  
Computational Models ◽  
Language Models ◽  
Generative Adversarial Networks ◽  
Scaling Properties ◽  
Gating Mechanism ◽  
Taylor’S Law

In this article, we evaluate computational models of natural language with respect to the universal statistical behaviors of natural language. Statistical mechanical analyses have revealed that natural language text is characterized by scaling properties, which quantify the global structure in the vocabulary population and the long memory of a text. We study whether five scaling properties (given by Zipf’s law, Heaps’ law, Ebeling’s method, Taylor’s law, and long-range correlation analysis) can serve for evaluation of computational models. Specifically, we test n-gram language models, a probabilistic context-free grammar, language models based on Simon/Pitman-Yor processes, neural language models, and generative adversarial networks for text generation. Our analysis reveals that language models based on recurrent neural networks with a gating mechanism (i.e., long short-term memory; a gated recurrent unit; and quasi-recurrent neural networks) are the only computational models that can reproduce the long memory behavior of natural language. Furthermore, through comparison with recently proposed model-based evaluation methods, we find that the exponent of Taylor’s law is a good indicator of model quality.

Locally Recurrent Neural Networks and Their Applications

2010 ◽  
pp. 195-222
Author(s):  
Todor D. Ganchev
Keyword(s):  
Neural Networks ◽  
Recurrent Neural Networks ◽  
Computational Models ◽  
Machine Learning Techniques ◽  
Spatial Correlations ◽  
Temporal Correlations ◽  
Learning Techniques ◽  
Locally Recurrent ◽  
The One ◽  
And Control

In this chapter we review various computational models of locally recurrent neurons and deliberate the architecture of some archetypal locally recurrent neural networks (LRNNs) that are based on them. Generalizations of these structures are discussed as well. Furthermore, we point at a number of realworld applications of LRNNs that have been reported in past and recent publications. These applications involve classification or prediction of temporal sequences, discovering and modeling of spatial and temporal correlations, process identification and control, etc. Validation experiments reported in these developments provide evidence that locally recurrent architectures are capable of identifying and exploiting temporal and spatial correlations (i.e., the context in which events occur), which is the main reason for their advantageous performance when compared with the one of their non-recurrent counterparts or other reasonable machine learning techniques.

SQL Generation from Natural Language Using Supervised Learning and Recurrent Neural Networks

2020 ◽  
pp. 175-183
Author(s):  
Youssef Mellah ◽  
El Hassane Ettifouri ◽  
Abdelkader Rhouati ◽  
Walid Dahhane ◽  
Toumi Bouchentouf ◽  
...  
Keyword(s):  
Neural Networks ◽  
Natural Language ◽  
Supervised Learning ◽  
Recurrent Neural Networks

scholarly journals Deep learning in Emergency Medicine: Recent contributions and methodological challenges

2020 ◽  
Vol 16 (1) ◽  
Author(s):  
Francesco Faita
Keyword(s):  
Artificial Intelligence ◽  
Neural Networks ◽  
Emergency Medicine ◽  
Deep Learning ◽  
Convolutional Neural Networks ◽  
Recurrent Neural Networks ◽  
Computational Models ◽  
Clinical Usefulness ◽  
Human Knowledge ◽  
Deep Convolutional Neural Networks

In the last few years, artificial intelligence (AI) technology has grown dramatically impacting several fields of human knowledge and medicine in particular. Among other approaches, deep learning, which is a subset of AI based on specific computational models, such as deep convolutional neural networks and recurrent neural networks, has shown exceptional performance in images and signals processing. Accordingly, emergency medicine will benefit from the adoption of this technology. However, a particular attention should be devoted to the review of these papers in order to exclude overoptimistic results from clinically transferable ones. We presented a group of studies recently published on PubMed and selected by keywords ‘deep learning emergency medicine’ and ‘artificial intelligence emergency medicine’ with the aim of highlighting their methodological strengths and weaknesses, as well as their clinical usefulness.

scholarly journals Sentence embeddings in NLI with iterative refinement encoders

2019 ◽  
Vol 25 (4) ◽  
pp. 467-482 ◽  
Author(s):  
Aarne Talman ◽  
Anssi Yli-Jyrä ◽  
Jörg Tiedemann
Keyword(s):  
Neural Networks ◽  
Natural Language Processing ◽  
Natural Language ◽  
Language Processing ◽  
Recurrent Neural Networks ◽  
State Of The Art ◽  
Iterative Refinement ◽  
Learning Tasks ◽  
Sentence Level ◽  
Refinement Strategy

AbstractSentence-level representations are necessary for various natural language processing tasks. Recurrent neural networks have proven to be very effective in learning distributed representations and can be trained efficiently on natural language inference tasks. We build on top of one such model and propose a hierarchy of bidirectional LSTM and max pooling layers that implements an iterative refinement strategy and yields state of the art results on the SciTail dataset as well as strong results for Stanford Natural Language Inference and Multi-Genre Natural Language Inference. We can show that the sentence embeddings learned in this way can be utilized in a wide variety of transfer learning tasks, outperforming InferSent on 7 out of 10 and SkipThought on 8 out of 9 SentEval sentence embedding evaluation tasks. Furthermore, our model beats the InferSent model in 8 out of 10 recently published SentEval probing tasks designed to evaluate sentence embeddings’ ability to capture some of the important linguistic properties of sentences.

scholarly journals Natural language processing for cognitive therapy: Extracting schemas from thought records

PLoS ONE ◽  
2021 ◽  
Vol 16 (10) ◽  
pp. e0257832
Author(s):  
Franziska Burger ◽  
Mark A. Neerincx ◽  
Willem-Paul Brinkman
Keyword(s):  
Mental Health ◽  
Neural Networks ◽  
Natural Language Processing ◽  
Natural Language ◽  
Language Processing ◽  
Recurrent Neural Networks ◽  
Support Vector ◽  
Free Text ◽  
Cognitive Approach ◽  
Text Input

The cognitive approach to psychotherapy aims to change patients’ maladaptive schemas, that is, overly negative views on themselves, the world, or the future. To obtain awareness of these views, they record their thought processes in situations that caused pathogenic emotional responses. The schemas underlying such thought records have, thus far, been largely manually identified. Using recent advances in natural language processing, we take this one step further by automatically extracting schemas from thought records. To this end, we asked 320 healthy participants on Amazon Mechanical Turk to each complete five thought records consisting of several utterances reflecting cognitive processes. Agreement between two raters on manually scoring the utterances with respect to how much they reflect each schema was substantial (Cohen’s κ = 0.79). Natural language processing software pretrained on all English Wikipedia articles from 2014 (GLoVE embeddings) was used to represent words and utterances, which were then mapped to schemas using k-nearest neighbors algorithms, support vector machines, and recurrent neural networks. For the more frequently occurring schemas, all algorithms were able to leverage linguistic patterns. For example, the scores assigned to the Competence schema by the algorithms correlated with the manually assigned scores with Spearman correlations ranging between 0.64 and 0.76. For six of the nine schemas, a set of recurrent neural networks trained separately for each of the schemas outperformed the other algorithms. We present our results here as a benchmark solution, since we conducted this research to explore the possibility of automatically processing qualitative mental health data and did not aim to achieve optimal performance with any of the explored models. The dataset of 1600 thought records comprising 5747 utterances is published together with this article for researchers and machine learning enthusiasts to improve upon our outcomes. Based on our promising results, we see further opportunities for using free-text input and subsequent natural language processing in other common therapeutic tools, such as ecological momentary assessments, automated case conceptualizations, and, more generally, as an alternative to mental health scales.

Sign in / Sign up

Export Citation Format

Share Document