scholarly journals Learning to Adaptively Scale Recurrent Neural Networks

2019 ◽  
Vol 33 ◽  
pp. 3822-3829
Author(s):  
Hao Hu ◽  
Liqiang Wang ◽  
Guo-Jun Qi
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Time Series ◽  
Recurrent Neural Network ◽  
Recurrent Neural Networks ◽  
Temporal Patterns ◽  
Multiple Sequence ◽  
Dynamical Scaling ◽  
Sequence Modeling ◽  
Multiscale Structures

Recent advancements in recurrent neural network (RNN) research have demonstrated the superiority of utilizing multiscale structures in learning temporal representations of time series. Currently, most of multiscale RNNs use fixed scales, which do not comply with the nature of dynamical temporal patterns among sequences. In this paper, we propose Adaptively Scaled Recurrent Neural Networks (ASRNN), a simple but efficient way to handle this problem. Instead of using predefined scales, ASRNNs are able to learn and adjust scales based on different temporal contexts, making them more flexible in modeling multiscale patterns. Compared with other multiscale RNNs, ASRNNs are bestowed upon dynamical scaling capabilities with much simpler structures, and are easy to be integrated with various RNN cells. The experiments on multiple sequence modeling tasks indicate ASRNNs can efficiently adapt scales based on different sequence contexts and yield better performances than baselines without dynamical scaling abilities.

scholarly journals Exploring Efficient Neural Architectures for Linguistic–Acoustic Mapping in Text-To-Speech

2019 ◽  
Vol 9 (16) ◽  
pp. 3391 ◽  
Author(s):  
Santiago Pascual ◽  
Joan Serrà ◽  
Antonio Bonafonte
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Recurrent Neural Network ◽  
Recurrent Neural Networks ◽  
Affine Transformations ◽  
Text To Speech ◽  
Recursive Structure ◽  
The One ◽  
Acoustic Mapping ◽  
Symbol Sequences

Conversion from text to speech relies on the accurate mapping from linguistic to acoustic symbol sequences, for which current practice employs recurrent statistical models such as recurrent neural networks. Despite the good performance of such models (in terms of low distortion in the generated speech), their recursive structure with intermediate affine transformations tends to make them slow to train and to sample from. In this work, we explore two different mechanisms that enhance the operational efficiency of recurrent neural networks, and study their performance–speed trade-off. The first mechanism is based on the quasi-recurrent neural network, where expensive affine transformations are removed from temporal connections and placed only on feed-forward computational directions. The second mechanism includes a module based on the transformer decoder network, designed without recurrent connections but emulating them with attention and positioning codes. Our results show that the proposed decoder networks are competitive in terms of distortion when compared to a recurrent baseline, whilst being significantly faster in terms of CPU and GPU inference time. The best performing model is the one based on the quasi-recurrent mechanism, reaching the same level of naturalness as the recurrent neural network based model with a speedup of 11.2 on CPU and 3.3 on GPU.

Adaptive Capability of Recurrent Neural Networks with Fixed Weights for Series-Parallel System Identification

2009 ◽  
Vol 21 (11) ◽  
pp. 3214-3227
Author(s):  
James Ting-Ho Lo
Keyword(s):  
Neural Network ◽  
Dynamical System ◽  
Neural Networks ◽  
System Identification ◽  
Recurrent Neural Network ◽  
Recurrent Neural Networks ◽  
Additional Input ◽  
Adaptive Capability ◽  
Environmental Process ◽  
Optimal Series

By a fundamental neural filtering theorem, a recurrent neural network with fixed weights is known to be capable of adapting to an uncertain environment. This letter reports some mathematical results on the performance of such adaptation for series-parallel identification of a dynamical system as compared with the performance of the best series-parallel identifier possible under the assumption that the precise value of the uncertain environmental process is given. In short, if an uncertain environmental process is observable (not necessarily constant) from the output of a dynamical system or constant (not necessarily observable), then a recurrent neural network exists as a series-parallel identifier of the dynamical system whose output approaches the output of an optimal series-parallel identifier using the environmental process as an additional input.

Relating the Slope of the Activation Function and the Learning Rate Within a Recurrent Neural Network

1999 ◽  
Vol 11 (5) ◽  
pp. 1069-1077 ◽  
Author(s):  
Danilo P. Mandic ◽  
Jonathon A. Chambers
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Recurrent Neural Network ◽  
Recurrent Neural Networks ◽  
Degrees Of Freedom ◽  
Learning Algorithm ◽  
Activation Function ◽  
Learning Rate ◽  
Optimization Task ◽  
Nonlinear Activation Function

A relationship between the learning rate η in the learning algorithm, and the slope β in the nonlinear activation function, for a class of recurrent neural networks (RNNs) trained by the real-time recurrent learning algorithm is provided. It is shown that an arbitrary RNN can be obtained via the referent RNN, with some deterministic rules imposed on its weights and the learning rate. Such relationships reduce the number of degrees of freedom when solving the nonlinear optimization task of finding the optimal RNN parameters.

scholarly journals Weighted Automata Extraction from Recurrent Neural Networks via Regression on State Spaces

2020 ◽  
Vol 34 (04) ◽  
pp. 5306-5314
Author(s):  
Takamasa Okudono ◽  
Masaki Waga ◽  
Taro Sekiyama ◽  
Ichiro Hasuo
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Recurrent Neural Network ◽  
Recurrent Neural Networks ◽  
Learning Algorithm ◽  
Internal State ◽  
State Spaces ◽  
Regression Methods ◽  
Weighted Automata ◽  
Equivalence Queries

We present a method to extract a weighted finite automaton (WFA) from a recurrent neural network (RNN). Our method is based on the WFA learning algorithm by Balle and Mohri, which is in turn an extension of Angluin's classic L* algorithm. Our technical novelty is in the use of regression methods for the so-called equivalence queries, thus exploiting the internal state space of an RNN to prioritize counterexample candidates. This way we achieve a quantitative/weighted extension of the recent work by Weiss, Goldberg and Yahav that extracts DFAs. We experimentally evaluate the accuracy, expressivity and efficiency of the extracted WFAs.

scholarly journals On Recurrent Neural Network Based Theorem Prover For First Order Minimal Logic

2021 ◽  
Vol 27 (11) ◽  
pp. 1193-1202
Author(s):  
Ashot Baghdasaryan ◽  
Hovhannes Bolibekyan
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Recurrent Neural Network ◽  
Theorem Proving ◽  
Recurrent Neural Networks ◽  
Selection Problem ◽  
Theorem Prover ◽  
Minimal Logic ◽  
Free System ◽  
First Order

There are three main problems for theorem proving with a standard cut-free system for the first order minimal logic. The first problem is the possibility of looping. Secondly, it might generate proofs which are permutations of each other. Finally, during the proof some choice should be made to decide which rules to apply and where to use them. New systems with history mechanisms were introduced for solving the looping problems of automated theorem provers in the first order minimal logic. In order to solve the rule selection problem, recurrent neural networks are deployed and they are used to determine which formula from the context should be used on further steps. As a result, it yields to the reduction of time during theorem proving.

scholarly journals Comparing classic time series models and the LSTM recurrent neural network: An application to S&P 500 stocks

2020 ◽  
Vol 6 (2) ◽  
pp. 137-148
Author(s):  
J. Oliver Muncharaz
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Time Series ◽  
Stock Market ◽  
Recurrent Neural Network ◽  
Stock Prices ◽  
Arima Model ◽  
Time Series Models ◽  
Prediction Errors ◽  
The Creation

In the financial literature, there is great interest in the prediction of stock prices. Stock prediction is necessary for the creation of different investment strategies, both speculative and hedging ones. The application of neural networks has involved a change in the creation of predictive models. In this paper, we analyze the capacity of recurrent neural networks, in particular the long short-term recurrent neural network (LSTM) as opposed to classic time series models such as the Exponential Smooth Time Series (ETS) and the Arima model (ARIMA). These models have been estimated for 284 stocks from the S&P 500 stock market index, comparing the MAE obtained from their predictions. The results obtained confirm a significant reduction in prediction errors when LSTM is applied. These results are consistent with other similar studies applied to stocks included in other stock market indices, as well as other financial assets such as exchange rates.

Evaluation of Two Recurrent Neural Network Methods for Prediction of Irrigation Rate and Timing

2020 ◽  
Vol 63 (5) ◽  
pp. 1327-1348
Author(s):  
Andrés F. Jiménez ◽  
Brenda V. Ortiz ◽  
Luca Bondesan ◽  
Guilherme Morata ◽  
Damianos Damianidis
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Recurrent Neural Network ◽  
Recurrent Neural Networks ◽  
Irrigation Scheduling ◽  
Soil Types ◽  
List Type ◽  
Soil Matric Potential ◽  
Matric Potential ◽  
Network Methods

HighlightsNARX and LSTM recurrent neural networks were evaluated for prediction of irrigation prescriptions.LSTM neural networks presented the best performance for irrigation scheduling using soil matric potential sensors.NARX neural networks had the best performance for predicting irrigation prescriptions using weather data.High performance for several time-ahead predictions using both recurrent neural networks, with R2 > 0.94.The results can be adopted as a decision-support tool in irrigation scheduling for fields with different types of soils.Abstract. The implementation of adequate irrigation strategies could be done through real-time monitoring of soil water status at several soil depths; however, this could also represent a complex nonlinear problem due to the plant-soil-weather relationships. In this study, two recurrent neural network (RNN) models were evaluated to estimate irrigation prescriptions. Data for this study were collected from an on-farm corn irrigation study conducted between 2017 and 2019 in Samson, Alabama. The study used hourly data of weather and soil matric potential (SMP) monitored at three soil depths from 13 sensor probes installed on a loamy fine sand soil and a sandy clay loam soil. Two neural network methods, i.e., a nonlinear autoregressive with exogenous (NARX) input system and long short-term memory (LSTM), were trained, validated, and tested with a maximum dataset of 20,052 records and a maximum of eight categorical attributes to estimate one-step irrigation prescriptions. The performance of both methods was evaluated by varying the model development parameters (neurons or blocks, dropout, and epochs) and determining their impact on the final model prediction. Results showed that both RNN models demonstrated good capability in the prediction of irrigation prescriptions for the soil types studied, with a coefficient of determination (R2) > 0.94 and root mean square error (RMSE) < 1.2 mm. The results of this study indicate that after training the RNNs using the dataset collected in the field, models using only SMP sensors at three soil depths obtained the best performance, followed by models that used only data of solar radiation, temperature, and relative humidity in the prediction of irrigation prescriptions. For future applicability, the RNN models can be extended using datasets from other places for training, which would allow the adoption of a unique data-driven soil moisture model for irrigation scheduling useful in a wide range of soil types. Keywords: Corn, Irrigation scheduling, Machine learning, Modeling, Soil matric potential sensor.

Recurrent Neural Networks for Fault Diagnosis and Severity Assessment of a Screw Compressor

1999 ◽  
Vol 121 (4) ◽  
pp. 724-729 ◽  
Author(s):  
C. James Li ◽  
Yimin Fan
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Recurrent Neural Network ◽  
Recurrent Neural Networks ◽  
Experimental Validation ◽  
Feedforward Neural Networks ◽  
Previous Method ◽  
Feedforward Neural Network ◽  
Screw Compressor ◽  
Severity Assessment

This paper describes a method to diagnose the most frequent faults of a screw compressor and assess magnitude of these faults by tracking changes in compressor’s dynamics. To determine the condition of the compressor, a feedforward neural network model is first employed to identify the dynamics of the compressor. A recurrent neural network is then used to classify the model into one of the three conditions including baseline, gaterotor wear and excessive friction. Finally, another recurrent neural network estimates the magnitude of a fault from the model. The method’s ability to generalize was evaluated. Experimental validation of the method was also performed. The results show significant improvement over the previous method which used only feedforward neural networks.

Convolutional Recurrent Neural Networks for Text Classification

2021 ◽  
Vol 32 (4) ◽  
pp. 65-82
Author(s):  
Shengfei Lyu ◽  
Jiaqi Liu
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Convolutional Neural Network ◽  
Recurrent Neural Network ◽  
Text Classification ◽  
Recurrent Neural Networks ◽  
Superior Performance ◽  
Novel Approach ◽  
Hidden States ◽  
Traditional Approaches

Recurrent neural network (RNN) and convolutional neural network (CNN) are two prevailing architectures used in text classification. Traditional approaches combine the strengths of these two networks by straightly streamlining them or linking features extracted from them. In this article, a novel approach is proposed to maintain the strengths of RNN and CNN to a great extent. In the proposed approach, a bi-directional RNN encodes each word into forward and backward hidden states. Then, a neural tensor layer is used to fuse bi-directional hidden states to get word representations. Meanwhile, a convolutional neural network is utilized to learn the importance of each word for text classification. Empirical experiments are conducted on several datasets for text classification. The superior performance of the proposed approach confirms its effectiveness.

Artificial Higher Order Pipeline Recurrent Neural Networks for Financial Time Series Prediction

2009 ◽  
pp. 164-189 ◽  
Author(s):  
Panos Liatsis ◽  
Abir Hussain ◽  
Efstathios Milonidis
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Recurrent Neural Network ◽  
Recurrent Neural Networks ◽  
Foreign Currency ◽  
Time Series Prediction ◽  
Higher Order ◽  
Second Order ◽  
Pipeline Architecture ◽  
Currency Exchange

The research described in this chapter is concerned with the development of a novel artificial higher order neural networks architecture called the second-order pipeline recurrent neural network. The proposed artificial neural network consists of a linear and a nonlinear section, extracting relevant features from the input signal. The structuring unit of the proposed neural network is the second-order recurrent neural network. The architecture consists of a series of second-order recurrent neural networks, which are concatenated with each other. Simulation results in one-step ahead predictions of the foreign currency exchange rates demonstrate the superior performance of the proposed pipeline architecture as compared to other feed-forward and recurrent structures.

Sign in / Sign up

Export Citation Format

Share Document