Partial AUC Optimisation Using Recurrent Neural Networks for Music Detection with Limited Training Data

Prognostics, such as remaining useful life (RUL) prediction, is a crucial task in condition-based maintenance. A major challenge in data-driven prognostics is the difficulty of obtaining a sufficient number of samples of failure progression. However, for traditional machine learning methods and deep neural networks, enough training data is a prerequisite to train good prediction models. In this work, we proposed a transfer learning algorithm based on Bi-directional Long Short-Term Memory (BLSTM) recurrent neural networks for RUL estimation, in which the models can be first trained on different but related datasets and then fine-tuned by the target dataset. Extensive experimental results show that transfer learning can in general improve the prediction models on the dataset with a small number of samples. There is one exception that when transferring from multi-type operating conditions to single operating conditions, transfer learning led to a worse result.

Download Full-text

A Novel Method for Training an Echo State Network with Feedback-Error Learning

Advances in Artificial Intelligence ◽

10.1155/2013/891501 ◽

2013 ◽

Vol 2013 ◽

pp. 1-9

Author(s):

Rikke Amilde Løvlid

Keyword(s):

Neural Networks ◽

Recurrent Neural Networks ◽

Training Data ◽

Echo State Network ◽

Training Method ◽

Computationally Efficient ◽

Echo State Networks ◽

Feedback Error Learning ◽

Feedback Error ◽

Error Learning

Echo state networks are a relatively new type of recurrent neural networks that have shown great potentials for solving non-linear, temporal problems. The basic idea is to transform the low dimensional temporal input into a higher dimensional state, and then train the output connection weights to make the system output the target information. Because only the output weights are altered, training is typically quick and computationally efficient compared to training of other recurrent neural networks. This paper investigates using an echo state network to learn the inverse kinematics model of a robot simulator with feedback-error-learning. In this scheme teacher forcing is not perfect, and joint constraints on the simulator makes the feedback error inaccurate. A novel training method which is less influenced by the noise in the training data is proposed and compared to the traditional ESN training method.

Download Full-text

Sample-level sound synthesis with recurrent neural networks and conceptors

10.7287/peerj.preprints.27361 ◽

2018 ◽

Author(s):

Chris Kiefer

Keyword(s):

Neural Networks ◽

Recurrent Neural Networks ◽

Frequency Control ◽

Synthesis Method ◽

Sound Synthesis ◽

Boolean Logic ◽

Training Data ◽

Reservoir Computing ◽

Research Questions ◽

Granular Synthesis

Conceptors are a recent development in the field of reservoir computing; they can be used to influence the dynamics of recurrent neural networks (RNNs), enabling generation of arbitrary patterns based on training data. Conceptors allow interpolation and extrapolation between patterns, and also provide a system of boolean logic for combining patterns together. Generation and manipulation of arbitrary patterns using conceptors has significant potential as a sound synthesis method for applications in computer music and procedural audio but has yet to be explored. Two novel methods of sound synthesis based on conceptors are introduced. Conceptular Synthesis is based on granular synthesis; sets of conceptors are trained to recall varying patterns from a single RNN, then a runtime mechanism switches between them, generating short patterns which are recombined into a longer sound. Conceptillators are trainable, pitch-controlled oscillators for harmonically rich waveforms, commonly used in a variety of sound synthesis applications. Both systems can exploit conceptor pattern morphing, boolean logic and manipulation of RNN dynamics, enabling new creative sonic possibilities. Experiments reveal how RNN runtime parameters can be used for pitch-independent timestretching and for precise frequency control of cyclic waveforms. They show how these techniques can create highly malleable sound synthesis models, trainable using short sound samples. Limitations are revealed with regards to reproduction quality, and pragmatic limitations are also shown, where exponential rises in computation and memory requirements preclude the use of these models for training with longer sound samples. The techniques presented here represent an initial exploration of the sound synthesis potential of conceptors; future possibilities and research questions are outlined, including possibilities in generative sound.

Download Full-text

Learning Brain Dynamics with Coupled Low-Dimensional Nonlinear Oscillators and Deep Recurrent Networks

Neural Computation ◽

10.1162/neco_a_01401 ◽

2021 ◽

pp. 1-40

Author(s):

Germán Abrevaya ◽

Guillaume Dumas ◽

Aleksandr Y. Aravkin ◽

Peng Zheng ◽

Jean-Christophe Gagnon-Audet ◽

...

Keyword(s):

Neural Networks ◽

Dynamical Systems ◽

Brain Imaging ◽

Recurrent Neural Networks ◽

Practical Importance ◽

Parameters Estimation ◽

Training Data ◽

Autoregressive Models ◽

Unseen Data ◽

Low Dimensional

Abstract Many natural systems, especially biological ones, exhibit complex multivariate nonlinear dynamical behaviors that can be hard to capture by linear autoregressive models. On the other hand, generic nonlinear models such as deep recurrent neural networks often require large amounts of training data, not always available in domains such as brain imaging; also, they often lack interpretability. Domain knowledge about the types of dynamics typically observed in such systems, such as a certain type of dynamical systems models, could complement purely data-driven techniques by providing a good prior. In this work, we consider a class of ordinary differential equation (ODE) models known as van der Pol (VDP) oscil lators and evaluate their ability to capture a low-dimensional representation of neural activity measured by different brain imaging modalities, such as calcium imaging (CaI) and fMRI, in different living organisms: larval zebrafish, rat, and human. We develop a novel and efficient approach to the nontrivial problem of parameters estimation for a network of coupled dynamical systems from multivariate data and demonstrate that the resulting VDP models are both accurate and interpretable, as VDP's coupling matrix reveals anatomically meaningful excitatory and inhibitory interactions across different brain subsystems. VDP outperforms linear autoregressive models (VAR) in terms of both the data fit accuracy and the quality of insight provided by the coupling matrices and often tends to generalize better to unseen data when predicting future brain activity, being comparable to and sometimes better than the recurrent neural networks (LSTMs). Finally, we demonstrate that our (generative) VDP model can also serve as a data-augmentation tool leading to marked improvements in predictive accuracy of recurrent neural networks. Thus, our work contributes to both basic and applied dimensions of neuroimaging: gaining scientific insights and improving brain-based predictive models, an area of potentially high practical importance in clinical diagnosis and neurotechnology.

Download Full-text

Emotion brain-computer interface using wavelet and recurrent neural networks

International Journal of Advances in Intelligent Informatics ◽

10.26555/ijain.v6i1.432 ◽

2020 ◽

Vol 6 (1) ◽

pp. 1 ◽

Cited By ~ 1

Author(s):

Esmeralda Contessa Djamal ◽

Hamid Fadhilah ◽

Asep Najmurrokhman ◽

Arlisa Wulandari ◽

Faiza Renaldi

Keyword(s):

Neural Networks ◽

Recurrent Neural Networks ◽

Short Term Memory ◽

Brain Computer Interface ◽

Computing Time ◽

Extraction Methods ◽

Training Data ◽

Computer Interface ◽

Eeg Signal ◽

Eeg Signals

Brain-Computer Interface (BCI) has an intermediate tool that is usually obtained from EEG signal information. This paper proposed the BCI to control a robot simulator based on three emotions for five seconds by extracting a wavelet function in advance with Recurrent Neural Networks (RNN). Emotion is amongst variables of the brain that can be used to move external devices. BCI's success depends on the ability to recognize one person’s emotions by extracting their EEG signals. One method to appropriately recognize EEG signals as a moving signal is wavelet transformation. Wavelet extracted EEG signal into theta, alpha, and beta wave, and consider them as the input of the RNN technique. Connectivity between sequences is accomplished with Long Short-Term Memory (LSTM). The study also compared frequency extraction methods using Fast Fourier Transform (FFT). The results showed that by extracting EEG signals using Wavelet transformations, we could achieve a confident accuracy of 100% for the training data and 70.54% of new data. While the same RNN configuration without pre-processing provided 39% accuracy, even adding FFT would only increase it to 52%. Furthermore, by using features of the frequency filter, we can increase its accuracy from 70.54% to 79.3%. These results showed the importance of selecting features because of RNNs concern to sequenced its inputs. The use of emotional variables is still relevant for instructions on BCI-based external devices, which provide an average computing time of merely 0.235 seconds.

Download Full-text

Physical Exercise Recommendation and Success Prediction Using Interconnected Recurrent Neural Networks (Preprint)

10.2196/preprints.27538 ◽

2021 ◽

Author(s):

Arash Mahyari ◽

Peter Pirolli

Keyword(s):

Neural Networks ◽

Recurrent Neural Networks ◽

Computational Models ◽

Recommendation System ◽

Developed Countries ◽

Training Data ◽

Published Data ◽

Large Sample Size ◽

Learning Approaches ◽

Success Rates

BACKGROUND Unhealthy behaviors, e.g., physical inactivity and unhealthful food choice, are the primary healthcare cost drivers in developed countries. Pervasive computational, sensing, and communication technology provided by smartphones and smartwatches have made it possible to support individuals in their everyday lives to develop healthier lifestyles. OBJECTIVE This paper proposes an exercise recommendation system to recommend daily exercises to elderly population. METHODS The system, consisting of two inter-connected recurrent neural networks (RNNs), uses the history of workouts to recommend the next workout activity for each individual. The system then predicts the probability of successful completion of the predicted activity by the individual. RESULTS The prediction accuracy of this interconnected-RNN model is assessed on previously published data from a four-week mobile health experiment and is shown to improve upon previous predictions from a computational cognitive model. The proposed system is able to predict the next exercise for each individual with 80% accuracy. CONCLUSIONS The dual-RNN system for recommending workout exercises along with predicting individual success rates achieves high accuracy for individuals from whom we do not have any training data. The proposed system was validated this achievement by training the proposed model on a set of users and testing on a new set of test users. Future studies will involve combinations of explanatory computational models such as ACT-R and machine learning approaches such as the dual-RNN system to address the shortcomings of existing recommendations systems in need of large sample size.

Download Full-text

Sample-level sound synthesis with recurrent neural networks and conceptors

PeerJ Computer Science ◽

10.7717/peerj-cs.205 ◽

2019 ◽

Vol 5 ◽

pp. e205 ◽

Cited By ~ 1

Author(s):

Chris Kiefer

Keyword(s):

Neural Networks ◽

Recurrent Neural Networks ◽

Synthesis Method ◽

Network Models ◽

Sound Synthesis ◽

Boolean Logic ◽

Training Data ◽

Sound Design ◽

Echo State Networks ◽

Granular Synthesis

Conceptors are a recent development in the field of reservoir computing; they can be used to influence the dynamics of recurrent neural networks (RNNs), enabling generation of arbitrary patterns based on training data. Conceptors allow interpolation and extrapolation between patterns, and also provide a system of boolean logic for combining patterns together. Generation and manipulation of arbitrary patterns using conceptors has significant potential as a sound synthesis method for applications in computer music but has yet to be explored. Conceptors are untested with the generation of multi-timbre audio patterns, and little testing has been done on scalability to longer patterns required for audio. A novel method of sound synthesis based on conceptors is introduced. Conceptular Synthesis is based on granular synthesis; sets of conceptors are trained to recall varying patterns from a single RNN, then a runtime mechanism switches between them, generating short patterns which are recombined into a longer sound. The quality of sound resynthesis using this technique is experimentally evaluated. Conceptor models are shown to resynthesise audio with a comparable quality to a close equivalent technique using echo state networks with stored patterns and output feedback. Conceptor models are also shown to excel in their malleability and potential for creative sound manipulation, in comparison to echo state network models which tend to fail when the same manipulations are applied. Examples are given demonstrating creative sonic possibilities, by exploiting conceptor pattern morphing, boolean conceptor logic and manipulation of RNN dynamics. Limitations of conceptor models are revealed with regards to reproduction quality, and pragmatic limitations are also shown, where rises in computation and memory requirements preclude the use of these models for training with longer sound samples. The techniques presented here represent an initial exploration of the sound synthesis potential of conceptors, demonstrating possible creative applications in sound design; future possibilities and research questions are outlined.

Download Full-text

Sample-level sound synthesis with recurrent neural networks and conceptors

10.7287/peerj.preprints.27361v1 ◽

2018 ◽

Author(s):

Chris Kiefer

Keyword(s):

Neural Networks ◽

Recurrent Neural Networks ◽

Frequency Control ◽

Synthesis Method ◽

Sound Synthesis ◽

Boolean Logic ◽

Training Data ◽

Reservoir Computing ◽

Research Questions ◽

Granular Synthesis

Conceptors are a recent development in the field of reservoir computing; they can be used to influence the dynamics of recurrent neural networks (RNNs), enabling generation of arbitrary patterns based on training data. Conceptors allow interpolation and extrapolation between patterns, and also provide a system of boolean logic for combining patterns together. Generation and manipulation of arbitrary patterns using conceptors has significant potential as a sound synthesis method for applications in computer music and procedural audio but has yet to be explored. Two novel methods of sound synthesis based on conceptors are introduced. Conceptular Synthesis is based on granular synthesis; sets of conceptors are trained to recall varying patterns from a single RNN, then a runtime mechanism switches between them, generating short patterns which are recombined into a longer sound. Conceptillators are trainable, pitch-controlled oscillators for harmonically rich waveforms, commonly used in a variety of sound synthesis applications. Both systems can exploit conceptor pattern morphing, boolean logic and manipulation of RNN dynamics, enabling new creative sonic possibilities. Experiments reveal how RNN runtime parameters can be used for pitch-independent timestretching and for precise frequency control of cyclic waveforms. They show how these techniques can create highly malleable sound synthesis models, trainable using short sound samples. Limitations are revealed with regards to reproduction quality, and pragmatic limitations are also shown, where exponential rises in computation and memory requirements preclude the use of these models for training with longer sound samples. The techniques presented here represent an initial exploration of the sound synthesis potential of conceptors; future possibilities and research questions are outlined, including possibilities in generative sound.

Download Full-text

3Cnet: Pathogenicity prediction of human variants using knowledge transfer with deep recurrent neural networks

10.1101/2020.09.27.302927 ◽

2020 ◽

Author(s):

Dhong-gun Won ◽

Kyoungyeul Lee

Keyword(s):

Neural Networks ◽

Knowledge Transfer ◽

Recurrent Neural Networks ◽

Performance Comparison ◽

Training Data ◽

Pathogenic Variants ◽

Pathogenicity Prediction ◽

Amount Of Knowledge ◽

The Cost ◽

Next Generation Sequencing Ngs

AbstractThanks to the improvement of Next Generation Sequencing (NGS), genome-based diagnosis for rare disease patients become possible. However, accurate interpretation of human variants requires massive amount of knowledge gathered from previous researches and clinical cases. Also, manual analysis for each variant in the genome of patients takes enormous time and effort of clinical experts and medical doctors. Therefore, to reduce the cost of diagnosis, various computational tools have been developed for the pathogenicity prediction of human variants. Nevertheless, there has been the circularity problem of conventional tools, which leads to the overlap of training data and eventually causes overfitting of algorithms. In this research, we developed a pathogenicity predictor, named as 3Cnet, using deep recurrent neural networks which analyzes the amino-acid context of a missense mutation. 3Cnet utilizes knowledge transfer of evolutionary conservation to train insufficient clinical data without overfitting. The performance comparison clearly shows that 3Cnet can find the true disease-causing variant from a large number of missense variants in the genome of a patient with higher sensitivity (recall = 13.9 %) compared to other prediction tools such as REVEL (recall = 7.5 %) or PrimateAI (recall = 6.4 %). Consequently, 3Cnet can improve the diagnostic rate for patients and discover novel pathogenic variants with high probability.

Download Full-text

Modeling Word Learning and Processing with Recurrent Neural Networks

Information ◽

10.3390/info11060320 ◽

2020 ◽

Vol 11 (6) ◽

pp. 320

Author(s):

Claudia Marzi

Keyword(s):

Neural Networks ◽

Recurrent Neural Networks ◽

Short Term Memory ◽

Morphological Structure ◽

Training Data ◽

Self Organizing Map ◽

Verb Forms ◽

Long Short Term Memory ◽

The One ◽

Self Organizing

The paper focuses on what two different types of Recurrent Neural Networks, namely a recurrent Long Short-Term Memory and a recurrent variant of self-organizing memories, a Temporal Self-Organizing Map, can tell us about speakers’ learning and processing a set of fully inflected verb forms selected from the top-frequency paradigms of Italian and German. Both architectures, due to the re-entrant layer of temporal connectivity, can develop a strong sensitivity to sequential patterns that are highly attested in the training data. The main goal is to evaluate learning and processing dynamics of verb inflection data in the two neural networks by focusing on the effects of morphological structure on word production and word recognition, as well as on word generalization for untrained verb forms. For both models, results show that production and recognition, as well as generalization, are facilitated for verb forms in regular paradigms. However, the two models are differently influenced by structural effects, with the Temporal Self-Organizing Map more prone to adaptively find a balance between processing issues of learnability and generalization, on the one side, and discriminability on the other side.

Download Full-text