scholarly journals Sample Complexity Bounds for RNNs with Application to Combinatorial Graph Problems (Student Abstract)

2020 ◽  
Vol 34 (10) ◽  
pp. 13745-13746
Author(s):  
Nil-Jana Akpinar ◽  
Bernhard Kratzwald ◽  
Stefan Feuerriegel
Keyword(s):  
Neural Networks ◽  
Recurrent Neural Networks ◽  
Theoretical Foundation ◽  
Single Layer ◽  
Sample Complexity ◽  
Complexity Bounds ◽  
Hard Edge ◽  
Graph Problems ◽  
Edge Clique Cover ◽  
Primary Contribution

Learning to predict solutions to real-valued combinatorial graph problems promises efficient approximations. As demonstrated based on the NP-hard edge clique cover number, recurrent neural networks (RNNs) are particularly suited for this task and can even outperform state-of-the-art heuristics. However, the theoretical framework for estimating real-valued RNNs is understood only poorly. As our primary contribution, this is the first work that upper bounds the sample complexity for learning real-valued RNNs. While such derivations have been made earlier for feed-forward and convolutional neural networks, our work presents the first such attempt for recurrent neural networks. Given a single-layer RNN with a rectified linear units and input of length b, we show that a population prediction error of ε can be realized with at most Õ(a4b/ε2) samples.1 We further derive comparable results for multi-layer RNNs. Accordingly, a size-adaptive RNN fed with graphs of at most n vertices can be learned in Õ(n6/ε2), i.,e., with only a polynomial number of samples. For combinatorial graph problems, this provides a theoretical foundation that renders RNNs competitive.

scholarly journals Bispectrum and Recurrent Neural Networks: Improved Classification of Interictal and Preictal States

2019 ◽  
Vol 9 (1) ◽  
Author(s):  
Laura Gagliano ◽  
Elie Bou Assi ◽  
Dang K. Nguyen ◽  
Mohamad Sawan
Keyword(s):  
Neural Networks ◽  
Multilayer Perceptron ◽  
Recurrent Neural Networks ◽  
Short Term Memory ◽  
Focal Epilepsy ◽  
Single Layer ◽  
Epileptic Seizures ◽  
Novel Approach ◽  
Multilayer Perceptron Networks

Abstract This work proposes a novel approach for the classification of interictal and preictal brain states based on bispectrum analysis and recurrent Long Short-Term Memory (LSTM) neural networks. Two features were first extracted from bilateral intracranial electroencephalography (iEEG) recordings of dogs with naturally occurring focal epilepsy. Single-layer LSTM networks were trained to classify 5-min long feature vectors as preictal or interictal. Classification performances were compared to previous work involving multilayer perceptron networks and higher-order spectral (HOS) features on the same dataset. The proposed LSTM network proved superior to the multilayer perceptron network and achieved an average classification accuracy of 86.29% on held-out data. Results imply the possibility of forecasting epileptic seizures using recurrent neural networks, with minimal feature extraction.

scholarly journals Size-independent sample complexity of neural networks

2020 ◽  
Vol 9 (2) ◽  
pp. 473-504 ◽  
Author(s):  
Noah Golowich ◽  
Alexander Rakhlin ◽  
Ohad Shamir
Keyword(s):  
Neural Networks ◽  
Network Size ◽  
Independent Interest ◽  
Sample Complexity ◽  
Complexity Bounds ◽  
Rademacher Complexity ◽  
Parameter Matrix ◽  
Additional Assumptions

Abstract We study the sample complexity of learning neural networks by providing new bounds on their Rademacher complexity, assuming norm constraints on the parameter matrix of each layer. Compared to previous work, these complexity bounds have improved dependence on the network depth and, under some additional assumptions, are fully independent of the network size (both depth and width). These results are derived using some novel techniques, which may be of independent interest.

SAMPLE COMPLEXITY FOR FUNCTION LEARNING TASKS THROUGH LINEAR NEURAL NETWORKS

2002 ◽  
Vol 11 (04) ◽  
pp. 499-511 ◽  
Author(s):  
ARTURO HERNÁNDEZ-AGUIRRE ◽  
CRIS KOUTSOUGERAS ◽  
BILL BUCKLES
Keyword(s):  
Neural Networks ◽  
Uniform Distribution ◽  
Real Function ◽  
Sample Complexity ◽  
Function Learning ◽  
Functional Link ◽  
Distribution Free ◽  
Complexity Bounds ◽  
Learning Tasks ◽  
Radial Basis Neural Networks

We find new sample complexity bounds for real function learning tasks in the uniform distribution by means of linear neural networks. These bounds, tighter than the distribution-free ones reported elsewhere in the literature, are applicable to simple functional link networks and radial basis neural networks.

scholarly journals An Algebraic Framework to Represent Finite State Machines in Single-Layer Recurrent Neural Networks

1995 ◽  
Vol 7 (5) ◽  
pp. 931-949 ◽  
Author(s):  
R. Alquézar ◽  
A. Sanfeliu
Keyword(s):  
Neural Networks ◽  
Recurrent Neural Networks ◽  
Single Layer ◽  
Finite State Machines ◽  
Grammatical Inference ◽  
State Machines ◽  
Linear System Of Equations ◽  
Wide Range ◽  
Finite State ◽  
Algebraic Framework

In this paper we present an algebraic framework to represent finite state machines (FSMs) in single-layer recurrent neural networks (SLRNNs), which unifies and generalizes some of the previous proposals. This framework is based on the formulation of both the state transition function and the output function of an FSM as a linear system of equations, and it permits an analytical explanation of the representational capabilities of first-order and higher-order SLRNNs. The framework can be used to insert symbolic knowledge in RNNs prior to learning from examples and to keep this knowledge while training the network. This approach is valid for a wide range of activation functions, whenever some stability conditions are met. The framework has already been used in practice in a hybrid method for grammatical inference reported elsewhere (Sanfeliu and Alquézar 1994).

First-order versus second-order single-layer recurrent neural networks

1994 ◽  
Vol 5 (3) ◽  
pp. 511-513 ◽  
Author(s):  
M.W. Goudreau ◽  
C.L. Giles ◽  
S.T. Chakradhar ◽  
D. Chen
Keyword(s):  
Neural Networks ◽  
Recurrent Neural Networks ◽  
Single Layer ◽  
Second Order ◽  
First Order

scholarly journals STABILITY OF MULTI-LAYER CELLULAR NEURAL/NONLINEAR NETWORKS

2004 ◽  
Vol 14 (10) ◽  
pp. 3567-3586 ◽  
Author(s):  
LEVENTE TÖRÖK ◽  
TAMÁS ROSKA
Keyword(s):  
Neural Networks ◽  
Recurrent Neural Networks ◽  
Single Layer ◽  
Engineering Applications ◽  
One Dimensional ◽  
Nonlinear Networks ◽  
Special Regard ◽  
Stability Theorems ◽  
The One ◽  
Template Size

We have found a formalism that lets us present generalizations of several stability theorems (see Chua & Roska, 1990; Chua & Wu, 1992; Gilli, 1993; Forti, 2002] on Multi-Layer Cellular Neural/Nonlinear Networks (MLCNN) formerly claimed for Single-Layer Cellular Neural/Nonlinear Networks (CNN). The theorems were selected with special regard to usefulness in engineering applications. Hence, in contrast to many works considering stability on recurrent neural networks, the criteria of the new theorems have clear indications that are easy to verify directly on the template values. Proofs of six new theorems on 2-Layer CNNs (2LCNN) related to symmetric, τ-symmetric, nonsymmetric, τ-nonsymmetric, and sign-symmetric cases are given. Furthermore, a theorem with a proof on a MLCNN with arbitrary template size and arbitrary layer number in relation to the sign-symmetric theorem is given, along with a conjecture for the one-dimensional, two-layer, nonreciprocal case.

Sign in / Sign up

Export Citation Format

Share Document