Combining State Space Reconstruction and Forecasting by Neural Networks

2000 ◽  
pp. 259-267
Author(s):  
Hans Georg Zimmermann ◽  
Ralph Neuneier
Keyword(s):  
Neural Networks ◽  
State Space ◽  
State Space Reconstruction

scholarly journals State-Space Representations of Deep Neural Networks

2019 ◽  
Vol 31 (3) ◽  
pp. 538-554
Author(s):  
Michael Hauser ◽  
Sean Gunn ◽  
Samer Saab ◽  
Asok Ray
Keyword(s):  
Neural Networks ◽  
State Space ◽  
Deep Neural Networks ◽  
The State ◽  
Network Architectures ◽  
Embedding Dimension ◽  
Dynamical Equations ◽  
Closed Form Solutions ◽  
Dense Networks ◽  
Finite Difference Equations

This letter deals with neural networks as dynamical systems governed by finite difference equations. It shows that the introduction of [Formula: see text]-many skip connections into network architectures, such as residual networks and additive dense networks, defines [Formula: see text]th order dynamical equations on the layer-wise transformations. Closed-form solutions for the state-space representations of general [Formula: see text]th order additive dense networks, where the concatenation operation is replaced by addition, as well as [Formula: see text]th order smooth networks, are found. The developed provision endows deep neural networks with an algebraic structure. Furthermore, it is shown that imposing [Formula: see text]th order smoothness on network architectures with [Formula: see text]-many nodes per layer increases the state-space dimension by a multiple of [Formula: see text], and so the effective embedding dimension of the data manifold by the neural network is [Formula: see text]-many dimensions. It follows that network architectures of these types reduce the number of parameters needed to maintain the same embedding dimension by a factor of [Formula: see text] when compared to an equivalent first-order, residual network. Numerical simulations and experiments on CIFAR10, SVHN, and MNIST have been conducted to help understand the developed theory and efficacy of the proposed concepts.

Sign in / Sign up

Export Citation Format

Share Document