Aspects of the numerical analysis of neural networks

Acta Numerica ◽  
1994 ◽  
Vol 3 ◽  
pp. 145-202 ◽  
Author(s):  
S.W. Ellacott
Keyword(s):  
Neural Networks ◽  
Degree Of Approximation ◽  
Compact Set ◽  
Backpropagation Algorithm ◽  
Feedforward Network ◽  
Delta Rule ◽  
Open Questions ◽  
Numerical Process ◽  
The Subject ◽  
Hidden Layer

This article starts with a brief introduction to neural networks for those unfamiliar with the basic concepts, together with a very brief overview of mathematical approaches to the subject. This is followed by a more detailed look at three areas of research which are of particular interest to numerical analysts.The first area is approximation theory. IfKis a compact set in ℝn, for somen, then it is proved that a semilinear feedforward network with one hidden layer can uniformly approximate any continuous function inC(K) to any required accuracy. A discussion of known results and open questions on the degree of approximation is included. We also consider the relevance of radial basis functions to neural networks.The second area considered is that of learning algorithms. A detailed analysis of one popular algorithm (the delta rule) will be given, indicating why one implementation leads to a stable numerical process, whereas an initially attractive variant (essentially a form of steepest descent) does not. Similar considerations apply to the backpropagation algorithm. The effect of filtering and other preprocessing of the input data will also be discussed systematically.Finally some applications of neural networks to numerical computation are considered.

scholarly journals Estimation of Approximating Rate for Neural Network inLwpSpaces

2012 ◽  
Vol 2012 ◽  
pp. 1-8
Author(s):  
Jian-Jun Wang ◽  
Chan-Yun Yang ◽  
Jia Jing
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Degree Of Approximation ◽  
Activation Functions ◽  
Feedforward Network ◽  
Topology Construction ◽  
And Topology ◽  
The Neural Network ◽  
Hidden Layer ◽  
Approximation Capability

A class of Soblove type multivariate function is approximated by feedforward network with one hidden layer of sigmoidal units and a linear output. By adopting a set of orthogonal polynomial basis and under certain assumptions for the governing activation functions of the neural network, the upper bound on the degree of approximation can be obtained for the class of Soblove functions. The results obtained are helpful in understanding the approximation capability and topology construction of the sigmoidal neural networks.

Simultaneous Approximations of Polynomials and Derivatives and Their Applications to Neural Networks

2008 ◽  
Vol 20 (11) ◽  
pp. 2757-2791 ◽  
Author(s):  
Yoshifusa Ito
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Uniform Norm ◽  
Degree Of Approximation ◽  
Probability Measures ◽  
Compact Sets ◽  
Network Parameters ◽  
Whole Space ◽  
Hidden Layer ◽  
Approximating Polynomials

We have constructed one-hidden-layer neural networks capable of approximating polynomials and their derivatives simultaneously. Generally, optimizing neural network parameters to be trained at later steps of the BP training is more difficult than optimizing those to be trained at the first step. Taking into account this fact, we suppressed the number of parameters of the former type. We measure degree of approximation in both the uniform norm on compact sets and the Lp-norm on the whole space with respect to probability measures.

A BACKPROPAGATION ALGORITHM FOR A NETWORK OF NEURONS WITH THRESHOLD CONTROLLED SYNAPSES

1991 ◽  
Vol 02 (01n02) ◽  
pp. 135-141 ◽  
Author(s):  
A. Hartstein
Keyword(s):  
Backpropagation Algorithm ◽  
Feedforward Network ◽  
Decision Space ◽  
Parity Problem ◽  
Vlsi Technology ◽  
Multiplicative Type ◽  
Hidden Layer

Neurons with threshold controlled synapses are easier to implement in VLSI technology than the more commonly studied multiplicative-type synapses. In this paper I derive a backpropagation algorithm which is suitable for networks using this type of neuron. The decision surface obtained from this type of network is composed out of elementary hyperoctahedra centered on each point in decision space. Simulations of a simple two-layer feedforward network are used to show that a network with one hidden layer can learn the logical AND, OR, and XOR functions, and in addition solve the eight-bit parity problem and the four-bit problem.

scholarly journals Neural Network Identifiability for a Family of Sigmoidal Nonlinearities

2021 ◽  
Author(s):  
Verner Vlačić ◽  
Helmut Bölcskei
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Network Architecture ◽  
Uniform Norm ◽  
Feed Forward Neural Network ◽  
The Subject ◽  
Hidden Layer ◽  
Arbitrary Precision ◽  
Necessary And Sufficient ◽  
Fully Connected

AbstractThis paper addresses the following question of neural network identifiability: Does the input–output map realized by a feed-forward neural network with respect to a given nonlinearity uniquely specify the network architecture, weights, and biases? The existing literature on the subject (Sussman in Neural Netw 5(4):589–593, 1992; Albertini et al. in Artificial neural networks for speech and vision, 1993; Fefferman in Rev Mat Iberoam 10(3):507–555, 1994) suggests that the answer should be yes, up to certain symmetries induced by the nonlinearity, and provided that the networks under consideration satisfy certain “genericity conditions.” The results in Sussman (1992) and Albertini et al. (1993) apply to networks with a single hidden layer and in Fefferman (1994) the networks need to be fully connected. In an effort to answer the identifiability question in greater generality, we derive necessary genericity conditions for the identifiability of neural networks of arbitrary depth and connectivity with an arbitrary nonlinearity. Moreover, we construct a family of nonlinearities for which these genericity conditions are minimal, i.e., both necessary and sufficient. This family is large enough to approximate many commonly encountered nonlinearities to within arbitrary precision in the uniform norm.

A Convergence Result for Learning in Recurrent Neural Networks

1994 ◽  
Vol 6 (3) ◽  
pp. 420-440 ◽  
Author(s):  
Chung-Ming Kuan ◽  
Kurt Hornik ◽  
Halbert White
Keyword(s):  
Neural Networks ◽  
Stochastic Processes ◽  
Recurrent Neural Networks ◽  
Convergence Result ◽  
Local Optimum ◽  
Recurrent Networks ◽  
Convergence Properties ◽  
Rigorous Analysis ◽  
Backpropagation Algorithm ◽  
Hidden Layer

We give a rigorous analysis of the convergence properties of a backpropagation algorithm for recurrent networks containing either output or hidden layer recurrence. The conditions permit data generated by stochastic processes with considerable dependence. Restrictions are offered that may help assure convergence of the network parameters to a local optimum, as some simulations illustrate.

A MODIFIED ERROR BACKPROPAGATION ALGORITHM FOR COMPLEX-VALUE NEURAL NETWORKS

2005 ◽  
Vol 15 (06) ◽  
pp. 435-443 ◽  
Author(s):  
XIAOMING CHEN ◽  
ZHENG TANG ◽  
CATHERINE VARIAPPAN ◽  
SONGSONG LI ◽  
TOSHIMI OKADA
Keyword(s):  
Neural Networks ◽  
Image Processing ◽  
Fourier Transformation ◽  
Error Function ◽  
Local Minima ◽  
Backpropagation Algorithm ◽  
Speed Up ◽  
Simulation Results ◽  
Hidden Layer ◽  
Complex Valued

The complex-valued backpropagation algorithm has been widely used in fields of dealing with telecommunications, speech recognition and image processing with Fourier transformation. However, the local minima problem usually occurs in the process of learning. To solve this problem and to speed up the learning process, we propose a modified error function by adding a term to the conventional error function, which is corresponding to the hidden layer error. The simulation results show that the proposed algorithm is capable of preventing the learning from sticking into the local minima and of speeding up the learning.

scholarly journals ReLU Network with Bounded Width Is a Universal Approximator in View of an Approximate Identity

2021 ◽  
Vol 11 (1) ◽  
pp. 427
Author(s):  
Sunghwan Moon
Keyword(s):  
Neural Networks ◽  
Early Stage ◽  
Expressive Power ◽  
Activation Function ◽  
Approximate Identity ◽  
Compact Set ◽  
Universal Approximator ◽  
Wide Range ◽  
Hidden Layer ◽  
Bounded Width

Deep neural networks have shown very successful performance in a wide range of tasks, but a theory of why they work so well is in the early stage. Recently, the expressive power of neural networks, important for understanding deep learning, has received considerable attention. Classic results, provided by Cybenko, Barron, etc., state that a network with a single hidden layer and suitable activation functions is a universal approximator. A few years ago, one started to study how width affects the expressiveness of neural networks, i.e., a universal approximation theorem for a deep neural network with a Rectified Linear Unit (ReLU) activation function and bounded width. Here, we show how any continuous function on a compact set of Rnin,nin∈N can be approximated by a ReLU network having hidden layers with at most nin+5 nodes in view of an approximate identity.

scholarly journals Application of artificial neural networks (ANN) in Lake Drwęckie water level modelling

2015 ◽  
Vol 15 (1) ◽  
pp. 21-30 ◽  
Author(s):  
Adam Piasecki ◽  
Jakub Jurasz ◽  
Rajmund Skowron
Keyword(s):  
Neural Networks ◽  
Artificial Neural Networks ◽  
Water Level ◽  
Meteorological Data ◽  
Water Level Fluctuations ◽  
Explanatory Variables ◽  
Real Value ◽  
The Subject ◽  
Artificial Neural ◽  
Hidden Layer

Abstract This paper presents an attempt to model water-level fluctuations in a lake based on artificial neural networks. The subject of research was the water level in Lake Drwęckie over the period 1980-2012. For modelling purposes, meteorological data from the weather station in Olsztyn were used. As a result of the research conducted, the model M_Meteo_Lag_3 was identified as the most accurate. This artificial neural network model has seven input neurons, four neurons in the hidden layer and one neuron in the output layer. As explanatory variables meteorological parameters (minimal, maximal and mean temperature, and humidity) and values of dependent variables from three earlier months were implemented. The paper claims that artificial neural networks performed well in terms of modelling the analysed phenomenon. In most cases (55%) the modelled value differed from the real value by an average of 7.25 cm. Only in two cases did a meaningful error occur, of 33 and 38 cm.

TRANSITION TO CONVOLUTIONAL NEURAL NETWORKS IN RADAR PROBLEMS

2019 ◽  
pp. 96-99
Author(s):  
K. Maystrenko ◽  
A. Budilov ◽  
D. Afanasev
Keyword(s):  
Neural Networks ◽  
Pattern Recognition ◽  
Target Detection ◽  
Convolutional Neural Networks ◽  
Network Topology ◽  
Subject Areas ◽  
The Subject ◽  
Printed Materials ◽  
Hardware Costs

Goal. Identify trends and prospects for the development of radar in terms of the use of convolutional neural networks for target detection. Materials and methods. Analysis of relevant printed materials related to the subject areas of radar and convolutional neural networks. Results. The transition to convolutional neural networks in the field of radar is considered. A review of papers on the use of convolutional neural networks in pattern recognition problems, in particular, in the radar problem, is carried out. Hardware costs for the implementation of convolutional neural networks are analyzed. Conclusion. The conclusion is made about the need to create a methodology for selecting a network topology depending on the parameters of the radar task.

Sign in / Sign up

Export Citation Format

Share Document