Characterization of Degree of Approximation for Neural Networks with One Hidden Layer

2006 ◽  
Author(s):  
Fei-long Cao ◽  
Zong-ben Xu ◽  
Man-xi He
Keyword(s):  
Neural Networks ◽  
Degree Of Approximation ◽  
Hidden Layer

Simultaneous Approximations of Polynomials and Derivatives and Their Applications to Neural Networks

2008 ◽  
Vol 20 (11) ◽  
pp. 2757-2791 ◽  
Author(s):  
Yoshifusa Ito
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Uniform Norm ◽  
Degree Of Approximation ◽  
Probability Measures ◽  
Compact Sets ◽  
Network Parameters ◽  
Whole Space ◽  
Hidden Layer ◽  
Approximating Polynomials

We have constructed one-hidden-layer neural networks capable of approximating polynomials and their derivatives simultaneously. Generally, optimizing neural network parameters to be trained at later steps of the BP training is more difficult than optimizing those to be trained at the first step. Taking into account this fact, we suppressed the number of parameters of the former type. We measure degree of approximation in both the uniform norm on compact sets and the Lp-norm on the whole space with respect to probability measures.

Aspects of the numerical analysis of neural networks

Acta Numerica ◽  
1994 ◽  
Vol 3 ◽  
pp. 145-202 ◽  
Author(s):  
S.W. Ellacott
Keyword(s):  
Neural Networks ◽  
Degree Of Approximation ◽  
Compact Set ◽  
Backpropagation Algorithm ◽  
Feedforward Network ◽  
Delta Rule ◽  
Open Questions ◽  
Numerical Process ◽  
The Subject ◽  
Hidden Layer

This article starts with a brief introduction to neural networks for those unfamiliar with the basic concepts, together with a very brief overview of mathematical approaches to the subject. This is followed by a more detailed look at three areas of research which are of particular interest to numerical analysts.The first area is approximation theory. IfKis a compact set in ℝn, for somen, then it is proved that a semilinear feedforward network with one hidden layer can uniformly approximate any continuous function inC(K) to any required accuracy. A discussion of known results and open questions on the degree of approximation is included. We also consider the relevance of radial basis functions to neural networks.The second area considered is that of learning algorithms. A detailed analysis of one popular algorithm (the delta rule) will be given, indicating why one implementation leads to a stable numerical process, whereas an initially attractive variant (essentially a form of steepest descent) does not. Similar considerations apply to the backpropagation algorithm. The effect of filtering and other preprocessing of the input data will also be discussed systematically.Finally some applications of neural networks to numerical computation are considered.

scholarly journals Estimation of Approximating Rate for Neural Network inLwpSpaces

2012 ◽  
Vol 2012 ◽  
pp. 1-8
Author(s):  
Jian-Jun Wang ◽  
Chan-Yun Yang ◽  
Jia Jing
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Degree Of Approximation ◽  
Activation Functions ◽  
Feedforward Network ◽  
Topology Construction ◽  
And Topology ◽  
The Neural Network ◽  
Hidden Layer ◽  
Approximation Capability

A class of Soblove type multivariate function is approximated by feedforward network with one hidden layer of sigmoidal units and a linear output. By adopting a set of orthogonal polynomial basis and under certain assumptions for the governing activation functions of the neural network, the upper bound on the degree of approximation can be obtained for the class of Soblove functions. The results obtained are helpful in understanding the approximation capability and topology construction of the sigmoidal neural networks.

Analysis of Non-Linear Activation Functions for Classification Tasks Using Convolutional Neural Networks

2019 ◽  
Vol 12 (3) ◽  
pp. 156-161 ◽  
Author(s):  
Aman Dureja ◽  
Payal Pahwa
Keyword(s):  
Neural Networks ◽  
Deep Neural Networks ◽  
Activation Function ◽  
Primary Objective ◽  
Experimental Comparison ◽  
Activation Functions ◽  
Practical Applications ◽  
Network Activation ◽  
Non Linear ◽  
Hidden Layer

Background: In making the deep neural network, activation functions play an important role. But the choice of activation functions also affects the network in term of optimization and to retrieve the better results. Several activation functions have been introduced in machine learning for many practical applications. But which activation function should use at hidden layer of deep neural networks was not identified. Objective: The primary objective of this analysis was to describe which activation function must be used at hidden layers for deep neural networks to solve complex non-linear problems. Methods: The configuration for this comparative model was used by using the datasets of 2 classes (Cat/Dog). The number of Convolutional layer used in this network was 3 and the pooling layer was also introduced after each layer of CNN layer. The total of the dataset was divided into the two parts. The first 8000 images were mainly used for training the network and the next 2000 images were used for testing the network. Results: The experimental comparison was done by analyzing the network by taking different activation functions on each layer of CNN network. The validation error and accuracy on Cat/Dog dataset were analyzed using activation functions (ReLU, Tanh, Selu, PRelu, Elu) at number of hidden layers. Overall the Relu gave best performance with the validation loss at 25th Epoch 0.3912 and validation accuracy at 25th Epoch 0.8320. Conclusion: It is found that a CNN model with ReLU hidden layers (3 hidden layers here) gives best results and improve overall performance better in term of accuracy and speed. These advantages of ReLU in CNN at number of hidden layers are helpful to effectively and fast retrieval of images from the databases.

scholarly journals Hardware implementation of radial-basis neural networks with Gaussian activation functions on FPGA

2021 ◽  
Author(s):  
Volodymyr Shymkovych ◽  
Sergii Telenyk ◽  
Petro Kravets
Keyword(s):  
Neural Networks ◽  
Hardware Implementation ◽  
Gaussian Function ◽  
Activation Function ◽  
Rbf Neural Networks ◽  
Activation Functions ◽  
Rbf Network ◽  
Combination Scheme ◽  
Radial Basis ◽  
Hidden Layer

AbstractThis article introduces a method for realizing the Gaussian activation function of radial-basis (RBF) neural networks with their hardware implementation on field-programmable gaits area (FPGAs). The results of modeling of the Gaussian function on FPGA chips of different families have been presented. RBF neural networks of various topologies have been synthesized and investigated. The hardware component implemented by this algorithm is an RBF neural network with four neurons of the latent layer and one neuron with a sigmoid activation function on an FPGA using 16-bit numbers with a fixed point, which took 1193 logic matrix gate (LUTs—LookUpTable). Each hidden layer neuron of the RBF network is designed on an FPGA as a separate computing unit. The speed as a total delay of the combination scheme of the block RBF network was 101.579 ns. The implementation of the Gaussian activation functions of the hidden layer of the RBF network occupies 106 LUTs, and the speed of the Gaussian activation functions is 29.33 ns. The absolute error is ± 0.005. The Spartan 3 family of chips for modeling has been used to get these results. Modeling on chips of other series has been also introduced in the article. RBF neural networks of various topologies have been synthesized and investigated. Hardware implementation of RBF neural networks with such speed allows them to be used in real-time control systems for high-speed objects.

scholarly journals Exploiting heterogeneity in operational neural networks by synaptic plasticity

2021 ◽  
Author(s):  
Serkan Kiranyaz ◽  
Junaid Malik ◽  
Habib Ben Abdallah ◽  
Turker Ince ◽  
Alexandros Iosifidis ◽  
...  
Keyword(s):  
Neural Networks ◽  
Synaptic Plasticity ◽  
Network Model ◽  
Neuron Model ◽  
Linear Operators ◽  
Training Data ◽  
Learning Performance ◽  
Minimal Network ◽  
Hidden Layer ◽  
Hidden Neurons

AbstractThe recently proposed network model, Operational Neural Networks (ONNs), can generalize the conventional Convolutional Neural Networks (CNNs) that are homogenous only with a linear neuron model. As a heterogenous network model, ONNs are based on a generalized neuron model that can encapsulate any set of non-linear operators to boost diversity and to learn highly complex and multi-modal functions or spaces with minimal network complexity and training data. However, the default search method to find optimal operators in ONNs, the so-called Greedy Iterative Search (GIS) method, usually takes several training sessions to find a single operator set per layer. This is not only computationally demanding, also the network heterogeneity is limited since the same set of operators will then be used for all neurons in each layer. To address this deficiency and exploit a superior level of heterogeneity, in this study the focus is drawn on searching the best-possible operator set(s) for the hidden neurons of the network based on the “Synaptic Plasticity” paradigm that poses the essential learning theory in biological neurons. During training, each operator set in the library can be evaluated by their synaptic plasticity level, ranked from the worst to the best, and an “elite” ONN can then be configured using the top-ranked operator sets found at each hidden layer. Experimental results over highly challenging problems demonstrate that the elite ONNs even with few neurons and layers can achieve a superior learning performance than GIS-based ONNs and as a result, the performance gap over the CNNs further widens.

scholarly journals Characterization of III/V Semiconductors on Silicon by Analyzing 4D-STEM Data with Convolutional Neural Networks

2021 ◽  
Vol 27 (S1) ◽  
pp. 450-452
Author(s):  
Damien Heimes ◽  
Jonas Scheunert ◽  
Andreas Beyer ◽  
Jürgen Belz ◽  
Saleh Firoozabadi ◽  
...  
Keyword(s):  
Neural Networks ◽  
Convolutional Neural Networks

Optimization Artificial Neural Network Using Artificial Bee Colony in Letter Recognition Classification

2020 ◽  
Vol 8 (4) ◽  
pp. 469
Author(s):  
I Gusti Ngurah Alit Indrawan ◽  
I Made Widiartha
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Artificial Neural Networks ◽  
Classification Accuracy ◽  
Artificial Bee Colony Algorithm ◽  
Artificial Bee Colony ◽  
Letter Recognition ◽  
Bee Colony ◽  
Artificial Neural ◽  
Hidden Layer

Artificial Neural Networks or commonly abbreviated as ANN is one branch of science from the field of artificial intelligence which is often used to solve various problems in fields that involve grouping and pattern recognition. This research aims to classify Letter Recognition datasets using Artificial Neural Networks which are weighted optimally using the Artificial Bee Colony algorithm. The best classification accuracy results from this study were 92.85% using a combination of 4 hidden layers with each hidden layer containing 10 neurons.

Failure Modes Model of MEMS Accelerometers Based on Neural Networks

2008 ◽  
Author(s):  
Yanping Bai ◽  
Ping An ◽  
Yilong Hao
Keyword(s):  
Neural Networks ◽  
Failure Modes ◽  
Late Phase ◽  
Primary Sources ◽  
Bp Network ◽  
Bp Networks ◽  
Mems Accelerometers ◽  
Perception Model ◽  
Hidden Layer ◽  
Training Success

Fabrication of a MEMS system involves design, testing, packaging and reliability related issues. However, reliability issues that are discovered at a late phase may cause major delays in the product development going together with high costs. In this paper we study the failure modes and Mechanisms of MEMS accelerometers products and present the classification modeling of failure modes based on neural networks. In ours MEMS accelerometers, there are six failure mechanisms that have been found to be the primary sources of failure nodes. We introduce nonlinear BP network with a hidden layer and linear perception to classify for MEMS accelerometers products. Classification results show that nonlinear BP network seem to be most appropriate to approach the problem of failure modes classification than linear perception. BP neural network is capable of learning the intrinsic relations of the patterns with which they were trained. For all experiments results, the training success of rate is 100% for both methods. BP networks obtained a high forecast success of rate of over 99.5%. The linear perception model obtained a success of rate of over 95.5%. We also analyze the technology stability of MEMS products.

Sign in / Sign up

Export Citation Format

Share Document