Functionally Equivalent Feedforward Neural Networks

1994 ◽  
Vol 6 (3) ◽  
pp. 543-558 ◽  
Author(s):  
Věra Kůrková ◽  
Paul C. Kainen
Keyword(s):  
Neural Networks ◽  
Feedforward Neural Networks ◽  
Activation Function ◽  
Output Function ◽  
Input Output ◽  
General Activation ◽  
Hidden Layer

For a feedforward perceptron type architecture with a single hidden layer but with a quite general activation function, we characterize the relation between pairs of weight vectors determining networks with the same input-output function.

Analysis of Non-Linear Activation Functions for Classification Tasks Using Convolutional Neural Networks

2019 ◽  
Vol 12 (3) ◽  
pp. 156-161 ◽  
Author(s):  
Aman Dureja ◽  
Payal Pahwa
Keyword(s):  
Neural Networks ◽  
Deep Neural Networks ◽  
Activation Function ◽  
Primary Objective ◽  
Experimental Comparison ◽  
Activation Functions ◽  
Practical Applications ◽  
Network Activation ◽  
Non Linear ◽  
Hidden Layer

Background: In making the deep neural network, activation functions play an important role. But the choice of activation functions also affects the network in term of optimization and to retrieve the better results. Several activation functions have been introduced in machine learning for many practical applications. But which activation function should use at hidden layer of deep neural networks was not identified. Objective: The primary objective of this analysis was to describe which activation function must be used at hidden layers for deep neural networks to solve complex non-linear problems. Methods: The configuration for this comparative model was used by using the datasets of 2 classes (Cat/Dog). The number of Convolutional layer used in this network was 3 and the pooling layer was also introduced after each layer of CNN layer. The total of the dataset was divided into the two parts. The first 8000 images were mainly used for training the network and the next 2000 images were used for testing the network. Results: The experimental comparison was done by analyzing the network by taking different activation functions on each layer of CNN network. The validation error and accuracy on Cat/Dog dataset were analyzed using activation functions (ReLU, Tanh, Selu, PRelu, Elu) at number of hidden layers. Overall the Relu gave best performance with the validation loss at 25th Epoch 0.3912 and validation accuracy at 25th Epoch 0.8320. Conclusion: It is found that a CNN model with ReLU hidden layers (3 hidden layers here) gives best results and improve overall performance better in term of accuracy and speed. These advantages of ReLU in CNN at number of hidden layers are helpful to effectively and fast retrieval of images from the databases.

scholarly journals Hardware implementation of radial-basis neural networks with Gaussian activation functions on FPGA

2021 ◽  
Author(s):  
Volodymyr Shymkovych ◽  
Sergii Telenyk ◽  
Petro Kravets
Keyword(s):  
Neural Networks ◽  
Hardware Implementation ◽  
Gaussian Function ◽  
Activation Function ◽  
Rbf Neural Networks ◽  
Activation Functions ◽  
Rbf Network ◽  
Combination Scheme ◽  
Radial Basis ◽  
Hidden Layer

AbstractThis article introduces a method for realizing the Gaussian activation function of radial-basis (RBF) neural networks with their hardware implementation on field-programmable gaits area (FPGAs). The results of modeling of the Gaussian function on FPGA chips of different families have been presented. RBF neural networks of various topologies have been synthesized and investigated. The hardware component implemented by this algorithm is an RBF neural network with four neurons of the latent layer and one neuron with a sigmoid activation function on an FPGA using 16-bit numbers with a fixed point, which took 1193 logic matrix gate (LUTs—LookUpTable). Each hidden layer neuron of the RBF network is designed on an FPGA as a separate computing unit. The speed as a total delay of the combination scheme of the block RBF network was 101.579 ns. The implementation of the Gaussian activation functions of the hidden layer of the RBF network occupies 106 LUTs, and the speed of the Gaussian activation functions is 29.33 ns. The absolute error is ± 0.005. The Spartan 3 family of chips for modeling has been used to get these results. Modeling on chips of other series has been also introduced in the article. RBF neural networks of various topologies have been synthesized and investigated. Hardware implementation of RBF neural networks with such speed allows them to be used in real-time control systems for high-speed objects.

scholarly journals Using Deep Learning to Predict Complex Systems: A Case Study in Wind Farm Generation

Complexity ◽  
2018 ◽  
Vol 2018 ◽  
pp. 1-10 ◽  
Author(s):  
J. M. Torres ◽  
R. M. Aguilar
Keyword(s):  
Neural Networks ◽  
Deep Learning ◽  
Electricity Market ◽  
Power Plants ◽  
Wind Farm ◽  
Polynomial Regression ◽  
Supply And Demand ◽  
Feedforward Neural Networks ◽  
Activation Function ◽  
Renewable Energies

Making every component of an electrical system work in unison is being made more challenging by the increasing number of renewable energies used, the electrical output of which is difficult to determine beforehand. In Spain, the daily electricity market opens with a 12-hour lead time, where the supply and demand expected for the following 24 hours are presented. When estimating the generation, energy sources like nuclear are highly stable, while peaking power plants can be run as necessary. Renewable energies, however, which should eventually replace peakers insofar as possible, are reliant on meteorological conditions. In this paper we propose using different deep-learning techniques and architectures to solve the problem of predicting wind generation in order to participate in the daily market, by making predictions 12 and 36 hours in advance. We develop and compare various estimators based on feedforward, convolutional, and recurrent neural networks. These estimators were trained and validated with data from a wind farm located on the island of Tenerife. We show that the best candidates for each type are more precise than the reference estimator and the polynomial regression currently used at the wind farm. We also conduct a sensitivity analysis to determine which estimator type is most robust to perturbations. An analysis of our findings shows that the most accurate and robust estimators are those based on feedforward neural networks with a SELU activation function and convolutional neural networks.

Evaluation of Parameter Settings for Training Neural Networks Using Backpropagation Algorithms

2022 ◽  
pp. 202-226
Author(s):  
Leema N. ◽  
Khanna H. Nehemiah ◽  
Elgin Christo V. R. ◽  
Kannan A.
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Activation Function ◽  
Neural Network Training ◽  
Network Parameter ◽  
Network Parameters ◽  
Network Training ◽  
Rate Minimum ◽  
Hidden Layer ◽  
Function Number

Artificial neural networks (ANN) are widely used for classification, and the training algorithm commonly used is the backpropagation (BP) algorithm. The major bottleneck faced in the backpropagation neural network training is in fixing the appropriate values for network parameters. The network parameters are initial weights, biases, activation function, number of hidden layers and the number of neurons per hidden layer, number of training epochs, learning rate, minimum error, and momentum term for the classification task. The objective of this work is to investigate the performance of 12 different BP algorithms with the impact of variations in network parameter values for the neural network training. The algorithms were evaluated with different training and testing samples taken from the three benchmark clinical datasets, namely, Pima Indian Diabetes (PID), Hepatitis, and Wisconsin Breast Cancer (WBC) dataset obtained from the University of California Irvine (UCI) machine learning repository.

scholarly journals Group $L_{1/2}$ Regularization for Pruning Hidden Layer Nodes of Feedforward Neural Networks

IEEE Access ◽  
2019 ◽  
Vol 7 ◽  
pp. 9540-9557 ◽  
Author(s):  
Habtamu Zegeye Alemu ◽  
Junhong Zhao ◽  
Feng Li ◽  
Wei Wu
Keyword(s):  
Neural Networks ◽  
Feedforward Neural Networks ◽  
Hidden Layer

scholarly journals Application of Artificial Neural Networks to Assess the Mycological State of Bulk Stored Rapeseeds

Agriculture ◽  
2020 ◽  
Vol 10 (11) ◽  
pp. 567
Author(s):  
Jolanta Wawrzyniak
Keyword(s):  
Neural Networks ◽  
Artificial Neural Networks ◽  
Control Systems ◽  
Multilayer Perceptron ◽  
Activation Function ◽  
Fungal Population ◽  
Support Tool ◽  
Artificial Neural ◽  
Hidden Layer ◽  
And Storage

Artificial neural networks (ANNs) constitute a promising modeling approach that may be used in control systems for postharvest preservation and storage processes. The study investigated the ability of multilayer perceptron and radial-basis function ANNs to predict fungal population levels in bulk stored rapeseeds with various temperatures (T = 12–30 °C) and water activity in seeds (aw = 0.75–0.90). The neural network model input included aw, temperature, and time, whilst the fungal population level was the model output. During the model construction, networks with a different number of hidden layer neurons and different configurations of activation functions in neurons of the hidden and output layers were examined. The best architecture was the multilayer perceptron ANN, in which the hyperbolic tangent function acted as an activation function in the hidden layer neurons, while the linear function was the activation function in the output layer neuron. The developed structure exhibits high prediction accuracy and high generalization capability. The model provided in the research may be readily incorporated into control systems for postharvest rapeseed preservation and storage as a support tool, which based on easily measurable on-line parameters can estimate the risk of fungal development and thus mycotoxin accumulation.

On Functional Approximation with Normalized Gaussian Units

1994 ◽  
Vol 6 (2) ◽  
pp. 319-333 ◽  
Author(s):  
Michel Benaim
Keyword(s):  
Neural Networks ◽  
Asymptotic Stability ◽  
Asymptotic Properties ◽  
Learning Rule ◽  
Adaptive Algorithms ◽  
Feedforward Neural Networks ◽  
Hybrid Learning ◽  
Universal Approximation ◽  
Ode Method ◽  
Hidden Layer

Feedforward neural networks with a single hidden layer using normalized gaussian units are studied. It is proved that such neural networks are capable of universal approximation in a satisfactory sense. Then, a hybrid learning rule as per Moody and Darken that combines unsupervised learning of hidden units and supervised learning of output units is considered. By using the method of ordinary differential equations for adaptive algorithms (ODE method) it is shown that the asymptotic properties of the learning rule may be studied in terms of an autonomous cascade of dynamical systems. Some recent results from Hirsch about cascades are used to show the asymptotic stability of the learning rule.

Sign in / Sign up

Export Citation Format

Share Document