Computing Higher Order Derivatives in Universal Learning Networks

1998 ◽  
Vol 2 (2) ◽  
pp. 47-53 ◽  
Author(s):  
Kotaro Hirasawa ◽  
◽  
Jinglu Hu ◽  
Masanao Ohbayashi ◽  
Junichi Murata
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Numerical Simulations ◽  
Higher Order ◽  
Order Derivative ◽  
Neural Network Training ◽  
Learning Networks ◽  
Network Training ◽  
Local Description ◽  
Higher Order Derivative

This paper discusses higher order derivative computing for universal learning networks that form a super set of all kinds of neural networks. Two computing algorithms, backward and forward propagation, are proposed. Using a technique called "local description" expresses the proposed algorithms very simply. Numerical simulations demonstrate the usefulness of higher order derivatives in neural network training.

Evolutionary Algorithm Training of Higher Order Neural Networks

2010 ◽  
pp. 57-85
Author(s):  
M. G. Epitropakis ◽  
V. P. Plagianakos ◽  
Michael N. Vrahatis
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Real Life ◽  
Higher Order ◽  
Neural Network Training ◽  
Activation Functions ◽  
Swarm Optimization ◽  
Network Training ◽  
Higher Order Neural Networks ◽  
Distributed Evolutionary Algorithms

This chapter aims to further explore the capabilities of the Higher Order Neural Networks class and especially the Pi-Sigma Neural Networks. The performance of Pi-Sigma Networks is evaluated through several well known neural network training benchmarks. In the experiments reported here, Distributed Evolutionary Algorithms are implemented for Pi-Sigma neural networks training. More specifically, the distributed versions of the Differential Evolution and the Particle Swarm Optimization algorithms have been employed. To this end, each processor of a distributed computing environment is assigned a subpopulation of potential solutions. The subpopulations are independently evolved in parallel and occasional migration is allowed to facilitate the cooperation between them. The novelty of the proposed approach is that it is applied to train Pi-Sigma networks using threshold activation functions, while the weights and biases were confined in a narrow band of integers (constrained in the range [-32, 32]). Thus, the trained Pi-Sigma neural networks can be represented by using only 6 bits. Such networks are better suited for hardware implementation than the real weight ones and this property is very important in real-life applications. Experimental results suggest that the proposed training process is fast, stable and reliable and the distributed trained Pi-Sigma networks exhibit good generalization capabilities.

scholarly journals Machine-learning in astronomy

2014 ◽  
Vol 10 (S306) ◽  
pp. 279-287 ◽  
Author(s):  
Michael Hobson ◽  
Philip Graff ◽  
Farhan Feroz ◽  
Anthony Lasenby
Keyword(s):  
Neural Network ◽  
Machine Learning ◽  
Neural Networks ◽  
Gamma Ray ◽  
Neural Network Training ◽  
Training Algorithm ◽  
Data Description ◽  
Astronomical Data ◽  
Machine Learning Methods ◽  
Network Training

AbstractMachine-learning methods may be used to perform many tasks required in the analysis of astronomical data, including: data description and interpretation, pattern recognition, prediction, classification, compression, inference and many more. An intuitive and well-established approach to machine learning is the use of artificial neural networks (NNs), which consist of a group of interconnected nodes, each of which processes information that it receives and then passes this product on to other nodes via weighted connections. In particular, I discuss the first public release of the generic neural network training algorithm, calledSkyNet, and demonstrate its application to astronomical problems focusing on its use in the BAMBI package for accelerated Bayesian inference in cosmology, and the identification of gamma-ray bursters. TheSkyNetand BAMBI packages, which are fully parallelised using MPI, are available athttp://www.mrao.cam.ac.uk/software/.

scholarly journals Parallelization of Neural Network Training for NLP with Hogwild!

2017 ◽  
Vol 109 (1) ◽  
pp. 29-38 ◽  
Author(s):  
Valentin Deyringer ◽  
Alexander Fraser ◽  
Helmut Schmid ◽  
Tsuyoshi Okita
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Suitable Method ◽  
Neural Network Training ◽  
Training Time ◽  
Pos Tagging ◽  
Network Training ◽  
Speed Up

Abstract Neural Networks are prevalent in todays NLP research. Despite their success for different tasks, training time is relatively long. We use Hogwild! to counteract this phenomenon and show that it is a suitable method to speed up training Neural Networks of different architectures and complexity. For POS tagging and translation we report considerable speedups of training, especially for the latter. We show that Hogwild! can be an important tool for training complex NLP architectures.

Evaluation of Parameter Settings for Training Neural Networks Using Backpropagation Algorithms

2022 ◽  
pp. 202-226
Author(s):  
Leema N. ◽  
Khanna H. Nehemiah ◽  
Elgin Christo V. R. ◽  
Kannan A.
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Activation Function ◽  
Neural Network Training ◽  
Network Parameter ◽  
Network Parameters ◽  
Network Training ◽  
Rate Minimum ◽  
Hidden Layer ◽  
Function Number

Artificial neural networks (ANN) are widely used for classification, and the training algorithm commonly used is the backpropagation (BP) algorithm. The major bottleneck faced in the backpropagation neural network training is in fixing the appropriate values for network parameters. The network parameters are initial weights, biases, activation function, number of hidden layers and the number of neurons per hidden layer, number of training epochs, learning rate, minimum error, and momentum term for the classification task. The objective of this work is to investigate the performance of 12 different BP algorithms with the impact of variations in network parameter values for the neural network training. The algorithms were evaluated with different training and testing samples taken from the three benchmark clinical datasets, namely, Pima Indian Diabetes (PID), Hepatitis, and Wisconsin Breast Cancer (WBC) dataset obtained from the University of California Irvine (UCI) machine learning repository.

scholarly journals Evaluation of Parameter Settings for Training Neural Networks Using Backpropagation Algorithms

2020 ◽  
Vol 11 (4) ◽  
pp. 62-85
Author(s):  
Leema N. ◽  
Khanna H. Nehemiah ◽  
Elgin Christo V. R. ◽  
Kannan A.
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Activation Function ◽  
Neural Network Training ◽  
Network Parameter ◽  
Network Parameters ◽  
Network Training ◽  
Hidden Layer ◽  
Function Number ◽  
The Impact

Artificial neural networks (ANN) are widely used for classification, and the training algorithm commonly used is the backpropagation (BP) algorithm. The major bottleneck faced in the backpropagation neural network training is in fixing the appropriate values for network parameters. The network parameters are initial weights, biases, activation function, number of hidden layers and the number of neurons per hidden layer, number of training epochs, learning rate, minimum error, and momentum term for the classification task. The objective of this work is to investigate the performance of 12 different BP algorithms with the impact of variations in network parameter values for the neural network training. The algorithms were evaluated with different training and testing samples taken from the three benchmark clinical datasets, namely, Pima Indian Diabetes (PID), Hepatitis, and Wisconsin Breast Cancer (WBC) dataset obtained from the University of California Irvine (UCI) machine learning repository.

Incremental Neural Network Training for Medical Diagnosis

2008 ◽  
pp. 720-731
Author(s):  
Sheng-Uei Guan ◽  
Ji Hua Ang ◽  
Kay Chen Tan ◽  
Abdullah Al Mamun
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Incremental Learning ◽  
Medical Diagnosis ◽  
Incremental Algorithm ◽  
Neural Network Training ◽  
Network Training ◽  
Novel Method ◽  
Partitioning Algorithms

This chapter proposes a novel method of incremental interference-free neural network training (IIFNNT) for medical datasets, which takes into consideration the interference each attribute has on the others. A specially designed network is used to determine if two attributes interfere with each other, after which the attributes are partitioned using some partitioning algorithms. These algorithms make sure that attributes beneficial to each other are trained in the same batch, thus sharing the same subnetwork while interfering attributes are separated to reduce interference. There are several incremental neural networks available in literature (Guan & Li, 2001; Su, Guan & Yeo, 2001). The architecture of IIFNNT employed some incremental algorithm: the ILIA1 and ILIA2 (incremental learning with respect to new incoming attributes) (Guan & Li, 2001).

Soil Moisture Retrieval Based on ASAR Data and Genetic Neural Networks

2012 ◽  
Vol 500 ◽  
pp. 198-203
Author(s):  
Chang Lin Xiao ◽  
Yan Chen ◽  
Lina Liu ◽  
Ling Tong ◽  
Ming Quan Jia
Keyword(s):  
Neural Network ◽  
Genetic Algorithm ◽  
Neural Networks ◽  
Soil Moisture ◽  
Training Data ◽  
Neural Network Training ◽  
Practical Test ◽  
Network Training

Genetic Algorithm can further optimize Neural Networks, and this optimized Algorithm has been used in many fields and made better results, but currently, it have not been used in inversion parameters. This paper used backscattering coefficients from ASAR, AIEM model to calculate data as neural network training data and through Genetic Algorithm Neural Networks to retrieve soil moisture. Finally compared with practical test and shows the validity and superiority of the Genetic Algorithm Neural Networks.

Using Artificial Bee Colony to Improve Functional Link Neural Network Training

2012 ◽  
Vol 263-266 ◽  
pp. 2102-2108 ◽  
Author(s):  
Yana Mazwin Mohmad Hassim ◽  
Rozaida Ghazali
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Artificial Neural Networks ◽  
Artificial Bee Colony ◽  
Neural Network Training ◽  
Functional Link ◽  
Bee Colony ◽  
Network Training ◽  
Artificial Neural ◽  
Functional Link Neural Network

Artificial Neural Networks have emerged as an important tool for classification and have been widely used to classify non-linearly separable pattern. The most popular artificial neural networks model is a Multilayer Perceptron (MLP) that is able to perform classification task with significant success. However due to the complexity of MLP structure and also problems such as local minima trapping, over fitting and weight interference have made neural network training difficult. Thus, the easy way to avoid these problems is by removing the hidden layers. This paper presents the ability of Functional Link Neural Network (FLNN) in overcoming the complexity structure of MLP, using it single layer architecture and proposes an Artificial Bee Colony (ABC) optimization for training the FLNN. The proposed technique is expected to provide better learning scheme for a classifier in order to get more accurate classification result.

scholarly journals S-DFP: shifted dynamic fixed point for quantized deep neural network training

2021 ◽  
Author(s):  
Yasufumi Sakai ◽  
Yutaka Tamiya
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Fixed Point ◽  
Deep Neural Networks ◽  
Data Representation ◽  
Training Methods ◽  
Neural Network Training ◽  
Training Time ◽  
Network Training ◽  
Complex Models

AbstractRecent advances in deep neural networks have achieved higher accuracy with more complex models. Nevertheless, they require much longer training time. To reduce the training time, training methods using quantized weight, activation, and gradient have been proposed. Neural network calculation by integer format improves the energy efficiency of hardware for deep learning models. Therefore, training methods for deep neural networks with fixed point format have been proposed. However, the narrow data representation range of the fixed point format degrades neural network accuracy. In this work, we propose a new fixed point format named shifted dynamic fixed point (S-DFP) to prevent accuracy degradation in quantized neural networks training. S-DFP can change the data representation range of dynamic fixed point format by adding bias to the exponent. We evaluated the effectiveness of S-DFP for quantized neural network training on the ImageNet task using ResNet-34, ResNet-50, ResNet-101 and ResNet-152. For example, the accuracy of quantized ResNet-152 is improved from 76.6% with conventional 8-bit DFP to 77.6% with 8-bit S-DFP.

Sign in / Sign up

Export Citation Format

Share Document