Dynamic sample size selection based quasi-Newton training for highly nonlinear function approximation using multilayer neural networks

2013 ◽  
Author(s):  
Hiroshi Ninomiya
Keyword(s):  
Neural Networks ◽  
Sample Size ◽  
Function Approximation ◽  
Nonlinear Function ◽  
Size Selection ◽  
Highly Nonlinear ◽  
Multilayer Neural Networks ◽  
Nonlinear Function Approximation ◽  
Quasi Newton

A MATLAB-Based Study on Approximation Performances of Improved Algorithms of Typical BP Neural Networks

2013 ◽  
Vol 313-314 ◽  
pp. 1353-1356 ◽  
Author(s):  
Shuo Ding ◽  
Qing Hui Wu
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Adaptive Learning ◽  
Function Approximation ◽  
Nonlinear Function ◽  
Bp Neural Networks ◽  
Advantages And Disadvantages ◽  
The Best Approximation ◽  
Bp Networks ◽  
Quasi Newton

BP neural networks are widely used and the algorithms are various. This paper studies the advantages and disadvantages of improved algorithms of five typical BP networks, based on artificial neural network theories. First, the learning processes of improved algorithms of the five typical BP networks are elaborated on mathematically. Then a specific network is designed on the platform of MATLAB 7.0 to conduct approximation test for a given nonlinear function. At last, a comparison is made between the training speeds and memory consumption of the five BP networks. The simulation results indicate that for small scaled and medium scaled networks, LM optimization algorithm has the best approximation ability, followed by Quasi-Newton algorithm, conjugate gradient method, resilient BP algorithm, adaptive learning rate algorithm. Keywords: BP neural network; Improved algorithm; Function approximation; MATLAB

scholarly journals Accelerating Symmetric Rank-1 Quasi-Newton Method with Nesterov’s Gradient for Training Neural Networks

2021 ◽  
Author(s):  
S. Indrapriyadarsini ◽  
Shahrzad Mahboubi ◽  
Hiroshi Ninomiya ◽  
Takeshi Kamio ◽  
Hideki Asai
Keyword(s):  
Neural Networks ◽  
Newton Method ◽  
Trust Region ◽  
Nonlinear Problems ◽  
Classification Problem ◽  
Second Order ◽  
Highly Nonlinear ◽  
Gradient Based ◽  
Quasi Newton ◽  
Second Order Methods

Gradient based methods are popularly used in training neural networks and can be broadly categorized into first and second order methods. Second order methods have shown to have better convergence compared to first order methods, especially in solving highly nonlinear problems. The BFGS quasi-Newton method is the most commonly studied second order method for neural network training. Recent methods have shown to speed up the convergence of the BFGS method using the Nesterov’s acclerated gradient and momentum terms. The SR1 quasi-Newton method though less commonly used in training neural networks, are known to have interesting properties and provide good Hessian approximations when used with a trust-region approach. Thus, this paper aims to investigate accelerating the Symmetric Rank-1 (SR1) quasi-Newton method with the Nesterov’s gradient for training neural networks and briefly discuss its convergence. The performance of the proposed method is evaluated on a function approximation and image classification problem.

scholarly journals Accelerating Symmetric Rank-1 Quasi-Newton Method with Nesterov’s Gradient for Training Neural Networks

2021 ◽  
Author(s):  
S. Indrapriyadarsini ◽  
Shahrzad Mahboubi ◽  
Hiroshi Ninomiya ◽  
Takeshi Kamio ◽  
Hideki Asai
Keyword(s):  
Neural Networks ◽  
Newton Method ◽  
Trust Region ◽  
Nonlinear Problems ◽  
Classification Problem ◽  
Second Order ◽  
Highly Nonlinear ◽  
Gradient Based ◽  
Quasi Newton ◽  
Second Order Methods

Gradient based methods are popularly used in training neural networks and can be broadly categorized into first and second order methods. Second order methods have shown to have better convergence compared to first order methods, especially in solving highly nonlinear problems. The BFGS quasi-Newton method is the most commonly studied second order method for neural network training. Recent methods have shown to speed up the convergence of the BFGS method using the Nesterov’s acclerated gradient and momentum terms. The SR1 quasi-Newton method though less commonly used in training neural networks, are known to have interesting properties and provide good Hessian approximations when used with a trust-region approach. Thus, this paper aims to investigate accelerating the Symmetric Rank-1 (SR1) quasi-Newton method with the Nesterov’s gradient for training neural networks and briefly discuss its convergence. The performance of the proposed method is evaluated on a function approximation and image classification problem.

Sign in / Sign up

Export Citation Format

Share Document