scholarly journals Minimizing Average of Loss Functions Using Gradient Descent and Stochastic Gradient Descent

2016 ◽  
Vol 64 (2) ◽  
pp. 141-145
Author(s):  
Md Rajib Arefin ◽  
M Asadujjaman
Keyword(s):  
Convex Functions ◽  
Numerical Experiments ◽  
Gradient Descent ◽  
Complexity Analysis ◽  
Stochastic Gradient ◽  
Loss Functions ◽  
Stochastic Gradient Descent ◽  
Smooth Convex

This paper deals with minimizing average of loss functions using Gradient Descent (GD) and Stochastic Gradient Descent (SGD). We present these two algorithms for minimizing average of a large number of smooth convex functions. We provide some discussions on their complexity analysis, also illustrate the algorithms geometrically. At the end, we compare their performance through numerical experiments. Dhaka Univ. J. Sci. 64(2): 141-145, 2016 (July)

scholarly journals Neural ODEs as the deep limit of ResNets with constant weights

2020 ◽  
pp. 1-41 ◽  
Author(s):  
Benny Avelin ◽  
Kaj Nyström
Keyword(s):  
Neural Network ◽  
Gradient Descent ◽  
Deep Neural Network ◽  
Theoretical Foundation ◽  
Stochastic Gradient ◽  
Loss Functions ◽  
Stochastic Gradient Descent ◽  
Decay Estimates ◽  
Value Loss ◽  
Fokker Planck Equations

In this paper, we prove that, in the deep limit, the stochastic gradient descent on a ResNet type deep neural network, where each layer shares the same weight matrix, converges to the stochastic gradient descent for a Neural ODE and that the corresponding value/loss functions converge. Our result gives, in the context of minimization by stochastic gradient descent, a theoretical foundation for considering Neural ODEs as the deep limit of ResNets. Our proof is based on certain decay estimates for associated Fokker–Planck equations.

Sign in / Sign up

Export Citation Format

Share Document