Gradient descent observer for on-line battery parameter and state coestimation

Several studies have shown that natural gradient descent for on-line learning is much more efficient than standard gradient descent. In this article, we derive natural gradients in a slightly different manner and discuss implications for batch-mode learning and pruning, linking them to existing algorithms such as Levenberg-Marquardt optimization and optimal brain surgeon. The Fisher matrix plays an important role in all these algorithms. The second half of the article discusses a layered approximation of the Fisher matrix specific to multilayered perceptrons. Using this approximation rather than the exact Fisher matrix, we arrive at much faster “natural” learning algorithms and more robust pruning procedures.

Download Full-text

On-Line Learning Theory of Soft Committee Machines with Correlated Hidden Units –Steepest Gradient Descent and Natural Gradient Descent–

Journal of the Physical Society of Japan ◽

10.1143/jpsj.72.805 ◽

2003 ◽

Vol 72 (4) ◽

pp. 805-810 ◽

Cited By ~ 18

Author(s):

Masato Inoue ◽

Hyeyoung Park ◽

Masato Okada

Keyword(s):

Learning Theory ◽

Gradient Descent ◽

Natural Gradient ◽

On Line ◽

On Line Learning

Download Full-text

Learning by on-line gradient descent

Journal of Physics A General Physics ◽

10.1088/0305-4470/28/3/018 ◽

1995 ◽

Vol 28 (3) ◽

pp. 643-656 ◽

Cited By ~ 93

Author(s):

M Biehl ◽

H Schwarze

Keyword(s):

Gradient Descent ◽

On Line

Download Full-text

An Adaptive Feedback Noise Control Approach Using Q-Parameterization

Noise Control and Acoustics ◽

10.1115/imece2002-33341 ◽

2002 ◽

Author(s):

Mark A. McEver ◽

Daniel G. Cole ◽

Robert L. Clark

Keyword(s):

Impulse Response ◽

Gradient Descent ◽

Noise Control ◽

Finite Impulse Response ◽

Open Loop ◽

Infinite Impulse Response ◽

Feedback Signal ◽

Control Approach ◽

Iir Filters ◽

On Line

An algorithm is presented which uses adaptive Q-parameterized compensators for control of sound. All stabilizing feedback compensators can be described in terms of plant coprime factors and a free parameter, Q, which can be any stable function. By generating a feedback signal containing only disturbance information, the parameterized compensator allows Q to be designed in an open-loop fashion. The problem of designing Q to yield desired noise reduction is formulated as an on-line gradient descent-based adaptation process. Coefficient update equations are derived for different forms of Q, including digital finite impulse response (FIR) and lattice infinite impulse response (IIR) filters. Simulations predict good performance for both tonal and broadband disturbances, and a duct feedback noise control experiment results in a 37 dB tonal reduction.

Download Full-text

Fast Curvature Matrix-Vector Products for Second-Order Gradient Descent

Neural Computation ◽

10.1162/08997660260028683 ◽

2002 ◽

Vol 14 (7) ◽

pp. 1723-1738 ◽

Cited By ~ 85

Author(s):

Nicol N. Schraudolph

Keyword(s):

Gradient Descent ◽

Linear Time ◽

Second Order ◽

Natural Gradient ◽

Acceleration Techniques ◽

On Line ◽

On Line Learning ◽

Matrix Vector ◽

Fresh Insight ◽

Insight Into

We propose a generic method for iteratively approximating various second-order gradient steps—-Newton, Gauss-Newton, Levenberg-Marquardt, and natural gradient—-in linear time per iteration, using special curvature matrix-vector products that can be computed in O(n). Two recent acceleration techniques for on-line learning, matrix momentum and stochastic meta-descent (SMD), implement this approach. Since both were originally derived by very different routes, this offers fresh insight into their operation, resulting in further improvements to SMD.

Download Full-text