Adaptive Levenberg–Marquardt Algorithm: A New Optimization Strategy for Levenberg–Marquardt Neural Networks

Engineering data are often highly nonlinear and contain high-frequency noise, so the Levenberg–Marquardt (LM) algorithm may not converge when a neural network optimized by the algorithm is trained with engineering data. In this work, we analyzed the reasons for the LM neural network’s poor convergence commonly associated with the LM algorithm. Specifically, the effects of different activation functions such as Sigmoid, Tanh, Rectified Linear Unit (RELU) and Parametric Rectified Linear Unit (PRLU) were evaluated on the general performance of LM neural networks, and special values of LM neural network parameters were found that could make the LM algorithm converge poorly. We proposed an adaptive LM (AdaLM) algorithm to solve the problem of the LM algorithm. The algorithm coordinates the descent direction and the descent step by the iteration number, which can prevent falling into the local minimum value and avoid the influence of the parameter state of LM neural networks. We compared the AdaLM algorithm with the traditional LM algorithm and its variants in terms of accuracy and speed in the context of testing common datasets and aero-engine data, and the results verified the effectiveness of the AdaLM algorithm.

Download Full-text

Modification of Levenberg-Marquardt Algorithm for Solve Two Dimension Partial Differential Equation

Journal of University of Babylon for Pure and Applied Sciences ◽

10.29196/jubpas.v26i7.1417 ◽

2018 ◽

Vol 26 (7) ◽

pp. 107-117

Author(s):

Khalid Mindeel M. Al-Abrahemee ◽

Rana T. Shwayaa

Keyword(s):

Neural Network ◽

Differential Equation ◽

Partial Differential Equation ◽

Partial Differential Equations ◽

Partial Differential ◽

Two Dimension ◽

Marquardt Algorithm ◽

Levenberg Marquardt ◽

In The Beginning ◽

Lm Algorithm

In this paper we presented a new way based on neural network has been developed for solutione of two dimension partial differential equations . A modified neural network use to over passing the Disadvantages of LM algorithm, in the beginning we suggest signaler value decompositions of Jacobin matrix (J) and inverse of Jacobin matrix( J-1), if a matrix rectangular or singular Secondly, we suggest new calculation of μk , that ismk=|| E (w)||2 look the nonlinear execution equations E(w) = 0 has not empty solution W* and we refer to the second norm in all cases ,whereE(w): is continuously differentiable and E(x) is Lipeschitz continuous, that is=|| E(w 2)- E(w 1)||£ L|| w 2- w 1|| ,where L is Lipeschitz constant.

Download Full-text

Improving groundwater level forecasting with a feedforward neural network and linearly regressed projected precipitation

Journal of Hydroinformatics ◽

10.2166/hydro.2008.006 ◽

2008 ◽

Vol 10 (4) ◽

pp. 317-330 ◽

Cited By ~ 34

Author(s):

Ioannis K. Tsanis ◽

Paulin Coulibaly ◽

Ioannis N. Daliakopoulos

Keyword(s):

Neural Network ◽

Neural Networks ◽

Groundwater Level ◽

Feedforward Neural Network ◽

Specific Yield ◽

Deterministic Component ◽

Levenberg Marquardt ◽

Input Variables ◽

Lm Algorithm ◽

Groundwater Level Changes

A module that uses neural networks was developed for forecasting the groundwater changes in an aquifer. A modified standard Feedforward Neural Network (FNN), trained with the Levenberg–Marquardt (LM) algorithm with five input variables (precipitation, temperature, runoff, groundwater level and specific yield) with a deterministic component, is used. The deterministic component links precipitation with the seasonal recharge of the aquifer and projects the seasonal average precipitations. A new algorithm is applied to forecast the groundwater level changes in Messara Valley, Crete, Greece, where groundwater level has been steadily decreasing due to overexploitation during the last 20 years. Results from the new algorithm show that the introduction of specific yield improved the groundwater level forecasting marginally but the linearly projected precipitation component drastically increased the window of forecasting up to 30 months, equivalent to five biannual time-steps.

Download Full-text

The atmospheric model of neural networks based on the improved Levenberg-Marquardt algorithm

Open Astronomy ◽

10.1515/astro-2021-0003 ◽

2021 ◽

Vol 30 (1) ◽

pp. 24-35

Author(s):

Wenhui Cui ◽

Wei Qu ◽

Min Jiang ◽

Gang Yao

Keyword(s):

Neural Network ◽

Neural Networks ◽

Influencing Factor ◽

Atmospheric Model ◽

Average Error ◽

Atmospheric Models ◽

Neural Network Models ◽

Model Based ◽

Marquardt Algorithm ◽

Levenberg Marquardt

Abstract Traditional atmospheric models are based on the analysis and fitting of various factors influencing the space atmosphere density. Neural network models do not specifically analyze the polynomials of each influencing factor in the atmospheric model, but use large data sets for network construction. Two traditional atmospheric model algorithms are analyzed, the main factors affecting the atmospheric model are identified, and an atmospheric model based on neural networks containing various influencing factors is proposed. According to the simulation error, the Levenberg-Marquardt algorithm is used to iteratively realize the rapid network weight correction, and the optimal neural network atmospheric model is obtained. The space atmosphere is simulated and calculated with an atmospheric model based on neural networks, and its average error rate is lower than that of traditional atmospheric models such as the DTM2013 model and the MSIS00 model. At the same time, the calculation complexity of the atmospheric model based on the neural networks is significantly simplified than that of the traditional atmospheric model.

Download Full-text

The Optimal Levenberg-Marquardt Algorithm to Predict Welding Quality Using Mahalanobis Distance Theory

Advanced Materials Research ◽

10.4028/www.scientific.net/amr.936.1873 ◽

2014 ◽

Vol 936 ◽

pp. 1873-1877

Author(s):

Ill Soo Kim ◽

Qian Qian Wu ◽

Ji Hye Lee ◽

Jong Pyo Lee ◽

Min Ho Park ◽

...

Keyword(s):

Neural Network ◽

Neural Networks ◽

Mahalanobis Distance ◽

Gma Welding ◽

Welding Process ◽

Welding Parameters ◽

Welding Quality ◽

Marquardt Algorithm ◽

Levenberg Marquardt ◽

On Line

With the development of computational technology, neural network has attracted the more and more attentions to reveal the relationships between the process parameters and welding geometry. However, the Gas Metal Arc (GMA) welding is complex and of multiple interactions so that mathematical model for welding parameters has not been achieved. Neural networks have been noted as being particularly advantageous for modeling systems which contain noisy, fuzzy and uncertain elements, while a sufficient algorithm is employed. In this study, Levenberg-Marquardt algorithm was employed into GMA welding process. Mahalanobis Distance (MD) was measured to determine the on-line welding quality to avoid joint failure as welding quality. To get an optimal neural network, cases with different configurations were carried out. The Root of the Mean sum of Squared (RMS) error was adopted to evaluate the accuracy of the prediction by neural networks with LM algorithm. The results presented that the proposed algorithm had the superiority of high accuracy that can be used in the on-line welding process.

Download Full-text

Local Levenberg-Marquardt Algorithm for Learning Feedforwad Neural Networks

Journal of Artificial Intelligence and Soft Computing Research ◽

10.2478/jaiscr-2020-0020 ◽

2020 ◽

Vol 10 (4) ◽

pp. 299-316 ◽

Cited By ~ 2

Author(s):

Jarosław Bilski ◽

Bartosz Kowalczyk ◽

Alina Marchlewska ◽

Jacek M. Zurada

Keyword(s):

Neural Networks ◽

Computational Complexity ◽

Function Approximation ◽

Classification Problems ◽

Local Computation ◽

Method Performance ◽

Marquardt Algorithm ◽

Levenberg Marquardt ◽

Local Modification ◽

Lm Algorithm

AbstractThis paper presents a local modification of the Levenberg-Marquardt algorithm (LM). First, the mathematical basics of the classic LM method are shown. The classic LM algorithm is very efficient for learning small neural networks. For bigger neural networks, whose computational complexity grows significantly, it makes this method practically inefficient. In order to overcome this limitation, local modification of the LM is introduced in this paper. The main goal of this paper is to develop a more complexity efficient modification of the LM method by using a local computation. The introduced modification has been tested on the following benchmarks: the function approximation and classification problems. The obtained results have been compared to the classic LM method performance. The paper shows that the local modification of the LM method significantly improves the algorithm’s performance for bigger networks. Several possible proposals for future works are suggested.

Download Full-text

Comparison of an Artificial Neural Network and a Multiple Linear Regression in Predicting the Heat of Combustion of Diesel Fuel Based on Hydrocarbon Groups

Revista de Chimie ◽

10.37358/rc.20.6.8171 ◽

2020 ◽

Vol 71 (6) ◽

pp. 66-74

Author(s):

Younis M. Younis ◽

Salman H. Abbas ◽

Farqad T. Najim ◽

Firas Hashim Kamar ◽

Gheorghe Nechifor

Keyword(s):

Neural Network ◽

Artificial Neural Network ◽

Diesel Fuel ◽

Back Propagation ◽

Second Step ◽

Heating Value ◽

Network Training ◽

Marquardt Algorithm ◽

Levenberg Marquardt ◽

Propagation Network

A comparison between artificial neural network (ANN) and multiple linear regression (MLR) models was employed to predict the heat of combustion, and the gross and net heat values, of a diesel fuel engine, based on the chemical composition of the diesel fuel. One hundred and fifty samples of Iraqi diesel provided data from chromatographic analysis. Eight parameters were applied as inputs in order to predict the gross and net heat combustion of the diesel fuel. A trial-and-error method was used to determine the shape of the individual ANN. The results showed that the prediction accuracy of the ANN model was greater than that of the MLR model in predicting the gross heat value. The best neural network for predicting the gross heating value was a back-propagation network (8-8-1), using the Levenberg�Marquardt algorithm for the second step of network training. R = 0.98502 for the test data. In the same way, the best neural network for predicting the net heating value was a back-propagation network (8-5-1), using the Levenberg�Marquardt algorithm for the second step of network training. R = 0.95112 for the test data.

Download Full-text

Predicting Seepage of Earth Dams Using Neural Network and Genetic Algorithm

Advanced Materials Research ◽

10.4028/www.scientific.net/amr.403-408.3081 ◽

2011 ◽

Vol 403-408 ◽

pp. 3081-3085 ◽

Cited By ~ 2

Author(s):

Xin Ying Miao ◽

Jin Kui Chu ◽

Jing Qiao ◽

Ling Han Zhang

Keyword(s):

Neural Network ◽

Genetic Algorithm ◽

Field Data ◽

Back Propagation ◽

Earth Dam ◽

Earth Dams ◽

Gradient Descent Algorithm ◽

Levenberg Marquardt ◽

Lm Algorithm ◽

Good Agreement

Measurements of seepage are fundamental for earth dam surveillance. However, it is difficult to establish an effective and practical dam seepage prediction model due to the nonlinearity between seepage and its influencing factors. Genetic Algorithm for Levenberg-Marquardt(GA-LM), a new neural network(NN) model has been developed for predicting the seepage of an earth dam in China using 381 databases of field data (of which 366 in 2008 were used for training and 15 in 2009 for testing). Genetic algorithm(GA) is an ecological system algorithm, which was adopted to optimize the NN structure. Levenberg-Marquardt (LM) algorithm was originally designed to serve as an intermediate optimization algorithm between the Gauss-Newton(GN) method and the gradient descent algorithm, which was used to train NN. The predicted seepage values using GA-LM model are in good agreement with the field data. It is demonstrated here that the model is capable of predicting the seepage of earth dams accurately. The performance of GA-LM has been compared with that of conventional Back-Propagation(BP) algorithm and LM algorithm with trial-and-error approach. The comparison indicates that the GA-LM model can offer stronger and better performance than conventional NNs when used as a quick interpolation and extrapolation tool.

Download Full-text