Smart Gradient - An adaptive technique for improving gradient estimation

<p style='text-indent:20px;'>Computing the gradient of a function provides fundamental information about its behavior. This information is essential for several applications and algorithms across various fields. One common application that requires gradients are optimization techniques such as stochastic gradient descent, Newton's method and trust region methods. However, these methods usually require a numerical computation of the gradient at every iteration of the method which is prone to numerical errors. We propose a simple limited-memory technique for improving the accuracy of a numerically computed gradient in this gradient-based optimization framework by exploiting (1) a coordinate transformation of the gradient and (2) the history of previously taken descent directions. The method is verified empirically by extensive experimentation on both test functions and on real data applications. The proposed method is implemented in the <inline-formula><tex-math id="M1">\begin{document}$\texttt{R} $\end{document}</tex-math></inline-formula> package <inline-formula><tex-math id="M2">\begin{document}$ \texttt{smartGrad}$\end{document}</tex-math></inline-formula> and in C<inline-formula><tex-math id="M3">\begin{document}$ \texttt{++} $\end{document}</tex-math></inline-formula>.</p>

Download Full-text

Meta-Descent for Online, Continual Prediction

Proceedings of the AAAI Conference on Artificial Intelligence ◽

10.1609/aaai.v33i01.33013943 ◽

2019 ◽

Vol 33 ◽

pp. 3943-3950

Author(s):

Andrew Jacobsen ◽

Matthew Schlegel ◽

Cameron Linke ◽

Thomas Degris ◽

Adam White ◽

...

Keyword(s):

Gradient Descent ◽

Large Scale ◽

Time Series Prediction ◽

Real Data ◽

Second Order ◽

Stochastic Gradient Descent ◽

Step Size ◽

Vector Approximation ◽

Prediction Problems ◽

Stationary Problems

This paper investigates different vector step-size adaptation approaches for non-stationary online, continual prediction problems. Vanilla stochastic gradient descent can be considerably improved by scaling the update with a vector of appropriately chosen step-sizes. Many methods, including AdaGrad, RMSProp, and AMSGrad, keep statistics about the learning process to approximate a second order update—a vector approximation of the inverse Hessian. Another family of approaches use meta-gradient descent to adapt the stepsize parameters to minimize prediction error. These metadescent strategies are promising for non-stationary problems, but have not been as extensively explored as quasi-second order methods. We first derive a general, incremental metadescent algorithm, called AdaGain, designed to be applicable to a much broader range of algorithms, including those with semi-gradient updates or even those with accelerations, such as RMSProp. We provide an empirical comparison of methods from both families. We conclude that methods from both families can perform well, but in non-stationary prediction problems the meta-descent methods exhibit advantages. Our method is particularly robust across several prediction problems, and is competitive with the state-of-the-art method on a large-scale, time-series prediction problem on real data from a mobile robot.

Download Full-text

Optimization of Intrusion Detection Systems Determined by Ameliorated HNADAM-SGD Algorithm

10.20944/preprints202112.0323.v1 ◽

2021 ◽

Author(s):

Shyla Shyla ◽

Vishal Bhatnagar ◽

Vikram Bali ◽

Shivani Bali

Keyword(s):

Intrusion Detection ◽

Gradient Descent ◽

Optimization Techniques ◽

Machine Learning Algorithms ◽

Intrusion Detection Systems ◽

Stochastic Gradient Descent ◽

Data Traffic ◽

Detection Systems ◽

Moment Estimation ◽

Classi Fication

A single Information security is of pivotal concern for consistently streaming information over the widespread internetwork. The bottleneck flow of incoming and outgoing data traffic introduces the issue of malicious activities taken place by intruders, hackers and attackers in the form of authenticity desecration, gridlocking data traffic, vandalizing data and crashing the established network. The issue of emerging suspicious activities is managed by the domain of Intrusion Detection Systems (IDS). The IDS consistently monitors the network for identifica-tion of suspicious activities and generates alarm and indication in presence of malicious threats and worms. The performance of IDS is improved by using different signature based machine learning algorithms. In this paper, the performance of IDS model is determined using hybridization of nestrov-accelerated adaptive moment estimation –stochastic gradient descent (HNADAM-SDG) algorithm. The performance of the algorithm is compared with other classi-fication algorithms as logistic regression, ridge classifier and ensemble algorithm by adapting feature selection and optimization techniques

Download Full-text

Performance of Gradient-Based Solutions versus Genetic Algorithms in the Correlation of Thermal Mathematical Models of Spacecrafts

International Journal of Aerospace Engineering ◽

10.1155/2017/7683457 ◽

2017 ◽

Vol 2017 ◽

pp. 1-12

Author(s):

Eva Anglada ◽

Laura Martinez-Jimenez ◽

Iñaki Garmendia

Keyword(s):

Genetic Algorithms ◽

Mathematical Models ◽

International Space Station ◽

Space Station ◽

Real Space ◽

Optimization Techniques ◽

International Space ◽

Thermal Test ◽

Gradient Based ◽

Pros And Cons

The correlation of the thermal mathematical models (TMMs) of spacecrafts with the results of the thermal test is a demanding task in terms of time and effort. Theoretically, it can be automatized by means of optimization techniques, although this is a challenging task. Previous studies have shown the ability of genetic algorithms to perform this task in several cases, although some limitations have been detected. In addition, gradient-based methods, although also presenting some limitations, have provided good solutions in other technical fields. For this reason, the performance of genetic algorithms and gradient-based methods in the correlation of TMMs is discussed in this paper to compare the pros and cons of them. The case of study used in the comparison is a real space instrument flown aboard the International Space Station.

Download Full-text

A new stochastic gradient descent possibilistic clustering algorithm

AI Communications ◽

10.3233/aic-210125 ◽

2021 ◽

pp. 1-18

Author(s):

Angeliki Koutsimpela ◽

Konstantinos D. Koutroumbas

Keyword(s):

Cost Function ◽

Gradient Descent ◽

Clustering Algorithm ◽

Clustering Algorithms ◽

Real Data ◽

Stochastic Gradient ◽

Stochastic Gradient Descent ◽

Data Sets ◽

Convergence Results ◽

Possibilistic Clustering

Several well known clustering algorithms have their own online counterparts, in order to deal effectively with the big data issue, as well as with the case where the data become available in a streaming fashion. However, very few of them follow the stochastic gradient descent philosophy, despite the fact that the latter enjoys certain practical advantages (such as the possibility of (a) running faster than their batch processing counterparts and (b) escaping from local minima of the associated cost function), while, in addition, strong theoretical convergence results have been established for it. In this paper a novel stochastic gradient descent possibilistic clustering algorithm, called O- PCM 2 is introduced. The algorithm is presented in detail and it is rigorously proved that the gradient of the associated cost function tends to zero in the L 2 sense, based on general convergence results established for the family of the stochastic gradient descent algorithms. Furthermore, an additional discussion is provided on the nature of the points where the algorithm may converge. Finally, the performance of the proposed algorithm is tested against other related algorithms, on the basis of both synthetic and real data sets.

Download Full-text

Efficient Structural Design Using a Numerical Optimization Technique

IABSE Congress, New York, New York 2019: The Evolving Metropolis ◽

10.2749/newyork.2019.0499 ◽

2019 ◽

Author(s):

Qian Wang ◽

Lucas Schmotzer ◽

Yongwook Kim

Keyword(s):

Structural Design ◽

Numerical Optimization ◽

Optimization Technique ◽

Optimization Method ◽

Optimization Techniques ◽

Convergence Criteria ◽

Efficient Design ◽

Concrete Building ◽

Gradient Based ◽

Structural Responses

<p>Structural designs of complex buildings and infrastructures have long been based on engineering experience and a trial-and-error approach. The structural performance is checked each time when a design is determined. An alternative strategy based on numerical optimization techniques can provide engineers an effective and efficient design approach. To achieve an optimal design, a finite element (FE) program is employed to calculate structural responses including forces and deformations. A gradient-based or gradient-free optimization method can be integrated with the FE program to guide the design iterations, until certain convergence criteria are met. Due to the iterative nature of the numerical optimization, a user programming is required to repeatedly access and modify input data and to collect output data of the FE program. In this study, an approximation method was developed so that the structural responses could be expressed as approximate functions, and that the accuracy of the functions could be adaptively improved. In the method, the FE program was not required to be directly looped in the optimization iterations. As a practical illustrative example, a 3D reinforced concrete building structure was optimized. The proposed method worked very well and optimal designs were found to reduce the torsional responses of the building.</p>

Download Full-text

Optimization of a Steam Reforming Plant Modeled with Artificial Neural Networks

Electronics ◽

10.3390/electronics9111923 ◽

2020 ◽

Vol 9 (11) ◽

pp. 1923

Author(s):

Eduardo G. Pardo ◽

Jaime Blanco-Linares ◽

David Velázquez ◽

Francisco Serradilla

Keyword(s):

Neural Networks ◽

Artificial Neural Networks ◽

Steam Reforming ◽

Real Data ◽

Memetic Algorithms ◽

Black Box ◽

Optimization Techniques ◽

Ann Model ◽

The Real ◽

Artificial Neural

The objective of this research is to improve the hydrogen production and total profit of a real Steam Reforming plant. Given the impossibility of tuning the real factory to optimize its operation, we propose modelling the plant using Artificial Neural Networks (ANNs). Particularly, we combine a set of independent ANNs into a single model. Each ANN uses different sets of inputs depending on the physical processes simulated. The model is then optimized as a black-box system using metaheuristics (Genetic and Memetic Algorithms). We demonstrate that the proposed ANN model presents a high correlation between the real output and the predicted one. Additionally, the performance of the proposed optimization techniques has been validated by the engineers of the plant, who reported a significant increase in the benefit that was obtained after optimization. Furthermore, this approach has been favorably compared with the results that were provided by a general black-box solver. All methods were tested over real data that were provided by the factory.

Download Full-text

Neural Networks for the Joint Development of Individual Payments and Claim Incurred

Risks ◽

10.3390/risks8020033 ◽

2020 ◽

Vol 8 (2) ◽

pp. 33

Author(s):

Łukasz Delong ◽

Mario V. Wüthrich

Keyword(s):

Neural Networks ◽

Regression Models ◽

Development Process ◽

Real Data ◽

Data Set ◽

Joint Development ◽

Chain Ladder Method ◽

History Of ◽

Feature Information ◽

Chain Ladder

The goal of this paper is to develop regression models and postulate distributions which can be used in practice to describe the joint development process of individual claim payments and claim incurred. We apply neural networks to estimate our regression models. As regressors we use the whole claim history of incremental payments and claim incurred, as well as any relevant feature information which is available to describe individual claims and their development characteristics. Our models are calibrated and tested on a real data set, and the results are benchmarked with the Chain-Ladder method. Our analysis focuses on the development of the so-called Reported But Not Settled (RBNS) claims. We show benefits of using deep neural network and the whole claim history in our prediction problem.

Download Full-text

System RBDO With Correlated Variables Using Probabilistic Re-Analysis and Local Metamodels

Volume 1: 36th Design Automation Conference, Parts A and B ◽

10.1115/detc2010-28130 ◽

2010 ◽

Cited By ~ 1

Author(s):

Ramon C. Kuczera ◽

Zissimos P. Mourelatos ◽

Efstratios Nikolaidis

Keyword(s):

System Reliability ◽

Random Variables ◽

Trust Region ◽

Optimization Approach ◽

Reliability Based Design Optimization ◽

Reliability Based Design ◽

Sensitivity Derivatives ◽

Simulation Based ◽

Gradient Based ◽

Mc Simulation

A simulation-based, system reliability-based design optimization (RBDO) method is presented that can handle problems with multiple failure regions and correlated random variables. Copulas are used to represent dependence between random variables. The method uses a Probabilistic Re-Analysis (PRRA) approach in conjunction with a sequential trust-region optimization approach and local metamodels covering each trust region. PRRA calculates very efficiently the system reliability of a design by performing a single Monte Carlo (MC) simulation per trust region. Although PRRA is based on MC simulation, it calculates “smooth” sensitivity derivatives, allowing the use of a gradient-based optimizer. The PRRA method is based on importance sampling. One requirement for providing accurate results is that the support of the sampling PDF must contain the support of the joint PDF of the input random variables. The trust-region optimization approach satisfies this requirement. Local metamodels are constructed sequentially for each trust region taking advantage of the potential overlap of the trust regions. The metamodels are used to determine the value of the indicator function in MC simulation. An example with correlated input random variables demonstrates the accuracy and efficiency of the proposed RBDO method.

Download Full-text

Selecting among Alternative Scenarios of Human Evolution by Simulated Genetic Gradients

Genes ◽

10.3390/genes9100506 ◽

2018 ◽

Vol 9 (10) ◽

pp. 506 ◽

Cited By ~ 2

Author(s):

Catarina Branco ◽

Miguel Arenas

Keyword(s):

Human Evolution ◽

Simulated Data ◽

Real Data ◽

Population Admixture ◽

Evolutionary Processes ◽

Long Distance ◽

History Of ◽

Pros And Cons ◽

Alternative Scenarios ◽

And Migration

Selecting among alternative scenarios of human evolution is nowadays a common methodology to investigate the history of our species. This strategy is usually based on computer simulations of genetic data under different evolutionary scenarios, followed by a fitting of the simulated data with the real data. A recent trend in the investigation of ancestral evolutionary processes of modern humans is the application of genetic gradients as a measure of fitting, since evolutionary processes such as range expansions, range contractions, and population admixture (among others) can lead to different genetic gradients. In addition, this strategy allows the analysis of the genetic causes of the observed genetic gradients. Here, we review recent findings on the selection among alternative scenarios of human evolution based on simulated genetic gradients, including pros and cons. First, we describe common methodologies to simulate genetic gradients and apply them to select among alternative scenarios of human evolution. Next, we review previous studies on the influence of range expansions, population admixture, last glacial period, and migration with long-distance dispersal on genetic gradients for some regions of the world. Finally, we discuss this analytical approach, including technical limitations, required improvements, and advice. Although here we focus on human evolution, this approach could be extended to study other species.

Download Full-text

Pyrolysis modeling, sensitivity analysis, and optimization techniques for combustible materials: A review

Journal of Fire Sciences ◽

10.1177/0734904119852740 ◽

2019 ◽

Vol 37 (4-6) ◽

pp. 377-433

Author(s):

Tatenda Nyazika ◽

Maude Jimenez ◽

Fabienne Samyn ◽

Serge Bourbigot

Keyword(s):

Sensitivity Analysis ◽

Material Properties ◽

Input Parameter ◽

Fire Behavior ◽

Sensitivity Study ◽

Optimization Techniques ◽

Decomposition Reactions ◽

Input Parameters ◽

Gradient Based ◽

Complete Set

Over the past years, pyrolysis models have moved from thermal models to comprehensive models with great flexibility including multi-step decomposition reactions. However, the downside is the need for a complete set of input data such as the material properties and the parameters related to the decomposition kinetics. Some of the parameters are not directly measurable or are difficult to determine and they carry a certain degree of uncertainty at high temperatures especially for materials that can melt, shrink, or swell. One can obtain input parameters by searching through the literature; however, certain materials may have the same nomenclature but the material properties may vary depending on the manufacturer, thereby inducing uncertainties in the model. Modelers have resorted to the use of optimization techniques such as gradient-based and direct search methods to estimate input parameters from experimental bench-scale data. As an integral part of the model, a sensitivity study allows to identify the role of each input parameter on the outputs. This work presents an overview of pyrolysis modeling, sensitivity analysis, and optimization techniques used to predict the fire behavior of combustible solids when exposed to an external heat flux.

Download Full-text