Meta-Heuristic Parameter Optimization for ANN and Real-Time Applications of ANN

2022 ◽  
pp. 166-201
Author(s):  
Asha Gowda Karegowda ◽  
Devika G.
Keyword(s):  
Neural Networks ◽  
Real Time ◽  
Parameter Optimization ◽  
Heuristic Algorithms ◽  
Search Space ◽  
Optimization Method ◽  
Classification Problems ◽  
Local Optima ◽  
Consistent Solution ◽  
Real Time Applications

Artificial neural networks (ANN) are often more suitable for classification problems. Even then, training of ANN is a surviving challenge task for large and high dimensional natured search space problems. These hitches are more for applications that involves process of fine tuning of ANN control parameters: weights and bias. There is no single search and optimization method that suits the weights and bias of ANN for all the problems. The traditional heuristic approach fails because of their poorer convergence speed and chances of ending up with local optima. In this connection, the meta-heuristic algorithms prove to provide consistent solution for optimizing ANN training parameters. This chapter will provide critics on both heuristics and meta-heuristic existing literature for training neural networks algorithms, applicability, and reliability on parameter optimization. In addition, the real-time applications of ANN will be presented. Finally, future directions to be explored in the field of ANN are presented which will of potential interest for upcoming researchers.

Meta-Heuristic Parameter Optimization for ANN and Real-Time Applications of ANN

2021 ◽  
pp. 227-269
Author(s):  
Asha Gowda Karegowda ◽  
Devika G.
Keyword(s):  
Neural Networks ◽  
Real Time ◽  
Parameter Optimization ◽  
Heuristic Algorithms ◽  
Search Space ◽  
Optimization Method ◽  
Classification Problems ◽  
Local Optima ◽  
Consistent Solution ◽  
Real Time Applications

Artificial neural networks (ANN) are often more suitable for classification problems. Even then, training of ANN is a surviving challenge task for large and high dimensional natured search space problems. These hitches are more for applications that involves process of fine tuning of ANN control parameters: weights and bias. There is no single search and optimization method that suits the weights and bias of ANN for all the problems. The traditional heuristic approach fails because of their poorer convergence speed and chances of ending up with local optima. In this connection, the meta-heuristic algorithms prove to provide consistent solution for optimizing ANN training parameters. This chapter will provide critics on both heuristics and meta-heuristic existing literature for training neural networks algorithms, applicability, and reliability on parameter optimization. In addition, the real-time applications of ANN will be presented. Finally, future directions to be explored in the field of ANN are presented which will of potential interest for upcoming researchers.

Analysis of Gravitation-Based Optimization Algorithms for Clustering and Classification

2020 ◽  
pp. 74-99 ◽  
Author(s):  
Sajad Ahmad Rather ◽  
P. Shanthi Bala
Keyword(s):  
Heuristic Algorithms ◽  
Optimization Problems ◽  
Search Algorithm ◽  
Clustering Algorithms ◽  
Optimization Algorithms ◽  
Gravitational Search Algorithm ◽  
Search Space ◽  
Classification Problems ◽  
Local Optima ◽  
Clustering And Classification

In recent years, various heuristic algorithms based on natural phenomena and swarm behaviors were introduced to solve innumerable optimization problems. These optimization algorithms show better performance than conventional algorithms. Recently, the gravitational search algorithm (GSA) is proposed for optimization which is based on Newton's law of universal gravitation and laws of motion. Within a few years, GSA became popular among the research community and has been applied to various fields such as electrical science, power systems, computer science, civil and mechanical engineering, etc. This chapter shows the importance of GSA, its hybridization, and applications in solving clustering and classification problems. In clustering, GSA is hybridized with other optimization algorithms to overcome the drawbacks such as curse of dimensionality, trapping in local optima, and limited search space of conventional data clustering algorithms. GSA is also applied to classification problems for pattern recognition, feature extraction, and increasing classification accuracy.

scholarly journals A New Hyper-Parameter Optimization Method for Power Load Forecast Based on Recurrent Neural Networks

Algorithms ◽  
2021 ◽  
Vol 14 (6) ◽  
pp. 163
Author(s):  
Yaru Li ◽  
Yulai Zhang ◽  
Yongping Cai
Keyword(s):  
Neural Networks ◽  
Parameter Optimization ◽  
Recurrent Neural Networks ◽  
Critical Role ◽  
Optimization Method ◽  
Bayesian Optimization ◽  
Power Load ◽  
Gradient Calculation ◽  
Truncated Newton ◽  
Selection Of

The selection of the hyper-parameters plays a critical role in the task of prediction based on the recurrent neural networks (RNN). Traditionally, the hyper-parameters of the machine learning models are selected by simulations as well as human experiences. In recent years, multiple algorithms based on Bayesian optimization (BO) are developed to determine the optimal values of the hyper-parameters. In most of these methods, gradients are required to be calculated. In this work, the particle swarm optimization (PSO) is used under the BO framework to develop a new method for hyper-parameter optimization. The proposed algorithm (BO-PSO) is free of gradient calculation and the particles can be optimized in parallel naturally. So the computational complexity can be effectively reduced which means better hyper-parameters can be obtained under the same amount of calculation. Experiments are done on real world power load data,where the proposed method outperforms the existing state-of-the-art algorithms,BO with limit-BFGS-bound (BO-L-BFGS-B) and BO with truncated-newton (BO-TNC),in terms of the prediction accuracy. The errors of the prediction result in different models show that BO-PSO is an effective hyper-parameter optimization method.

scholarly journals Real-time Neural Networks Implementation Proposal for Microcontrollers

Electronics ◽  
2020 ◽  
Vol 9 (10) ◽  
pp. 1597
Author(s):  
Caio José B. V. Guimarães ◽  
Marcelo A. C. Fernandes
Keyword(s):  
Neural Networks ◽  
Real Time ◽  
Intelligent Systems ◽  
Implementation Strategy ◽  
Processing Time ◽  
Mean Squared Error ◽  
Low Cost ◽  
Real Time Applications ◽  
Time Required ◽  
Process Classification

The adoption of intelligent systems with Artificial Neural Networks (ANNs) embedded in hardware for real-time applications currently faces a growing demand in fields such as the Internet of Things (IoT) and Machine to Machine (M2M). However, the application of ANNs in this type of system poses a significant challenge due to the high computational power required to process its basic operations. This paper aims to show an implementation strategy of a Multilayer Perceptron (MLP)-type neural network, in a microcontroller (a low-cost, low-power platform). A modular matrix-based MLP with the full classification process was implemented as was the backpropagation training in the microcontroller. The testing and validation were performed through Hardware-In-the-Loop (HIL) of the Mean Squared Error (MSE) of the training process, classification results, and the processing time of each implementation module. The results revealed a linear relationship between the values of the hyperparameters and the processing time required for classification, also the processing time concurs with the required time for many applications in the fields mentioned above. These findings show that this implementation strategy and this platform can be applied successfully in real-time applications that require the capabilities of ANNs.

scholarly journals A Homogeneous Algorithm for Motion Estimation and Compensation by Using Cellular Neural Networks

2010 ◽  
Vol 5 (5) ◽  
pp. 719 ◽  
Author(s):  
Cristian Grava ◽  
Alexandru Gacsádi ◽  
Ioan Buciu
Keyword(s):  
Neural Networks ◽  
Motion Estimation ◽  
Real Time ◽  
Cellular Neural Networks ◽  
Image Sequences ◽  
Artificial Vision ◽  
Real Time Image ◽  
Field Programmable ◽  
Real Time Applications ◽  
Time Image

In this paper we present an original implementation of a homogeneous algorithm for motion estimation and compensation in image sequences, by using Cellular Neural Networks (CNN). The CNN has been proven their efficiency in real-time image processing, because they can be implemented on a CNN chip or they can be emulated on Field Programmable Gate Array (FPGA). The motion information is obtained by using a CNN implementation of the well-known Horn & Schunck method. This information is further used in a CNN implementation of a motion-compensation method. Through our algorithm we obtain a homogeneous implementation for real-time applications in artificial vision or medical imaging. The algorithm is illustrated on some classical sequences and the results confirm the validity of our algorithm.

scholarly journals A hybrid cuckoo–harmony search algorithm for optimal design of water distribution systems

2015 ◽  
Vol 18 (3) ◽  
pp. 544-563 ◽  
Author(s):  
Razi Sheikholeslami ◽  
Aaron C. Zecchin ◽  
Feifei Zheng ◽  
Siamak Talatahari
Keyword(s):  
Distribution System ◽  
Distribution Systems ◽  
Water Distribution ◽  
Heuristic Algorithms ◽  
Optimization Problems ◽  
Search Algorithm ◽  
Harmony Search ◽  
Cuckoo Search ◽  
Search Space ◽  
Optimization Method

Meta-heuristic algorithms have been broadly used to deal with a range of water resources optimization problems over the past decades. One issue that exists in the use of these algorithms is the requirement of large computational resources, especially when handling real-world problems. To overcome this challenge, this paper develops a hybrid optimization method, the so-called CSHS, in which a cuckoo search (CS) algorithm is combined with a harmony search (HS) scheme. Within this hybrid framework, the CS is employed to find the promising regions of the search space within the initial explorative stages of the search, followed by a thorough exploitation phase using the combined CS and HS algorithms. The utility of the proposed CSHS is demonstrated using four water distribution system design problems with increased scales and complexity. The obtained results reveal that the CSHS method outperforms the standard CS, as well as the majority of other meta-heuristics that have previously been applied to the case studies investigated, in terms of efficiently seeking optimal solutions. Furthermore, the CSHS has two control parameters that need to be fine-tuned compared to many other algorithms, which is appealing for its practical application as an extensive parameter-calibration process is typically computationally very demanding.

scholarly journals NICE: Noise Injection and Clamping Estimation for Neural Network Quantization

Mathematics ◽  
2021 ◽  
Vol 9 (17) ◽  
pp. 2144
Author(s):  
Chaim Baskin ◽  
Evgenii Zheltonozhkii ◽  
Tal Rozen ◽  
Natan Liss ◽  
Yoav Chai ◽  
...  
Keyword(s):  
Neural Networks ◽  
Power Systems ◽  
Low Power ◽  
Real Time ◽  
Language Processing ◽  
Power Efficient ◽  
Noise Injection ◽  
Classification Tasks ◽  
Real Time Applications ◽  
The Cost

Convolutional Neural Networks (CNNs) are very popular in many fields including computer vision, speech recognition, natural language processing, etc. Though deep learning leads to groundbreaking performance in those domains, the networks used are very computationally demanding and are far from being able to perform in real-time applications even on a GPU, which is not power efficient and therefore does not suit low power systems such as mobile devices. To overcome this challenge, some solutions have been proposed for quantizing the weights and activations of these networks, which accelerate the runtime significantly. Yet, this acceleration comes at the cost of a larger error unless spatial adjustments are carried out. The method proposed in this work trains quantized neural networks by noise injection and a learned clamping, which improve accuracy. This leads to state-of-the-art results on various regression and classification tasks, e.g., ImageNet classification with architectures such as ResNet-18/34/50 with as low as 3 bit weights and activations. We implement the proposed solution on an FPGA to demonstrate its applicability for low-power real-time applications. The quantization code will become publicly available upon acceptance.

scholarly journals An improved hybrid Aquila Optimizer and Harris Hawks Optimization for global optimization

2021 ◽  
Vol 18 (6) ◽  
pp. 7076-7109
Author(s):  
Shuang Wang ◽  
◽  
Heming Jia ◽  
Qingxin Liu ◽  
Rong Zheng ◽  
...  
Keyword(s):  
Global Optimization ◽  
Heuristic Algorithms ◽  
Optimization Problems ◽  
Search Space ◽  
Search Performance ◽  
Exploration And Exploitation ◽  
Design Problems ◽  
Local Optima ◽  
Opposition Based Learning ◽  
Engineering Design Problems

<abstract> <p>This paper introduces an improved hybrid Aquila Optimizer (AO) and Harris Hawks Optimization (HHO) algorithm, namely IHAOHHO, to enhance the searching performance for global optimization problems. In the IHAOHHO, valuable exploration and exploitation capabilities of AO and HHO are retained firstly, and then representative-based hunting (RH) and opposition-based learning (OBL) strategies are added in the exploration and exploitation phases to effectively improve the diversity of search space and local optima avoidance capability of the algorithm, respectively. To verify the optimization performance and the practicability, the proposed algorithm is comprehensively analyzed on standard and CEC2017 benchmark functions and three engineering design problems. The experimental results show that the proposed IHAOHHO has more superior global search performance and faster convergence speed compared to the basic AO and HHO and selected state-of-the-art meta-heuristic algorithms.</p> </abstract>

scholarly journals Noise and Performances Analysis of Commerical Aircrafts using Artificial Neural Networks

2015 ◽  
Vol 2 (1-2.) ◽  
Author(s):  
Şahin Yildirim
Keyword(s):  
Neural Networks ◽  
Artificial Neural Networks ◽  
Real Time ◽  
High Technology ◽  
Measured Parameter ◽  
The World ◽  
Real Time Applications ◽  
Artificial Neural ◽  
Parameter Values

Commerical aircrafts are very important part for airway travelling. In spite of high technology on aircrafts, there is still fatality accidents in the world. Because of this reason, it is very important criteria to analyse noises of main elements of the air-craft systems. In tis study, an aircraft’s main disturbances are analysed with proposed neural networks. Firstly, the noises of the jet, turbine and fan were measured from the aircraft. Secondly, the measured parameter values were predicted the proposed neural networks. The results of the proposed neuarl approaches were shown that this type of neural predictors will be employed to predict aircrafts unpredicted disturbances in real time applications.

scholarly journals Evolutionary Spiking Neural Networks for Solving Supervised Classification Problems

2019 ◽  
Vol 2019 ◽  
pp. 1-13 ◽  
Author(s):  
G. López-Vázquez ◽  
M. Ornelas-Rodriguez ◽  
A. Espinal ◽  
J. A. Soria-Alcaraz ◽  
A. Rojas-Domínguez ◽  
...  
Keyword(s):  
Neural Networks ◽  
Supervised Classification ◽  
Fitness Function ◽  
Statistical Tests ◽  
Search Space ◽  
Spiking Neural Networks ◽  
Classification Problems ◽  
Benchmark Datasets ◽  
Explicit Training ◽  
Better Than

This paper presents a grammatical evolution (GE)-based methodology to automatically design third generation artificial neural networks (ANNs), also known as spiking neural networks (SNNs), for solving supervised classification problems. The proposal performs the SNN design by exploring the search space of three-layered feedforward topologies with configured synaptic connections (weights and delays) so that no explicit training is carried out. Besides, the designed SNNs have partial connections between input and hidden layers which may contribute to avoid redundancies and reduce the dimensionality of input feature vectors. The proposal was tested on several well-known benchmark datasets from the UCI repository and statistically compared against a similar design methodology for second generation ANNs and an adapted version of that methodology for SNNs; also, the results of the two methodologies and the proposed one were improved by changing the fitness function in the design process. The proposed methodology shows competitive and consistent results, and the statistical tests support the conclusion that the designs produced by the proposal perform better than those produced by other methodologies.

Sign in / Sign up

Export Citation Format

Share Document