The Whale Optimization Algorithm Approach for Deep Neural Networks

One of the biggest challenge in the field of deep learning is the parameter selection and optimization process. In recent years different algorithms have been proposed including bio-inspired solutions to solve this problem, however, there are many challenges including local minima, saddle points, and vanishing gradients. In this paper, we introduce the Whale Optimisation Algorithm (WOA) based on the swarm foraging behavior of humpback whales to optimise neural network hyperparameters. We wish to stress that to the best of our knowledge this is the first attempt that uses Whale Optimisation Algorithm for the optimisation task of hyperparameters. After a detailed description of the WOA algorithm we formulate and explain the application in deep learning, present the implementation, and compare the proposed algorithm with other well-known algorithms including widely used Grid and Random Search methods. Additionally, we have implemented a third dimension feature analysis to the original WOA algorithm to utilize 3D search space (3D-WOA). Simulations show that the proposed algorithm can be successfully used for hyperparameters optimization, achieving accuracy of 89.85% and 80.60% for Fashion MNIST and Reuters datasets, respectively.

Download Full-text

Automatic Detection of Arrhythmia Based on Multi-Resolution Representation of ECG Signal

Sensors ◽

10.3390/s20061579 ◽

2020 ◽

Vol 20 (6) ◽

pp. 1579

Author(s):

Dongqi Wang ◽

Qinghua Meng ◽

Dongming Chen ◽

Hupo Zhang ◽

Lisheng Xu

Keyword(s):

Neural Networks ◽

Deep Learning ◽

Deep Neural Networks ◽

Channel Model ◽

Expert Knowledge ◽

Automatic Detection ◽

Data Representation ◽

Learning Technology ◽

Arrhythmia Detection ◽

Automatic Feature Extraction

Automatic detection of arrhythmia is of great significance for early prevention and diagnosis of cardiovascular disease. Traditional feature engineering methods based on expert knowledge lack multidimensional and multi-view information abstraction and data representation ability, so the traditional research on pattern recognition of arrhythmia detection cannot achieve satisfactory results. Recently, with the increase of deep learning technology, automatic feature extraction of ECG data based on deep neural networks has been widely discussed. In order to utilize the complementary strength between different schemes, in this paper, we propose an arrhythmia detection method based on the multi-resolution representation (MRR) of ECG signals. This method utilizes four different up to date deep neural networks as four channel models for ECG vector representations learning. The deep learning based representations, together with hand-crafted features of ECG, forms the MRR, which is the input of the downstream classification strategy. The experimental results of big ECG dataset multi-label classification confirm that the F1 score of the proposed method is 0.9238, which is 1.31%, 0.62%, 1.18% and 0.6% higher than that of each channel model. From the perspective of architecture, this proposed method is highly scalable and can be employed as an example for arrhythmia recognition.

Download Full-text

Explaining deep neural networks for knowledge discovery in electrocardiogram analysis

Scientific Reports ◽

10.1038/s41598-021-90285-5 ◽

2021 ◽

Vol 11 (1) ◽

Author(s):

Steven A. Hicks ◽

Jonas L. Isaksen ◽

Vajira Thambawita ◽

Jonas Ghouse ◽

Gustav Ahlberg ◽

...

Keyword(s):

Decision Making ◽

Deep Learning ◽

Deep Neural Networks ◽

Clinical Decision Making ◽

Medical Knowledge ◽

Clinical Decision ◽

Medical Data ◽

Medical Doctors ◽

Medical Tests ◽

Deep Learning Model

AbstractDeep learning-based tools may annotate and interpret medical data more quickly, consistently, and accurately than medical doctors. However, as medical doctors are ultimately responsible for clinical decision-making, any deep learning-based prediction should be accompanied by an explanation that a human can understand. We present an approach called electrocardiogram gradient class activation map (ECGradCAM), which is used to generate attention maps and explain the reasoning behind deep learning-based decision-making in ECG analysis. Attention maps may be used in the clinic to aid diagnosis, discover new medical knowledge, and identify novel features and characteristics of medical tests. In this paper, we showcase how ECGradCAM attention maps can unmask how a novel deep learning model measures both amplitudes and intervals in 12-lead electrocardiograms, and we show an example of how attention maps may be used to develop novel ECG features.

Download Full-text

Whale optimization algorithm based on lateral inhibition for image matching and vision-guided AUV docking

Journal of Intelligent & Fuzzy Systems ◽

10.3233/jifs-200365 ◽

2020 ◽

pp. 1-12

Author(s):

Zheping Yan ◽

Jinzhong Zhang ◽

Jialing Tang

Keyword(s):

Lateral Inhibition ◽

Optimization Algorithm ◽

Image Matching ◽

Autonomous Underwater Vehicle ◽

Optimal Solution ◽

Search Space ◽

Whale Optimization Algorithm ◽

Inhibition Mechanism ◽

Whale Optimization ◽

Accuracy And Stability

The accuracy and stability of relative pose estimation of an autonomous underwater vehicle (AUV) and a target depend on whether the characteristics of the underwater image can be accurately and quickly extracted. In this paper, a whale optimization algorithm (WOA) based on lateral inhibition (LI) is proposed to solve the image matching and vision-guided AUV docking problem. The proposed method is named the LI-WOA. The WOA is motivated by the behavior of humpback whales, and it mainly imitates encircling prey, bubble-net attacking and searching for prey to obtain the globally optimal solution in the search space. The WOA not only balances exploration and exploitation but also has a faster convergence speed, higher calculation accuracy and stronger robustness than other approaches. The lateral inhibition mechanism can effectively perform image enhancement and image edge extraction to improve the accuracy and stability of image matching. The LI-WOA combines the optimization efficiency of the WOA and the matching accuracy of the LI mechanism to improve convergence accuracy and the correct matching rate. To verify its effectiveness and feasibility, the WOA is compared with other algorithms by maximizing the similarity between the original image and the template image. The experimental results show that the LI-WOA has a better average value, a higher correct rate, less execution time and stronger robustness than other algorithms. The LI-WOA is an effective and stable method for solving the image matching and vision-guided AUV docking problem.

Download Full-text

Prediction of the critical temperature of a superconductor by using the WOA/MARS, Ridge, Lasso and Elastic-net machine learning techniques

Neural Computing and Applications ◽

10.1007/s00521-021-06304-z ◽

2021 ◽

Author(s):

Paulino José García-Nieto ◽

Esperanza García-Gonzalo ◽

José Pablo Paredes-Sánchez

Keyword(s):

Critical Temperature ◽

Optimization Algorithm ◽

Regression Models ◽

Chemical Properties ◽

Atomic Radius ◽

Elastic Net ◽

Multivariate Adaptive Regression Splines ◽

Machine Learning Techniques ◽

Humpback Whales ◽

Whale Optimization

AbstractThis study builds a predictive model capable of estimating the critical temperature of a superconductor from experimentally determined physico-chemical properties of the material (input variables): features extracted from the thermal conductivity, atomic radius, valence, electron affinity and atomic mass. This original model is built using a novel hybrid algorithm relied on the multivariate adaptive regression splines (MARS) technique in combination with a nature-inspired meta-heuristic optimization algorithm termed the whale optimization algorithm (WOA) that mimics the social behavior of humpback whales. Additionally, the Ridge, Lasso and Elastic-net regression models were fitted to the same experimental data for comparison purposes. The results of the current investigation indicate that the critical temperature of a superconductor can be successfully predicted using this proposed hybrid WOA/MARS-based model. Furthermore, the results obtained with the Ridge, Lasso and Elastic-net regression models are clearly worse than those obtained with the WOA/MARS-based model.

Download Full-text

GMBO: Group Mean-Based Optimizer for Solving Various Optimization Problems

Mathematics ◽

10.3390/math9111190 ◽

2021 ◽

Vol 9 (11) ◽

pp. 1190

Author(s):

Mohammad Dehghani ◽

Zeinab Montazeri ◽

Štěpán Hubálovský

Keyword(s):

Optimization Algorithm ◽

Optimization Problems ◽

Search Algorithm ◽

Random Search ◽

Optimization Algorithms ◽

Gravitational Search Algorithm ◽

Main Idea ◽

Population Based ◽

Grey Wolf Optimizer ◽

Whale Optimization

There are many optimization problems in the different disciplines of science that must be solved using the appropriate method. Population-based optimization algorithms are one of the most efficient ways to solve various optimization problems. Population-based optimization algorithms are able to provide appropriate solutions to optimization problems based on a random search of the problem-solving space without the need for gradient and derivative information. In this paper, a new optimization algorithm called the Group Mean-Based Optimizer (GMBO) is presented; it can be applied to solve optimization problems in various fields of science. The main idea in designing the GMBO is to use more effectively the information of different members of the algorithm population based on two selected groups, with the titles of the good group and the bad group. Two new composite members are obtained by averaging each of these groups, which are used to update the population members. The various stages of the GMBO are described and mathematically modeled with the aim of being used to solve optimization problems. The performance of the GMBO in providing a suitable quasi-optimal solution on a set of 23 standard objective functions of different types of unimodal, high-dimensional multimodal, and fixed-dimensional multimodal is evaluated. In addition, the optimization results obtained from the proposed GMBO were compared with eight other widely used optimization algorithms, including the Marine Predators Algorithm (MPA), the Tunicate Swarm Algorithm (TSA), the Whale Optimization Algorithm (WOA), the Grey Wolf Optimizer (GWO), Teaching–Learning-Based Optimization (TLBO), the Gravitational Search Algorithm (GSA), Particle Swarm Optimization (PSO), and the Genetic Algorithm (GA). The optimization results indicated the acceptable performance of the proposed GMBO, and, based on the analysis and comparison of the results, it was determined that the GMBO is superior and much more competitive than the other eight algorithms.

Download Full-text

Enabling deeper learning on big data for materials informatics applications

Scientific Reports ◽

10.1038/s41598-021-83193-1 ◽

2021 ◽

Vol 11 (1) ◽

Author(s):

Dipendra Jha ◽

Vishu Gupta ◽

Logan Ward ◽

Zijiang Yang ◽

Christopher Wolverton ◽

...

Keyword(s):

Neural Networks ◽

Big Data ◽

Deep Learning ◽

Deep Neural Networks ◽

Materials Science ◽

Prediction Models ◽

Model Performance ◽

Materials Informatics ◽

Learning Framework ◽

Significant Attention

AbstractThe application of machine learning (ML) techniques in materials science has attracted significant attention in recent years, due to their impressive ability to efficiently extract data-driven linkages from various input materials representations to their output properties. While the application of traditional ML techniques has become quite ubiquitous, there have been limited applications of more advanced deep learning (DL) techniques, primarily because big materials datasets are relatively rare. Given the demonstrated potential and advantages of DL and the increasing availability of big materials datasets, it is attractive to go for deeper neural networks in a bid to boost model performance, but in reality, it leads to performance degradation due to the vanishing gradient problem. In this paper, we address the question of how to enable deeper learning for cases where big materials data is available. Here, we present a general deep learning framework based on Individual Residual learning (IRNet) composed of very deep neural networks that can work with any vector-based materials representation as input to build accurate property prediction models. We find that the proposed IRNet models can not only successfully alleviate the vanishing gradient problem and enable deeper learning, but also lead to significantly (up to 47%) better model accuracy as compared to plain deep neural networks and traditional ML techniques for a given input materials representation in the presence of big data.

Download Full-text

Whale optimization algorithm fused with SVM to detect stress in EEG signals

Intelligent Decision Technologies ◽

10.3233/idt-200047 ◽

2021 ◽

Vol 15 (1) ◽

pp. 87-97

Author(s):

Richa Gupta ◽

M. Afshar Alam ◽

Parul Agarwal

Keyword(s):

Optimization Algorithm ◽

Whale Optimization Algorithm ◽

Humpback Whales ◽

Support Vector ◽

Eeg Signals ◽

Hunting Behavior ◽

Positron Emission ◽

Whale Optimization ◽

Signal Activity ◽

Work Done

Identifying stress and its level has always been a challenging area for researchers. A lot of work is going on around the world on the same. An attempt has been made by the authors in this paper as they present a methodology for detecting stress in EEG signals. Electroencephalogram (EEG) is commonly used to acquire brain signal activity. Though there exist other techniques to extract the same like Functional magnetic resonance imaging (fMRI), positron emission tomography (PET) we have used EEG as it is economical. We have used an open-source dataset for EEG data. Various images are used as the target stressor for collecting EEG signals. After feature selection and extraction, a support vector machine (SVM) with a whale optimization algorithm (WOA) in its kernel function for classification is used. WOA is a bio-inspired meta-heuristic algorithm, based on the hunting behavior of humpback whales. Using this method, we had obtained 91% accuracy for detecting the stress. The paper also compared the previous work done in detecting stress with the work proposed in this paper.

Download Full-text

Representing Deep Neural Networks Latent Space Geometries with Graphs

Algorithms ◽

10.3390/a14020039 ◽

2021 ◽

Vol 14 (2) ◽

pp. 39

Author(s):

Carlos Lassance ◽

Vincent Gripon ◽

Antonio Ortega

Keyword(s):

Machine Learning ◽

Neural Networks ◽

Deep Learning ◽

Objective Function ◽

Learning Process ◽

Deep Neural Networks ◽

State Of The Art ◽

The Core ◽

Learning Tasks ◽

Latent Space

Deep Learning (DL) has attracted a lot of attention for its ability to reach state-of-the-art performance in many machine learning tasks. The core principle of DL methods consists of training composite architectures in an end-to-end fashion, where inputs are associated with outputs trained to optimize an objective function. Because of their compositional nature, DL architectures naturally exhibit several intermediate representations of the inputs, which belong to so-called latent spaces. When treated individually, these intermediate representations are most of the time unconstrained during the learning process, as it is unclear which properties should be favored. However, when processing a batch of inputs concurrently, the corresponding set of intermediate representations exhibit relations (what we call a geometry) on which desired properties can be sought. In this work, we show that it is possible to introduce constraints on these latent geometries to address various problems. In more detail, we propose to represent geometries by constructing similarity graphs from the intermediate representations obtained when processing a batch of inputs. By constraining these Latent Geometry Graphs (LGGs), we address the three following problems: (i) reproducing the behavior of a teacher architecture is achieved by mimicking its geometry, (ii) designing efficient embeddings for classification is achieved by targeting specific geometries, and (iii) robustness to deviations on inputs is achieved via enforcing smooth variation of geometry between consecutive latent spaces. Using standard vision benchmarks, we demonstrate the ability of the proposed geometry-based methods in solving the considered problems.

Download Full-text

Data-Driven Structural Health Monitoring and Damage Detection through Deep Learning: State-of-the-Art Review

Sensors ◽

10.3390/s20102778 ◽

2020 ◽

Vol 20 (10) ◽

pp. 2778 ◽

Cited By ~ 12

Author(s):

Mohsen Azimi ◽

Armin Eslamlou ◽

Gokhan Pekcan

Keyword(s):

Deep Learning ◽

Structural Health Monitoring ◽

Health Monitoring ◽

High Speed ◽

Deep Neural Networks ◽

State Of The Art ◽

Data Driven ◽

Structural Health ◽

Promising Tool ◽

Significant Attention

Data-driven methods in structural health monitoring (SHM) is gaining popularity due to recent technological advancements in sensors, as well as high-speed internet and cloud-based computation. Since the introduction of deep learning (DL) in civil engineering, particularly in SHM, this emerging and promising tool has attracted significant attention among researchers. The main goal of this paper is to review the latest publications in SHM using emerging DL-based methods and provide readers with an overall understanding of various SHM applications. After a brief introduction, an overview of various DL methods (e.g., deep neural networks, transfer learning, etc.) is presented. The procedure and application of vibration-based, vision-based monitoring, along with some of the recent technologies used for SHM, such as sensors, unmanned aerial vehicles (UAVs), etc. are discussed. The review concludes with prospects and potential limitations of DL-based methods in SHM applications.

Download Full-text

Off-the-shelf deep learning is not enough, and requires parsimony, Bayesianity, and causality

npj Computational Materials ◽

10.1038/s41524-020-00487-0 ◽

2021 ◽

Vol 7 (1) ◽

Author(s):

Rama K. Vasudevan ◽

Maxim Ziatdinov ◽

Lukas Vlcek ◽

Sergei V. Kalinin

Keyword(s):

Neural Networks ◽

Computer Vision ◽

Deep Learning ◽

Bayesian Methods ◽

Deep Neural Networks ◽

Applied Research ◽

Modern Science ◽

Generative Models ◽

Knowledge Development ◽

Physical Constraints

AbstractDeep neural networks (‘deep learning’) have emerged as a technology of choice to tackle problems in speech recognition, computer vision, finance, etc. However, adoption of deep learning in physical domains brings substantial challenges stemming from the correlative nature of deep learning methods compared to the causal, hypothesis driven nature of modern science. We argue that the broad adoption of Bayesian methods incorporating prior knowledge, development of solutions with incorporated physical constraints and parsimonious structural descriptors and generative models, and ultimately adoption of causal models, offers a path forward for fundamental and applied research.

Download Full-text