Complete and representative training of neural networks: A generalization study using double noise injection and natural images

Geophysics ◽  
2021 ◽  
pp. 1-43
Author(s):  
Chao Zhang ◽  
Mirko van der Baan
Keyword(s):  
Machine Learning ◽  
Neural Networks ◽  
Seismic Data ◽  
Learning Algorithm ◽  
Random Noise ◽  
Solution Space ◽  
Natural Images ◽  
Training Data ◽  
Noisy Input ◽  
Noise Injection

Neural networks hold substantial promise to automate various processing and interpretation tasks. Yet their performance is often sub-optimal compared with standard but more closely guided approaches. Lack of performance is often attributed to poor generalization, in particular if fewer training examples are provided than free parameters exist in the machine learning algorithm. In this case the training data are typically memorized instead of the algorithm learning the underlying general trends. Network generalization is improved if the provided samples are representative, in that they describe all features of interest well. We argue that a more subtle condition preventing poor performance is that the provided examples must also be complete; the examples must span the full solution space. Ensuring completeness during training is challenging unless the target application is well understood. We illustrate that one possible solution is to make the problem more general if this greatly increases the number of available training data. For instance, if seismic images are treated as a subclass of natural images, then a deep-learning-based denoiser for seismic data can be trained using exclusively natural images. The latter are widely available. The resulting denoising algorithm has never seen any seismic data during the training stage; yet it displays a performance comparable to standard and advanced random-noise reduction methods. We exclude any seismic data during training to demonstrate the natural images are both complete and representative for this specific task. Furthermore, we apply a novel approach to increase the amount of training data known as double noise injection, providing both noisy input and output images during the training process. Given the importance of network generalization, we hope that insights gained in this study may help improve the performance of a range of machine learning applications in geophysics.

scholarly journals A Natural Images Pre-Trained Deep Learning Method for Seismic Random Noise Attenuation

2022 ◽  
Vol 14 (2) ◽  
pp. 263
Author(s):  
Haixia Zhao ◽  
Tingting Bai ◽  
Zhiqiang Wang
Keyword(s):  
Deep Learning ◽  
Seismic Data ◽  
Field Data ◽  
Noise Suppression ◽  
Signal To Noise Ratio ◽  
Random Noise ◽  
Natural Images ◽  
Training Data ◽  
Learning Approaches ◽  
Learning Method

Seismic field data are usually contaminated by random or complex noise, which seriously affect the quality of seismic data contaminating seismic imaging and seismic interpretation. Improving the signal-to-noise ratio (SNR) of seismic data has always been a key step in seismic data processing. Deep learning approaches have been successfully applied to suppress seismic random noise. The training examples are essential in deep learning methods, especially for the geophysical problems, where the complete training data are not easy to be acquired due to high cost of acquisition. In this work, we propose a natural images pre-trained deep learning method to suppress seismic random noise through insight of the transfer learning. Our network contains pre-trained and post-trained networks: the former is trained by natural images to obtain the preliminary denoising results, while the latter is trained by a small amount of seismic images to fine-tune the denoising effects by semi-supervised learning to enhance the continuity of geological structures. The results of four types of synthetic seismic data and six field data demonstrate that our network has great performance in seismic random noise suppression in terms of both quantitative metrics and intuitive effects.

scholarly journals Identification of Neuronal Polarity by Node-Based Machine Learning

2021 ◽  
Author(s):  
Chen-Zhi Su ◽  
Kuan-Ting Chou ◽  
Hsuan-Pei Huang ◽  
Chiau-Jou Li ◽  
Ching-Che Charng ◽  
...  
Keyword(s):  
Machine Learning ◽  
Neural Networks ◽  
Spatial Information ◽  
Learning Algorithm ◽  
Training Data ◽  
Neuronal Polarity ◽  
Projection Neurons ◽  
Branch Points ◽  
Spatial Correlations ◽  
Information Dynamics

AbstractIdentifying the direction of signal flows in neural networks is important for understanding the intricate information dynamics of a living brain. Using a dataset of 213 projection neurons distributed in more than 15 neuropils of a Drosophila brain, we develop a powerful machine learning algorithm: node-based polarity identifier of neurons (NPIN). The proposed model is trained only by information specific to nodes, the branch points on the skeleton, and includes both Soma Features (which contain spatial information from a given node to a soma) and Local Features (which contain morphological information of a given node). After including the spatial correlations between nodal polarities, our NPIN provided extremely high accuracy (>96.0%) for the classification of neuronal polarity, even for complex neurons with more than two dendrite/axon clusters. Finally, we further apply NPIN to classify the neuronal polarity of neurons in other species (Blowfly and Moth), which have much less neuronal data available. Our results demonstrate the potential of NPIN as a powerful tool to identify the neuronal polarity of insects and to map out the signal flows in the brain’s neural networks if more training data become available in the future.

Scalable Approach to High Coverages on Oxides via Iterative Training of a Machine-Learning Algorithm

2019 ◽  
Author(s):  
Andrew Medford ◽  
Shengchun Yang ◽  
Fuzhu Liu
Keyword(s):  
Machine Learning ◽  
Chemical Potential ◽  
Learning Algorithm ◽  
Absolute Error ◽  
Low Energy ◽  
Training Data ◽  
High Coverage ◽  
Metal Compounds ◽  
Adsorption Energies ◽  
The Stability

Understanding the interaction of multiple types of adsorbate molecules on solid surfaces is crucial to establishing the stability of catalysts under various chemical environments. Computational studies on the high coverage and mixed coverages of reaction intermediates are still challenging, especially for transition-metal compounds. In this work, we present a framework to predict differential adsorption energies and identify low-energy structures under high- and mixed-adsorbate coverages on oxide materials. The approach uses Gaussian process machine-learning models with quantified uncertainty in conjunction with an iterative training algorithm to actively identify the training set. The framework is demonstrated for the mixed adsorption of CH<sub>x</sub>, NH<sub>x</sub> and OH<sub>x</sub> species on the oxygen vacancy and pristine rutile TiO<sub>2</sub>(110) surface sites. The results indicate that the proposed algorithm is highly efficient at identifying the most valuable training data, and is able to predict differential adsorption energies with a mean absolute error of ~0.3 eV based on <25% of the total DFT data. The algorithm is also used to identify 76% of the low-energy structures based on <30% of the total DFT data, enabling construction of surface phase diagrams that account for high and mixed coverage as a function of the chemical potential of C, H, O, and N. Furthermore, the computational scaling indicates the algorithm scales nearly linearly (N<sup>1.12</sup>) as the number of adsorbates increases. This framework can be directly extended to metals, metal oxides, and other materials, providing a practical route toward the investigation of the behavior of catalysts under high-coverage conditions.

scholarly journals Transfer Learning with Deep Recurrent Neural Networks for Remaining Useful Life Estimation

2018 ◽  
Vol 8 (12) ◽  
pp. 2416 ◽  
Author(s):  
Ansi Zhang ◽  
Honglei Wang ◽  
Shaobo Li ◽  
Yuxin Cui ◽  
Zhonghao Liu ◽  
...  
Keyword(s):  
Neural Networks ◽  
Transfer Learning ◽  
Recurrent Neural Networks ◽  
Prediction Models ◽  
Learning Algorithm ◽  
Remaining Useful Life ◽  
Operating Conditions ◽  
Training Data ◽  
Good Prediction ◽  
Useful Life

Prognostics, such as remaining useful life (RUL) prediction, is a crucial task in condition-based maintenance. A major challenge in data-driven prognostics is the difficulty of obtaining a sufficient number of samples of failure progression. However, for traditional machine learning methods and deep neural networks, enough training data is a prerequisite to train good prediction models. In this work, we proposed a transfer learning algorithm based on Bi-directional Long Short-Term Memory (BLSTM) recurrent neural networks for RUL estimation, in which the models can be first trained on different but related datasets and then fine-tuned by the target dataset. Extensive experimental results show that transfer learning can in general improve the prediction models on the dataset with a small number of samples. There is one exception that when transferring from multi-type operating conditions to single operating conditions, transfer learning led to a worse result.

Water Quality Prediction Using Statistical Tool and Machine Learning Algorithm

2020 ◽  
pp. 609-623
Author(s):  
Arun Kumar Beerala ◽  
Gobinath R. ◽  
Shyamala G. ◽  
Siribommala Manvitha
Keyword(s):  
Machine Learning ◽  
Learning Algorithm ◽  
Training Data ◽  
Machine Learning Techniques ◽  
Statistical Tool ◽  
Data Set ◽  
Water Quality Prediction ◽  
Living Things ◽  
Sampling Locations ◽  
Different Seasons

Water is the most valuable natural resource for all living things and the ecosystem. The quality of groundwater is changed due to change in ecosystem, industrialisation, and urbanisation, etc. In the study, 60 samples were taken and analysed for various physio-chemical parameters. The sampling locations were located using global positioning system (GPS) and were taken for two consecutive years for two different seasons, monsoon (Nov-Dec) and post-monsoon (Jan-Mar). In 2016-2017 and 2017-2018 pH, EC, and TDS were obtained in the field. Hardness and Chloride are determined using titration method. Nitrate and Sulphate were determined using Spectrophotometer. Machine learning techniques were used to train the data set and to predict the unknown values. The dominant elements of groundwater are as follows: Ca2, Mg2 for cation and Cl-, SO42, NO3− for anions. The regression value for the training data set was found to be 0.90596, and for the entire network, it was found to be 0.81729. The best performance was observed as 0.0022605 at epoch 223.

Convolutional Neural Network

2022 ◽  
pp. 1559-1575
Author(s):  
Mário Pereira Véstias
Keyword(s):  
Neural Network ◽  
Machine Learning ◽  
Neural Networks ◽  
Artificial Neural Networks ◽  
Deep Learning ◽  
Convolutional Neural Network ◽  
Machine Learning Algorithms ◽  
Training Data ◽  
Machine Learning Model ◽  
Artificial Neural

Machine learning is the study of algorithms and models for computing systems to do tasks based on pattern identification and inference. When it is difficult or infeasible to develop an algorithm to do a particular task, machine learning algorithms can provide an output based on previous training data. A well-known machine learning model is deep learning. The most recent deep learning models are based on artificial neural networks (ANN). There exist several types of artificial neural networks including the feedforward neural network, the Kohonen self-organizing neural network, the recurrent neural network, the convolutional neural network, the modular neural network, among others. This article focuses on convolutional neural networks with a description of the model, the training and inference processes and its applicability. It will also give an overview of the most used CNN models and what to expect from the next generation of CNN models.

Design-Oriented Multifidelity Fluid Simulation Using Machine Learned Fidelity Mapping

2019 ◽  
Author(s):  
Kazuko Fuchi ◽  
Eric M. Wolf ◽  
David S. Makhija ◽  
Nathan A. Wukie ◽  
Christopher R. Schrock ◽  
...  
Keyword(s):  
Machine Learning ◽  
Learning Algorithm ◽  
Fluid Simulation ◽  
Machine Learning Algorithms ◽  
Training Data ◽  
Supervised Machine Learning ◽  
High Fidelity ◽  
Computational Domain ◽  
Symmetry Properties ◽  
High Fidelity Simulations

Abstract A machine learning algorithm that performs multifidelity domain decomposition is introduced. While the design of complex systems can be facilitated by numerical simulations, the determination of appropriate physics couplings and levels of model fidelity can be challenging. The proposed method automatically divides the computational domain into subregions and assigns required fidelity level, using a small number of high fidelity simulations to generate training data and low fidelity solutions as input data. Unsupervised and supervised machine learning algorithms are used to correlate features from low fidelity solutions to fidelity assignment. The effectiveness of the method is demonstrated in a problem of viscous fluid flow around a cylinder at Re ≈ 20. Ling et al. built physics-informed invariance and symmetry properties into machine learning models and demonstrated improved model generalizability. Along these lines, we avoid using problem dependent features such as coordinates of sample points, object geometry or flow conditions as explicit inputs to the machine learning model. Use of pointwise flow features generates large data sets from only one or two high fidelity simulations, and the fidelity predictor model achieved 99.5% accuracy at training points. The trained model was shown to be capable of predicting a fidelity map for a problem with an altered cylinder radius. A significant improvement in the prediction performance was seen when inputs are expanded to include multiscale features that incorporate neighborhood information.

scholarly journals Interpretation of Neural Networks Is Fragile

2019 ◽  
Vol 33 ◽  
pp. 3681-3688 ◽  
Author(s):  
Amirata Ghorbani ◽  
Abubakar Abid ◽  
James Zou
Keyword(s):  
Neural Network ◽  
Machine Learning ◽  
Neural Networks ◽  
Input Data ◽  
Learning Algorithm ◽  
Hessian Matrix ◽  
Machine Learning Algorithm ◽  
Feature Importance ◽  
Adversarial Attack ◽  
Measurement Biases

In order for machine learning to be trusted in many applications, it is critical to be able to reliably explain why the machine learning algorithm makes certain predictions. For this reason, a variety of methods have been developed recently to interpret neural network predictions by providing, for example, feature importance maps. For both scientific robustness and security reasons, it is important to know to what extent can the interpretations be altered by small systematic perturbations to the input data, which might be generated by adversaries or by measurement biases. In this paper, we demonstrate how to generate adversarial perturbations that produce perceptively indistinguishable inputs that are assigned the same predicted label, yet have very different interpretations. We systematically characterize the robustness of interpretations generated by several widely-used feature importance interpretation methods (feature importance maps, integrated gradients, and DeepLIFT) on ImageNet and CIFAR-10. In all cases, our experiments show that systematic perturbations can lead to dramatically different interpretations without changing the label. We extend these results to show that interpretations based on exemplars (e.g. influence functions) are similarly susceptible to adversarial attack. Our analysis of the geometry of the Hessian matrix gives insight on why robustness is a general challenge to current interpretation approaches.

scholarly journals Big data, machine learning and artificial intelligence: a neurologist’s guide

2020 ◽  
pp. practneurol-2020-002688
Author(s):  
Stephen D Auger ◽  
Benjamin M Jacobs ◽  
Ruth Dobson ◽  
Charles R Marshall ◽  
Alastair J Noyce
Keyword(s):  
Artificial Intelligence ◽  
Machine Learning ◽  
Neural Networks ◽  
Big Data ◽  
Clinical Practice ◽  
Clinical Data ◽  
Learning Algorithm ◽  
Machine Learning Algorithm ◽  
Basic Principles ◽  
Biological Neural Networks

Modern clinical practice requires the integration and interpretation of ever-expanding volumes of clinical data. There is, therefore, an imperative to develop efficient ways to process and understand these large amounts of data. Neurologists work to understand the function of biological neural networks, but artificial neural networks and other forms of machine learning algorithm are likely to be increasingly encountered in clinical practice. As their use increases, clinicians will need to understand the basic principles and common types of algorithm. We aim to provide a coherent introduction to this jargon-heavy subject and equip neurologists with the tools to understand, critically appraise and apply insights from this burgeoning field.

Attenuation of random noise using denoising convolutional neural networks

2019 ◽  
Vol 7 (3) ◽  
pp. SE269-SE280
Author(s):  
Xu Si ◽  
Yijun Yuan ◽  
Tinghua Si ◽  
Shiwen Gao
Keyword(s):  
Neural Network ◽  
Data Processing ◽  
Seismic Data ◽  
Random Noise ◽  
Training Data ◽  
Prediction Errors ◽  
Network Learning ◽  
Deep Network ◽  
Residual Image ◽  
And Training

Random noise often contaminates seismic data and reduces its signal-to-noise ratio. Therefore, the removal of random noise has been an essential step in seismic data processing. The [Formula: see text]-[Formula: see text] predictive filtering method is one of the most widely used methods in suppressing random noise. However, when the subsurface structure becomes complex, this method suffers from higher prediction errors owing to the large number of different dip components that need to be predicted. Here, we used a denoising convolutional neural network (DnCNN) algorithm to attenuate random noise in seismic data. This method does not assume the linearity and stationarity of the signal in the conventional [Formula: see text]-[Formula: see text] domain prediction technique, and it involves creating a set of training data that are obtained by data processing, feeding the neural network with the training data obtained, and deep network learning and training. During deep network learning and training, the activation function and batch normalization are used to solve the gradient vanishing and gradient explosion problems, and the residual learning technique is used to improve the calculation precision, respectively. After finishing deep network learning and training, the network will have the ability to separate the residual image from the seismic data with noise. Then, clean images can be obtained by subtracting the residual image from the raw data with noise. Tests on the synthetic and real data demonstrate that the DnCNN algorithm is very effective for random noise attenuation in seismic data.

Sign in / Sign up

Export Citation Format

Share Document