Asynchronous stochastic optimization for sequence training of deep neural networks: towards big data

2014 ◽  
Author(s):  
Erik McDermott ◽  
Georg Heigold ◽  
Pedro J. Moreno ◽  
Andrew Senior ◽  
Michiel Bacchiani
Keyword(s):  
Neural Networks ◽  
Big Data ◽  
Stochastic Optimization ◽  
Deep Neural Networks

scholarly journals Enabling deeper learning on big data for materials informatics applications

2021 ◽  
Vol 11 (1) ◽  
Author(s):  
Dipendra Jha ◽  
Vishu Gupta ◽  
Logan Ward ◽  
Zijiang Yang ◽  
Christopher Wolverton ◽  
...  
Keyword(s):  
Neural Networks ◽  
Big Data ◽  
Deep Learning ◽  
Deep Neural Networks ◽  
Materials Science ◽  
Prediction Models ◽  
Model Performance ◽  
Materials Informatics ◽  
Learning Framework ◽  
Significant Attention

AbstractThe application of machine learning (ML) techniques in materials science has attracted significant attention in recent years, due to their impressive ability to efficiently extract data-driven linkages from various input materials representations to their output properties. While the application of traditional ML techniques has become quite ubiquitous, there have been limited applications of more advanced deep learning (DL) techniques, primarily because big materials datasets are relatively rare. Given the demonstrated potential and advantages of DL and the increasing availability of big materials datasets, it is attractive to go for deeper neural networks in a bid to boost model performance, but in reality, it leads to performance degradation due to the vanishing gradient problem. In this paper, we address the question of how to enable deeper learning for cases where big materials data is available. Here, we present a general deep learning framework based on Individual Residual learning (IRNet) composed of very deep neural networks that can work with any vector-based materials representation as input to build accurate property prediction models. We find that the proposed IRNet models can not only successfully alleviate the vanishing gradient problem and enable deeper learning, but also lead to significantly (up to 47%) better model accuracy as compared to plain deep neural networks and traditional ML techniques for a given input materials representation in the presence of big data.

scholarly journals Critical Assessment of Artificial Intelligence Methods for Prediction of hERG Channel Inhibition in the ‘Big Data’ Era

2020 ◽  
Author(s):  
Vishal Babu Siramshetty ◽  
Dac-Trung Nguyen ◽  
Natalia J. Martinez ◽  
Anton Simeonov ◽  
Noel T. Southall ◽  
...  
Keyword(s):  
Artificial Intelligence ◽  
Neural Networks ◽  
Big Data ◽  
Recurrent Neural Networks ◽  
Deep Neural Networks ◽  
Prediction Models ◽  
Chemical Space ◽  
Superior Performance ◽  
Gradient Boosting ◽  
Artificial Intelligence Methods

The rise of novel artificial intelligence methods necessitates a comparison of this wave of new approaches with classical machine learning for a typical drug discovery project. Inhibition of the potassium ion channel, whose alpha subunit is encoded by human Ether-à-go-go-Related Gene (hERG), leads to prolonged QT interval of the cardiac action potential and is a significant safety pharmacology target for the development of new medicines. Several computational approaches have been employed to develop prediction models for assessment of hERG liabilities of small molecules including recent work using deep learning methods. Here we perform a comprehensive comparison of prediction models based on classical (random forests and gradient boosting) and modern (deep neural networks and recurrent neural networks) artificial intelligence methods. The training set (~9000 compounds) was compiled by integrating hERG bioactivity data from ChEMBL database with experimental data generated from an in-house, high-throughput thallium flux assay. We utilized different molecular descriptors including the latent descriptors, which are real-valued continuous vectors derived from chemical autoencoders trained on a large chemical space (> 1.5 million compounds). The models were prospectively validated on ~840 in-house compounds screened in the same thallium flux assay. The deep neural networks performed significantly better than the classical methods with the latent descriptors. The recurrent neural networks that operate on SMILES provided highest model sensitivity. The best models were merged into a consensus model that offered superior performance compared to reference models from academic and commercial domains. Further, we shed light on the potential of artificial intelligence methods to exploit the chemistry big data and generate novel chemical representations useful in predictive modeling and tailoring new chemical space.<br>

PID Controller-Based Stochastic Optimization Acceleration for Deep Neural Networks

2020 ◽  
Vol 31 (12) ◽  
pp. 5079-5091 ◽  
Author(s):  
Haoqian Wang ◽  
Yi Luo ◽  
Wangpeng An ◽  
Qingyun Sun ◽  
Jun Xu ◽  
...  
Keyword(s):  
Neural Networks ◽  
Stochastic Optimization ◽  
Pid Controller ◽  
Deep Neural Networks

scholarly journals Chatbots Employing Deep Learning for Big Data

2019 ◽  
Vol 8 (11) ◽  
pp. 1005-1010
Keyword(s):  
Artificial Intelligence ◽  
Neural Networks ◽  
Big Data ◽  
Deep Learning ◽  
Natural Language Processing ◽  
Language Processing ◽  
Deep Neural Networks ◽  
Heterogeneous Data ◽  
Instructive Feedback ◽  
Technical Specifications

With the evolution of artificial intelligence to deep learning, the age of perspicacious machines has pioneered that can even mimic as a human. A Conversational software agent is one of the best-suited examples of such intuitive machines which are also commonly known as chatbot actuated with natural language processing. The paper enlisted some existing popular chatbots along with their details, technical specifications, and functionalities. Research shows that most of the customers have experienced penurious service. Also, the inception of meaningful cum instructive feedback endure a demanding and exigent assignment as enactment for chatbots builtout reckon mostly upon templates and hand-written rules. Current chatbot models lack in generating required responses and thus contradict the quality conversation. So involving deep learning amongst these models can overcome this lack and can fill up the paucity with deep neural networks. Some of the deep Neural networks utilized for this till now are Stacked Auto-Encoder, sparse auto-encoders, predictive sparse and denoising auto-encoders. But these DNN are unable to handle big data involving large amounts of heterogeneous data. While Tensor Auto Encoder which overcomes this drawback is time-consuming. This paper has proposed the Chatbot to handle the big data in a manageable time.

scholarly journals Adaptive Learning Rate via Covariance Matrix Based Preconditioning for Deep Neural Networks

2017 ◽  
Author(s):  
Yasutoshi Ida ◽  
Yasuhiro Fujiwara ◽  
Sotetsu Iwamura
Keyword(s):  
Neural Networks ◽  
Stochastic Optimization ◽  
Covariance Matrix ◽  
Adaptive Learning ◽  
Deep Neural Networks ◽  
Learning Rate ◽  
First Order ◽  
Adaptive Learning Rate ◽  
Approximate Hessian

Adaptive learning rate algorithms such as RMSProp are widely used for training deep neural networks. RMSProp offers efficient training since it uses first order gradients to approximate Hessian-based preconditioning. However, since the first order gradients include noise caused by stochastic optimization, the approximation may be inaccurate. In this paper, we propose a novel adaptive learning rate algorithm called SDProp. Its key idea is effective handling of the noise by preconditioning based on covariance matrix. For various neural networks, our approach is more efficient and effective than RMSProp and its variant.

Sign in / Sign up

Export Citation Format

Share Document