scholarly journals Combination of deep neural network with attention mechanism enhances the explainability of protein contact prediction

2020 ◽  
Author(s):  
Chen Chen ◽  
Tianqi Wu ◽  
Zhiye Guo ◽  
Jianlin Cheng
Keyword(s):  
Neural Network ◽  
Deep Learning ◽  
Attention Mechanism ◽  
Contact Prediction ◽  
Residue Contact ◽  
Complementary Effect ◽  
Essential Components ◽  
Internal Mechanism ◽  
Contact Predictions ◽  
The Relationship

AbstractDeep learning has emerged as a revolutionary technology for protein residue-residue contact prediction since the 2012 CASP10 competition. Considerable advancements in the predictive power of the deep learning-based contact predictions have been achieved since then. However, little effort has been put into interpreting the black-box deep learning methods. Algorithms that can interpret the relationship between predicted contact maps and the internal mechanism of the deep learning architectures are needed to explore the essential components of contact inference and improve their explainability. In this study, we present an attention-based convolutional neural network for protein contact prediction, which consists of two attention mechanism-based modules: sequence attention and regional attention. Our benchmark results on the CASP13 free-modeling (FM) targets demonstrate that the two attention modules added on top of existing typical deep learning models exhibit a complementary effect that contributes to predictive improvements. More importantly, the inclusion of the attention mechanism provides interpretable patterns that contain useful insights into the key fold-determining residues in proteins. We expect the attention-based model can provide a reliable and practically interpretable technique that helps break the current bottlenecks in explaining deep neural networks for contact prediction.

scholarly journals PconsC4: fast, free, easy, and accurate contact predictions

2018 ◽  
Author(s):  
Mirco Michel ◽  
David Menéndez Hurtado ◽  
Arne Elofsson
Keyword(s):  
Deep Learning ◽  
State Of The Art ◽  
Prediction Methods ◽  
Coupling Analysis ◽  
Learning Methods ◽  
Contact Prediction ◽  
Residue Contact ◽  
Direct Coupling Analysis ◽  
Computationally Expensive ◽  
Contact Predictions

AbstractMotivationResidue contact prediction was revolutionized recently by the introduction of direct coupling analysis (DCA). Further improvements, in particular for small families, have been obtained by the combination of DCA and deep learning methods. However, existing deep learning contact prediction methods often rely on a number of external programs and are therefore computationally expensive.ResultsHere, we introduce a novel contact predictor, PconsC4, which performs on par with state of the art methods. PconsC4 is heavily optimized, does not use any external programs and therefore is significantly faster and easier to use than other methods.AvailabilityPconsC4 is freely available under the GPL license from https://github.com/ElofssonLab/PconsC4. Installation is easy using the pip command and works on any system with Python 3.5 or later and a modern GCC [email protected]

PconsC4: fast, accurate and hassle-free contact predictions

2018 ◽  
Vol 35 (15) ◽  
pp. 2677-2679 ◽  
Author(s):  
Mirco Michel ◽  
David Menéndez Hurtado ◽  
Arne Elofsson
Keyword(s):  
Deep Learning ◽  
State Of The Art ◽  
Supplementary Information ◽  
Prediction Methods ◽  
Coupling Analysis ◽  
Contact Prediction ◽  
Residue Contact ◽  
Direct Coupling Analysis ◽  
Computationally Expensive ◽  
Contact Predictions

Abstract Motivation Residue contact prediction was revolutionized recently by the introduction of direct coupling analysis (DCA). Further improvements, in particular for small families, have been obtained by the combination of DCA and deep learning methods. However, existing deep learning contact prediction methods often rely on a number of external programs and are therefore computationally expensive. Results Here, we introduce a novel contact predictor, PconsC4, which performs on par with state of the art methods. PconsC4 is heavily optimized, does not use any external programs and therefore is significantly faster and easier to use than other methods. Availability and implementation PconsC4 is freely available under the GPL license from https://github.com/ElofssonLab/PconsC4. Installation is easy using the pip command and works on any system with Python 3.5 or later and a GCC compiler. It does not require a GPU nor special hardware. Supplementary information Supplementary data are available at Bioinformatics online.

scholarly journals An Attention Mechanism Oriented Hybrid CNN-RNN Deep Learning Architecture of Container Terminal Liner Handling Conditions Prediction

2021 ◽  
Vol 2021 ◽  
pp. 1-17
Author(s):  
Bin Li ◽  
Yuqing He
Keyword(s):  
Neural Network ◽  
Deep Learning ◽  
Container Terminal ◽  
Computational Thinking ◽  
Handling Time ◽  
Attention Mechanism ◽  
Operating Conditions ◽  
Forecasting Error ◽  
Design Paradigm ◽  
Architecture And Design

The booming computational thinking and deep learning make it possible to construct agile, efficient, and robust deep learning-driven decision-making support engine for the operation of container terminal handling systems (CTHSs). Within the conceptual framework of computational logistics, an attention mechanism oriented hybrid convolutional neural network and recurrent neural network deep learning architecture (AMO-HCR-DLA) is proposed technically to predict the container terminal liner handling conditions that mainly include liner handling time (LHT) and total working time of quay crane farm (TWT-QCF) for a calling liner. Consequently, the container terminal oriented logistics generalized computation (CTO-LGC) automation and intelligence are established tentatively by AMO-HCR-DLA. A typical regional container terminal hub of China is selected to design, implement, execute, and evaluate the AMO-HCR-DLA with the actual production data. In the case of severe vibration of LHT and TWT-QCF, while forecasting the handling conditions of 210 ships based on the CTO-LGC running log of four years, the forecasting error of LHT within one hour is more than 97% and that of TWT-QCF within six hours accounts for 89.405%. When predicting the operating conditions of 300 liners by the log of five years, the forecasting deviation of LHT within one hour is more than striking 99% and that of TWT-QCF within six hours reaches up to 94.010% as well. All are far superior to the predicting outcomes by the classical algorithms of machine learning and deep learning. Hence, the AMO-HCR-DLA shows excellent performance for the prediction of CTHS with the low and stable computational consuming. It also demonstrates the feasibility, credibility, and realizability of the computing architecture and design paradigm of AMO-HCR-DLA preliminarily.

High accuracy offering attention mechanisms based deep learning approach using CNN/bi-LSTM for sentiment analysis

2021 ◽  
Vol ahead-of-print (ahead-of-print) ◽  
Author(s):  
Venkateswara Rao Kota ◽  
Shyamala Devi Munisamy
Keyword(s):  
Neural Network ◽  
Deep Learning ◽  
Natural Language Processing ◽  
Natural Language ◽  
Sentiment Analysis ◽  
Language Processing ◽  
Attention Mechanism ◽  
Supervised Machine Learning ◽  
Method Performance ◽  
Content Type

PurposeNeural network (NN)-based deep learning (DL) approach is considered for sentiment analysis (SA) by incorporating convolutional neural network (CNN), bi-directional long short-term memory (Bi-LSTM) and attention methods. Unlike the conventional supervised machine learning natural language processing algorithms, the authors have used unsupervised deep learning algorithms.Design/methodology/approachThe method presented for sentiment analysis is designed using CNN, Bi-LSTM and the attention mechanism. Word2vec word embedding is used for natural language processing (NLP). The discussed approach is designed for sentence-level SA which consists of one embedding layer, two convolutional layers with max-pooling, one LSTM layer and two fully connected (FC) layers. Overall the system training time is 30 min.FindingsThe method performance is analyzed using metrics like precision, recall, F1 score, and accuracy. CNN is helped to reduce the complexity and Bi-LSTM is helped to process the long sequence input text.Originality/valueThe attention mechanism is adopted to decide the significance of every hidden state and give a weighted sum of all the features fed as input.

scholarly journals On the Relationship between Generalization and Robustness to Adversarial Examples

Symmetry ◽  
2021 ◽  
Vol 13 (5) ◽  
pp. 817
Author(s):  
Anibal Pedraza ◽  
Oscar Deniz ◽  
Gloria Bueno
Keyword(s):  
Neural Network ◽  
Deep Learning ◽  
Network Architecture ◽  
Neural Network Architecture ◽  
Trade Off ◽  
Adversarial Examples ◽  
Simultaneous Loss ◽  
The Relationship

One of the most intriguing phenomenons related to deep learning is the so-called adversarial examples. These samples are visually equivalent to normal inputs, undetectable for humans, yet they cause the networks to output wrong results. The phenomenon can be framed as a symmetry/asymmetry problem, whereby inputs to a neural network with a similar/symmetric appearance to regular images, produce an opposite/asymmetric output. Some researchers are focused on developing methods for generating adversarial examples, while others propose defense methods. In parallel, there is a growing interest in characterizing the phenomenon, which is also the focus of this paper. From some well known datasets of common images, like CIFAR-10 and STL-10, a neural network architecture is first trained in a normal regime, where training and validation performances increase, reaching generalization. Additionally, the same architectures and datasets are trained in an overfitting regime, where there is a growing disparity in training and validation performances. The behaviour of these two regimes against adversarial examples is then compared. From the results, we observe greater robustness to adversarial examples in the overfitting regime. We explain this simultaneous loss of generalization and gain in robustness to adversarial examples as another manifestation of the well-known fitting-generalization trade-off.

scholarly journals Application of an emotional classification model in e-commerce text based on an improved transformer model

PLoS ONE ◽  
2021 ◽  
Vol 16 (3) ◽  
pp. e0247984
Author(s):  
Xuyang Wang ◽  
Yixuan Tong
Keyword(s):  
Neural Network ◽  
Deep Learning ◽  
Text Classification ◽  
Rapid Development ◽  
Attention Mechanism ◽  
Mobile Internet ◽  
Sentiment Classification ◽  
Classification Model ◽  
Test Model ◽  
Transformer Model

With the rapid development of the mobile internet, people are becoming more dependent on the internet to express their comments on products or stores; meanwhile, text sentiment classification of these comments has become a research hotspot. In existing methods, it is fairly popular to apply a deep learning method to the text classification task. Aiming at solving information loss, weak context and other problems, this paper makes an improvement based on the transformer model to reduce the difficulty of model training and training time cost and achieve higher overall model recall and accuracy in text sentiment classification. The transformer model replaces the traditional convolutional neural network (CNN) and the recurrent neural network (RNN) and is fully based on the attention mechanism; therefore, the transformer model effectively improves the training speed and reduces training difficulty. This paper selects e-commerce reviews as research objects and applies deep learning theory. First, the text is preprocessed by word vectorization. Then the IN standardized method and the GELUs activation function are applied based on the original model to analyze the emotional tendencies of online users towards stores or products. The experimental results show that our method improves by 9.71%, 6.05%, 5.58% and 5.12% in terms of recall and approaches the peak level of the F1 value in the test model by comparing BiLSTM, Naive Bayesian Model, the serial BiLSTM_CNN model and BiLSTM with an attention mechanism model. Therefore, this finding proves that our method can be used to improve the text sentiment classification accuracy and effectively apply the method to text classification.

scholarly journals An attention-based deep learning model for clinical named entity recognition of Chinese electronic medical records

2019 ◽  
Vol 19 (S5) ◽  
Author(s):  
Luqi Li ◽  
Jie Zhao ◽  
Li Hou ◽  
Yunkai Zhai ◽  
Jinming Shi ◽  
...  
Keyword(s):  
Neural Network ◽  
Deep Learning ◽  
Electronic Medical Records ◽  
Medical Records ◽  
Named Entity Recognition ◽  
Attention Mechanism ◽  
Entity Recognition ◽  
Long Distance ◽  
Named Entity ◽  
Text Features

Abstract Background Clinical named entity recognition (CNER) is important for medical information mining and establishment of high-quality knowledge map. Due to the different text features from natural language and a large number of professional and uncommon clinical terms in Chinese electronic medical records (EMRs), there are still many difficulties in clinical named entity recognition of Chinese EMRs. It is of great importance to eliminate semantic interference and improve the ability of autonomous learning of internal features of the model under the small training corpus. Methods From the perspective of deep learning, we integrated the attention mechanism into neural network, and proposed an improved clinical named entity recognition method for Chinese electronic medical records called BiLSTM-Att-CRF, which could capture more useful information of the context and avoid the problem of missing information caused by long-distance factors. In addition, medical dictionaries and part-of-speech (POS) features were also introduced to improve the performance of the model. Results Based on China Conference on Knowledge Graph and Semantic Computing (CCKS) 2017 and 2018 Chinese EMRs corpus, our BiLSTM-Att-CRF model finally achieved better performance than other widely-used models without additional features(F1-measure of 85.4% in CCKS 2018, F1-measure of 90.29% in CCKS 2017), and achieved the best performance with POS and dictionary features (F1-measure of 86.11% in CCKS 2018, F1-measure of 90.48% in CCKS 2017). In particular, the BiLSTM-Att-CRF model had significant effect on the improvement of Recall. Conclusions Our work preliminarily confirmed the validity of attention mechanism in discovering key information and mining text features, which might provide useful ideas for future research in clinical named entity recognition of Chinese electronic medical records. In the future, we will explore the deeper application of attention mechanism in neural network.

scholarly journals COMTOP: Protein Residue–Residue Contact Prediction through Mixed Integer Linear Optimization

Membranes ◽  
2021 ◽  
Vol 11 (7) ◽  
pp. 503
Author(s):  
Md. Selim Reza ◽  
Huiling Zhang ◽  
Md. Tofazzal Hossain ◽  
Langxi Jin ◽  
Shengzhong Feng ◽  
...  
Keyword(s):  
Prediction Accuracy ◽  
Linear Optimization ◽  
Mixed Integer ◽  
Consensus Method ◽  
Contact Prediction ◽  
Residue Contact ◽  
Mixed Integer Linear Optimization ◽  
Test Sets ◽  
Contact Predictions ◽  
Integer Linear Optimization

Protein contact prediction helps reconstruct the tertiary structure that greatly determines a protein’s function; therefore, contact prediction from the sequence is an important problem. Recently there has been exciting progress on this problem, but many of the existing methods are still low quality of prediction accuracy. In this paper, we present a new mixed integer linear programming (MILP)-based consensus method: a Consensus scheme based On a Mixed integer linear opTimization method for prOtein contact Prediction (COMTOP). The MILP-based consensus method combines the strengths of seven selected protein contact prediction methods, including CCMpred, EVfold, DeepCov, NNcon, PconsC4, plmDCA, and PSICOV, by optimizing the number of correctly predicted contacts and achieving a better prediction accuracy. The proposed hybrid protein residue–residue contact prediction scheme was tested in four independent test sets. For 239 highly non-redundant proteins, the method showed a prediction accuracy of 59.68%, 70.79%, 78.86%, 89.04%, 94.51%, and 97.35% for top-5L, top-3L, top-2L, top-L, top-L/2, and top-L/5 contacts, respectively. When tested on the CASP13 and CASP14 test sets, the proposed method obtained accuracies of 75.91% and 77.49% for top-L/5 predictions, respectively. COMTOP was further tested on 57 non-redundant ɑ-helical transmembrane proteins and achieved prediction accuracies of 64.34% and 73.91% for top-L/2 and top-L/5 predictions, respectively. For all test datasets, the improvement of COMTOP in accuracy over the seven individual methods increased with the increasing number of predicted contacts. For example, COMTOP performed much better for large number of contact predictions (such as top-5L and top-3L) than for small number of contact predictions such as top-L/2 and top-L/5. The results and analysis demonstrate that COMTOP can significantly improve the performance of the individual methods; therefore, COMTOP is more robust against different types of test sets. COMTOP also showed better/comparable predictions when compared with the state-of-the-art predictors.

scholarly journals Protein model accuracy estimation empowered by deep learning and inter-residue distance prediction in CASP14

2021 ◽  
Author(s):  
Xiao Chen ◽  
Jian Liu ◽  
Zhiye Guo ◽  
Tianqi Wu ◽  
Jie Hou ◽  
...  
Keyword(s):  
Deep Learning ◽  
Structure Prediction ◽  
Structural Models ◽  
Single Model ◽  
Model Accuracy ◽  
Model Quality ◽  
Residue Contact ◽  
Contact Distance ◽  
Protein Model ◽  
Contact Predictions

AbstractThe inter-residue contact prediction and deep learning showed the promise to improve the estimation of protein model accuracy (EMA) in the 13th Critical Assessment of Protein Structure Prediction (CASP13). During the 2020 CASP14 experiment, we developed and tested several EMA predictors that used deep learning with the new features based on inter-residue distance/contact predictions as well as the existing model quality features. The average global distance test (GDT-TS) score loss of ranking CASP14 structural models by three multi-model MULTICOM EMA predictors (MULTICOM-CONSTRUCT, MULTICOM-AI, and MULTICOM-CLUSTER) is 0.073, 0.079, and 0.081, respectively, which are ranked first, second, and third places out of 68 CASP14 EMA predictors. The single-model EMA predictor (MULTICOM-DEEP) is ranked 10th place among all the single-model EMA methods in terms of GDT_TS score loss. The results show that deep learning and contact/distance predictions are useful in ranking and selecting protein structural models.

Sign in / Sign up

Export Citation Format

Share Document