scholarly journals Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences

2021 ◽  
Vol 118 (15) ◽  
pp. e2016239118
Author(s):  
Alexander Rives ◽  
Joshua Meier ◽  
Tom Sercu ◽  
Siddharth Goyal ◽  
Zeming Lin ◽  
...  
Keyword(s):  
Artificial Intelligence ◽  
Amino Acids ◽  
Unsupervised Learning ◽  
Tertiary Structure ◽  
State Of The Art ◽  
Sequence Data ◽  
Protein Sequences ◽  
Representation Learning ◽  
Biological Properties ◽  
Biochemical Properties

In the field of artificial intelligence, a combination of scale in data and model capacity enabled by unsupervised learning has led to major advances in representation learning and statistical generation. In the life sciences, the anticipated growth of sequencing promises unprecedented data on natural sequence diversity. Protein language modeling at the scale of evolution is a logical step toward predictive and generative artificial intelligence for biology. To this end, we use unsupervised learning to train a deep contextual language model on 86 billion amino acids across 250 million protein sequences spanning evolutionary diversity. The resulting model contains information about biological properties in its representations. The representations are learned from sequence data alone. The learned representation space has a multiscale organization reflecting structure from the level of biochemical properties of amino acids to remote homology of proteins. Information about secondary and tertiary structure is encoded in the representations and can be identified by linear projections. Representation learning produces features that generalize across a range of applications, enabling state-of-the-art supervised prediction of mutational effect and secondary structure and improving state-of-the-art features for long-range contact prediction.

scholarly journals Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences

2019 ◽  
Author(s):  
Alexander Rives ◽  
Joshua Meier ◽  
Tom Sercu ◽  
Siddharth Goyal ◽  
Zeming Lin ◽  
...  
Keyword(s):  
Artificial Intelligence ◽  
Amino Acids ◽  
Unsupervised Learning ◽  
Tertiary Structure ◽  
State Of The Art ◽  
Sequence Data ◽  
Protein Sequences ◽  
Representation Learning ◽  
Biological Properties ◽  
Biochemical Properties

In the field of artificial intelligence, a combination of scale in data and model capacity enabled by unsupervised learning has led to major advances in representation learning and statistical generation. In the life sciences, the anticipated growth of sequencing promises unprecedented data on natural sequence diversity. Protein language modeling at the scale of evolution is a logical step toward predictive and generative artificial intelligence for biology. To this end we use unsupervised learning to train a deep contextual language model on 86 billion amino acids across 250 million protein sequences spanning evolutionary diversity. The resulting model contains information about biological properties in its representations. The representations are learned from sequence data alone. The learned representation space has a multi-scale organization reflecting structure from the level of biochemical properties of amino acids to remote homology of proteins. Information about secondary and tertiary structure is encoded in the representations and can be identified by linear projections. Representation learning produces features that generalize across a range of applications, enabling state-of-the-art supervised prediction of mutational effect and secondary structure, and improving state-of-the-art features for long-range contact prediction.

scholarly journals Protein sequence profile prediction using ProtAlbert transformer1

2021 ◽  
Author(s):  
Fatemeh Zare-Mirakabad ◽  
Armin Behjati ◽  
Seyed Shahriar Arab ◽  
Abbas Nowzari-Dalini
Keyword(s):  
Amino Acids ◽  
Protein Sequence ◽  
Nearest Neighbor ◽  
Tertiary Structure ◽  
Query Sequence ◽  
Protein Secondary Structure ◽  
Protein Sequences ◽  
Family Characteristics ◽  
Sequence Profile ◽  
Protein Sequence Profile

Protein sequences can be viewed as a language; therefore, we benefit from using the models initially developed for natural languages such as transformers. ProtAlbert is one of the best pre-trained transformers on protein sequences, and its efficiency enables us to run the model on longer sequences with less computation power while having similar performance with the other pre-trained transformers. This paper includes two main parts: transformer analysis and profile prediction. In the first part, we propose five algorithms to assess the attention heads in different layers of ProtAlbert for five protein characteristics, nearest-neighbor interactions, type of amino acids, biochemical and biophysical properties of amino acids, protein secondary structure, and protein tertiary structure. These algorithms are performed on 55 proteins extracted from CASP13 and three case study proteins whose sequences, experimental tertiary structures, and HSSP profiles are available. This assessment shows that although the model is only pre-trained on protein sequences, attention heads in the layers of ProtAlbert are representative of some protein family characteristics. This conclusion leads to the second part of our work. We propose an algorithm called PA_SPP for protein sequence profile prediction by pre-trained ProtAlbert using masked-language modeling. PA_SPP algorithm can help the researchers to predict an HSSP profile while there are no similar sequences to a query sequence in the database for making the HSSP profile.

scholarly journals Unsupervised Disentangled Representation Learning with Analogical Relations

2018 ◽  
Author(s):  
Zejian Li ◽  
Yongchuan Tang ◽  
Yongxing He
Keyword(s):  
Artificial Intelligence ◽  
Cognitive Processes ◽  
State Of The Art ◽  
Representation Learning ◽  
Subspace Learning ◽  
Generative Models ◽  
The State ◽  
The Other ◽  
Learning Methods ◽  
Training Strategy

Learning the disentangled representation of interpretable generative factors of data is one of the foundations to allow artificial intelligence to think like people. In this paper, we propose the analogical training strategy for the unsupervised disentangled representation learning in generative models. The analogy is one of the typical cognitive processes, and our proposed strategy is based on the observation that sample pairs in which one is different from the other in one specific generative factor show the same analogical relation. Thus, the generator is trained to generate sample pairs from which a designed classifier can identify the underlying analogical relation. In addition, we propose a disentanglement metric called the subspace score, which is inspired by subspace learning methods and does not require supervised information. Experiments show that our proposed training strategy allows the generative models to find the disentangled factors, and that our methods can give competitive performances as compared with the state-of-the-art methods.

scholarly journals Improving Generalizability of Protein Sequence Models with Data Augmentations

2021 ◽  
Author(s):  
Hongyu Shen ◽  
Layne C. Price ◽  
Taha Bahadori ◽  
Franziska Seeger
Keyword(s):  
Machine Learning ◽  
Protein Sequence ◽  
Data Augmentation ◽  
Sequence Data ◽  
Protein Sequences ◽  
Representation Learning ◽  
Amino Acid Replacement ◽  
Fine Tuning ◽  
Protein Sequence Data ◽  
Tuning Methods

AbstractWhile protein sequence data is an emerging application domain for machine learning methods, small modifications to protein sequences can result in difficult-to-predict changes to the protein’s function. Consequently, protein machine learning models typically do not use randomized data augmentation procedures analogous to those used in computer vision or natural language, e.g., cropping or synonym substitution. In this paper, we empirically explore a set of simple string manipulations, which we use to augment protein sequence data when fine-tuning semi-supervised protein models. We provide 276 different comparisons to the Tasks Assessing Protein Embeddings (TAPE) baseline models, with Transformer-based models and training datasets that vary from the baseline methods only in the data augmentations and representation learning procedure. For each TAPE validation task, we demonstrate improvements to the baseline scores when the learned protein representation is fixed between tasks. We also show that contrastive learning fine-tuning methods typically outperform masked-token prediction in these models, with increasing amounts of data augmentation generally improving performance for contrastive learning protein methods. We find the most consistent results across TAPE tasks when using domain-motivated transformations, such as amino acid replacement, as well as restricting the Transformer attention to randomly sampled sub-regions of the protein sequence. In rarer cases, we even find that information-destroying augmentations, such as randomly shuffling entire protein sequences, can improve downstream performance.

scholarly journals Unsupervised Representation Learning by Predicting Random Distances

2020 ◽  
Author(s):  
Hu Wang ◽  
Guansong Pang ◽  
Chunhua Shen ◽  
Congbo Ma
Keyword(s):  
Neural Networks ◽  
Anomaly Detection ◽  
Unsupervised Learning ◽  
Large Scale ◽  
Deep Neural Networks ◽  
State Of The Art ◽  
Representation Learning ◽  
Great Success ◽  
Learning Tasks ◽  
Real World Datasets

Deep neural networks have gained great success in a broad range of tasks due to its remarkable capability to learn semantically rich features from high-dimensional data. However, they often require large-scale labelled data to successfully learn such features, which significantly hinders their adaption in unsupervised learning tasks, such as anomaly detection and clustering, and limits their applications to critical domains where obtaining massive labelled data is prohibitively expensive. To enable unsupervised learning on those domains, in this work we propose to learn features without using any labelled data by training neural networks to predict data distances in a randomly projected space. Random mapping is a theoretically proven approach to obtain approximately preserved distances. To well predict these distances, the representation learner is optimised to learn genuine class structures that are implicitly embedded in the randomly projected space. Empirical results on 19 real-world datasets show that our learned representations substantially outperform a few state-of-the-art methods for both anomaly detection and clustering tasks. Code is available at: \url{https://git.io/RDP}

scholarly journals Language models enable zero-shot prediction of the effects of mutations on protein function

2021 ◽  
Author(s):  
Joshua Meier ◽  
Roshan Rao ◽  
Robert Verkuil ◽  
Jason Liu ◽  
Tom Sercu ◽  
...  
Keyword(s):  
Experimental Data ◽  
Protein Function ◽  
Sequence Variation ◽  
State Of The Art ◽  
Sequence Data ◽  
Fundamental Problem ◽  
Protein Sequences ◽  
Language Models ◽  
New Model ◽  
Prediction Task

Modeling the effect of sequence variation on function is a fundamental problem for understanding and designing proteins. Since evolution encodes information about function into patterns in protein sequences, unsupervised models of variant effects can be learned from sequence data. The approach to date has been to fit a model to a family of related sequences. The conventional setting is limited, since a new model must be trained for each prediction task. We show that using only zero-shot inference, without any supervision from experimental data or additional training, protein language models capture the functional effects of sequence variation, performing at state-of-the-art.

Multi-Resolution Autoregressive Graph-to-Graph Translation for Molecules

2019 ◽  
Author(s):  
Wengong Jin ◽  
Regina Barzilay ◽  
Tommi S Jaakkola
Keyword(s):  
Drug Discovery ◽  
State Of The Art ◽  
Molecular Graph ◽  
Biochemical Properties ◽  
Large Margin ◽  
Previous State ◽  
Translation Methods ◽  
Atom Level ◽  
Precursor Molecules ◽  
Prior State

The problem of accelerating drug discovery relies heavily on automatic tools to optimize precursor molecules to afford them with better biochemical properties. Our work in this paper substantially extends prior state-of-the-art on graph-to-graph translation methods for molecular optimization. In particular, we realize coherent multi-resolution representations by interweaving trees over substructures with the atom-level encoding of the original molecular graph. Moreover, our graph decoder is fully autoregressive, and interleaves each step of adding a new substructure with the process of resolving its connectivity to the emerging molecule. We evaluate our model on multiple molecular optimization tasks and show that our model outperforms previous state-of-the-art baselines by a large margin.

Bio-organometallic Peptide Conjugates: Recent Advances in Their Synthesis and Prospects for Biomedical Application

2020 ◽  
Vol 24 (21) ◽  
pp. 2508-2523
Author(s):  
Johana Gómez ◽  
Diego Sierra ◽  
Constanza Cárdenas ◽  
Fanny Guzmán
Keyword(s):  
Amino Acids ◽  
Biomedical Application ◽  
Structural Diversity ◽  
Biological Properties ◽  
Therapeutic Agents ◽  
Organometallic Complexes ◽  
Chemical Behavior ◽  
Metal Compounds ◽  
Bioorganometallic Chemistry ◽  
Pharmacological Control

One area of organometallic chemistry that has attracted great interest in recent years is the syntheses, characterization and study of organometallic complexes conjugated to biomolecules with different steric and electronic properties as potential therapeutic agents against cancer and malaria, as antibiotics and as radiopharmaceuticals. This minireview focuses on the unique structural diversity that has recently been discovered in α- amino acids and the reactions of metallocene complexes with peptides having different chemical behavior and potential medical applications. Replacing α-amino acids with metallocene fragments is an effective way of selectively influencing the physicochemical, structural, electrochemical and biological properties of the peptides. Consequently, research in the field of bioorganometallic chemistry offers the opportunity to develop bioactive metal compounds as an innovative and promising approach in the search for pharmacological control of different diseases.

A Study on Host Tropism Determinants of Influenza Virus Using Machine Learning

2020 ◽  
Vol 15 (2) ◽  
pp. 121-134 ◽  
Author(s):  
Eunmi Kwon ◽  
Myeongji Cho ◽  
Hayeon Kim ◽  
Hyeon S. Son
Keyword(s):  
Machine Learning ◽  
Amino Acids ◽  
Influenza Virus ◽  
Random Forest ◽  
Physicochemical Properties ◽  
Protein Sequences ◽  
Influenza Viruses ◽  
Host Tropism ◽  
Post Hoc ◽  
Ha Protein

Background: The host tropism determinants of influenza virus, which cause changes in the host range and increase the likelihood of interaction with specific hosts, are critical for understanding the infection and propagation of the virus in diverse host species. Methods: Six types of protein sequences of influenza viral strains isolated from three classes of hosts (avian, human, and swine) were obtained. Random forest, naïve Bayes classification, and knearest neighbor algorithms were used for host classification. The Java language was used for sequence analysis programming and identifying host-specific position markers. Results: A machine learning technique was explored to derive the physicochemical properties of amino acids used in host classification and prediction. HA protein was found to play the most important role in determining host tropism of the influenza virus, and the random forest method yielded the highest accuracy in host prediction. Conserved amino acids that exhibited host-specific differences were also selected and verified, and they were found to be useful position markers for host classification. Finally, ANOVA analysis and post-hoc testing revealed that the physicochemical properties of amino acids, comprising protein sequences combined with position markers, differed significantly among hosts. Conclusion: The host tropism determinants and position markers described in this study can be used in related research to classify, identify, and predict the hosts of influenza viruses that are currently susceptible or likely to be infected in the future.

Sign in / Sign up

Export Citation Format

Share Document