Memory-assisted reinforcement learning for diverse molecular de novo design

Abstract In de novo molecular design, recurrent neural networks (RNN) have been shown to be effective methods for sampling and generating novel chemical structures. Using a technique called reinforcement learning (RL), an RNN can be tuned to target a particular section of chemical space with optimized desirable properties using a scoring function. However, ligands generated by current RL methods so far tend to have relatively low diversity, and sometimes even result in duplicate structures when optimizing towards desired properties. Here, we propose a new method to address the low diversity issue in RL for molecular design. Memory-assisted RL is an extension of the known RL, with the introduction of a so-called memory unit. As proof of concept, we applied our method to generate structures with a desired AlogP value. In a second case study, we applied our method to design ligands for the dopamine type 2 receptor and the 5-hydroxytryptamine type 1A receptor. For both receptors, a machine learning model was developed to predict whether generated molecules were active or not for the receptor. In both case studies, it was found that memory-assisted RL led to the generation of more compounds predicted to be active having higher chemical diversity, thus achieving better coverage of chemical space of known ligands compared to established RL methods.

Download Full-text

Memory-assisted Reinforcement Learning for Diverse Molecular De Novo Design

10.21203/rs.3.rs-52871/v1 ◽

2020 ◽

Author(s):

Thomas Blaschke ◽

Ola Engkvist ◽

Jürgen Bajorath ◽

Hongming Chen

Keyword(s):

Reinforcement Learning ◽

De Novo ◽

Molecular Design ◽

Chemical Space ◽

Scoring Function ◽

Chemical Diversity ◽

Chemical Structures ◽

Low Diversity ◽

De Novo Molecular Design ◽

Dopamine 2 Receptor

Abstract In de novo molecular design, recurrent neural networks (RNN) have been shown to be effective methods for sampling and generating novel chemical structures. Using a technique called reinforcement learning (RL), an RNN can be tuned to target a particular section of chemical space with optimized desirable properties using a scoring function. However, ligands generated by current RL methods so far tend to have relatively low diversity, and sometimes even result in duplicate structures when optimizing towards particular properties. Here, we propose a new method to address the low diversity issue in RL. Memory-assisted RL is an extension of the known RL, with the introduction of a so-called memory unit. As proof of concept, we applied our method to generate structures with an optimized logP. In a second case study, we applied our method to design ligands for the dopamine 2 receptor and the 5-hydroxytryptamine 1A receptor. For both receptors, a machine learning model was developed to predict whether generated molecules were active or not for the receptor. In both case studies, it was found that memory-assisted RL led to the generation of more active compounds and with higher chemical diversity, thus achieving better coverage of chemical space of known ligands compared to established RL method.

Download Full-text

Memory-Assisted Reinforcement Learning for Diverse Molecular De Novo Design

10.26434/chemrxiv.12693152.v1 ◽

2020 ◽

Cited By ~ 1

Author(s):

Thomas Blaschke ◽

Ola Engkvist ◽

Jürgen Bajorath ◽

Hongming Chen

Keyword(s):

Reinforcement Learning ◽

De Novo ◽

Molecular Design ◽

Chemical Space ◽

Scoring Function ◽

De Novo Design ◽

Memory Unit ◽

Chemical Structures ◽

Low Diversity ◽

De Novo Molecular Design

<div><div><div><p>In de novo molecular design, recurrent neural networks (RNN) have been shown to be effective methods for sampling and generating novel chemical structures. Using a technique called reinforcement learning (RL), an RNN can be tuned to target a particular section of chemical space with optimized desirable properties using a scoring function. However, ligands generated by current RL methods so far tend to have relatively low diversity, and sometimes even result in duplicate structures when optimizing towards particular properties. Here, we propose a new method to address the low diversity issue in RL. Memory-assisted RL is an extension of the known RL, with the introduction of a so-called memory unit.</p></div></div></div>

Download Full-text

Memory-Assisted Reinforcement Learning for Diverse Molecular De Novo Design

10.26434/chemrxiv.12693152 ◽

2020 ◽

Author(s):

Thomas Blaschke ◽

Ola Engkvist ◽

Jürgen Bajorath ◽

Hongming Chen

Keyword(s):

Reinforcement Learning ◽

De Novo ◽

Molecular Design ◽

Chemical Space ◽

Scoring Function ◽

De Novo Design ◽

Memory Unit ◽

Chemical Structures ◽

Low Diversity ◽

De Novo Molecular Design

<div><div><div><p>In de novo molecular design, recurrent neural networks (RNN) have been shown to be effective methods for sampling and generating novel chemical structures. Using a technique called reinforcement learning (RL), an RNN can be tuned to target a particular section of chemical space with optimized desirable properties using a scoring function. However, ligands generated by current RL methods so far tend to have relatively low diversity, and sometimes even result in duplicate structures when optimizing towards particular properties. Here, we propose a new method to address the low diversity issue in RL. Memory-assisted RL is an extension of the known RL, with the introduction of a so-called memory unit.</p></div></div></div>

Download Full-text

An Exploration Strategy Improves the Diversity of de novo Ligands Using Deep Reinforcement Learning – A Case for the Adenosine A2A Receptor

10.26434/chemrxiv.7436789 ◽

2019 ◽

Author(s):

Xuhan Liu ◽

Kai Ye ◽

Herman Van Vlijmen ◽

Adriaan P. IJzerman ◽

Gerard JP Van westen

Keyword(s):

Reinforcement Learning ◽

Language Processing ◽

De Novo ◽

Chemical Space ◽

Chemical Diversity ◽

Reward Function ◽

Chemical Structures ◽

Exploration Strategy ◽

Machine Learning Model ◽

Adenosine A2a

<p></p><p>Over the last five years deep learning has progressed tremendously in both image recognition and natural language processing. Now it is increasingly applied to other data rich fields. In drug discovery, recurrent neural networks (RNNs) have been shown to be an effective method to generate novel chemical structures in the form of SMILES. However, ligands generated by current methods have so far provided relatively low diversity and do not fully cover the whole chemical space occupied by known ligands. Here, we propose a new method (DrugEx) to discover <i>de novo</i> drug-like molecules. DrugEx is an RNN model (generator) trained through reinforcement learning which was integrated with a special exploration strategy. As a case study we applied our method to design ligands against the adenosine A<sub>2A</sub> receptor. From ChEMBL data, a machine learning model (predictor) was created to predict whether generated molecules are active or not. Based on this predictor as the reward function, the generator was trained by reinforcement learning without any further data. We then compared the performance of our method with two previously published methods, REINVENT and ORGANIC. We found that candidate molecules our model designed, and predicted to be active, had a larger chemical diversity, and better covered the chemical space of known ligands compared to the state-of-the-art.</p><p></p>

Download Full-text

An Exploration Strategy Improves the Diversity of de novo Ligands Using Deep Reinforcement Learning – A Case for the 3 Adenosine A2A Receptor

10.26434/chemrxiv.7436789.v1 ◽

2018 ◽

Author(s):

Xuhan Liu ◽

Kai Ye ◽

Herman Van Vlijmen ◽

Adriaan P. IJzerman ◽

Gerard JP Van westen

Keyword(s):

Reinforcement Learning ◽

Language Processing ◽

De Novo ◽

Chemical Space ◽

Chemical Diversity ◽

Reward Function ◽

Chemical Structures ◽

Exploration Strategy ◽

Machine Learning Model ◽

Adenosine A2a

<p>Over the last five years deep learning has progressed tremendously in both image recognition and natural language processing. Now it is increasingly applied to other data rich fields. In drug discovery, recurrent neural networks (RNNs) have been shown to be an effective method to generate novel chemical structures in the form of SMILES. However, ligands generated by current methods used to provide relatively little diversity and do not fully cover the whole chemical space occupied by known ligands. Here, we propose a new method (DrugEx) to discover <i>de novo</i> drug-like molecules. DrugEx is an RNN model (generator) trained through a special exploration strategy integrated into reinforcement learning. As a case study we applied our method to design ligands against the adenosine A<sub>2A</sub> receptor. From ChEMBL data, a machine learning model (predictor) was created to predict whether generated molecules are active or not. Based on this predictor as the reward function, the generator was trained by reinforcement learning without any further data. We then compared the performance of our method with two previously published methods, REINVENT and ORGANIC. We found that candidate molecules our model designed that were predicted to be active, had a larger chemical diversity, and better covered the chemical space of known ligands compared to the state-of-the-art.</p>

Download Full-text

An Exploration Strategy Improves the Diversity of de novo Ligands Using Deep Reinforcement Learning – A Case for the Adenosine A2A Receptor

10.26434/chemrxiv.7436789.v3 ◽

2019 ◽

Author(s):

Xuhan Liu ◽

Kai Ye ◽

Herman Van Vlijmen ◽

Adriaan P. IJzerman ◽

Gerard JP Van westen

Keyword(s):

Reinforcement Learning ◽

Language Processing ◽

De Novo ◽

Chemical Space ◽

Chemical Diversity ◽

Reward Function ◽

Chemical Structures ◽

Exploration Strategy ◽

Machine Learning Model ◽

Adenosine A2a

<p></p><p>Over the last five years deep learning has progressed tremendously in both image recognition and natural language processing. Now it is increasingly applied to other data rich fields. In drug discovery, recurrent neural networks (RNNs) have been shown to be an effective method to generate novel chemical structures in the form of SMILES. However, ligands generated by current methods have so far provided relatively low diversity and do not fully cover the whole chemical space occupied by known ligands. Here, we propose a new method (DrugEx) to discover <i>de novo</i> drug-like molecules. DrugEx is an RNN model (generator) trained through reinforcement learning which was integrated with a special exploration strategy. As a case study we applied our method to design ligands against the adenosine A<sub>2A</sub> receptor. From ChEMBL data, a machine learning model (predictor) was created to predict whether generated molecules are active or not. Based on this predictor as the reward function, the generator was trained by reinforcement learning without any further data. We then compared the performance of our method with two previously published methods, REINVENT and ORGANIC. We found that candidate molecules our model designed, and predicted to be active, had a larger chemical diversity, and better covered the chemical space of known ligands compared to the state-of-the-art.</p><p></p>

Download Full-text

An Exploration Strategy Improves the Diversity of de novo Ligands Using Deep Reinforcement Learning – A Case for the Adenosine A2A Receptor

10.26434/chemrxiv.7436789.v2 ◽

2018 ◽

Author(s):

Xuhan Liu ◽

Kai Ye ◽

Herman Van Vlijmen ◽

Adriaan P. IJzerman ◽

Gerard JP Van westen

Keyword(s):

Reinforcement Learning ◽

Language Processing ◽

De Novo ◽

Chemical Space ◽

Chemical Diversity ◽

Reward Function ◽

Chemical Structures ◽

Exploration Strategy ◽

Machine Learning Model ◽

Adenosine A2a

<p>Over the last five years deep learning has progressed tremendously in both image recognition and natural language processing. Now it is increasingly applied to other data rich fields. In drug discovery, recurrent neural networks (RNNs) have been shown to be an effective method to generate novel chemical structures in the form of SMILES. However, ligands generated by current methods used to provide relatively little diversity and do not fully cover the whole chemical space occupied by known ligands. Here, we propose a new method (DrugEx) to discover <i>de novo</i> drug-like molecules. DrugEx is an RNN model (generator) trained through a special exploration strategy integrated into reinforcement learning. As a case study we applied our method to design ligands against the adenosine A<sub>2A</sub> receptor. From ChEMBL data, a machine learning model (predictor) was created to predict whether generated molecules are active or not. Based on this predictor as the reward function, the generator was trained by reinforcement learning without any further data. We then compared the performance of our method with two previously published methods, REINVENT and ORGANIC. We found that candidate molecules our model designed that were predicted to be active, had a larger chemical diversity, and better covered the chemical space of known ligands compared to the state-of-the-art.</p>

Download Full-text

A De Novo Molecular Generation Method Using Latent Vector Based Generative Adversarial Network

10.26434/chemrxiv.8299544.v3 ◽

2019 ◽

Cited By ~ 1

Author(s):

Oleksii Prykhodko ◽

Simon Viet Johansson ◽

Panagiotis-Christos Kotsias ◽

Josep Arús-Pous ◽

Esben Jannik Bjerrum ◽

...

Keyword(s):

Neural Network ◽

Deep Learning ◽

De Novo ◽

Molecular Design ◽

Chemical Space ◽

Training Set ◽

Generative Adversarial Network ◽

Adversarial Network ◽

De Novo Molecular Design ◽

Novel Structures

<p> </p><p>Deep learning methods applied to drug discovery have been used to generate novel structures. In this study, we propose a new deep learning architecture, LatentGAN, which combines an autoencoder and a generative adversarial neural network for de novo molecular design. We applied the method in two scenarios: one to generate random drug-like compounds and another to generate target-biased compounds. Our results show that the method works well in both cases: sampled compounds from the trained model can largely occupy the same chemical space as the training set and also generate a substantial fraction of novel compounds. Moreover, the drug-likeness score of compounds sampled from LatentGAN is also similar to that of the training set. Lastly, generated compounds differ from those obtained with a Recurrent Neural Network-based generative model approach, indicating that both methods can be used complementarily.</p><p> </p>

Download Full-text

A de novo molecular generation method using latent vector based generative adversarial network

Journal of Cheminformatics ◽

10.1186/s13321-019-0397-9 ◽

2019 ◽

Vol 11 (1) ◽

Cited By ~ 24

Author(s):

Oleksii Prykhodko ◽

Simon Viet Johansson ◽

Panagiotis-Christos Kotsias ◽

Josep Arús-Pous ◽

Esben Jannik Bjerrum ◽

...

Keyword(s):

Neural Network ◽

Deep Learning ◽

De Novo ◽

Molecular Design ◽

Chemical Space ◽

Training Set ◽

Generative Adversarial Network ◽

Adversarial Network ◽

De Novo Molecular Design ◽

Novel Structures

AbstractDeep learning methods applied to drug discovery have been used to generate novel structures. In this study, we propose a new deep learning architecture, LatentGAN, which combines an autoencoder and a generative adversarial neural network for de novo molecular design. We applied the method in two scenarios: one to generate random drug-like compounds and another to generate target-biased compounds. Our results show that the method works well in both cases. Sampled compounds from the trained model can largely occupy the same chemical space as the training set and also generate a substantial fraction of novel compounds. Moreover, the drug-likeness score of compounds sampled from LatentGAN is also similar to that of the training set. Lastly, generated compounds differ from those obtained with a Recurrent Neural Network-based generative model approach, indicating that both methods can be used complementarily.

Download Full-text