Exploring Local Chemical Space in De Novo Molecular Generation Using Multi-Agent Deep Reinforcement Learning

Due to the large drug-like chemical space available to search for feasible drug-like molecules, rational drug design often starts from specific scaffolds to which side chains/substituents are added or modified. With the rapid growth of the application of deep learning in drug discovery, a variety of effective approaches have been developed for de novo drug design. In previous work, we proposed a method named DrugEx, which can be applied in polypharmacology based on multi-objective deep reinforcement learning. However, the previous version is trained under fixed objectives similar to other known methods and does not allow users to input any prior information (i.e. a desired scaffold). In order to improve the general applicability, we updated DrugEx to design drug molecules based on scaffolds which consist of multiple fragments provided by users. In this work, the Transformer model was employed to generate molecular structures. The Transformer is a multi-head self-attention deep learning model containing an encoder to receive scaffolds as input and a decoder to generate molecules as output. In order to deal with the graph representation of molecules we proposed a novel positional encoding for each atom and bond based on an adjacency matrix to extend the architecture of the Transformer. Each molecule was generated by growing and connecting procedures for the fragments in the given scaffold that were unified into one model. Moreover, we trained this generator under a reinforcement learning framework to increase the number of desired ligands. As a proof of concept, our proposed method was applied to design ligands for the adenosine A2A receptor (A2AAR) and compared with SMILES-based methods. The results demonstrated the effectiveness of our method in that 100% of the generated molecules are valid and most of them had a high predicted affinity value towards A2AAR with given scaffolds.

Download Full-text

Memory-assisted reinforcement learning for diverse molecular de novo design

10.21203/rs.3.rs-52871/v2 ◽

2020 ◽

Author(s):

Thomas Blaschke ◽

Ola Engkvist ◽

Jürgen Bajorath ◽

Hongming Chen

Keyword(s):

Reinforcement Learning ◽

De Novo ◽

Molecular Design ◽

Chemical Space ◽

Scoring Function ◽

Chemical Diversity ◽

Chemical Structures ◽

Machine Learning Model ◽

Low Diversity ◽

De Novo Molecular Design

Abstract In de novo molecular design, recurrent neural networks (RNN) have been shown to be effective methods for sampling and generating novel chemical structures. Using a technique called reinforcement learning (RL), an RNN can be tuned to target a particular section of chemical space with optimized desirable properties using a scoring function. However, ligands generated by current RL methods so far tend to have relatively low diversity, and sometimes even result in duplicate structures when optimizing towards desired properties. Here, we propose a new method to address the low diversity issue in RL for molecular design. Memory-assisted RL is an extension of the known RL, with the introduction of a so-called memory unit. As proof of concept, we applied our method to generate structures with a desired AlogP value. In a second case study, we applied our method to design ligands for the dopamine type 2 receptor and the 5-hydroxytryptamine type 1A receptor. For both receptors, a machine learning model was developed to predict whether generated molecules were active or not for the receptor. In both case studies, it was found that memory-assisted RL led to the generation of more compounds predicted to be active having higher chemical diversity, thus achieving better coverage of chemical space of known ligands compared to established RL methods.

Download Full-text

An Exploration Strategy Improves the Diversity of de novo Ligands Using Deep Reinforcement Learning – A Case for the Adenosine A2A Receptor

10.26434/chemrxiv.7436789 ◽

2019 ◽

Author(s):

Xuhan Liu ◽

Kai Ye ◽

Herman Van Vlijmen ◽

Adriaan P. IJzerman ◽

Gerard JP Van westen

Keyword(s):

Reinforcement Learning ◽

Language Processing ◽

De Novo ◽

Chemical Space ◽

Chemical Diversity ◽

Reward Function ◽

Chemical Structures ◽

Exploration Strategy ◽

Machine Learning Model ◽

Adenosine A2a

<p></p><p>Over the last five years deep learning has progressed tremendously in both image recognition and natural language processing. Now it is increasingly applied to other data rich fields. In drug discovery, recurrent neural networks (RNNs) have been shown to be an effective method to generate novel chemical structures in the form of SMILES. However, ligands generated by current methods have so far provided relatively low diversity and do not fully cover the whole chemical space occupied by known ligands. Here, we propose a new method (DrugEx) to discover <i>de novo</i> drug-like molecules. DrugEx is an RNN model (generator) trained through reinforcement learning which was integrated with a special exploration strategy. As a case study we applied our method to design ligands against the adenosine A<sub>2A</sub> receptor. From ChEMBL data, a machine learning model (predictor) was created to predict whether generated molecules are active or not. Based on this predictor as the reward function, the generator was trained by reinforcement learning without any further data. We then compared the performance of our method with two previously published methods, REINVENT and ORGANIC. We found that candidate molecules our model designed, and predicted to be active, had a larger chemical diversity, and better covered the chemical space of known ligands compared to the state-of-the-art.</p><p></p>

Download Full-text

Memory-Assisted Reinforcement Learning for Diverse Molecular De Novo Design

10.26434/chemrxiv.12693152.v1 ◽

2020 ◽

Cited By ~ 1

Author(s):

Thomas Blaschke ◽

Ola Engkvist ◽

Jürgen Bajorath ◽

Hongming Chen

Keyword(s):

Reinforcement Learning ◽

De Novo ◽

Molecular Design ◽

Chemical Space ◽

Scoring Function ◽

De Novo Design ◽

Memory Unit ◽

Chemical Structures ◽

Low Diversity ◽

De Novo Molecular Design

<div><div><div><p>In de novo molecular design, recurrent neural networks (RNN) have been shown to be effective methods for sampling and generating novel chemical structures. Using a technique called reinforcement learning (RL), an RNN can be tuned to target a particular section of chemical space with optimized desirable properties using a scoring function. However, ligands generated by current RL methods so far tend to have relatively low diversity, and sometimes even result in duplicate structures when optimizing towards particular properties. Here, we propose a new method to address the low diversity issue in RL. Memory-assisted RL is an extension of the known RL, with the introduction of a so-called memory unit.</p></div></div></div>

Download Full-text

An Exploration Strategy Improves the Diversity of de novo Ligands Using Deep Reinforcement Learning – A Case for the 3 Adenosine A2A Receptor

10.26434/chemrxiv.7436789.v1 ◽

2018 ◽

Author(s):

Xuhan Liu ◽

Kai Ye ◽

Herman Van Vlijmen ◽

Adriaan P. IJzerman ◽

Gerard JP Van westen

Keyword(s):

Reinforcement Learning ◽

Language Processing ◽

De Novo ◽

Chemical Space ◽

Chemical Diversity ◽

Reward Function ◽

Chemical Structures ◽

Exploration Strategy ◽

Machine Learning Model ◽

Adenosine A2a

<p>Over the last five years deep learning has progressed tremendously in both image recognition and natural language processing. Now it is increasingly applied to other data rich fields. In drug discovery, recurrent neural networks (RNNs) have been shown to be an effective method to generate novel chemical structures in the form of SMILES. However, ligands generated by current methods used to provide relatively little diversity and do not fully cover the whole chemical space occupied by known ligands. Here, we propose a new method (DrugEx) to discover <i>de novo</i> drug-like molecules. DrugEx is an RNN model (generator) trained through a special exploration strategy integrated into reinforcement learning. As a case study we applied our method to design ligands against the adenosine A<sub>2A</sub> receptor. From ChEMBL data, a machine learning model (predictor) was created to predict whether generated molecules are active or not. Based on this predictor as the reward function, the generator was trained by reinforcement learning without any further data. We then compared the performance of our method with two previously published methods, REINVENT and ORGANIC. We found that candidate molecules our model designed that were predicted to be active, had a larger chemical diversity, and better covered the chemical space of known ligands compared to the state-of-the-art.</p>

Download Full-text

Memory-assisted Reinforcement Learning for Diverse Molecular De Novo Design

10.21203/rs.3.rs-52871/v1 ◽

2020 ◽

Author(s):

Thomas Blaschke ◽

Ola Engkvist ◽

Jürgen Bajorath ◽

Hongming Chen

Keyword(s):

Reinforcement Learning ◽

De Novo ◽

Molecular Design ◽

Chemical Space ◽

Scoring Function ◽

Chemical Diversity ◽

Chemical Structures ◽

Low Diversity ◽

De Novo Molecular Design ◽

Dopamine 2 Receptor

Abstract In de novo molecular design, recurrent neural networks (RNN) have been shown to be effective methods for sampling and generating novel chemical structures. Using a technique called reinforcement learning (RL), an RNN can be tuned to target a particular section of chemical space with optimized desirable properties using a scoring function. However, ligands generated by current RL methods so far tend to have relatively low diversity, and sometimes even result in duplicate structures when optimizing towards particular properties. Here, we propose a new method to address the low diversity issue in RL. Memory-assisted RL is an extension of the known RL, with the introduction of a so-called memory unit. As proof of concept, we applied our method to generate structures with an optimized logP. In a second case study, we applied our method to design ligands for the dopamine 2 receptor and the 5-hydroxytryptamine 1A receptor. For both receptors, a machine learning model was developed to predict whether generated molecules were active or not for the receptor. In both case studies, it was found that memory-assisted RL led to the generation of more active compounds and with higher chemical diversity, thus achieving better coverage of chemical space of known ligands compared to established RL method.

Download Full-text

An Exploration Strategy Improves the Diversity of de novo Ligands Using Deep Reinforcement Learning – A Case for the Adenosine A2A Receptor

10.26434/chemrxiv.7436789.v3 ◽

2019 ◽

Author(s):

Xuhan Liu ◽

Kai Ye ◽

Herman Van Vlijmen ◽

Adriaan P. IJzerman ◽

Gerard JP Van westen

Keyword(s):

Reinforcement Learning ◽

Language Processing ◽

De Novo ◽

Chemical Space ◽

Chemical Diversity ◽

Reward Function ◽

Chemical Structures ◽

Exploration Strategy ◽

Machine Learning Model ◽

Adenosine A2a

<p></p><p>Over the last five years deep learning has progressed tremendously in both image recognition and natural language processing. Now it is increasingly applied to other data rich fields. In drug discovery, recurrent neural networks (RNNs) have been shown to be an effective method to generate novel chemical structures in the form of SMILES. However, ligands generated by current methods have so far provided relatively low diversity and do not fully cover the whole chemical space occupied by known ligands. Here, we propose a new method (DrugEx) to discover <i>de novo</i> drug-like molecules. DrugEx is an RNN model (generator) trained through reinforcement learning which was integrated with a special exploration strategy. As a case study we applied our method to design ligands against the adenosine A<sub>2A</sub> receptor. From ChEMBL data, a machine learning model (predictor) was created to predict whether generated molecules are active or not. Based on this predictor as the reward function, the generator was trained by reinforcement learning without any further data. We then compared the performance of our method with two previously published methods, REINVENT and ORGANIC. We found that candidate molecules our model designed, and predicted to be active, had a larger chemical diversity, and better covered the chemical space of known ligands compared to the state-of-the-art.</p><p></p>

Download Full-text

An Exploration Strategy Improves the Diversity of de novo Ligands Using Deep Reinforcement Learning – A Case for the Adenosine A2A Receptor

10.26434/chemrxiv.7436789.v2 ◽

2018 ◽

Author(s):

Xuhan Liu ◽

Kai Ye ◽

Herman Van Vlijmen ◽

Adriaan P. IJzerman ◽

Gerard JP Van westen

Keyword(s):

Reinforcement Learning ◽

Language Processing ◽

De Novo ◽

Chemical Space ◽

Chemical Diversity ◽

Reward Function ◽

Chemical Structures ◽

Exploration Strategy ◽

Machine Learning Model ◽

Adenosine A2a

<p>Over the last five years deep learning has progressed tremendously in both image recognition and natural language processing. Now it is increasingly applied to other data rich fields. In drug discovery, recurrent neural networks (RNNs) have been shown to be an effective method to generate novel chemical structures in the form of SMILES. However, ligands generated by current methods used to provide relatively little diversity and do not fully cover the whole chemical space occupied by known ligands. Here, we propose a new method (DrugEx) to discover <i>de novo</i> drug-like molecules. DrugEx is an RNN model (generator) trained through a special exploration strategy integrated into reinforcement learning. As a case study we applied our method to design ligands against the adenosine A<sub>2A</sub> receptor. From ChEMBL data, a machine learning model (predictor) was created to predict whether generated molecules are active or not. Based on this predictor as the reward function, the generator was trained by reinforcement learning without any further data. We then compared the performance of our method with two previously published methods, REINVENT and ORGANIC. We found that candidate molecules our model designed that were predicted to be active, had a larger chemical diversity, and better covered the chemical space of known ligands compared to the state-of-the-art.</p>

Download Full-text

Memory-Assisted Reinforcement Learning for Diverse Molecular De Novo Design

10.26434/chemrxiv.12693152 ◽

2020 ◽

Author(s):

Thomas Blaschke ◽

Ola Engkvist ◽

Jürgen Bajorath ◽

Hongming Chen

Keyword(s):

Reinforcement Learning ◽

De Novo ◽

Molecular Design ◽

Chemical Space ◽

Scoring Function ◽

De Novo Design ◽

Memory Unit ◽

Chemical Structures ◽

Low Diversity ◽

De Novo Molecular Design

<div><div><div><p>In de novo molecular design, recurrent neural networks (RNN) have been shown to be effective methods for sampling and generating novel chemical structures. Using a technique called reinforcement learning (RL), an RNN can be tuned to target a particular section of chemical space with optimized desirable properties using a scoring function. However, ligands generated by current RL methods so far tend to have relatively low diversity, and sometimes even result in duplicate structures when optimizing towards particular properties. Here, we propose a new method to address the low diversity issue in RL. Memory-assisted RL is an extension of the known RL, with the introduction of a so-called memory unit.</p></div></div></div>

Download Full-text

Memory-assisted reinforcement learning for diverse molecular de novo design

Journal of Cheminformatics ◽

10.1186/s13321-020-00473-0 ◽

2020 ◽

Vol 12 (1) ◽

Cited By ~ 1

Author(s):

Thomas Blaschke ◽

Ola Engkvist ◽

Jürgen Bajorath ◽

Hongming Chen

Keyword(s):

Reinforcement Learning ◽

De Novo ◽

Molecular Design ◽

Chemical Space ◽

Scoring Function ◽

Chemical Diversity ◽

Chemical Structures ◽

Machine Learning Model ◽

Low Diversity ◽

De Novo Molecular Design

Abstract In de novo molecular design, recurrent neural networks (RNN) have been shown to be effective methods for sampling and generating novel chemical structures. Using a technique called reinforcement learning (RL), an RNN can be tuned to target a particular section of chemical space with optimized desirable properties using a scoring function. However, ligands generated by current RL methods so far tend to have relatively low diversity, and sometimes even result in duplicate structures when optimizing towards desired properties. Here, we propose a new method to address the low diversity issue in RL for molecular design. Memory-assisted RL is an extension of the known RL, with the introduction of a so-called memory unit. As proof of concept, we applied our method to generate structures with a desired AlogP value. In a second case study, we applied our method to design ligands for the dopamine type 2 receptor and the 5-hydroxytryptamine type 1A receptor. For both receptors, a machine learning model was developed to predict whether generated molecules were active or not for the receptor. In both case studies, it was found that memory-assisted RL led to the generation of more compounds predicted to be active having higher chemical diversity, thus achieving better coverage of chemical space of known ligands compared to established RL methods.

Download Full-text