Reinforcement Learning in Healthcare: A Survey

As a subfield of machine learning, reinforcement learning (RL) aims at optimizing decision making by using interaction samples of an agent with its environment and the potentially delayed feedbacks. In contrast to traditional supervised learning that typically relies on one-shot, exhaustive, and supervised reward signals, RL tackles sequential decision-making problems with sampled, evaluative, and delayed feedbacks simultaneously. Such a distinctive feature makes RL techniques a suitable candidate for developing powerful solutions in various healthcare domains, where diagnosing decisions or treatment regimes are usually characterized by a prolonged period with delayed feedbacks. By first briefly examining theoretical foundations and key methods in RL research, this survey provides an extensive overview of RL applications in a variety of healthcare domains, ranging from dynamic treatment regimes in chronic diseases and critical care, automated medical diagnosis, and many other control or scheduling problems that have infiltrated every aspect of the healthcare system. In addition, we discuss the challenges and open issues in the current research and highlight some potential solutions and directions for future research.

A Privacy-Preserving Reinforcement Learning Approach for Dynamic Treatment Regimes on Health Data

Based on the clinical states of the patient, dynamic treatment regime technology can provide various therapeutic methods, which is helpful for medical treatment policymaking. Reinforcement learning is an important approach for developing this technology. In order to implement the reinforcement learning algorithm efficiently, the computation of health data is usually outsourced to the untrustworthy cloud server. However, it may leak, falsify, or delete private health data. Encryption is a common method for solving this problem. But the cloud server is difficult to calculate encrypted health data. In this paper, based on Cheon et al.’s approximate homomorphic encryption scheme, we first propose secure computation protocols for implementing comparison, maximum, exponentiation, and division. Next, we design a homomorphic reciprocal of square root protocol firstly, which only needs one approximate computation. Based on the proposed secure computation protocols, we design a secure asynchronous advantage actor-critic reinforcement learning algorithm for the first time. Then, it is used to implement a secure treatment decision-making algorithm. Simulation results show that our secure computation protocols and algorithms are feasible.

Identification of a novel antifungal backbone of naphthalimide thiazoles with synergistic potential for chemical and dynamic treatment

Aim: The high incidence and prevalence of fungal infections call for new antifungal drugs. This work was to develop naphthalimide thiazoles as potential antifungal agents. Results & methodology: These compounds showed significant antifungal potency toward some tested fungi. Especially, naphthalimide thiazole 4h with excellent anti- Candida tropicalis efficacy possessed good hemolysis level, low toxicity and no obvious resistance. Deciphering the mechanism showed that 4h interacted with DNA and disrupted the antioxidant defense system of C. tropicalis. Compound 4h also triggered membrane depolarization, leakage of cytoplasmic contents and LDH inhibition. Simultaneously, 4h rendered metabolic inactivation and eradicated the formed biofilms of C. tropicalis. Conclusion: The multifaceted synergistic effect initiated by naphthalimide thiazoles is a reasonable treatment window for prospective development.

Individualized resuscitation strategy for septic shock formalized by finite mixture modeling and dynamic treatment regimen

Background Septic shock comprises a heterogeneous population, and individualized resuscitation strategy is of vital importance. The study aimed to identify subclasses of septic shock with non-supervised learning algorithms, so as to tailor resuscitation strategy for each class. Methods Patients with septic shock in 25 tertiary care teaching hospitals in China from January 2016 to December 2017 were enrolled in the study. Clinical and laboratory variables were collected on days 0, 1, 2, 3 and 7 after ICU admission. Subclasses of septic shock were identified by both finite mixture modeling and K-means clustering. Individualized fluid volume and norepinephrine dose were estimated using dynamic treatment regime (DTR) model to optimize the final mortality outcome. DTR models were validated in the eICU Collaborative Research Database (eICU-CRD) dataset. Results A total of 1437 patients with a mortality rate of 29% were included for analysis. The finite mixture modeling and K-means clustering robustly identified five classes of septic shock. Class 1 (baseline class) accounted for the majority of patients over all days; class 2 (critical class) had the highest severity of illness; class 3 (renal dysfunction) was characterized by renal dysfunction; class 4 (respiratory failure class) was characterized by respiratory failure; and class 5 (mild class) was characterized by the lowest mortality rate (21%). The optimal fluid infusion followed the resuscitation/de-resuscitation phases with initial large volume infusion and late restricted volume infusion. While class 1 transitioned to de-resuscitation phase on day 3, class 3 transitioned on day 1. Classes 1 and 3 might benefit from early use of norepinephrine, and class 2 can benefit from delayed use of norepinephrine while waiting for adequate fluid infusion. Conclusions Septic shock comprises a heterogeneous population that can be robustly classified into five phenotypes. These classes can be easily identified with routine clinical variables and can help to tailor resuscitation strategy in the context of precise medicine.