Personalized Dual-Hormone Control for Type 1 Diabetes Using Deep Reinforcement Learning

(1) Background: People living with type 1 diabetes (T1D) require self-management to maintain blood glucose (BG) levels in a therapeutic range through the delivery of exogenous insulin. However, due to the various variability, uncertainty and complex glucose dynamics, optimizing the doses of insulin delivery to minimize the risk of hyperglycemia and hypoglycemia is still an open problem. (2) Methods: In this work, we propose a novel insulin bolus advisor which uses deep reinforcement learning (DRL) and continuous glucose monitoring to optimize insulin dosing at mealtime. In particular, an actor-critic model based on deep deterministic policy gradient is designed to compute mealtime insulin doses. The proposed system architecture uses a two-step learning framework, in which a population model is first obtained and then personalized by subject-specific data. Prioritized memory replay is adopted to accelerate the training process in clinical practice. To validate the algorithm, we employ a customized version of the FDA-accepted UVA/Padova T1D simulator to perform in silico trials on 10 adult subjects and 10 adolescent subjects. (3) Results: Compared to a standard bolus calculator as the baseline, the DRL insulin bolus advisor significantly improved the average percentage time in target range (70–180 mg/dL) from 74.1%±8.4% to 80.9%±6.9% (p<0.01) and 54.9%±12.4% to 61.6%±14.1% (p<0.01) in the the adult and adolescent cohorts, respectively, while reducing hypoglycemia. (4) Conclusions: The proposed algorithm has the potential to improve mealtime bolus insulin delivery in people with T1D and is a feasible candidate for future clinical validation.

Download Full-text

Glucagon sensitivity and clearance in type 1 diabetes: insights from in vivo and in silico experiments

AJP Endocrinology and Metabolism ◽

10.1152/ajpendo.00236.2015 ◽

2015 ◽

Vol 309 (5) ◽

pp. E474-E486 ◽

Cited By ~ 11

Author(s):

Ling Hinshaw ◽

Ashwini Mallad ◽

Chiara Dalla Man ◽

Rita Basu ◽

Claudio Cobelli ◽

...

Keyword(s):

Type 1 Diabetes ◽

In Silico ◽

Artificial Pancreas ◽

Glucose Production ◽

Endogenous Glucose Production ◽

Action Model ◽

The Difference ◽

Hormone Control

Glucagon use in artificial pancreas for type 1 diabetes (T1D) is being explored for prevention and rescue from hypoglycemia. However, the relationship between glucagon stimulation of endogenous glucose production (EGP) viz., hepatic glucagon sensitivity, and prevailing glucose concentrations has not been examined. To test the hypothesis that glucagon sensitivity is increased at hypoglycemia vs. euglycemia, we studied 29 subjects with T1D randomized to a hypoglycemia or euglycemia clamp. Each subject was studied at three glucagon doses at euglycemia or hypoglycemia, with EGP measured by isotope dilution technique. The peak EGP increments and the integrated EGP response increased with increasing glucagon dose during euglycemia and hypoglycemia. However, the difference in dose response based on glycemia was not significant despite higher catecholamine concentrations in the hypoglycemia group. Knowledge of glucagon's effects on EGP was used to develop an in silico glucagon action model. The model-derived output fitted the obtained data at both euglycemia and hypoglycemia for all glucagon doses tested. Glucagon clearance did not differ between glucagon doses studied in both groups. Therefore, the glucagon controller of a dual hormone control system may not need to adjust glucagon sensitivity, and hence glucagon dosing, based on glucose concentrations during euglycemia and hypoglycemia.

Download Full-text

Reinforcement Learning Based Method for Managing Type 1 Diabetes (Preprint)

10.2196/preprints.12905 ◽

2018 ◽

Author(s):

Mahsa Oroojeni Mohammad Javad ◽

Stephen Olusegun Agboola ◽

Kamal Jethwani ◽

Ibrahim Zeid ◽

Sagar Kamarthi

Keyword(s):

Type 1 Diabetes ◽

Reinforcement Learning ◽

Blood Glucose ◽

Blood Glucose Level ◽

Diabetic Patient ◽

Glucose Level ◽

Dosage Level ◽

Insulin Dosage ◽

Diabetic Patients

BACKGROUND Diabetes is a serious chronic disease marked by high levels of blood glucose. It results from issues related to how insulin is produced and/or how insulin functions in the body. In the long run, uncontrolled blood sugar can damage the vessels that supply blood to important organs such as heart, kidneys, eyes, and nerves. Currently there are no effective algorithms to automatically recommend insulin dosage level considering the characteristics of a diabetic patient. OBJECTIVE The objective of this work is to develop and validate a general reinforcement learning framework and a related learning model for personalized treatment and management of Type 1 diabetes and its complications. METHODS This research presents a model-free reinforcement learning (RL) algorithm to recommend insulin level to regulate the blood glucose level of a diabetic patient considering his/her state defined by A1C level, alcohol usage, activity level, and BMI value. In this approach, an RL agent learns from its exploration and response of diabetic patients when they are subject to different actions in terms of insulin dosage level. As a result of a treatment action at time step t, the RL agent receives a numeric reward depending on the response of the patient’s blood glucose level. At each stage the reward for the learning agent is calculated as a function of the difference between the glucose level in the patient body and its target level. The RL algorithm is trained on ten years of the clinical data of 87 patients obtained from the Mass General Hospital. Demographically, 59% of patients are male and 41% of patients are female; the median of age is 54 years and mean is 52.92 years; 86% of patients are white and 47% of 87 patients are married. RESULTS The performance of the algorithm is evaluated on 60 test cases. Further the performance of Support Vector Machine (SVM) has been applied for Lantus class prediction and results has been compared with Q-learning algorithm recommendation. The results show that the RL recommendations of insulin levels for test patients match with the actual prescriptions of the test patients. The RL gave prediction with an accuracy of 88% and SVM shows 80% accuracy. CONCLUSIONS Since the RL algorithm can select actions that improve patient condition by taking into account delayed effects, it has a good potential to control blood glucose level in diabetic patients.

Download Full-text

Basal Glucose Control in Type 1 Diabetes using Deep Reinforcement Learning: An In Silico Validation

IEEE Journal of Biomedical and Health Informatics ◽

10.1109/jbhi.2020.3014556 ◽

2020 ◽

pp. 1-1 ◽

Cited By ~ 1

Author(s):

Taiyu Zhu ◽

Kezhi Li ◽

Pau Herrero ◽

Pantelis Georgiou

Keyword(s):

Type 1 Diabetes ◽

Reinforcement Learning ◽

Glucose Control ◽

In Silico

Download Full-text

A Reinforcement Learning–Based Method for Management of Type 1 Diabetes: Exploratory Study

JMIR Diabetes ◽

10.2196/12905 ◽

2019 ◽

Vol 4 (3) ◽

pp. e12905 ◽

Cited By ~ 2

Author(s):

Mahsa Oroojeni Mohammad Javad ◽

Stephen Olusegun Agboola ◽

Kamal Jethwani ◽

Abe Zeid ◽

Sagar Kamarthi

Keyword(s):

Type 1 Diabetes ◽

Reinforcement Learning ◽

Blood Glucose ◽

Blood Glucose Level ◽

Glucose Level ◽

Clinical Data ◽

Exploratory Study ◽

Insulin Dosage

Background Type 1 diabetes mellitus (T1DM) is characterized by chronic insulin deficiency and consequent hyperglycemia. Patients with T1DM require long-term exogenous insulin therapy to regulate blood glucose levels and prevent the long-term complications of the disease. Currently, there are no effective algorithms that consider the unique characteristics of T1DM patients to automatically recommend personalized insulin dosage levels. Objective The objective of this study was to develop and validate a general reinforcement learning (RL) framework for the personalized treatment of T1DM using clinical data. Methods This research presents a model-free data-driven RL algorithm, namely Q-learning, that recommends insulin doses to regulate the blood glucose level of a T1DM patient, considering his or her state defined by glycated hemoglobin (HbA1c) levels, body mass index, engagement in physical activity, and alcohol usage. In this approach, the RL agent identifies the different states of the patient by exploring the patient’s responses when he or she is subjected to varying insulin doses. On the basis of the result of a treatment action at time step t, the RL agent receives a numeric reward, positive or negative. The reward is calculated as a function of the difference between the actual blood glucose level achieved in response to the insulin dose and the targeted HbA1c level. The RL agent was trained on 10 years of clinical data of patients treated at the Mass General Hospital. Results A total of 87 patients were included in the training set. The mean age of these patients was 53 years, 59% (51/87) were male, 86% (75/87) were white, and 47% (41/87) were married. The performance of the RL agent was evaluated on 60 test cases. RL agent–recommended insulin dosage interval includes the actual dose prescribed by the physician in 53 out of 60 cases (53/60, 88%). Conclusions This exploratory study demonstrates that an RL algorithm can be used to recommend personalized insulin doses to achieve adequate glycemic control in patients with T1DM. However, further investigation in a larger sample of patients is needed to confirm these findings.

Download Full-text

Controlling Blood Glucose For Patients With Type 1 DiabetesUsing Deep Reinforcement Learning – The Influence OfChanging The Reward Function

Proceedings of the Northern Lights Deep Learning Workshop ◽

10.7557/18.5166 ◽

2020 ◽

Vol 1 ◽

pp. 6

Author(s):

Miguel Angel Tejedor Hernandez ◽

Jonas Nordhaug Myhre

Keyword(s):

Type 1 Diabetes ◽

Reinforcement Learning ◽

Blood Glucose ◽

In Silico ◽

Critical Component ◽

Promising Direction ◽

Reward Function ◽

Reward Functions

Reinforcement learning (RL) is a promising direction in adaptive and personalized type 1 diabetes (T1D) treatment. However, the reward function – a most critical component in RL – is a component that is in most cases hand designed and often overlooked. In this paper we show that different reward functions can dramatically influence the final result when using RL to treat in-silico T1D patients.

Download Full-text

In-Silico Evaluation of Glucose Regulation Using Policy Gradient Reinforcement Learning for Patients with Type 1 Diabetes Mellitus

Applied Sciences ◽

10.3390/app10186350 ◽

2020 ◽

Vol 10 (18) ◽

pp. 6350

Author(s):

Jonas Nordhaug Myhre ◽

Miguel Tejedor ◽

Ilkka Kalervo Launonen ◽

Anas El Fathi ◽

Fred Godtliebsen

Keyword(s):

Diabetes Mellitus ◽

Type 1 Diabetes ◽

Reinforcement Learning ◽

Blood Glucose ◽

State Of The Art ◽

Blood Glucose Control ◽

Glucose Regulation ◽

Gradient Algorithms ◽

Policy Gradient

In this paper, we test and evaluate policy gradient reinforcement learning for automated blood glucose control in patients with Type 1 Diabetes Mellitus. Recent research has shown that reinforcement learning is a promising approach to accommodate the need for individualized blood glucose level control algorithms. The motivation for using policy gradient algorithms comes from the fact that adaptively administering insulin is an inherently continuous task. Policy gradient algorithms are known to be superior in continuous high-dimensional control tasks. Previously, most of the approaches for automated blood glucose control using reinforcement learning has used a finite set of actions. We use the Trust-Region Policy Optimization algorithm in this work. It represents the state of the art for deep policy gradient algorithms. The experiments are carried out in-silico using the Hovorka model, and stochastic behavior is modeled through simulated carbohydrate counting errors to illustrate the full potential of the framework. Furthermore, we use a model-free approach where no prior information about the patient is given to the algorithm. Our experiments show that the reinforcement learning agent is able to compete with and sometimes outperform state-of-the-art model predictive control in blood glucose regulation.

Download Full-text

Control of Blood Glucose for Type-1 Diabetes by Using Reinforcement Learning with Feedforward Algorithm

Computational and Mathematical Methods in Medicine ◽

10.1155/2018/4091497 ◽

2018 ◽

Vol 2018 ◽

pp. 1-8 ◽

Cited By ~ 1

Author(s):

Phuong D. Ngo ◽

Susan Wei ◽

Anna Holubová ◽

Jan Muzik ◽

Fred Godtliebsen

Keyword(s):

Type 1 Diabetes ◽

Control System ◽

Reinforcement Learning ◽

Blood Glucose ◽

State Variables ◽

Meal Time ◽

Feedforward Controller ◽

Bolus Insulin ◽

Nonlinear Nature

Background. Type-1 diabetes is a condition caused by the lack of insulin hormone, which leads to an excessive increase in blood glucose level. The glucose kinetics process is difficult to control due to its complex and nonlinear nature and with state variables that are difficult to measure. Methods. This paper proposes a method for automatically calculating the basal and bolus insulin doses for patients with type-1 diabetes using reinforcement learning with feedforward controller. The algorithm is designed to keep the blood glucose stable and directly compensate for the external events such as food intake. Its performance was assessed using simulation on a blood glucose model. The usage of the Kalman filter with the controller was demonstrated to estimate unmeasurable state variables. Results. Comparison simulations between the proposed controller with the optimal reinforcement learning and the proportional-integral-derivative controller show that the proposed methodology has the best performance in regulating the fluctuation of the blood glucose. The proposed controller also improved the blood glucose responses and prevented hypoglycemia condition. Simulation of the control system in different uncertain conditions provided insights on how the inaccuracies of carbohydrate counting and meal-time reporting affect the performance of the control system. Conclusion. The proposed controller is an effective tool for reducing postmeal blood glucose rise and for countering the effects of external known events such as meal intake and maintaining blood glucose at a healthy level under uncertainties.

Download Full-text