Self-Supervised Mixture-of-Experts by Uncertainty Estimation

Learning related tasks in various domains and transferring exploited knowledge to new situations is a significant challenge in Reinforcement Learning (RL). However, most RL algorithms are data inefficient and fail to generalize in complex environments, limiting their adaptability and applicability in multi-task scenarios. In this paper, we propose SelfSupervised Mixture-of-Experts (SUM), an effective algorithm driven by predictive uncertainty estimation for multitask RL. SUM utilizes a multi-head agent with shared parameters as experts to learn a series of related tasks simultaneously by Deep Deterministic Policy Gradient (DDPG). Each expert is extended by predictive uncertainty estimation on known and unknown states to enhance the Q-value evaluation capacity against overfitting and the overall generalization ability. These enable the agent to capture and diffuse the common knowledge across different tasks improving sample efficiency in each task and the effectiveness of expert scheduling across multiple tasks. Instead of task-specific design as common MoEs, a self-supervised gating network is adopted to determine a potential expert to handle each interaction from unseen environments and calibrated completely by the uncertainty feedback from the experts without explicit supervision. To alleviate the imbalanced expert utilization as the crux of MoE, optimization is accomplished via decayedmasked experience replay, which encourages both diversification and specialization of experts during different periods. We demonstrate that our approach learns faster and achieves better performance by efficient transfer and robust generalization, outperforming several related methods on extended OpenAI Gym’s MuJoCo multi-task environments.

Download Full-text

Deep Deterministic Policy Gradient Based on Double Network Prioritized Experience Replay

IEEE Access ◽

10.1109/access.2021.3074535 ◽

2021 ◽

pp. 1-1

Author(s):

Chaohai Kang ◽

Chuiting Rong ◽

Weijian Ren ◽

Fengcai Huo ◽

Pengyun Liu

Keyword(s):

Double Network ◽

Policy Gradient ◽

Experience Replay ◽

Gradient Based

Download Full-text

Predictive Uncertainty Estimation Using Deep Learning for Soft Robot Multimodal Sensing

IEEE Robotics and Automation Letters ◽

10.1109/lra.2021.3056066 ◽

2021 ◽

Vol 6 (2) ◽

pp. 951-957

Author(s):

Ze Yang Ding ◽

Junn Yong Loo ◽

Vishnu Monn Baskaran ◽

Surya Girinatha Nurzaman ◽

Chee Pin Tan

Keyword(s):

Deep Learning ◽

Uncertainty Estimation ◽

Soft Robot ◽

Predictive Uncertainty ◽

Multimodal Sensing

Download Full-text

Predictive Uncertainty Estimation for Tractable Deep Probabilistic Models

Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence ◽

10.24963/ijcai.2020/745 ◽

2020 ◽

Author(s):

Julissa Villanueva Llerena

Keyword(s):

Probabilistic Models ◽

Linear Time ◽

Generative Models ◽

Uncertainty Estimation ◽

Image Completion ◽

Predictive Uncertainty ◽

Learning Tasks ◽

Challenging Tasks ◽

Statistical Support ◽

Marginal Inference

Tractable Deep Probabilistic Models (TPMs) are generative models based on arithmetic circuits that allow for exact marginal inference in linear time. These models have obtained promising results in several machine learning tasks. Like many other models, TPMs can produce over-confident incorrect inferences, especially on regions with small statistical support. In this work, we will develop efficient estimators of the predictive uncertainty that are robust to data scarcity and outliers. We investigate two approaches. The first approach measures the variability of the output to perturbations of the model weights. The second approach captures the variability of the prediction to changes in the model architecture. We will evaluate the approaches on challenging tasks such as image completion and multilabel classification.

Download Full-text

Improving predictive uncertainty estimation using Dropout–Hamiltonian Monte Carlo

Soft Computing ◽

10.1007/s00500-019-04195-w ◽

2019 ◽

Vol 24 (6) ◽

pp. 4307-4322 ◽

Cited By ~ 1

Author(s):

Sergio Hernández ◽

Diego Vergara ◽

Matías Valdenegro-Toro ◽

Felipe Jorquera

Keyword(s):

Monte Carlo ◽

Uncertainty Estimation ◽

Hamiltonian Monte Carlo ◽

Predictive Uncertainty

Download Full-text

Deep Reinforcement Learning Based Left-Turn Connected and Automated Vehicle Control at Signalized Intersection in Vehicle-to-Infrastructure Environment

Information ◽

10.3390/info11020077 ◽

2020 ◽

Vol 11 (2) ◽

pp. 77 ◽

Cited By ~ 1

Author(s):

Juan Chen ◽

Zhengxuan Xue ◽

Daiqian Fan

Keyword(s):

Reinforcement Learning ◽

Control Method ◽

Signalized Intersection ◽

Signal Control ◽

Left Turn ◽

Automated Vehicle ◽

Whole Process ◽

Policy Gradient ◽

Experience Replay ◽

Automated Vehicle Control

In order to solve the problem of vehicle delay caused by stops at signalized intersections, a micro-control method of a left-turning connected and automated vehicle (CAV) based on an improved deep deterministic policy gradient (DDPG) is designed in this paper. In this paper, the micro-control of the whole process of a left-turn vehicle approaching, entering, and leaving a signalized intersection is considered. In addition, in order to solve the problems of low sampling efficiency and overestimation of the critic network of the DDPG algorithm, a positive and negative reward experience replay buffer sampling mechanism and multi-critic network structure are adopted in the DDPG algorithm in this paper. Finally, the effectiveness of the signal control method, six DDPG-based methods (DDPG, PNRERB-1C-DDPG, PNRERB-3C-DDPG, PNRERB-5C-DDPG, PNRERB-5CNG-DDPG, and PNRERB-7C-DDPG), and four DQN-based methods (DQN, Dueling DQN, Double DQN, and Prioritized Replay DQN) are verified under 0.2, 0.5, and 0.7 saturation degrees of left-turning vehicles at a signalized intersection within a VISSIM simulation environment. The results show that the proposed deep reinforcement learning method can get a number of stops benefits ranging from 5% to 94%, stop time benefits ranging from 1% to 99%, and delay benefits ranging from −17% to 93%, respectively compared with the traditional signal control method.

Download Full-text

UAV Maneuvering Target Tracking in Uncertain Environments Based on Deep Reinforcement Learning and Meta-Learning

Remote Sensing ◽

10.3390/rs12223789 ◽

2020 ◽

Vol 12 (22) ◽

pp. 3789

Author(s):

Bo Li ◽

Zhigang Gan ◽

Daqing Chen ◽

Dyachenko Sergey Aleksandrovich

Keyword(s):

Reinforcement Learning ◽

Target Tracking ◽

Uncertain Environments ◽

Target Movement ◽

Maneuvering Target Tracking ◽

Novel Approach ◽

Policy Gradient ◽

Meta Learning ◽

Experience Replay ◽

Task Experience

This paper combines deep reinforcement learning (DRL) with meta-learning and proposes a novel approach, named meta twin delayed deep deterministic policy gradient (Meta-TD3), to realize the control of unmanned aerial vehicle (UAV), allowing a UAV to quickly track a target in an environment where the motion of a target is uncertain. This approach can be applied to a variety of scenarios, such as wildlife protection, emergency aid, and remote sensing. We consider a multi-task experience replay buffer to provide data for the multi-task learning of the DRL algorithm, and we combine meta-learning to develop a multi-task reinforcement learning update method to ensure the generalization capability of reinforcement learning. Compared with the state-of-the-art algorithms, namely the deep deterministic policy gradient (DDPG) and twin delayed deep deterministic policy gradient (TD3), experimental results show that the Meta-TD3 algorithm has achieved a great improvement in terms of both convergence value and convergence rate. In a UAV target tracking problem, Meta-TD3 only requires a few steps to train to enable a UAV to adapt quickly to a new target movement mode more and maintain a better tracking effectiveness.

Download Full-text

Predictive Uncertainty Estimation in Water Demand Forecasting Using the Model Conditional Processor

Water ◽

10.3390/w10040475 ◽

2018 ◽

Vol 10 (4) ◽

pp. 475 ◽

Cited By ~ 10

Author(s):

Amos Anele ◽

Ezio Todini ◽

Yskandar Hamam ◽

Adnan Abu-Mahfouz

Keyword(s):

Water Demand ◽

Demand Forecasting ◽

Uncertainty Estimation ◽

Predictive Uncertainty ◽

Water Demand Forecasting

Download Full-text

Exploration with Multiple Random ε-Buffers in Off-Policy Deep Reinforcement Learning

Symmetry ◽

10.3390/sym11111352 ◽

2019 ◽

Vol 11 (11) ◽

pp. 1352 ◽

Cited By ~ 1

Author(s):

Kim ◽

Park

Keyword(s):

Reinforcement Learning ◽

Experimental Comparison ◽

Continuous Control ◽

Policy Gradient ◽

Experience Replay ◽

Discrete Action ◽

Original Goal ◽

Efficient Exploration ◽

Greedy Policy ◽

Theoretical Results

In terms of deep reinforcement learning (RL), exploration is highly significant in achieving better generalization. In benchmark studies, ε-greedy random actions have been used to encourage exploration and prevent over-fitting, thereby improving generalization. Deep RL with random ε-greedy policies, such as deep Q-networks (DQNs), can demonstrate efficient exploration behavior. A random ε-greedy policy exploits additional replay buffers in an environment of sparse and binary rewards, such as in the real-time online detection of network securities by verifying whether the network is “normal or anomalous.” Prior studies have illustrated that a prioritized replay memory attributed to a complex temporal difference error provides superior theoretical results. However, another implementation illustrated that in certain environments, the prioritized replay memory is not superior to the randomly-selected buffers of random ε-greedy policy. Moreover, a key challenge of hindsight experience replay inspires our objective by using additional buffers corresponding to each different goal. Therefore, we attempt to exploit multiple random ε-greedy buffers to enhance explorations for a more near-perfect generalization with one original goal in off-policy RL. We demonstrate the benefit of off-policy learning from our method through an experimental comparison of DQN and a deep deterministic policy gradient in terms of discrete action, as well as continuous control for complete symmetric environments.

Download Full-text