scholarly journals A Multi-Agent Reinforcement Learning Approach to Price and Comfort Optimization in HVAC-Systems

Energies ◽  
2021 ◽  
Vol 14 (22) ◽  
pp. 7491
Author(s):  
Christian Blad ◽  
Simon Bøgh ◽  
Carsten Kallesøe
Keyword(s):  
Reinforcement Learning ◽  
Control Method ◽  
Single Agent ◽  
Electric Energy ◽  
Heating System ◽  
Hvac Systems ◽  
Hvac System ◽  
Data Set ◽  
Training Time ◽  
Multi Agent

This paper addresses the challenge of minimizing training time for the control of Heating, Ventilation, and Air-conditioning (HVAC) systems with online Reinforcement Learning (RL). This is done by developing a novel approach to Multi-Agent Reinforcement Learning (MARL) to HVAC systems. In this paper, the environment formed by the HVAC system is formulated as a Markov Game (MG) in a general sum setting. The MARL algorithm is designed in a decentralized structure, where only relevant states are shared between agents, and actions are shared in a sequence, which are sensible from a system’s point of view. The simulation environment is a domestic house located in Denmark and designed to resemble an average house. The heat source in the house is an air-to-water heat pump, and the HVAC system is an Underfloor Heating system (UFH). The house is subjected to weather changes from a data set collected in Copenhagen in 2006, spanning the entire year except for June, July, and August, where heat is not required. It is shown that: (1) When comparing Single Agent Reinforcement Learning (SARL) and MARL, training time can be reduced by 70% for a four temperature-zone UFH system, (2) the agent can learn and generalize over seasons, (3) the cost of heating can be reduced by 19% or the equivalent to 750 kWh of electric energy per year for an average Danish domestic house compared to a traditional control method, and (4) oscillations in the room temperature can be reduced by 40% when comparing the RL control methods with a traditional control method.

scholarly journals Multi-Agent Reinforcement Learning: A Review of Challenges and Applications

2021 ◽  
Vol 11 (11) ◽  
pp. 4948
Author(s):  
Lorenzo Canese ◽  
Gian Carlo Cardarilli ◽  
Luca Di Di Nunzio ◽  
Rocco Fazzolari ◽  
Daniele Giardino ◽  
...  
Keyword(s):  
Reinforcement Learning ◽  
Mathematical Models ◽  
Learning Algorithms ◽  
Single Agent ◽  
Critical Issues ◽  
Multi Agent ◽  
Pros And Cons ◽  
Application Fields

In this review, we present an analysis of the most used multi-agent reinforcement learning algorithms. Starting with the single-agent reinforcement learning algorithms, we focus on the most critical issues that must be taken into account in their extension to multi-agent scenarios. The analyzed algorithms were grouped according to their features. We present a detailed taxonomy of the main multi-agent approaches proposed in the literature, focusing on their related mathematical models. For each algorithm, we describe the possible application fields, while pointing out its pros and cons. The described multi-agent algorithms are compared in terms of the most important characteristics for multi-agent reinforcement learning applications—namely, nonstationarity, scalability, and observability. We also describe the most common benchmark environments used to evaluate the performances of the considered methods.

scholarly journals On-Demand Channel Bonding in Heterogeneous WLANs: A Multi-Agent Deep Reinforcement Learning Approach

Sensors ◽  
2020 ◽  
Vol 20 (10) ◽  
pp. 2789 ◽  
Author(s):  
Hang Qi ◽  
Hao Huang ◽  
Zhiqun Hu ◽  
Xiangming Wen ◽  
Zhaoming Lu
Keyword(s):  
Reinforcement Learning ◽  
Transmission Rate ◽  
Single Agent ◽  
Time Of Day ◽  
Action Space ◽  
Traffic Load ◽  
Traffic Demand ◽  
Channel Bonding ◽  
On Demand ◽  
Multi Agent

In order to meet the ever-increasing traffic demand of Wireless Local Area Networks (WLANs), channel bonding is introduced in IEEE 802.11 standards. Although channel bonding effectively increases the transmission rate, the wider channel reduces the number of non-overlapping channels and is more susceptible to interference. Meanwhile, the traffic load differs from one access point (AP) to another and changes significantly depending on the time of day. Therefore, the primary channel and channel bonding bandwidth should be carefully selected to meet traffic demand and guarantee the performance gain. In this paper, we proposed an On-Demand Channel Bonding (O-DCB) algorithm based on Deep Reinforcement Learning (DRL) for heterogeneous WLANs to reduce transmission delay, where the APs have different channel bonding capabilities. In this problem, the state space is continuous and the action space is discrete. However, the size of action space increases exponentially with the number of APs by using single-agent DRL, which severely affects the learning rate. To accelerate learning, Multi-Agent Deep Deterministic Policy Gradient (MADDPG) is used to train O-DCB. Real traffic traces collected from a campus WLAN are used to train and test O-DCB. Simulation results reveal that the proposed algorithm has good convergence and lower delay than other algorithms.

Multi-agent reinforcement learning using ordinal action selection and approximate policy iteration

2016 ◽  
Vol 14 (06) ◽  
pp. 1650053
Author(s):  
Daxue Liu ◽  
Jun Wu ◽  
Xin Xu
Keyword(s):  
Reinforcement Learning ◽  
Single Agent ◽  
Action Selection ◽  
Policy Iteration ◽  
Common Interest ◽  
Policy Space ◽  
Markov Games ◽  
Approximate Policy Iteration ◽  
Multi Agent ◽  
Agent Coordination

Multi-agent reinforcement learning (MARL) provides a useful and flexible framework for multi-agent coordination in uncertain dynamic environments. However, the generalization ability and scalability of algorithms to large problem sizes, already problematic in single-agent RL, is an even more formidable obstacle in MARL applications. In this paper, a new MARL method based on ordinal action selection and approximate policy iteration called OAPI (Ordinal Approximate Policy Iteration), is presented to address the scalability issue of MARL algorithms in common-interest Markov Games. In OAPI, an ordinal action selection and learning strategy is integrated with distributed approximate policy iteration not only to simplify the policy space and eliminate the conflicts in multi-agent coordination, but also to realize the approximation of near-optimal policies for Markov Games with large state spaces. Based on the simplified policy space using ordinal action selection, the OAPI algorithm implements distributed approximate policy iteration utilizing online least-squares policy iteration (LSPI). This resulted in multi-agent coordination with good convergence properties with reduced computational complexity. The simulation results of a coordinated multi-robot navigation task illustrate the feasibility and effectiveness of the proposed approach.

scholarly journals An Edge Based Multi-Agent Auto Communication Method for Traffic Light Control

Sensors ◽  
2020 ◽  
Vol 20 (15) ◽  
pp. 4291 ◽  
Author(s):  
Qiang Wu ◽  
Jianqing Wu ◽  
Jun Shen ◽  
Binbin Yong ◽  
Qingguo Zhou
Keyword(s):  
Reinforcement Learning ◽  
Control Method ◽  
Communication Protocol ◽  
Traffic Signal ◽  
Signal Control ◽  
Traffic Signal Control ◽  
Light Control ◽  
Traffic Light ◽  
Traffic Light Control ◽  
Multi Agent

With smart city infrastructures growing, the Internet of Things (IoT) has been widely used in the intelligent transportation systems (ITS). The traditional adaptive traffic signal control method based on reinforcement learning (RL) has expanded from one intersection to multiple intersections. In this paper, we propose a multi-agent auto communication (MAAC) algorithm, which is an innovative adaptive global traffic light control method based on multi-agent reinforcement learning (MARL) and an auto communication protocol in edge computing architecture. The MAAC algorithm combines multi-agent auto communication protocol with MARL, allowing an agent to communicate the learned strategies with others for achieving global optimization in traffic signal control. In addition, we present a practicable edge computing architecture for industrial deployment on IoT, considering the limitations of the capabilities of network transmission bandwidth. We demonstrate that our algorithm outperforms other methods over 17% in experiments in a real traffic simulation environment.

Special Issue on New Trends in Reinforcement Learning

2009 ◽  
Vol 13 (6) ◽  
pp. 599-599
Author(s):  
Kazuteru Miyazaki ◽  
◽  
Keiki Takadama ◽  
Keyword(s):  
Reinforcement Learning ◽  
Single Agent ◽  
Large Contribution ◽  
Special Issue ◽  
Learning Classifier System ◽  
Wide Range ◽  
Important Approach ◽  
Directed Learning ◽  
Multi Agent ◽  
Institute Of Technology

Recently, the tailor-made system that grants an individual request has been recognized as the important approach. Such a system requires the ggoal-directed learningh through interaction between user and system, which is mainly addressed in greinforcement learningh domain. This special issue on gNew Trends in Reinforcement Learningh called for papers on the cuttingedge research exploring the goal-directed learning, which represents reinforcement learning. Many contributions were forthcoming, but we finally selected 12 works for publication. Although greinforcement learningh is included in the title of this special issue, the research works do not necessarily have to be on reinforcement learning itself, so long as the theme coincides with that of this special issue. In making our final selections, we gave special consideration to the kinds of research which can actively lead to new trends in reinforcement learning. Of the 12 papers in this special issue, the first four mainly deal with the expansion of the reinforcement learning method in single agent environments. These cover a broad range of research, from works based on dynamic programming to exploitation-oriented methods. The next two works deal with the Learning Classifier System (LCS), which applies the rule discovery mechanism to reinforcement learning. LCS is a technique with a long history, but for this issue, we were able to publish two theoretical works. We are also grateful to Prof. Toshio Fukuda, Nagoya University, and Prof. Kaoru Hirota, Tokyo Institute of Technology, the editors-in-chief, and the NASTEC 2008 conference staff for inviting us to guest-edit this Journal. The next four papers mainly deal with multi agent environments. We were able to draw from a wide range of research: from measuring interaction, through the expansion of techniques incorporating simultaneous learning, to research leading to application in multi agent environments. The last two contributions mainly deal with application. We publish one paper on exemplar generalization and another detailing the successful application to government bond trading. Each of these researches can be considered to be at the cutting-edge of reinforcement learning. We would like to end by saying that we hope this special issue constitutes a large contribution to the development of the field while holding a wide international appeal.

Plan-based reward shaping for multi-agent reinforcement learning

2016 ◽  
Vol 31 (1) ◽  
pp. 44-58 ◽  
Author(s):  
Sam Devlin ◽  
Daniel Kudenko
Keyword(s):  
Reinforcement Learning ◽  
Potential Function ◽  
Single Agent ◽  
Individual Agent ◽  
Reward Shaping ◽  
Multi Agent ◽  
Theoretical Results ◽  
Planning Knowledge

AbstractRecent theoretical results have justified the use of potential-based reward shaping as a way to improve the performance of multi-agent reinforcement learning (MARL). However, the question remains of how to generate a useful potential function.Previous research demonstrated the use of STRIPS operator knowledge to automatically generate a potential function for single-agent reinforcement learning. Following up on this work, we investigate the use of STRIPS planning knowledge in the context of MARL.Our results show that a potential function based on joint or individual plan knowledge can significantly improve MARL performance compared with no shaping. In addition, we investigate the limitations of individual plan knowledge as a source of reward shaping in cases where the combination of individual agent plans causes conflict.

scholarly journals Value Function Transfer for Deep Multi-Agent Reinforcement Learning Based on N-Step Returns

2019 ◽  
Author(s):  
Yong Liu ◽  
Yujing Hu ◽  
Yang Gao ◽  
Yingfeng Chen ◽  
Changjie Fan
Keyword(s):  
Reinforcement Learning ◽  
Knowledge Transfer ◽  
Value Function ◽  
Single Agent ◽  
Multi Agent Systems ◽  
Agent Systems ◽  
Markov Decision ◽  
Dimensional State Space ◽  
Multi Agent ◽  
Function Transfer

Many real-world problems, such as robot control and soccer game, are naturally modeled as sparse-interaction multi-agent systems. Reutilizing single-agent knowledge in multi-agent systems with sparse interactions can greatly accelerate the multi-agent learning process. Previous works rely on bisimulation metric to define Markov decision process (MDP) similarity for controlling knowledge transfer. However, bisimulation metric is costly to compute and is not suitable for high-dimensional state space problems. In this work, we propose more scalable transfer learning methods based on a novel MDP similarity concept. We start by defining the MDP similarity based on the N-step return (NSR) values of an MDP. Then, we propose two knowledge transfer methods based on deep neural networks called direct value function transfer and NSR-based value function transfer. We conduct experiments in image-based grid world, multi-agent particle environment (MPE) and Ms. Pac-Man game. The results indicate that the proposed methods can significantly accelerate multi-agent reinforcement learning and meanwhile get better asymptotic performance.

scholarly journals A Novel Heterogeneous Swarm Reinforcement Learning Method for Sequential Decision Making Problems

2019 ◽  
Vol 1 (2) ◽  
pp. 590-610
Author(s):  
Zohreh Akbari ◽  
Rainer Unland
Keyword(s):  
Decision Making ◽  
Reinforcement Learning ◽  
Single Agent ◽  
Sequential Decision Making ◽  
Multi Agent Systems ◽  
Sequential Decision ◽  
Agent Systems ◽  
Novel Approach ◽  
Markov Decision ◽  
Multi Agent

Sequential Decision Making Problems (SDMPs) that can be modeled as Markov Decision Processes can be solved using methods that combine Dynamic Programming (DP) and Reinforcement Learning (RL). Depending on the problem scenarios and the available Decision Makers (DMs), such RL algorithms may be designed for single-agent systems or multi-agent systems that either consist of agents with individual goals and decision making capabilities, which are influenced by other agent’s decisions, or behave as a swarm of agents that collaboratively learn a single objective. Many studies have been conducted in this area; however, when concentrating on available swarm RL algorithms, one obtains a clear view of the areas that still require attention. Most of the studies in this area focus on homogeneous swarms and so far, systems introduced as Heterogeneous Swarms (HetSs) merely include very few, i.e., two or three sub-swarms of homogeneous agents, which either, according to their capabilities, deal with a specific sub-problem of the general problem or exhibit different behaviors in order to reduce the risk of bias. This study introduces a novel approach that allows agents, which are originally designed to solve different problems and hence have higher degrees of heterogeneity, to behave as a swarm when addressing identical sub-problems. In fact, the affinity between two agents, which measures the compatibility of agents to work together towards solving a specific sub-problem, is used in designing a Heterogeneous Swarm RL (HetSRL) algorithm that allows HetSs to solve the intended SDMPs.

Sign in / Sign up

Export Citation Format

Share Document