A Nonlinear Approach to Robust Routing Based on Reinforcement Learning with State Space Compression and Adaptive Basis Construction

AbstractThis work analyses the performance of Reinforcement Learning (RL) versus Swarm Intelligence (SI) for coordinating multiple unmanned High Altitude Platform Stations (HAPS) for communications area coverage. It builds upon previous work which looked at various elements of both algorithms. The main aim of this paper is to address the continuous state-space challenge within this work by using partitioning to manage the high dimensionality problem. This enabled comparing the performance of the classical cases of both RL and SI establishing a baseline for future comparisons of improved versions. From previous work, SI was observed to perform better across various key performance indicators. However, after tuning parameters and empirically choosing suitable partitioning ratio for the RL state space, it was observed that the SI algorithm still maintained superior coordination capability by achieving higher mean overall user coverage (about 20% better than the RL algorithm), in addition to faster convergence rates. Though the RL technique showed better average peak user coverage, the unpredictable coverage dip was a key weakness, making SI a more suitable algorithm within the context of this work.

Download Full-text

Anomaly detection using state‐space models and reinforcement learning

Structural Control and Health Monitoring ◽

10.1002/stc.2720 ◽

2021 ◽

Author(s):

Shervin Khazaeli ◽

Luong Ha Nguyen ◽

James A. Goulet

Keyword(s):

Reinforcement Learning ◽

Anomaly Detection ◽

State Space ◽

State Space Models

Download Full-text

Adaptive state space partitioning for reinforcement learning

Engineering Applications of Artificial Intelligence ◽

10.1016/j.engappai.2004.08.005 ◽

2004 ◽

Vol 17 (6) ◽

pp. 577-588 ◽

Cited By ~ 22

Author(s):

Ivan S.K. Lee ◽

Henry Y.K. Lau

Keyword(s):

Reinforcement Learning ◽

State Space ◽

Space Partitioning

Download Full-text

Effect of human guidance and state space size on Interactive Reinforcement Learning

2011 RO-MAN ◽

10.1109/roman.2011.6005223 ◽

2011 ◽

Cited By ~ 37

Author(s):

Halit Bener Suay ◽

Sonia Chernova

Keyword(s):

Reinforcement Learning ◽

State Space ◽

Space Size

Download Full-text

Generalized Representation Learning Methods for Deep Reinforcement Learning

Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence ◽

10.24963/ijcai.2020/748 ◽

2020 ◽

Author(s):

Hanhua Zhu

Keyword(s):

Reinforcement Learning ◽

State Space ◽

Representation Learning ◽

Learning Methods ◽

New Methods ◽

Compact State

Deep reinforcement learning (DRL) increases the successful applications of reinforcement learning (RL) techniques but also brings challenges such as low sample efficiency. In this work, I propose generalized representation learning methods to obtain compact state space suitable for RL from a raw observation state. I expect my new methods will increase sample efficiency of RL by understandable representations of state and therefore improve the performance of RL.

Download Full-text

Hierarchical Neuro-Fuzzy Systems Part II

Encyclopedia of Artificial Intelligence ◽

10.4018/978-1-59904-849-9.ch121 ◽

2011 ◽

pp. 817-824

Author(s):

Marley Vellasco ◽

Marco Pacheco ◽

Karla Figueiredo ◽

Flavio Souza

Keyword(s):

Reinforcement Learning ◽

State Space ◽

Learning Process ◽

Fuzzy Systems ◽

Space Partitioning ◽

Learning Methods ◽

New Class ◽

Neuro Fuzzy ◽

Binary Space Partitioning ◽

Priori Information

This paper describes a new class of neuro-fuzzy models, called Reinforcement Learning Hierarchical Neuro- Fuzzy Systems (RL-HNF). These models employ the BSP (Binary Space Partitioning) and Politree partitioning of the input space [Chrysanthou,1992] and have been developed in order to bypass traditional drawbacks of neuro-fuzzy systems: the reduced number of allowed inputs and the poor capacity to create their own structure and rules (ANFIS [Jang,1997], NEFCLASS [Kruse,1995] and FSOM [Vuorimaa,1994]). These new models, named Reinforcement Learning Hierarchical Neuro-Fuzzy BSP (RL-HNFB) and Reinforcement Learning Hierarchical Neuro-Fuzzy Politree (RL-HNFP), descend from the original HNFB that uses Binary Space Partitioning (see Hierarchical Neuro-Fuzzy Systems Part I). By using hierarchical partitioning, together with the Reinforcement Learning (RL) methodology, a new class of Neuro-Fuzzy Systems (SNF) was obtained, which executes, in addition to automatically learning its structure, the autonomous learning of the actions to be taken by an agent, dismissing a priori information (number of rules, fuzzy rules and sets) relative to the learning process. These characteristics represent an important differential when compared with existing intelligent agents learning systems, because in applications involving continuous environments and/or environments considered to be highly dimensional, the use of traditional Reinforcement Learning methods based on lookup tables (a table that stores value functions for a small or discrete state space) is no longer possible, since the state space becomes too large. This second part of hierarchical neuro-fuzzy systems focus on the use of reinforcement learning process. The first part presented HNFB models based on supervised learning methods. The RL-HNFB and RL-HNFP models were evaluated in a benchmark control application and a simulated Khepera robot environment with multiple obstacles.

Download Full-text