Exploration-exploitation in multi-agent learning: Catastrophe theory meets game theory

AbstractA dynamical systems perspective on multi-agent learning, based on the link between evolutionary game theory and reinforcement learning, provides an improved, qualitative understanding of the emerging collective learning dynamics. However, confusion exists with respect to how this dynamical systems account of multi-agent learning should be interpreted. In this article, I propose to embed the dynamical systems description of multi-agent learning into different abstraction levels of cognitive analysis. The purpose of this work is to make the connections between these levels explicit in order to gain improved insight into multi-agent learning. I demonstrate the usefulness of this framework with the general and widespread class of temporal-difference reinforcement learning. I find that its deterministic dynamical systems description follows a minimum free-energy principle and unifies a boundedly rational account of game theory with decision-making under uncertainty. I then propose an on-line sample-batch temporal-difference algorithm which is characterized by the combination of applying a memory-batch and separated state-action value estimation. I find that this algorithm serves as a micro-foundation of the deterministic learning equations by showing that its learning trajectories approach the ones of the deterministic learning equations under large batch sizes. Ultimately, this framework of embedding a dynamical systems description into different abstraction levels gives guidance on how to unleash the full potential of the dynamical systems approach to multi-agent learning.

Download Full-text

Hierarchical Reinforcement Learning

ACM Computing Surveys ◽

10.1145/3453160 ◽

2021 ◽

Vol 54 (5) ◽

pp. 1-35

Author(s):

Shubham Pateria ◽

Budhitama Subagdja ◽

Ah-hwee Tan ◽

Chai Quek

Keyword(s):

Reinforcement Learning ◽

Future Research ◽

Comprehensive Overview ◽

Open Problems ◽

Practical Applications ◽

Hierarchical Reinforcement Learning ◽

The Past ◽

Agent Learning ◽

Multi Agent ◽

Supplementary Material

Hierarchical Reinforcement Learning (HRL) enables autonomous decomposition of challenging long-horizon decision-making tasks into simpler subtasks. During the past years, the landscape of HRL research has grown profoundly, resulting in copious approaches. A comprehensive overview of this vast landscape is necessary to study HRL in an organized manner. We provide a survey of the diverse HRL approaches concerning the challenges of learning hierarchical policies, subtask discovery, transfer learning, and multi-agent learning using HRL. The survey is presented according to a novel taxonomy of the approaches. Based on the survey, a set of important open problems is proposed to motivate the future research in HRL. Furthermore, we outline a few suitable task domains for evaluating the HRL approaches and a few interesting examples of the practical applications of HRL in the Supplementary Material.

Download Full-text

The possible and the impossible in multi-agent learning

Artificial Intelligence ◽

10.1016/j.artint.2006.10.015 ◽

2007 ◽

Vol 171 (7) ◽

pp. 429-433 ◽

Cited By ~ 16

Author(s):

H. Peyton Young

Keyword(s):

Agent Learning ◽

Multi Agent

Download Full-text

Multi-agent Learning Algorithms

Encyclopedia of Machine Learning and Data Mining ◽

10.1007/978-1-4899-7687-1_569 ◽

2017 ◽

pp. 860-863

Author(s):

Yoav Shoham ◽

Rob Powers

Keyword(s):

Learning Algorithms ◽

Agent Learning ◽

Multi Agent

Download Full-text

The Role Of Cooperation in Multi-Agent Learning

CKBS ’90 ◽

10.1007/978-1-4471-1831-2_9 ◽

1991 ◽

pp. 164-180 ◽

Cited By ~ 1

Author(s):

Sati S. Sian

Keyword(s):

Agent Learning ◽

Multi Agent

Download Full-text

GAME THEORY BASED MULTI CRITERIA DECISION MAKING PROBLEM UNDER UNCERTAINTY: A CASE STUDY ON INDIAN TEA INDUSTRY

Journal of Business Economics and Management ◽

10.3846/16111699.2017.1401553 ◽

2018 ◽

Vol 19 (1) ◽

pp. 154-175 ◽

Cited By ~ 6

Author(s):

Animesh DEBNATH ◽

Abhirup BANDYOPADHYAY ◽

Jagannath ROY ◽

Samarjit KAR

Keyword(s):

Game Theory ◽

Decision Making ◽

Evolutionary Game ◽

Market Volatility ◽

Evolutionary Strategies ◽

Multi Criteria Decision Making ◽

Tea Industry ◽

Multi Agent ◽

The Impact ◽

Indian Tea

The long-term evolution of multi agent multi criteria decision making (MCDM) and to obtain sustainable decision a novel methodology is proposed based on evolutionary game theory. In this paper multi agent MCDM is represented as an evolutionary game and the evolutionary strategies are defined as sustainable decisions. Here we consider the problem of decision making in Indian Tea Industry. The agents in this game are essentially Indian Tea Estate owner and Indian Tea board. The replicator dynamics of the evolutionary game are studied to obtain evolutionary strategies which could be defined as sustainable strategies. The multi agent MCDM in Indian Tea Industry is considered under different socio-political and Corporate Social Responsibility scenario and groups of Indian Tea Industry. Again, the impacts of imprecision and market volatility on the outcome of some strategies (decisions) are studied. In this paper the imprecision on the impact of the strategies are modelled as fuzzy numbers whereas the market volatility is taken into account as white noise. Hence the MCDM problem for Indian Tea Industry is modelled as a hybrid evolutionary game. The probabilities of strategies are obtained by solving hybrid evolutionary game and could be represented as a Dempster-Shafer belief structure. The simulation results facilitate the Decision Makers to choose the strategies (decisions) under different type of uncertainty.

Download Full-text