scholarly journals Explainable Reinforcement Learning through a Causal Lens

2020 ◽  
Vol 34 (03) ◽  
pp. 2493-2500
Author(s):  
Prashan Madumal ◽  
Tim Miller ◽  
Liz Sonenberg ◽  
Frank Vetere
Keyword(s):  
Reinforcement Learning ◽  
Causal Model ◽  
Causal Models ◽  
Causal Relationships ◽  
Causal Explanations ◽  
Making Sense ◽  
Model Free ◽  
The World ◽  
Strategy Game ◽  
Real Time Strategy Game

Prominent theories in cognitive science propose that humans understand and represent the knowledge of the world through causal relationships. In making sense of the world, we build causal models in our mind to encode cause-effect relations of events and use these to explain why new events happen by referring to counterfactuals — things that did not happen. In this paper, we use causal models to derive causal explanations of the behaviour of model-free reinforcement learning agents. We present an approach that learns a structural causal model during reinforcement learning and encodes causal relationships between variables of interest. This model is then used to generate explanations of behaviour based on counterfactual analysis of the causal model. We computationally evaluate the model in 6 domains and measure performance and task prediction accuracy. We report on a study with 120 participants who observe agents playing a real-time strategy game (Starcraft II) and then receive explanations of the agents' behaviour. We investigate: 1) participants' understanding gained by explanations through task prediction; 2) explanation satisfaction and 3) trust. Our results show that causal model explanations perform better on these measures compared to two other baseline explanation models.

scholarly journals ANALYSIS OF THE APPLICATION OF REINFORCEMENT LEARNING ALGORITHMS ON THE STARCRAFT II VIDEO GAME

2020 ◽  
Vol 11 (4) ◽  
Author(s):  
Leandro Vian ◽  
Marcelo De Gomensoro Malheiros
Keyword(s):  
Reinforcement Learning ◽  
Video Game ◽  
Cost Effective ◽  
Machine Learning Techniques ◽  
Specific Training ◽  
Board Games ◽  
Learning Techniques ◽  
Technical Issues ◽  
Strategy Game ◽  
Real Time Strategy Game

In recent years Machine Learning techniques have become the driving force behind the worldwide emergence of Artificial Intelligence, producing cost-effective and precise tools for pattern recognition and data analysis. A particular approach for the training of neural networks, Reinforcement Learning (RL), achieved prominence creating almost unbeatable artificial opponents in board games like Chess or Go, and also on video games. This paper gives an overview of Reinforcement Learning and tests this approach against a very popular real-time strategy game, Starcraft II. Our goal is to examine the tools and algorithms readily available for RL, also addressing different scenarios where a neural network can be linked to Starcraft II to learn by itself. This work describes both the technical issues involved and the preliminary results obtained by the application of two specific training strategies, A2C and DQN.

scholarly journals Learning Macromanagement in Starcraft by Deep Reinforcement Learning

Sensors ◽  
2021 ◽  
Vol 21 (10) ◽  
pp. 3332
Author(s):  
Wenzhen Huang ◽  
Qiyue Yin ◽  
Junge Zhang ◽  
Kaiqi Huang
Keyword(s):  
Reinforcement Learning ◽  
Imperfect Information ◽  
Expert Knowledge ◽  
High Rate ◽  
Current State ◽  
Policy Gradient ◽  
Novel Method ◽  
Strategy Game ◽  
Real Time Strategy Game ◽  
Very High

StarCraft is a real-time strategy game that provides a complex environment for AI research. Macromanagement, i.e., selecting appropriate units to build depending on the current state, is one of the most important problems in this game. To reduce the requirements for expert knowledge and enhance the coordination of the systematic bot, we select reinforcement learning (RL) to tackle the problem of macromanagement. We propose a novel deep RL method, Mean Asynchronous Advantage Actor-Critic (MA3C), which computes the approximate expected policy gradient instead of the gradient of sampled action to reduce the variance of the gradient, and encode the history queue with recurrent neural network to tackle the problem of imperfect information. The experimental results show that MA3C achieves a very high rate of winning, approximately 90%, against the weaker opponents and it improves the win rate about 30% against the stronger opponents. We also propose a novel method to visualize and interpret the policy learned by MA3C. Combined with the visualized results and the snapshots of games, we find that the learned macromanagement not only adapts to the game rules and the policy of the opponent bot, but also cooperates well with the other modules of MA3C-Bot.

What We Call Misconceptions May Be Necessary Stepping-Stones Toward Making Sense of the World

Science Scope ◽  
2016 ◽  
Vol 039 (07) ◽  
Author(s):  
Todd Campbell ◽  
Christina Schwarz ◽  
Mark Windschitl
Keyword(s):  
Stepping Stones ◽  
Making Sense ◽  
The World

Model-Based and Model-Free Social Cognition

2019 ◽  
Author(s):  
Leor M Hackel ◽  
Jeffrey Jordan Berg ◽  
Björn Lindström ◽  
David Amodio
Keyword(s):  
Reinforcement Learning ◽  
Social Cognition ◽  
Learning Strategies ◽  
Memory Systems ◽  
Learning Task ◽  
Financial Advisors ◽  
Model Based ◽  
Model Free ◽  
Systems Model ◽  
Task Assessment

Do habits play a role in our social impressions? To investigate the contribution of habits to the formation of social attitudes, we examined the roles of model-free and model-based reinforcement learning in social interactions—computations linked in past work to habit and planning, respectively. Participants in this study learned about novel individuals in a sequential reinforcement learning paradigm, choosing financial advisors who led them to high- or low-paying stocks. Results indicated that participants relied on both model-based and model-free learning, such that each independently predicted choice during the learning task and self-reported liking in a post-task assessment. Specifically, participants liked advisors who could provide large future rewards as well as advisors who had provided them with large rewards in the past. Moreover, participants varied in their use of model-based and model-free learning strategies, and this individual difference influenced the way in which learning related to self-reported attitudes: among participants who relied more on model-free learning, model-free social learning related more to post-task attitudes. We discuss implications for attitudes, trait impressions, and social behavior, as well as the role of habits in a memory systems model of social cognition.

Sign in / Sign up

Export Citation Format

Share Document