scholarly journals Improving Policies via Search in Cooperative Partially Observable Games

2020 ◽  
Vol 34 (05) ◽  
pp. 7187-7194
Author(s):  
Adam Lerer ◽  
Hengyuan Hu ◽  
Jakob Foerster ◽  
Noam Brown
Keyword(s):  
Social Interactions ◽  
State Of The Art ◽  
Common Knowledge ◽  
Single Agent ◽  
Approximation Error ◽  
Original Performance ◽  
Search Technique ◽  
Multi Agent ◽  
Zero Sum ◽  
Partially Observable

Recent superhuman results in games have largely been achieved in a variety of zero-sum settings, such as Go and Poker, in which agents need to compete against others. However, just like humans, real-world AI systems have to coordinate and communicate with other agents in cooperative partially observable environments as well. These settings commonly require participants to both interpret the actions of others and to act in a way that is informative when being interpreted. Those abilities are typically summarized as theory of mind and are seen as crucial for social interactions. In this paper we propose two different search techniques that can be applied to improve an arbitrary agreed-upon policy in a cooperative partially observable game. The first one, single-agent search, effectively converts the problem into a single agent setting by making all but one of the agents play according to the agreed-upon policy. In contrast, in multi-agent search all agents carry out the same common-knowledge search procedure whenever doing so is computationally feasible, and fall back to playing according to the agreed-upon policy otherwise. We prove that these search procedures are theoretically guaranteed to at least maintain the original performance of the agreed-upon policy (up to a bounded approximation error). In the benchmark challenge problem of Hanabi, our search technique greatly improves the performance of every agent we tested and when applied to a policy trained using RL achieves a new state-of-the-art score of 24.61 / 25 in the game, compared to a previous-best of 24.08 / 25.

scholarly journals IPOMDP-Net: A Deep Neural Network for Partially Observable Multi-Agent Planning Using Interactive POMDPs

2019 ◽  
Vol 33 ◽  
pp. 6062-6069 ◽  
Author(s):  
Yanlin Han ◽  
Piotr Gmytrasiewicz
Keyword(s):  
Neural Network ◽  
Network Architecture ◽  
State Of The Art ◽  
Neural Computing ◽  
Neural Network Architecture ◽  
Markov Decision ◽  
Planning Algorithm ◽  
Multi Agent ◽  
Partially Observable ◽  
Multi Agent Planning

This paper introduces the IPOMDP-net, a neural network architecture for multi-agent planning under partial observability. It embeds an interactive partially observable Markov decision process (I-POMDP) model and a QMDP planning algorithm that solves the model in a neural network architecture. The IPOMDP-net is fully differentiable and allows for end-to-end training. In the learning phase, we train an IPOMDP-net on various fixed and randomly generated environments in a reinforcement learning setting, assuming observable reinforcements and unknown (randomly initialized) model functions. In the planning phase, we test the trained network on new, unseen variants of the environments under the planning setting, using the trained model to plan without reinforcements. Empirical results show that our model-based IPOMDP-net outperforms the other state-of-the-art modelfree network and generalizes better to larger, unseen environments. Our approach provides a general neural computing architecture for multi-agent planning using I-POMDPs. It suggests that, in a multi-agent setting, having a model of other agents benefits our decision-making, resulting in a policy of higher quality and better generalizability.

Exploring communication protocols and centralized critics in multi-agent deep learning

2020 ◽  
Vol 27 (4) ◽  
pp. 333-351
Author(s):  
David Simões ◽  
Nuno Lau ◽  
Luís Paulo Reis
Keyword(s):  
Deep Learning ◽  
Human Performance ◽  
Learning Algorithm ◽  
Single Agent ◽  
Communication Protocols ◽  
Relevant Information ◽  
Team Members ◽  
Deep Learning Algorithm ◽  
Multi Agent ◽  
Partially Observable

Tackling multi-agent environments where each agent has a local limited observation of the global state is a non-trivial task that often requires hand-tuned solutions. A team of agents coordinating in such scenarios must handle the complex underlying environment, while each agent only has partial knowledge about the environment. Deep reinforcement learning has been shown to achieve super-human performance in single-agent environments, and has since been adapted to the multi-agent paradigm. This paper proposes A3C3, a multi-agent deep learning algorithm, where agents are evaluated by a centralized referee during the learning phase, but remain independent from each other in actual execution. This referee’s neural network is augmented with a permutation invariance architecture to increase its scalability to large teams. A3C3 also allows agents to learn communication protocols with which agents share relevant information to their team members, allowing them to overcome their limited knowledge, and achieve coordination. A3C3 and its permutation invariant augmentation is evaluated in multiple multi-agent test-beds, which include partially-observable scenarios, swarm environments, and complex 3D soccer simulations.

scholarly journals Multi Agent Deep Learning with Cooperative Communication

2020 ◽  
Vol 10 (3) ◽  
pp. 189-207
Author(s):  
David Simões ◽  
Nuno Lau ◽  
Luís Paulo Reis
Keyword(s):  
Deep Learning ◽  
Cooperative Communication ◽  
State Of The Art ◽  
Communication Protocols ◽  
Relevant Information ◽  
Distributed Execution ◽  
Multi Agent ◽  
Multiple Environments ◽  
Partially Observable ◽  
Multi Agents

AbstractWe consider the problem of multi agents cooperating in a partially-observable environment. Agents must learn to coordinate and share relevant information to solve the tasks successfully. This article describes Asynchronous Advantage Actor-Critic with Communication (A3C2), an end-to-end differentiable approach where agents learn policies and communication protocols simultaneously. A3C2 uses a centralized learning, distributed execution paradigm, supports independent agents, dynamic team sizes, partially-observable environments, and noisy communications. We compare and show that A3C2 outperforms other state-of-the-art proposals in multiple environments.

scholarly journals Multi-Agent Distributed Deep Deterministic Policy Gradient for Partially Observable Tracking

Actuators ◽  
2021 ◽  
Vol 10 (10) ◽  
pp. 268
Author(s):  
Dongyu Fan ◽  
Haikuo Shen ◽  
Lijing Dong
Keyword(s):  
Single Agent ◽  
Training Time ◽  
Practical Applications ◽  
Tracking Tasks ◽  
Learning Tasks ◽  
Policy Gradient ◽  
Multi Agent ◽  
Decentralized Execution ◽  
Partially Observable ◽  
Distributed Tracking

In many existing multi-agent reinforcement learning tasks, each agent observes all the other agents from its own perspective. In addition, the training process is centralized, namely the critic of each agent can access the policies of all the agents. This scheme has certain limitations since every single agent can only obtain the information of its neighbor agents due to the communication range in practical applications. Therefore, in this paper, a multi-agent distributed deep deterministic policy gradient (MAD3PG) approach is presented with decentralized actors and distributed critics to realize multi-agent distributed tracking. The distinguishing feature of the proposed framework is that we adopted the multi-agent distributed training with decentralized execution, where each critic only takes the agent’s and the neighbor agents’ policies into account. Experiments were conducted in the distributed tracking tasks based on multi-agent particle environments where N(N=3,N=5) agents track a target agent with partial observation. The results showed that the proposed method achieves a higher reward with a shorter training time compared to other methods, including MADDPG, DDPG, PPO, and DQN. The proposed novel method leads to a more efficient and effective multi-agent tracking.

scholarly journals Column Generation Algorithms for Constrained POMDPs

2018 ◽  
Vol 62 ◽  
pp. 489-533 ◽  
Author(s):  
Erwin Walraven ◽  
Matthijs T. J. Spaan
Keyword(s):  
Column Generation ◽  
Decision Process ◽  
State Of The Art ◽  
Single Agent ◽  
Current State ◽  
Exact Evaluation ◽  
Markov Decision ◽  
Secondary Objective ◽  
Partially Observable ◽  
Planning Problems

In several real-world domains it is required to plan ahead while there are finite resources available for executing the plan. The limited availability of resources imposes constraints on the plans that can be executed, which need to be taken into account while computing a plan. A Constrained Partially Observable Markov Decision Process (Constrained POMDP) can be used to model resource-constrained planning problems which include uncertainty and partial observability. Constrained POMDPs provide a framework for computing policies which maximize expected reward, while respecting constraints on a secondary objective such as cost or resource consumption. Column generation for linear programming can be used to obtain Constrained POMDP solutions. This method incrementally adds columns to a linear program, in which each column corresponds to a POMDP policy obtained by solving an unconstrained subproblem. Column generation requires solving a potentially large number of POMDPs, as well as exact evaluation of the resulting policies, which is computationally difficult. We propose a method to solve subproblems in a two-stage fashion using approximation algorithms. First, we use a tailored point-based POMDP algorithm to obtain an approximate subproblem solution. Next, we convert this approximate solution into a policy graph, which we can evaluate efficiently. The resulting algorithm is a new approximate method for Constrained POMDPs in single-agent settings, but also in settings in which multiple independent agents share a global constraint. Experiments based on several domains show that our method outperforms the current state of the art.

Browser Security Attacks and Detection Techniques: A Case of Tabnabbing

2020 ◽  
Vol 8 (1) ◽  
pp. 33-41
Author(s):  
Dr. S. Sarika ◽  
Keyword(s):  
Credit Card ◽  
State Of The Art ◽  
Experimental Results ◽  
Detection Technique ◽  
Security Attacks ◽  
Agent Based ◽  
Detection Techniques ◽  
Browser Security ◽  
Cyber Threats ◽  
Multi Agent

Phishing is a malicious and deliberate act of sending counterfeit messages or mimicking a webpage. The goal is either to steal sensitive credentials like login information and credit card details or to install malware on a victim’s machine. Browser-based cyber threats have become one of the biggest concerns in networked architectures. The most prolific form of browser attack is tabnabbing which happens in inactive browser tabs. In a tabnabbing attack, a fake page disguises itself as a genuine page to steal data. This paper presents a multi agent based tabnabbing detection technique. The method detects heuristic changes in a webpage when a tabnabbing attack happens and give a warning to the user. Experimental results show that the method performs better when compared with state of the art tabnabbing detection techniques.

scholarly journals Multi-Agent Reinforcement Learning: A Review of Challenges and Applications

2021 ◽  
Vol 11 (11) ◽  
pp. 4948
Author(s):  
Lorenzo Canese ◽  
Gian Carlo Cardarilli ◽  
Luca Di Di Nunzio ◽  
Rocco Fazzolari ◽  
Daniele Giardino ◽  
...  
Keyword(s):  
Reinforcement Learning ◽  
Mathematical Models ◽  
Learning Algorithms ◽  
Single Agent ◽  
Critical Issues ◽  
Multi Agent ◽  
Pros And Cons ◽  
Application Fields

In this review, we present an analysis of the most used multi-agent reinforcement learning algorithms. Starting with the single-agent reinforcement learning algorithms, we focus on the most critical issues that must be taken into account in their extension to multi-agent scenarios. The analyzed algorithms were grouped according to their features. We present a detailed taxonomy of the main multi-agent approaches proposed in the literature, focusing on their related mathematical models. For each algorithm, we describe the possible application fields, while pointing out its pros and cons. The described multi-agent algorithms are compared in terms of the most important characteristics for multi-agent reinforcement learning applications—namely, nonstationarity, scalability, and observability. We also describe the most common benchmark environments used to evaluate the performances of the considered methods.

A Simulation Software for the Evaluation of Vulnerabilities in Reputation Management Systems

2021 ◽  
Vol 37 (1-4) ◽  
pp. 1-30
Author(s):  
Vincenzo Agate ◽  
Alessandra De Paola ◽  
Giuseppe Lo Re ◽  
Marco Morana
Keyword(s):  
Quality Of Service ◽  
Distributed Systems ◽  
State Of The Art ◽  
Simulation Software ◽  
Reputation Management ◽  
Management Systems ◽  
Security Attacks ◽  
Vulnerability Evaluation ◽  
Multi Agent

Multi-agent distributed systems are characterized by autonomous entities that interact with each other to provide, and/or request, different kinds of services. In several contexts, especially when a reward is offered according to the quality of service, individual agents (or coordinated groups) may act in a selfish way. To prevent such behaviours, distributed Reputation Management Systems (RMSs) provide every agent with the capability of computing the reputation of the others according to direct past interactions, as well as indirect opinions reported by their neighbourhood. This last point introduces a weakness on gossiped information that makes RMSs vulnerable to malicious agents’ intent on disseminating false reputation values. Given the variety of application scenarios in which RMSs can be adopted, as well as the multitude of behaviours that agents can implement, designers need RMS evaluation tools that allow them to predict the robustness of the system to security attacks, before its actual deployment. To this aim, we present a simulation software for the vulnerability evaluation of RMSs and illustrate three case studies in which this tool was effectively used to model and assess state-of-the-art RMSs.

scholarly journals EXPRESS: Standing out from the crowd: Both cue numerosity and social information affect attention in multi-agent contexts

2021 ◽  
pp. 174702182110130
Author(s):  
Francesca Capozzi ◽  
Andrew Paul Bayliss ◽  
Jelena Ristic
Keyword(s):  
Social Interactions ◽  
Social Information ◽  
Test Phase ◽  
Learning Phase ◽  
Complex Interplay ◽  
Two Phase ◽  
Attentional Allocation ◽  
Social Cue ◽  
Gaze Cuing ◽  
Multi Agent

Groups of people offer abundant opportunities for social interactions. We used a two-phase task to investigate how social cue numerosity and social information about an individual affected attentional allocation in such multi-agent settings. The learning phase was a standard gaze-cuing procedure in which a stimulus face could be either uninformative or informative about the upcoming target. The test phase was a group-cuing procedure in which the stimulus faces from the learning phase were presented in groups of three. The target could either be cued by the group minority (i.e., one face) or majority (i.e., two faces) or by uninformative or informative stimulus faces. Results showed an effect of cue numerosity, whereby responses were faster to targets cued by the group majority than the group minority. However, responses to targets cued by informative identities included in the group minority were as fast as responses to targets cued by the group majority. Thus, previously learned social information about an individual was able to offset the general enhancement of cue numerosity, revealing a complex interplay between cue numerosity and social information in guiding attention in multi-agent settings.

Sign in / Sign up

Export Citation Format

Share Document