An Improved Approach towards Multi-Agent Pursuit–Evasion Game Decision-Making Using Deep Reinforcement Learning

A pursuit–evasion game is a classical maneuver confrontation problem in the multi-agent systems (MASs) domain. An online decision technique based on deep reinforcement learning (DRL) was developed in this paper to address the problem of environment sensing and decision-making in pursuit–evasion games. A control-oriented framework developed from the DRL-based multi-agent deep deterministic policy gradient (MADDPG) algorithm was built to implement multi-agent cooperative decision-making to overcome the limitation of the tedious state variables required for the traditionally complicated modeling process. To address the effects of errors between a model and a real scenario, this paper introduces adversarial disturbances. It also proposes a novel adversarial attack trick and adversarial learning MADDPG (A2-MADDPG) algorithm. By introducing an adversarial attack trick for the agents themselves, uncertainties of the real world are modeled, thereby optimizing robust training. During the training process, adversarial learning was incorporated into our algorithm to preprocess the actions of multiple agents, which enabled them to properly respond to uncertain dynamic changes in MASs. Experimental results verified that the proposed approach provides superior performance and effectiveness for pursuers and evaders, and both can learn the corresponding confrontational strategy during training.

Two-Stage Pursuit Strategy for Incomplete-Information Impulsive Space Pursuit-Evasion Mission Using Reinforcement Learning

This paper presents a novel and robust two-stage pursuit strategy for the incomplete-information impulsive space pursuit-evasion missions considering the J2 perturbation. The strategy firstly models the impulsive pursuit-evasion game problem into a far-distance rendezvous stage and a close-distance game stage according to the perception range of the evader. For the far-distance rendezvous stage, it is transformed into a rendezvous trajectory optimization problem and a new objective function is proposed to obtain the pursuit trajectory with the optimal terminal pursuit capability. For the close-distance game stage, a closed-loop pursuit approach is proposed using one of the reinforcement learning algorithms, i.e., the deep deterministic policy gradient algorithm, to solve and update the pursuit trajectory for the incomplete-information impulsive pursuit-evasion missions. The feasibility of this novel strategy and its robustness to different initial states of the pursuer and evader and to the evasion strategies are demonstrated for the sun-synchronous orbit pursuit-evasion game scenarios. The results of the Monte Carlo tests show that the successful pursuit ratio of the proposed method is over 91% for all the given scenarios.

Does Whistleblowing on Tax Evaders Reduce Ingroup Cooperation?

Whistleblowing is a powerful and rather inexpensive instrument to deter tax evasion. Despite the deterrent effects on tax evasion, whistleblowing can reduce trust and undermine agents’ attitude to cooperate with group members. Yet, no study has investigated the potential spillover effects of whistleblowing on ingroup cooperation. This paper reports results of a laboratory experiment in which subjects participate in two consecutive phases in unchanging groups: a tax evasion game, followed by a generalized gift exchange game. Two dimensions are manipulated in our experiment: the inclusion of a whistleblowing stage in which, after observing others’ declared incomes, subjects can signal other group members to the tax authority, and the provision of information about the content of the second phase before the tax evasion game is played. Our results show that whistleblowing is effective in both curbing tax evasion and improving the precision of tax auditing. Moreover, we detect no statistically significant spillover effects of whistleblowing on ingroup cooperation in the subsequent generalized gift exchange game, with this result being unaffected by the provision of information about the experimental task in the second phase. Finally, the provision of information does not significantly alter subjects’ (tax and whistleblowing) choices in the tax evasion game: thus, knowledge about perspective ingroup cooperation did not alter attitude toward whistleblowing.

Incomplete Information Pursuit-Evasion Game Control for a Space Non-Cooperative Target

Aiming to solve the optimal control problem for the pursuit-evasion game with a space non-cooperative target under the condition of incomplete information, a new method degenerating the game into a strong tracking problem is proposed, where the unknown target maneuver is processed as colored noise. First, the relative motion is modeled in the rotating local vertical local horizontal (LVLH) frame originated at a virtual Chief based on the Hill-Clohessy-Wiltshire relative dynamics, while the measurement models for three different sensor schemes (i.e., single LOS (line-of-sight) sensor, LOS range sensor and double LOS sensor) are established and an extended Kalman Filter (EKF) is used to obtain the relative state of target. Next, under the assumption that the unknown maneuver of the target is colored noise, the game control law of chaser is derived based on the linear quadratic differential game theory. Furthermore, the optimal control law considering the thrust limitation is obtained. After that, the observability of the relative orbit state is analyzed, where the relative orbit is weakly observable in a short period of time in the case of only LOS angle measurements, fully observable in the cases of LOS range and double LOS measurement schemes. Finally, numerical simulations are conducted to verify the proposed method. The results show that by using the single LOS scheme, the chaser would firstly approach the target but then would lose the game because of the existence of the target’s unknown maneuver. Conversely, the chaser can successfully win the game in the cases of LOS range and double LOS sensor schemes.

Solving Partially Observable Stochastic Shortest-Path Games

We study the two-player zero-sum extension of the partially observable stochastic shortest-path problem where one agent has only partial information about the environment. We formulate this problem as a partially observable stochastic game (POSG): given a set of target states and negative rewards for each transition, the player with imperfect information maximizes the expected undiscounted total reward until a target state is reached. The second player with the perfect information aims for the opposite. We base our formalism on POSGs with one-sided observability (OS-POSGs) and give the following contributions: (1) we introduce a novel heuristic search value iteration algorithm that iteratively solves depth-limited variants of the game, (2) we derive the bound on the depth guaranteeing an arbitrary precision, (3) we propose a novel upper-bound estimation that allows early terminations, and (4) we experimentally evaluate the algorithm on a pursuit-evasion game.

Revisiting a Three-Player Pursuit-Evasion Game

We consider the game of a holonomic evader passing between two holonomic pursuers. The optimal trajectories of this game are known. We give a detailed explanation of the game of kind’s solution and present a computationally efficient way to obtain trajectories numerically by integrating the retrograde path equations. Additionally, we propose a method for calculating the partial derivatives of the Value function in the game of degree. This latter result applies to differential games with homogeneous Value.