Shapley Q-Value: A Local Reward Approach to Solve Global Reward Games

Cooperative game is a critical research area in the multi-agent reinforcement learning (MARL). Global reward game is a subclass of cooperative games, where all agents aim to maximize the global reward. Credit assignment is an important problem studied in the global reward game. Most of previous works stood by the view of non-cooperative-game theoretical framework with the shared reward approach, i.e., each agent being assigned a shared global reward directly. This, however, may give each agent an inaccurate reward on its contribution to the group, which could cause inefficient learning. To deal with this problem, we i) introduce a cooperative-game theoretical framework called extended convex game (ECG) that is a superset of global reward game, and ii) propose a local reward approach called Shapley Q-value. Shapley Q-value is able to distribute the global reward, reflecting each agent's own contribution in contrast to the shared reward approach. Moreover, we derive an MARL algorithm called Shapley Q-value deep deterministic policy gradient (SQDDPG), using Shapley Q-value as the critic for each agent. We evaluate SQDDPG on Cooperative Navigation, Prey-and-Predator and Traffic Junction, compared with the state-of-the-art algorithms, e.g., MADDPG, COMA, Independent DDPG and Independent A2C. In the experiments, SQDDPG shows a significant improvement on the convergence rate. Finally, we plot Shapley Q-value and validate the property of fair credit assignment.

Complexity of inheritance of ℱ-convexity for restricted games induced by minimum partitions

Let G = (N, E, w) be a weighted communication graph. For any subset A ⊆ N, we delete all minimum-weight edges in the subgraph induced by A. The connected components of the resultant subgraph constitute the partition 𝒫min(A) of A. Then, for every cooperative game (N, v), the 𝒫min-restricted game (N, v̅) is defined by v̅(A)=∑F∈𝒫min(A)v(F) for all A ⊆ N. We prove that we can decide in polynomial time if there is inheritance of ℱ-convexity, i.e., if for every ℱ-convex game the 𝒫min-restricted game is ℱ-convex, where ℱ-convexity is obtained by restricting convexity to connected subsets. This implies that we can also decide in polynomial time for any unweighted graph if there is inheritance of convexity for Myerson’s graph-restricted game.

LARGENESS OF THE CORE OF k-CONVEX SYMMETRIC GAMES

A cooperative TU game is said to posses a large core as defined by Sharkey [1982] if for every acceptable vector there is a smaller core vector in the game. This paper is devoted to characterization(s) of largeness of the core of a subclass of games known as k-convex games (containing the convex games in case k = n). The k-convex games were defined by Driessen [1988] because of the core structure they possess, which is the same as that of a suitably defined convex game. The main goal is to show that the totally balanced symmetric k-convex games possess a large core if and only if the game is convex.

CONVEX GAMES VERSUS CLAN GAMES

In this paper we provide characterizations of convex games and total clan games by using properties of their corresponding marginal games. We show that a "dualize and restrict" procedure transforms total clan games with zero worth for the clan into monotonic convex games. Furthermore, each monotonic convex game generates a total clan game with zero worth for the clan by a "dualize and extend" procedure. These procedures are also useful for relating core elements and elements of the Weber set of the corresponding games.

CONVEXITY IN STOCHASTIC COOPERATIVE SITUATIONS

This paper introduces a new model concerning cooperative situations in which the payoffs are modeled by random variables. We analyze these situations by means of cooperative games with random payoffs. Special attention is paid to three types of convexity, namely coalitional-merge, individual-merge and marginal convexity. The relations between these types are studied and in particular, as opposed to their deterministic counterparts for TU games, we show that these three types of convexity are not equivalent. However, all types imply that the core of the game is nonempty. Sufficient conditions on the preferences are derived such that the Shapley value, defined as the average of the marginal vectors, is an element of the core of a convex game.