LeDeepChef Deep Reinforcement Learning Agent for Families of Text-Based Games

While Reinforcement Learning (RL) approaches lead to significant achievements in a variety of areas in recent history, natural language tasks remained mostly unaffected, due to the compositional and combinatorial nature that makes them notoriously hard to optimize. With the emerging field of Text-Based Games (TBGs), researchers try to bridge this gap. Inspired by the success of RL algorithms on Atari games, the idea is to develop new methods in a restricted game world and then gradually move to more complex environments. Previous work in the area of TBGs has mainly focused on solving individual games. We, however, consider the task of designing an agent that not just succeeds in a single game, but performs well across a whole family of games, sharing the same theme. In this work, we present our deep RL agent—LeDeepChef—that shows generalization capabilities to never-before-seen games of the same family with different environments and task descriptions. The agent participated in Microsoft Research's First TextWorld Problems: A Language and Reinforcement Learning Challenge and outperformed all but one competitor on the final test set. The games from the challenge all share the same theme, namely cooking in a modern house environment, but differ significantly in the arrangement of the rooms, the presented objects, and the specific goal (recipe to cook). To build an agent that achieves high scores across a whole family of games, we use an actor-critic framework and prune the action-space by using ideas from hierarchical reinforcement learning and a specialized module trained on a recipe database.

Complexity of inheritance of ℱ-convexity for restricted games induced by minimum partitions

Let G = (N, E, w) be a weighted communication graph. For any subset A ⊆ N, we delete all minimum-weight edges in the subgraph induced by A. The connected components of the resultant subgraph constitute the partition 𝒫min(A) of A. Then, for every cooperative game (N, v), the 𝒫min-restricted game (N, v̅) is defined by v̅(A)=∑F∈𝒫min(A)v(F) for all A ⊆ N. We prove that we can decide in polynomial time if there is inheritance of ℱ-convexity, i.e., if for every ℱ-convex game the 𝒫min-restricted game is ℱ-convex, where ℱ-convexity is obtained by restricting convexity to connected subsets. This implies that we can also decide in polynomial time for any unweighted graph if there is inheritance of convexity for Myerson’s graph-restricted game.

Convexity of graph-restricted games induced by minimum partitions

We consider restricted games on weighted graphs associated with minimum partitions. We replace in the classical definition of Myerson restricted game the connected components of any subgraph by the sub-components corresponding to a minimum partition. This minimum partition 𝒫min is i nduced by the deletion of the minimum weight edges. We provide five necessary conditions on the graph edge-weights to have inheritance of convexity from the underlying game to the restricted game associated with 𝒫min. Then, we establish that these conditions are also sufficient for a weaker condition, called ℱ-convexity, obtained by restriction of convexity to connected subsets. Moreover, we prove that inheritance of convexity for Myerson restricted game associated with a given graph G is equivalent to inheritance of ℱ-convexity for the 𝒫min-restricted game associated with a particular weighted graph G′ built from G by adding a dominating vertex, and with only two different edge-weights. Then, we prove that G is cycle-complete if and only if a specific condition on adjacent cycles is satisfied on G′.

An Exact Double-Oracle Algorithm for Zero-Sum Extensive-Form Games with Imperfect Information

Developing scalable solution algorithms is one of the central problems in computational game theory. We present an iterative algorithm for computing an exact Nash equilibrium for two-player zero-sum extensive-form games with imperfect information. Our approach combines two key elements: (1) the compact sequence-form representation of extensive-form games and (2) the algorithmic framework of double-oracle methods. The main idea of our algorithm is to restrict the game by allowing the players to play only selected sequences of available actions. After solving the restricted game, new sequences are added by finding best responses to the current solution using fast algorithms. We experimentally evaluate our algorithm on a set of games inspired by patrolling scenarios, board, and card games. The results show significant runtime improvements in games admitting an equilibrium with small support, and substantial improvement in memory use even on games with large support. The improvement in memory use is particularly important because it allows our algorithm to solve much larger game instances than existing linear programming methods. Our main contributions include (1) a generic sequence-form double-oracle algorithm for solving zero-sum extensive-form games; (2) fast methods for maintaining a valid restricted game model when adding new sequences; (3) a search algorithm and pruning methods for computing best-response sequences; (4) theoretical guarantees about the convergence of the algorithm to a Nash equilibrium; (5) experimental analysis of our algorithm on several games, including an approximate version of the algorithm.