Simulation-based policy generation using large-scale Markov decision processes

We propose and analyze a temporal concatenation heuristic for solving large-scale finite-horizon Markov decision processes (MDP), which divides the MDP into smaller sub-problems along the time horizon and generates an overall solution by simply concatenating the optimal solutions from these sub-problems. As a “black box” architecture, temporal concatenation works with a wide range of existing MDP algorithms. Our main results characterize the regret of temporal concatenation compared to the optimal solution. We provide upper bounds for general MDP instances, as well as a family of MDP instances in which the upper bounds are shown to be tight. Together, our results demonstrate temporal concatenation's potential of substantial speed-up at the expense of some performance degradation.

Download Full-text

PAC bounds for simulation-based optimization of Markov decision processes

2007 46th IEEE Conference on Decision and Control ◽

10.1109/cdc.2007.4435050 ◽

2007 ◽

Keyword(s):

Markov Decision Processes ◽

Decision Processes ◽

Simulation Based ◽

Simulation Based Optimization ◽

Markov Decision

Download Full-text

Simulation‐based Uniform Value Function Estimates of Markov Decision Processes

SIAM Journal on Control and Optimization ◽

10.1137/040619508 ◽

2006 ◽

Vol 45 (5) ◽

pp. 1633-1656 ◽

Cited By ~ 13

Author(s):

Rahul Jain ◽

Pravin P. Varaiya

Keyword(s):

Markov Decision Processes ◽

Value Function ◽

Decision Processes ◽

Uniform Value ◽

Simulation Based ◽

Markov Decision

Download Full-text

Decentralized control of multi-robot partially observable Markov decision processes using belief space macro-actions

The International Journal of Robotics Research ◽

10.1177/0278364917692864 ◽

2017 ◽

Vol 36 (2) ◽

pp. 231-258 ◽

Cited By ~ 14

Author(s):

Shayegan Omidshafiei ◽

Ali–Akbar Agha–Mohammadi ◽

Christopher Amato ◽

Shih–Yuan Liu ◽

Jonathan P How ◽

...

Keyword(s):

Markov Decision Processes ◽

Large Scale ◽

Decision Processes ◽

Delivery Problem ◽

Package Delivery ◽

Markov Decision ◽

Partially Observable Markov ◽

High Level ◽

Partially Observable ◽

Multi Robot

This work focuses on solving general multi-robot planning problems in continuous spaces with partial observability given a high-level domain description. Decentralized Partially Observable Markov Decision Processes (Dec-POMDPs) are general models for multi-robot coordination problems. However, representing and solving Dec-POMDPs is often intractable for large problems. This work extends the Dec-POMDP model to the Decentralized Partially Observable Semi-Markov Decision Process (Dec-POSMDP) to take advantage of the high-level representations that are natural for multi-robot problems and to facilitate scalable solutions to large discrete and continuous problems. The Dec-POSMDP formulation uses task macro-actions created from lower-level local actions that allow for asynchronous decision-making by the robots, which is crucial in multi-robot domains. This transformation from Dec-POMDPs to Dec-POSMDPs with a finite set of automatically-generated macro-actions allows use of efficient discrete-space search algorithms to solve them. The paper presents algorithms for solving Dec-POSMDPs, which are more scalable than previous methods since they can incorporate closed-loop belief space macro-actions in planning. These macro-actions are automatically constructed to produce robust solutions. The proposed algorithms are then evaluated on a complex multi-robot package delivery problem under uncertainty, showing that our approach can naturally represent realistic problems and provide high-quality solutions for large-scale problems.

Download Full-text