Approximation algorithms for restless bandit problems

2010 ◽  
Vol 58 (1) ◽  
pp. 1-50 ◽  
Author(s):  
Sudipto Guha ◽  
Kamesh Munagala ◽  
Peng Shi
Keyword(s):  
Approximation Algorithms ◽  
Bandit Problems ◽  
Restless Bandit

scholarly journals Online Pandora’s Boxes and Bandits

2019 ◽  
Vol 33 ◽  
pp. 1885-1892 ◽  
Author(s):  
Hossein Esfandiari ◽  
MohammadTaghi HajiAghayi ◽  
Brendan Lucier ◽  
Michael Mitzenmacher
Keyword(s):  
Decision Making ◽  
Reinforcement Learning ◽  
Approximation Algorithms ◽  
Standard Model ◽  
Decision Process ◽  
Bandit Problems ◽  
Knapsack Constraints ◽  
Feasibility Constraints ◽  
The Cost

We consider online variations of the Pandora’s box problem (Weitzman 1979), a standard model for understanding issues related to the cost of acquiring information for decision-making. Our problem generalizes both the classic Pandora’s box problem and the prophet inequality framework. Boxes are presented online, each with a random value and cost drawn jointly from some known distribution. Pandora chooses online whether to open each box given its cost, and then chooses irrevocably whether to keep the revealed prize or pass on it. We aim for approximation algorithms against adversaries that can choose the largest prize over any opened box, and use optimal offline policies to decide which boxes to open (without knowledge of the value inside)1. We consider variations where Pandora can collect multiple prizes subject to feasibility constraints, such as cardinality, matroid, or knapsack constraints. We also consider variations related to classic multi-armed bandit problems from reinforcement learning. Our results use a reduction-based framework where we separate the issues of the cost of acquiring information from the online decision process of which prizes to keep. Our work shows that in many scenarios, Pandora can achieve a good approximation to the best possible performance.

scholarly journals Some indexable families of restless bandit problems

2006 ◽  
Vol 38 (3) ◽  
pp. 643-672 ◽  
Author(s):  
K. D. Glazebrook ◽  
D. Ruiz-Hernandez ◽  
C. Kirkbride
Keyword(s):  
Index Theory ◽  
Stochastic Scheduling ◽  
Gittins Index ◽  
Scheduling Problems ◽  
Bandit Problems ◽  
Index Policy ◽  
Restless Bandit ◽  
Machine Maintenance ◽  
State Evolution ◽  
Strong Performance

In 1988 Whittle introduced an important but intractable class of restless bandit problems which generalise the multiarmed bandit problems of Gittins by allowing state evolution for passive projects. Whittle's account deployed a Lagrangian relaxation of the optimisation problem to develop an index heuristic. Despite a developing body of evidence (both theoretical and empirical) which underscores the strong performance of Whittle's index policy, a continuing challenge to implementation is the need to establish that the competing projects all pass an indexability test. In this paper we employ Gittins' index theory to establish the indexability of (inter alia) general families of restless bandits which arise in problems of machine maintenance and stochastic scheduling problems with switching penalties. We also give formulae for the resulting Whittle indices. Numerical investigations testify to the outstandingly strong performance of the index heuristics concerned.

Index policies for a class of discounted restless bandits

2002 ◽  
Vol 34 (04) ◽  
pp. 754-774 ◽  
Author(s):  
K. D. Glazebrook ◽  
J. Niño-Mora ◽  
P. S. Ansell
Keyword(s):  
Conservation Laws ◽  
Special Class ◽  
Computational Study ◽  
Bandit Problems ◽  
Index Policy ◽  
Restless Bandit ◽  
Restless Bandits ◽  
Index Policies ◽  
Strong Performance ◽  
Dual Speed

The paper concerns a class of discounted restless bandit problems which possess an indexability property. Conservation laws yield an expression for the reward suboptimality of a general policy. These results are utilised to study the closeness to optimality of an index policy for a special class of simple and natural dual speed restless bandits for which indexability is guaranteed. The strong performance of the index policy is confirmed by a computational study.

scholarly journals Some indexable families of restless bandit problems

2006 ◽  
Vol 38 (03) ◽  
pp. 643-672 ◽  
Author(s):  
K. D. Glazebrook ◽  
D. Ruiz-Hernandez ◽  
C. Kirkbride
Keyword(s):  
Index Theory ◽  
Stochastic Scheduling ◽  
Gittins Index ◽  
Scheduling Problems ◽  
Bandit Problems ◽  
Index Policy ◽  
Restless Bandit ◽  
Machine Maintenance ◽  
State Evolution ◽  
Strong Performance

In 1988 Whittle introduced an important but intractable class of restless bandit problems which generalise the multiarmed bandit problems of Gittins by allowing state evolution for passive projects. Whittle's account deployed a Lagrangian relaxation of the optimisation problem to develop an index heuristic. Despite a developing body of evidence (both theoretical and empirical) which underscores the strong performance of Whittle's index policy, a continuing challenge to implementation is the need to establish that the competing projects all pass an indexability test. In this paper we employ Gittins' index theory to establish the indexability of (inter alia) general families of restless bandits which arise in problems of machine maintenance and stochastic scheduling problems with switching penalties. We also give formulae for the resulting Whittle indices. Numerical investigations testify to the outstandingly strong performance of the index heuristics concerned.

Index policies for a class of discounted restless bandits

2002 ◽  
Vol 34 (4) ◽  
pp. 754-774 ◽  
Author(s):  
K. D. Glazebrook ◽  
J. Niño-Mora ◽  
P. S. Ansell
Keyword(s):  
Conservation Laws ◽  
Special Class ◽  
Computational Study ◽  
Bandit Problems ◽  
Index Policy ◽  
Restless Bandit ◽  
Restless Bandits ◽  
Index Policies ◽  
Strong Performance ◽  
Dual Speed

The paper concerns a class of discounted restless bandit problems which possess an indexability property. Conservation laws yield an expression for the reward suboptimality of a general policy. These results are utilised to study the closeness to optimality of an index policy for a special class of simple and natural dual speed restless bandits for which indexability is guaranteed. The strong performance of the index policy is confirmed by a computational study.

Learning, risk attitude and hot stoves in restless bandit problems

2009 ◽  
Vol 53 (3) ◽  
pp. 155-167 ◽  
Author(s):  
Guido Biele ◽  
Ido Erev ◽  
Eyal Ert
Keyword(s):  
Risk Attitude ◽  
Bandit Problems ◽  
Restless Bandit

scholarly journals INDEXABILITY AND OPTIMAL INDEX POLICIES FOR A CLASS OF REINITIALISING RESTLESS BANDITS

2015 ◽  
Vol 30 (1) ◽  
pp. 1-23 ◽  
Author(s):  
Sofía S. Villar
Keyword(s):  
Closed Form ◽  
Surveillance Systems ◽  
Bandit Problems ◽  
Index Policy ◽  
Initial State ◽  
Restless Bandit ◽  
Index Policies ◽  
Markov Decision ◽  
Whittle Index ◽  
Partially Observable

Motivated by a class of Partially Observable Markov Decision Processes with application in surveillance systems in which a set of imperfectly observed state processes is to be inferred from a subset of available observations through a Bayesian approach, we formulate and analyze a special family of multi-armed restless bandit problems. We consider the problem of finding an optimal policy for observing the processes that maximizes the total expected net rewards over an infinite time horizon subject to the resource availability. From the Lagrangian relaxation of the original problem, an index policy can be derived, as long as the existence of the Whittle index is ensured. We demonstrate that such a class of reinitializing bandits in which the projects' state deteriorates while active and resets to its initial state when passive until its completion possesses the structural property of indexability and we further show how to compute the index in closed form. In general, the Whittle index rule for restless bandit problems does not achieve optimality. However, we show that the proposed Whittle index rule is optimal for the problem under study in the case of stochastically heterogenous arms under the expected total criterion, and it is further recovered by a simple tractable rule referred to as the 1-limited Round Robin rule. Moreover, we illustrate the significant suboptimality of other widely used heuristic: the Myopic index rule, by computing in closed form its suboptimality gap. We present numerical studies which illustrate for the more general instances the performance advantages of the Whittle index rule over other simple heuristics.

Sign in / Sign up

Export Citation Format

Share Document