Independently Expiring Multiarmed Bandits

A multiarmed bandit problem is studied when the arms are not always available. The arms are first assumed to be intermittently available with some state/action-dependent probabilities. It is proven that no index policy can attain the maximum expected total discounted reward in every instance of that problem. The Whittle index policy is derived, and its properties are studied. Then it is assumed that the arms may break down, but repair is an option at some cost, and the new Whittle index policy is derived. Both problems are indexable. The proposed index policies cannot be dominated by any other index policy over all multiarmed bandit problems considered here. Whittle indices are evaluated for Bernoulli arms with unknown success probabilities.

Download Full-text

A Marginal Productivity Index Policy for the Finite-Horizon Multiarmed Bandit Problem

Proceedings of the 44th IEEE Conference on Decision and Control ◽

10.1109/cdc.2005.1582407 ◽

2006 ◽

Cited By ~ 2

Author(s):

J. Nino-Mora

Keyword(s):

Finite Horizon ◽

Productivity Index ◽

Bandit Problem ◽

Marginal Productivity ◽

Index Policy ◽

Multiarmed Bandit

Download Full-text

A Note on the Equivalence of Upper Confidence Bounds and Gittins Indices for Patient Agents

Operations Research ◽

10.1287/opre.2020.1987 ◽

2020 ◽

Author(s):

Daniel Russo

Keyword(s):

Posterior Distribution ◽

Error Term ◽

Discount Factor ◽

Gittins Index ◽

Confidence Bound ◽

Bandit Problem ◽

Confidence Bounds ◽

Upper Confidence Bound ◽

Gittins Indices ◽

Multiarmed Bandit

This note gives a short, self-contained proof of a sharp connection between Gittins indices and Bayesian upper confidence bound algorithms. I consider a Gaussian multiarmed bandit problem with discount factor [Formula: see text]. The Gittins index of an arm is shown to equal the [Formula: see text]-quantile of the posterior distribution of the arm's mean plus an error term that vanishes as [Formula: see text]. In this sense, for sufficiently patient agents, a Gittins index measures the highest plausible mean-reward of an arm in a manner equivalent to an upper confidence bound.

Download Full-text

Index policies for discounted bandit problems with availability constraints

Advances in Applied Probability ◽

10.1239/aap/1214950209 ◽

2008 ◽

Vol 40 (2) ◽

pp. 377-400 ◽

Cited By ~ 5

Author(s):

Savas Dayanik ◽

Warren Powell ◽

Kazutoshi Yamazaki

Keyword(s):

Bandit Problem ◽

Bandit Problems ◽

Index Policy ◽

State Action ◽

Index Policies ◽

Availability Constraints ◽

Whittle Index ◽

Multiarmed Bandit

A multiarmed bandit problem is studied when the arms are not always available. The arms are first assumed to be intermittently available with some state/action-dependent probabilities. It is proven that no index policy can attain the maximum expected total discounted reward in every instance of that problem. The Whittle index policy is derived, and its properties are studied. Then it is assumed that the arms may break down, but repair is an option at some cost, and the new Whittle index policy is derived. Both problems are indexable. The proposed index policies cannot be dominated by any other index policy over all multiarmed bandit problems considered here. Whittle indices are evaluated for Bernoulli arms with unknown success probabilities.

Download Full-text

Open Bandit Processes with Uncountable States and Time-Backward Effects

Journal of Applied Probability ◽

10.1239/jap/1371648948 ◽

2013 ◽

Vol 50 (2) ◽

pp. 388-402 ◽

Cited By ~ 1

Author(s):

Xianyi Wu ◽

Xian Zhou

Keyword(s):

Dynamic Scheduling ◽

Limited Resources ◽

Sufficient Condition ◽

Common Belief ◽

Gittins Index ◽

Bandit Problem ◽

Present Value ◽

Index Policy ◽

Processing Times ◽

Negative Time

Bandit processes and the Gittins index have provided powerful and elegant theory and tools for the optimization of allocating limited resources to competitive demands. In this paper we extend the Gittins theory to more general branching bandit processes, also referred to as open bandit processes, that allow uncountable states and backward times. We establish the optimality of the Gittins index policy with uncountably many states, which is useful in such problems as dynamic scheduling with continuous random processing times. We also allow negative time durations for discounting a reward to account for the present value of the reward that was received before the present time, which we refer to as time-backward effects. This could model the situation of offering bonus rewards for completing jobs above expectation. Moreover, we discover that a common belief on the optimality of the Gittins index in the generalized bandit problem is not always true without additional conditions, and provide a counterexample. We further apply our theory of open bandit processes with time-backward effects to prove the optimality of the Gittins index in the generalized bandit problem under a sufficient condition.

Download Full-text

Open Bandit Processes with Uncountable States and Time-Backward Effects

Journal of Applied Probability ◽

10.1017/s0021900200013437 ◽

2013 ◽

Vol 50 (02) ◽

pp. 388-402

Author(s):

Xianyi Wu ◽

Xian Zhou

Keyword(s):

Dynamic Scheduling ◽

Limited Resources ◽

Sufficient Condition ◽

Common Belief ◽

Gittins Index ◽

Bandit Problem ◽

Present Value ◽

Index Policy ◽

Processing Times ◽

Negative Time

Bandit processes and the Gittins index have provided powerful and elegant theory and tools for the optimization of allocating limited resources to competitive demands. In this paper we extend the Gittins theory to more general branching bandit processes, also referred to as open bandit processes, that allow uncountable states and backward times. We establish the optimality of the Gittins index policy with uncountably many states, which is useful in such problems as dynamic scheduling with continuous random processing times. We also allow negative time durations for discounting a reward to account for the present value of the reward that was received before the present time, which we refer to as time-backward effects. This could model the situation of offering bonus rewards for completing jobs above expectation. Moreover, we discover that a common belief on the optimality of the Gittins index in the generalized bandit problem is not always true without additional conditions, and provide a counterexample. We further apply our theory of open bandit processes with time-backward effects to prove the optimality of the Gittins index in the generalized bandit problem under a sufficient condition.

Download Full-text

Fast Two-Stage Computation of an Index Policy for Multi-Armed Bandits with Setup Delays

Mathematics ◽

10.3390/math9010052 ◽

2020 ◽

Vol 9 (1) ◽

pp. 52

Author(s):

José Niño-Mora

Keyword(s):

Numerical Study ◽

Arithmetic Operation ◽

Bandit Problem ◽

Index Policy ◽

Two Stage ◽

Second Stage ◽

Whittle Index ◽

Set Up ◽

Computing Method ◽

Special Case

We consider the multi-armed bandit problem with penalties for switching that include setup delays and costs, extending the former results of the author for the special case with no switching delays. A priority index for projects with setup delays that characterizes, in part, optimal policies was introduced by Asawa and Teneketzis in 1996, yet without giving a means of computing it. We present a fast two-stage index computing method, which computes the continuation index (which applies when the project has been set up) in a first stage and certain extra quantities with cubic (arithmetic-operation) complexity in the number of project states and then computes the switching index (which applies when the project is not set up), in a second stage, with quadratic complexity. The approach is based on new methodological advances on restless bandit indexation, which are introduced and deployed herein, being motivated by the limitations of previous results, exploiting the fact that the aforementioned index is the Whittle index of the project in its restless reformulation. A numerical study demonstrates substantial runtime speed-ups of the new two-stage index algorithm versus a general one-stage Whittle index algorithm. The study further gives evidence that, in a multi-project setting, the index policy is consistently nearly optimal.

Download Full-text

On transforming an index for generalised bandit problems

Journal of Applied Probability ◽

10.2307/3214927 ◽

1995 ◽

Vol 32 (1) ◽

pp. 168-182 ◽

Cited By ~ 4

Author(s):

K. D. Glazebrook ◽

S. Greatrix

Keyword(s):

Dynamic Programming ◽

Policy Evaluation ◽

Gittins Index ◽

Bandit Problem ◽

Bandit Problems ◽

Index Policies

Nash (1980) demonstrated that index policies are optimal for a class of generalised bandit problem. A transform of the index concerned has many of the attributes of the Gittins index. The transformed index is positive-valued, with maximal values yielding optimal actions. It may be characterised as the value of a restart problem and is hence computable via dynamic programming methodologies. The transformed index can also be used in procedures for policy evaluation.

Download Full-text