An Index Policy for Multiarmed Multimode Restless Bandits

We show that the fluid approximation to Whittle's index policy for restless bandits has a globally asymptotically stable equilibrium point when the bandits move on just three states. It follows that in this case the index policy is asymptotic optimal.

Download Full-text

Opportunistic Scheduling Revisited Using Restless Bandits: Indexability and Index Policy

IEEE Transactions on Wireless Communications ◽

10.1109/twc.2019.2931690 ◽

2019 ◽

Vol 18 (10) ◽

pp. 4997-5010

Author(s):

Kehao Wang ◽

Jihong Yu ◽

Lin Chen ◽

Pan Zhou ◽

Xiaohu Ge ◽

...

Keyword(s):

Opportunistic Scheduling ◽

Index Policy ◽

Restless Bandits

Download Full-text

Index policies for a class of discounted restless bandits

Advances in Applied Probability ◽

10.1017/s0001867800011903 ◽

2002 ◽

Vol 34 (04) ◽

pp. 754-774 ◽

Cited By ~ 8

Author(s):

K. D. Glazebrook ◽

J. Niño-Mora ◽

P. S. Ansell

Keyword(s):

Conservation Laws ◽

Special Class ◽

Computational Study ◽

Bandit Problems ◽

Index Policy ◽

Restless Bandit ◽

Restless Bandits ◽

Index Policies ◽

Strong Performance ◽

Dual Speed

The paper concerns a class of discounted restless bandit problems which possess an indexability property. Conservation laws yield an expression for the reward suboptimality of a general policy. These results are utilised to study the closeness to optimality of an index policy for a special class of simple and natural dual speed restless bandits for which indexability is guaranteed. The strong performance of the index policy is confirmed by a computational study.

Download Full-text

A Fast-Pivoting Algorithm for Whittle’s Restless Bandit Index

Mathematics ◽

10.3390/math8122226 ◽

2020 ◽

Vol 8 (12) ◽

pp. 2226 ◽

Cited By ~ 1

Author(s):

José Niño-Mora

Keyword(s):

Numerical Study ◽

Index Policy ◽

State Spaces ◽

Restless Bandit ◽

Restless Bandits ◽

Pivoting Algorithm ◽

Markov Decision ◽

Whittle Index ◽

Decision Epoch ◽

Change State

The Whittle index for restless bandits (two-action semi-Markov decision processes) provides an intuitively appealing optimal policy for controlling a single generic project that can be active (engaged) or passive (rested) at each decision epoch, and which can change state while passive. It further provides a practical heuristic priority-index policy for the computationally intractable multi-armed restless bandit problem, which has been widely applied over the last three decades in multifarious settings, yet mostly restricted to project models with a one-dimensional state. This is due in part to the difficulty of establishing indexability (existence of the index) and of computing the index for projects with large state spaces. This paper draws on the author’s prior results on sufficient indexability conditions and an adaptive-greedy algorithmic scheme for restless bandits to obtain a new fast-pivoting algorithm that computes the n Whittle index values of an n-state restless bandit by performing, after an initialization stage, n steps that entail (2/3)n3+O(n2) arithmetic operations. This algorithm also draws on the parametric simplex method, and is based on elucidating the pattern of parametric simplex tableaux, which allows to exploit special structure to substantially simplify and reduce the complexity of simplex pivoting steps. A numerical study demonstrates substantial runtime speed-ups versus alternative algorithms.

Download Full-text

Addendum to ‘On an index policy for restless bandits'

Advances in Applied Probability ◽

10.1017/s0001867800023582 ◽

1991 ◽

Vol 23 (02) ◽

pp. 429-430 ◽

Cited By ~ 1

Author(s):

Richard R. Weber ◽

Gideon Weiss

Keyword(s):

Equilibrium Point ◽

Stable Equilibrium ◽

Stable Equilibrium Point ◽

Asymptotically Stable ◽

Index Policy ◽

Fluid Approximation ◽

Globally Asymptotically Stable ◽

Restless Bandits

We show that the fluid approximation to Whittle's index policy for restless bandits has a globally asymptotically stable equilibrium point when the bandits move on just three states. It follows that in this case the index policy is asymptotic optimal.

Download Full-text

Index policies for a class of discounted restless bandits

Advances in Applied Probability ◽

10.1239/aap/1037990952 ◽

2002 ◽

Vol 34 (4) ◽

pp. 754-774 ◽

Cited By ~ 18

Author(s):

K. D. Glazebrook ◽

J. Niño-Mora ◽

P. S. Ansell

Keyword(s):

Conservation Laws ◽

Special Class ◽

Computational Study ◽

Bandit Problems ◽

Index Policy ◽

Restless Bandit ◽

Restless Bandits ◽

Index Policies ◽

Strong Performance ◽

Dual Speed

The paper concerns a class of discounted restless bandit problems which possess an indexability property. Conservation laws yield an expression for the reward suboptimality of a general policy. These results are utilised to study the closeness to optimality of an index policy for a special class of simple and natural dual speed restless bandits for which indexability is guaranteed. The strong performance of the index policy is confirmed by a computational study.

Download Full-text

Restless bandits, partial conservation laws and indexability

Advances in Applied Probability ◽

10.1017/s0001867800010648 ◽

2001 ◽

Vol 33 (1) ◽

pp. 76-98 ◽

Cited By ~ 29

Author(s):

José Niño-Mora

Keyword(s):

Conservation Laws ◽

Greedy Algorithm ◽

Stochastic Scheduling ◽

Model Parameters ◽

Scheduling Problems ◽

Index Policy ◽

Restless Bandits ◽

Partial Conservation ◽

Finite State ◽

Achievable Region

We show that if performance measures in a general stochastic scheduling problem satisfy partial conservation laws (PCL), which extend the generalized conservation laws (GCL) introduced by Bertsimas and Niño-Mora (1996), then the problem is solved optimally by a priority-index policy under a range of admissible linear performance objectives, with both this range and the optimal indices being determined by a one-pass adaptive-greedy algorithm that extends Klimov's: we call such scheduling problems PCL-indexable. We further apply the PCL framework to investigate the indexability property of restless bandits (two-action finite-state Markov decision chains) introduced by Whittle, obtaining the following results: (i) we present conditions on model parameters under which a single restless bandit is PCL-indexable, and hence indexable; membership of the class of PCL-indexable bandits is tested through a single run of the adaptive-greedy algorithm, which further computes the Whittle indices when the test is positive; this provides a tractable sufficient condition for indexability; (ii) we further introduce the subclass of GCL-indexable bandits (including classical bandits), which are indexable under arbitrary linear rewards. Our analysis is based on the achievable region approach to stochastic optimization, as the results follow from deriving and exploiting a new linear programming reformulation for single restless bandits.

Download Full-text

Scheduling in wireless networks with spatial reuse of spectrum as restless bandits

Performance Evaluation ◽

10.1016/j.peva.2021.102208 ◽

2021 ◽

pp. 102208

Author(s):

Vivek S. Borkar ◽

Shantanu Choudhary ◽

Vaibhav Kumar Gupta ◽

Gaurav S. Kasbekar

Keyword(s):

Wireless Networks ◽

Spatial Reuse ◽

Restless Bandits

Download Full-text

Fast Two-Stage Computation of an Index Policy for Multi-Armed Bandits with Setup Delays

Mathematics ◽

10.3390/math9010052 ◽

2020 ◽

Vol 9 (1) ◽

pp. 52

Author(s):

José Niño-Mora

Keyword(s):

Numerical Study ◽

Arithmetic Operation ◽

Bandit Problem ◽

Index Policy ◽

Two Stage ◽

Second Stage ◽

Whittle Index ◽

Set Up ◽

Computing Method ◽

Special Case

We consider the multi-armed bandit problem with penalties for switching that include setup delays and costs, extending the former results of the author for the special case with no switching delays. A priority index for projects with setup delays that characterizes, in part, optimal policies was introduced by Asawa and Teneketzis in 1996, yet without giving a means of computing it. We present a fast two-stage index computing method, which computes the continuation index (which applies when the project has been set up) in a first stage and certain extra quantities with cubic (arithmetic-operation) complexity in the number of project states and then computes the switching index (which applies when the project is not set up), in a second stage, with quadratic complexity. The approach is based on new methodological advances on restless bandit indexation, which are introduced and deployed herein, being motivated by the limitations of previous results, exploiting the fact that the aforementioned index is the Whittle index of the project in its restless reformulation. A numerical study demonstrates substantial runtime speed-ups of the new two-stage index algorithm versus a general one-stage Whittle index algorithm. The study further gives evidence that, in a multi-project setting, the index policy is consistently nearly optimal.

Download Full-text