A novel reinforcement learning algorithm is introduced for multiarmed restless bandits with average reward, using the paradigms of Q-learning and Whittle index. Specifically, we leverage the structure of the Whittle index policy to reduce the search space of Q-learning, resulting in major computational gains. Rigorous convergence analysis is provided, supported by numerical experiments. The numerical experiments show excellent empirical performance of the proposed scheme.

Whittle index based Q-learning for restless bandits with average reward

In this paper, we consider energy-efficient scheduling in a multiuser setting, where each user has a separate queue and there is a cost associated with holding packets (jobs) in each queue (modeling the delay constraints). The packets of each user need to be sent over a common channel. The channel qualities seen by the users are time-varying and differ across users. Also, the cost incurred, i.e., energy consumed, in packet transmission is a function of the channel quality. We pose the problem as an average cost Markov decision problem and prove that this problem is Whittle-like indexable. Based on this result, we propose an algorithm in which the Whittle-like index of each user is computed and the user who has the lowest value is selected for transmission. We evaluate the performance of this algorithm via simulations and show that it achieves a lower average cost than the maximum-weight scheduling and weighted fair scheduling strategies.

Opportunistic Scheduling as Restless Bandits

We consider the multi-armed restless bandit problem (RMABP) with an infinite horizon average cost objective. Each arm of the RMABP is associated with a Markov process that operates in two modes: active and passive. At each time slot a controller needs to designate a subset of the arms to be active, of which the associated processes will evolve differently from the passive case. Treated as an optimal control problem, the optimal solution of the RMABP is known to be computationally intractable. In many cases, the Whittle index policy achieves near optimal performance and can be tractably found. Nevertheless, computation of the Whittle indices requires knowledge of the transition matrices of the underlying processes, which are sometimes hidden from decision makers. In this paper, we take first steps towards a tractable and efficient reinforcement learning algorithm for controlling such a system. We setup parallel Q-learning recursions, with each recursion mapping to individual possible values of the Whittle index. We then update these recursions as we control the system, learning an approximation of the Whittle index as time evolves. Tested on several examples, our control outperforms naive priority allocations and nears the performance of the fully-informed Whittle index policy.

Towards Q-learning the Whittle Index for Restless Bandits

A vertex subset $S$ of a graph $G$ is a general position set of $G$ if no vertex of $S$ lies on a geodesic between two other vertices of $S$. The cardinality of a largest general position set of $G$ is the general position number (gp-number) ${\rm gp}(G)$ of $G$. The gp-number is determined for some families of Kneser graphs, in particular for $K(n,2)$ and $K(n,3)$. A sharp lower bound on the gp-number is proved for Cartesian products of graphs. The gp-number is also determined for joins of graphs, coronas over graphs, and line graphs of complete graphs.

The general position problem on Kneser graphs and on some graph operations

/pdf/the-general-position-problem-on-kneser-graphs-and-on-some-390fje1dtw.pdf

We propose and analyze a reinforcement learning algorithm for learning Whittle index for a class of indexable restless bandits based on linear function approximation and illustrate its use using as an example a restless bandit problem arising in scheduling of web crawlers for ephemeral content.

A reinforcement learning algorithm for restless bandits

We extend the Approximate-Proximal Point (APROX) family of model-based methods for solving stochastic convex optimization problems, including stochastic subgradient, proximal point, and bundle methods, to the minibatch setting. To do this, we propose two minibatched algorithms for which we prove a non-asymptotic upper bound on the rate of convergence, revealing a linear speedup in minibatch size. In contrast to standard stochastic gradient methods, these methods may have linear speedup in the minibatch setting even for non-smooth functions. Our algorithms maintain the desirable traits characteristic of the APROX family, such as robustness to initial step size choice. Additionally, we show improved convergence rates for "interpolation" problems, which (for example) gives a new parallelization strategy for alternating projections. We corroborate our theoretical results with extensive empirical testing, which demonstrates the gains provided by accurate modeling and minibatching.

/pdf/minibatch-stochastic-approximate-proximal-point-methods-2gg8o9pze0.pdf

Minibatch Stochastic Approximate Proximal Point Methods

In recent work (Pandit and Kulkarni [Discrete Applied Mathematics, 244 (2018), pp. 155--169]), the independence number of a graph was characterized as the maximum of the $\ell_1$ norm of solutions of a Linear Complementarity Problem (\LCP) defined suitably using parameters of the graph. Solutions of this LCP have another relation, namely, that they corresponded to Nash equilibria of a public goods game. Motivated by this, we consider a perturbation of this LCP and identify the combinatorial structures on the graph that correspond to the maximum $\ell_1$ norm of solutions of the new LCP. We introduce a new concept called independent clique solutions which are solutions of the LCP that are supported on independent cliques and show that for small perturbations, such solutions attain the maximum $\ell_1$ norm amongst all solutions of the new LCP.

On Independent Cliques and Linear Complementarity Problems

The increasing presence of decentralized renewable generation in the power grid has motivated consumers to install batteries to save excess energy for future use. The high price of energy storage calls for a shared storage system, but careful battery management is required so that the battery is operated in a manner that is fair to all and as efficiently as possible. In this paper, we study the tradeoffs between efficiency and fairness in operating a shared battery. We develop a framework based on constrained Markov decision processes to study both regimes, namely, optimizing efficiency under a hard fairness constraint and optimizing fairness under hard efficiency constraint. Our results show that there are fundamental limits to efficiency under fairness and vice-versa, and, in general, the two cannot be achieved simultaneously. We characterize these fundamental limits via absolute bounds on these quantities, and via the notion of price of fairness that we introduce in this paper.

Efficiency Fairness Tradeoff in Battery Sharing

We extend the Approximate-Proximal Point (aProx) family of model-based methods for solving stochastic convex optimization problems, including stochastic subgradient, proximal point, and bundle methods, to the minibatch and accelerated setting. To do so, we propose specific model-based algorithms and an acceleration scheme for which we provide non-asymptotic convergence guarantees, which are order-optimal in all problem-dependent constants and provide linear speedup in minibatch size, while maintaining the desirable robustness traits (e.g. to stepsize) of the aProx family. Additionally, we show improved convergence rates and matching lower bounds identifying new fundamental constants for "interpolation" problems, whose importance in statistical machine learning is growing; this, for example, gives a parallelization strategy for alternating projections. We corroborate our theoretical results with empirical testing to demonstrate the gains accurate modeling, acceleration, and minibatching provide.

Karan N. Chadha

Papers

A reinforcement learning algorithm for restless bandits

Minibatch Stochastic Approximate Proximal Point Methods

On Independent Cliques and Linear Complementarity Problems

Efficiency Fairness Tradeoff in Battery Sharing

Accelerated, Optimal, and Parallel: Some Results on Model-Based Stochastic Optimization.