Bandit Problems, Sequential Allocation of Experiments

Ingredients of Locational Analysis (J. Krarup & P. Pruzan). The p-Median Problem and Generalizations (P. Mirchandani). The Uncapacitated Facility Location Problem (G. Cornuejols, et al.). Multiperiod Capacitated Location Models (S. Jacobsen). Decomposition Methods for Facility Location Problems (T. Magnanti & R. Wong). Covering Problems (A. Kolen & A. Tamir). p-Center Problems (G. Handler). Duality: Covering and Constraining p-Center Problems on Trees (B. Tansel, et al.). Locations with Spatial Interactions: The Quadratic Assignment Problem (R. Burkard). Locations with Spatial Interactions: Competitive Locations and Games (S. Hakimi). Equilibrium Analysis for Voting and Competitive Location Problems (P. Hansen, et al.). Location of Mobile Units in a Stochastic Environment (O. Berman, et al.). Index.

https://link.springer.com/content/pdf/10.1057%2Fjors.1991.208.pdf

Discrete location theory

The topic of dynamic pricing and learning has received a considerable amount of attention in recent years, from different scientific communities. We survey these literature streams: we provide a brief introduction to the historical origins of quantitative research on pricing and demand estimation, point to different subfields in the area of dynamic pricing, and provide an in-depth overview of the available literature on dynamic pricing and learning. Our focus is on the operations research and management science literature, but we also discuss relevant contributions from marketing, economics, econometrics, and computer science. We discuss relations with methodologically related research areas, and identify directions for future research.

Dynamic pricing and learning: Historical origins, current research, and new directions

Thompson sampling is an algorithm for online decision problemswhere actions are taken sequentially in a manner thatmust balance between exploiting what is known to maximizeimmediate performance and investing to accumulatenew information that may improve future performance. Thealgorithm addresses a broad range of problems in a computationallyefficient manner and is therefore enjoying wideuse. This tutorial covers the algorithm and its application,illustrating concepts through a range of examples, includingBernoulli bandit problems, shortest path problems, productrecommendation, assortment, active learning with neuralnetworks, and reinforcement learning in Markov decisionprocesses. Most of these problems involve complex informationstructures, where information revealed by taking anaction informs beliefs about other actions. We will also discusswhen and why Thompson sampling is or is not effectiveand relations to alternative algorithms.

/pdf/a-tutorial-on-thompson-sampling-36i9srnsgd.pdf

A Tutorial on Thompson Sampling

This work addresses decentralized online optimization in nonstationary environments. A network of agents aim to track the minimizer of a global, time-varying, and convex function. The minimizer follows a known linear dynamics corrupted by unknown unstructured noise. At each time, the global function (which could be a tracking error) can be cast as a sum of a finite number of local functions, each of which is assigned to one agent in the network. Moreover, the local functions become available to agents sequentially, and agents do not have prior knowledge of the future cost functions. Therefore, agents must communicate with each other to build an online approximation of the global function. We propose a decentralized variation of the celebrated mirror descent algorithm, according to which agents perform a consensus step to follow the global function and take into account the dynamics of the global minimizer. In order to measure the performance of the proposed online algorithm, we compare it to its offline counterpart, where the global functions are available a priori . The gap between the two losses is defined as dynamic regret. We establish a regret bound that scales inversely in the spectral gap of the network and represents the deviation of minimizer sequence with respect to the given dynamics. We show that our framework subsumes a number of results in distributed optimization.

Distributed Online Optimization in Dynamic Environments Using Mirror Descent

In a multi-armed bandit (MAB) problem a gambler needs to choose at each round of play one of K arms, each characterized by an unknown reward distribution. Reward realizations are only observed when an arm is selected, and the gambler's objective is to maximize his cumulative expected earnings over some given horizon of play T. To do this, the gambler needs to acquire information about arms (exploration) while simultaneously optimizing immediate rewards (exploitation); the price paid due to this trade off is often referred to as the regret, and the main question is how small can this price be as a function of the horizon length T. This problem has been studied extensively when the reward distributions do not change over time; an assumption that supports a sharp characterization of the regret, yet is often violated in practical settings. In this paper, we focus on a MAB formulation which allows for a broad range of temporal uncertainties in the rewards, while still maintaining mathematical tractability. We fully characterize the (regret) complexity of this class of MAB problems by establishing a direct link between the extent of allowable reward "variation" and the minimal achievable regret, and by establishing a connection between the adversarial and the stochastic MAB frameworks.

/pdf/stochastic-multi-armed-bandit-problem-with-non-stationary-5ohx63yxxl.pdf

Stochastic Multi-Armed-Bandit Problem with Non-stationary Rewards

We consider a non-stationary variant of a sequential stochastic optimization problem, in which the underlying cost functions may change along the horizon. We propose a measure, termed variation budget, that controls the extent of said change, and study how restrictions on this budget impact achievable performance. We identify sharp conditions under which it is possible to achieve long-run-average optimality and more refined performance measures such as rate optimality that fully characterize the complexity of such problems. In doing so, we also establish a strong connection between two rather disparate strands of literature: adversarial online convex optimization; and the more traditional stochastic approximation paradigm (couched in a non-stationary setting). This connection is the key to deriving well performing policies in the latter, by leveraging structure of optimal policies in the former. Finally, tight bounds on the minimax regret allow us to quantify the "price of non-stationarity," which mathematically captures the added complexity embedded in a temporally changing environment versus a stationary one.

Non-stationary Stochastic Optimization

We consider a non-stationary variant of a sequential stochastic optimization problem, in which the underlying cost functions may change along the horizon. We propose a measure, termed variation budget, that controls the extent of said change, and study how restrictions on this budget impact achievable performance. We identify sharp conditions under which it is possible to achieve long-run average optimality and more refined performance measures such as rate optimality that fully characterize the complexity of such problems. In doing so, we also establish a strong connection between two rather disparate strands of literature: (1) adversarial online convex optimization and (2) the more traditional stochastic approximation paradigm (couched in a non-stationary setting). This connection is the key to deriving well-performing policies in the latter, by leveraging structure of optimal policies in the former. Finally, tight bounds on the minimax regret allow us to quantify the “price of non-stationarity,” which ...

/pdf/non-stationary-stochastic-optimization-4bfqjy516k.pdf

Non-Stationary Stochastic Optimization

In online advertising markets, advertisers often purchase ad placements through bidding in repeated auctions based on realized viewer information. We study how budget-constrained advertisers may bid in the presence of competition, when there is uncertainty about future bidding opportunities as well as competitors' heterogenous preferences and budgets. We formulate this problem as a sequential game of incomplete information, where bidders know neither their own valuation distribution, nor the budgets and valuation distributions of their competitors. We introduce a family of dynamic bidding strategies we refer to as "adaptive pacing" strategies, in which advertisers adjust their bids throughout the campaign according to the sample path of observed expenditures. We analyze the performance of this class of strategies under different assumptions on competitors' behavior. Under arbitrary competitors' bids, we establish through matching lower and upper bounds the asymptotic optimality of this class of strategies as the number of auctions grows large. When adopted by all the bidders, the dynamics converge to a tractable and meaningful steady state. Moreover, we show that these strategies constitute an approximate Nash equilibrium in dynamic strategies: The benefit of unilaterally deviating to other strategies, including ones with access to complete information, becomes negligible as the number of auctions and competitors grows large. This establishes a connection between regret minimization and market stability, by which advertisers can essentially follow equilibrium bidding strategies that also ensure the best performance that can be guaranteed off-equilibrium.

Learning in Repeated Auctions with Budgets: Regret Minimization and Equilibrium

In online advertising markets, advertisers often purchase ad placements through bidding in repeated auctions based on realized viewer information. We study how budget-constrained advertisers may co...

Yonatan Gur

Papers

Stochastic Multi-Armed-Bandit Problem with Non-stationary Rewards

Non-stationary Stochastic Optimization

Non-Stationary Stochastic Optimization

Learning in Repeated Auctions with Budgets: Regret Minimization and Equilibrium

Learning in Repeated Auctions with Budgets: Regret Minimization and Equilibrium