Today's Web-enabled deluge of electronic data calls for automated methods of data analysis. Machine learning provides these, developing methods that can automatically detect patterns in data and then use the uncovered patterns to predict future data. This textbook offers a comprehensive and self-contained introduction to the field of machine learning, based on a unified, probabilistic approach. The coverage combines breadth and depth, offering necessary background material on such topics as probability, optimization, and linear algebra as well as discussion of recent developments in the field, including conditional random fields, L1 regularization, and deep learning. The book is written in an informal, accessible style, complete with pseudo-code for the most important algorithms. All topics are copiously illustrated with color images and worked examples drawn from such application domains as biology, text processing, computer vision, and robotics. Rather than providing a cookbook of different heuristic methods, the book stresses a principled model-based approach, often using the language of graphical models to specify models in a concise and intuitive way. Almost all the models described have been implemented in a MATLAB software package--PMTK (probabilistic modeling toolkit)--that is freely available online. The book is suitable for upper-level undergraduates with an introductory-level college math background and beginning graduate students.

Machine Learning : A Probabilistic Perspective

Thompson sampling is one of oldest heuristic to address the exploration / exploitation trade-off, but it is surprisingly unpopular in the literature. We present here some empirical results using Thompson sampling on simulated and real data, and show that it is highly competitive. And since this heuristic is very easy to implement, we argue that it should be part of the standard baselines to compare against.

/pdf/an-empirical-evaluation-of-thompson-sampling-cmerwqpc37.pdf

An Empirical Evaluation of Thompson Sampling

In this paper we study the contextual bandit problem (also known as the multi-armed bandit problem with expert advice) for linear payoff functions. For T rounds, K actions, and d dimensional feature vectors, we prove an O (√ Td ln(KT ln(T )/δ) ) regret bound that holds with probability 1− δ for the simplest known (both conceptually and computationally) efficient upper confidence bound algorithm for this problem. We also prove a lower bound of Ω( √ Td) for this setting, matching the upper bound up to logarithmic factors.

/pdf/contextual-bandits-with-linear-payoff-functions-o4e1zb2qgr.pdf

Contextual bandits with linear Payoff functions

Thompson Sampling is one of the oldest heuristics for multi-armed bandit problems. It is a randomized algorithm based on Bayesian ideas, and has recently generated significant interest after several studies demonstrated it to have better empirical performance compared to the state-of-the-art methods. However, many questions regarding its theoretical performance remained open. In this paper, we design and analyze a generalization of Thompson Sampling algorithm for the stochastic contextual multi-armed bandit problem with linear payoff functions, when the contexts are provided by an adaptive adversary. This is among the most important and widely studied version of the contextual bandits problem. We prove a high probability regret bound of O(d2/e√T1+e) in time T for any 0 < e < 1, where d is the dimension of each context vector and e is a parameter used by the algorithm. Our results provide the first theoretical guarantees for the contextual version of Thompson Sampling, and are close to the lower bound of Ω(d√T) for this problem. This essentially solves a COLT open problem of Chapelle and Li [COLT 2012].

/pdf/thompson-sampling-for-contextual-bandits-with-linear-payoffs-1u65b6udmm.pdf

Thompson Sampling for Contextual Bandits with Linear Payoffs

We give a brief review of the developments in several classical problems of sequential analysis and their applications to biomedicine, economics and engi- neering. Even though it can only focus on a limited number of topics, the review shows that sequential analysis is still a vibrant subject after six decades of contin- ual development, with fresh ideas brought in from various fields of application and through interactions with other branches of statistics and probability. We conclude with some remarks on the opportunities and challenges ahead.

/pdf/sequential-analysis-some-classical-problems-and-new-4nma4mdg86.pdf

Sequential analysis: some classical problems and new challenges

Where to locate in a circular city

Spatial Cournot competition and agglomeration in a model of location choice

As does Woodroofe, we consider a Bayesian sequential allocation between two treatments that incorporates a covariate. The goal is to maximize the total discounted expected reward from an infinite population of patients. Although our model is more general than Woodroofe's, we are able to duplicate his main result: The myopic rule is asymptotically optimal.

/pdf/one-armed-bandit-problems-with-covariates-4haoi27z3x.pdf

Jyotirmoy Sarkar

Papers

Where to locate in a circular city

Spatial Cournot competition and agglomeration in a model of location choice

One-Armed Bandit Problems with Covariates

Availability of a periodically inspected system under perfect repair

Availability of a system maintained through several imperfect repairs before a replacement or a perfect repair