The paper proposes a novel upper confidence bound (UCB) procedure for identifying the arm with the largest mean in a multi-armed bandit game in the fixed confidence setting using a small number of total samples. The procedure cannot be improved in the sense that the number of samples required to identify the best arm is within a constant factor of a lower bound based on the law of the iterated logarithm (LIL). Inspired by the LIL, we construct our confidence bounds to explicitly account for the infinite time horizon of the algorithm. In addition, by using a novel stopping time for the algorithm we avoid a union bound over the arms that has been observed in other UCBtype algorithms. We prove that the algorithm is optimal up to constants and also show through simulations that it provides superior performance with respect to the state-of-the-art.

/pdf/lil-ucb-an-optimal-exploration-algorithm-for-multi-armed-1l53749f1b.pdf

lil' UCB : An Optimal Exploration Algorithm for Multi-Armed Bandits

1 Extension and applications of an inequality of Ville and Wald Let x 1… be a sequence of random variables with a specified joint probability distribution P We shall give a method for obtaining probability inequalities and related limit theorems concerning the behavior of the entire sequence of x’s

https://projecteuclid.org/download/pdf_1/euclid.aoms/1177696786

Statistical Methods Related to the Law of the Iterated Logarithm

1. Introduction and summary. Let W(t) denote a standard Wiener process for 0 ≦ t 0 (or for some t > 0) for a certain class of functions g(t), including functions which are ~ (2t log log t)½ as y → ∞. We also prove an invariance theorem which states that this probability is the limit as m → ∞ of the probability that S n ≦m ½ g(n/m) for some n ≦ τm (or for some n ≦ 1), where S n is the nth partial sum of any sequence x 1, x 2, … of independent and identically distributed (i.i.d.) random variables with mean 0 and variance 1.

/pdf/boundary-crossing-probabilities-for-the-wiener-process-and-514jzzq13z.pdf

Boundary Crossing Probabilities for the Wiener Process and Sample Sums

A confidence sequence is a sequence of confidence intervals that is uniformly valid over an unbounded time horizon. Our work develops confidence sequences whose widths go to zero, with nonasymptotic coverage guarantees under nonparametric conditions. We draw connections between the Cramer–Chernoff method for exponential concentration, the law of the iterated logarithm (LIL) and the sequential probability ratio test—our confidence sequences are time-uniform extensions of the first; provide tight, nonasymptotic characterizations of the second; and generalize the third to nonparametric settings, including sub-Gaussian and Bernstein conditions, self-normalized processes and matrix martingales. We illustrate the generality of our proof techniques by deriving an empirical-Bernstein bound growing at a LIL rate, as well as a novel upper LIL for the maximum eigenvalue of a sum of random matrices. Finally, we apply our methods to covariance matrix estimation and to estimation of sample average treatment effect under the Neyman–Rubin potential outcomes model.

/pdf/time-uniform-nonparametric-nonasymptotic-confidence-4kzk8lnxqo.pdf

Time-uniform, nonparametric, nonasymptotic confidence sequences

We develop a class of exponential bounds for the probability that a martingale sequence crosses a time-dependent linear threshold. Our key insight is that it is both natural and fruitful to formulate exponential concentration inequalities in this way. We illustrate this point by presenting a single assumption and theorem that together unify and strengthen many tail bounds for martingales, including classical inequalities (1960-80) by Bernstein, Bennett, Hoeffding, and Freedman; contemporary inequalities (1980-2000) by Shorack and Wellner, Pinelis, Blackwell, van de Geer, and de la Pe\~na; and several modern inequalities (post-2000) by Khan, Tropp, Bercu and Touati, Delyon, and others. In each of these cases, we give the strongest and most general statements to date, quantifying the time-uniform concentration of scalar, matrix, and Banach-space-valued martingales, under a variety of nonparametric assumptions in discrete and continuous time. In doing so, we bridge the gap between existing line-crossing inequalities, the sequential probability ratio test, the Cram\'er-Chernoff method, self-normalized processes, and other parts of the literature.

Time-uniform Chernoff bounds via nonnegative supermartingales

1. Introduction—Let x,x 1, x 2 … be a sequence of independent, identically distributed random variables with mean 0, variance 1, and moment generating function ϕ(t) = E(etz) finite in some neighborhood of t= 0, and put S n = x 1, + … + x n, \(\bar x\) n = S n/n. For any sequence of positive constants a n, n ≥ 1, let P m = P(|\(\bar x\) n| ≥ a n for some n ≥ m).

https://www.pnas.org/content/pnas/57/5/1188.full.pdf

Iterated logarithm inequalities.

There are fixed sample size methods for estimating the unknown size N of a finite population by tagging the elements of a first sample and then counting the number of tagged elements of a second sample [1, p. 43]. Less well known are sequential methods [2], which have the advantage that the total sample size automatically adjusts itself to the unknown N to assure a desired accuracy of the estimate. All these methods only provide estimates for which the relative error is likely to be small. Suppose, however, that we want PN (estimate = N)⪴α =.99, say, no matter what the value N = 1, 2, • • •. How can this be done?

/pdf/finding-the-size-of-a-finite-population-1qxox3ivth.pdf

Herbert Robbins

Papers

Iterated logarithm inequalities.

Finding the Size of a Finite Population