It has long been customary to measure the adequacy of an estimator by the smallness of its mean squared error. The least squares estimators were studied by Gauss and by other authors later in the nineteenth century. A proof that the best unbiased estimator of a linear function of the means of a set of observed random variables is the least squares estimator was given by Markov [12], a modified version of whose proof is given by David and Neyman [4]. A slightly more general theorem is given by Aitken [1]. Fisher [5] indicated that for large samples the maximum likelihood estimator approximately minimizes the mean squared error when compared with other reasonable estimators. This paper will be concerned with optimum properties or failure of optimum properties of the natural estimator in certain special problems with the risk usually measured by the mean squared error or, in the case of several parameters, by a quadratic function of the estimators. We shall first mention some recent papers on this subject and then give some results, mostly unpublished, in greater detail.

/pdf/estimation-with-quadratic-loss-314aqwx15u.pdf

Estimation with Quadratic Loss

/pdf/admissible-estimators-recurrent-diffusions-and-insoluble-4hhdwthac1.pdf

Admissible Estimators, Recurrent Diffusions, and Insoluble Boundary Value Problems

Preface. Acknowledgments. 1 Introduction. 1.1 Controversies. 1.2 A guided tour of decision theory. Part One: Foundations. 2 Coherence. 2.1 The "Dutch Book" theorem. 2.2 Temporal coherence. 2.3 Scoring rules and the axioms of probabilities. 2.4 Exercises. 3 Utility. 3.1 St. Petersburg paradox. 3.2 Expected utility theory and the theory of means. 3.3 The expected utility principle. 3.4 The von Neumann-Morgenstern representation theorem. 3.5 Allais' criticism. 3.6 Extensions. 3.7 Exercises. 4 Utility in action. 4.1 The "standard gamble". 4.2 Utility of money. 4.3 Utility functions for medical decisions. 4.4 Exercises. 5 Ramsey and Savage. 5.1 Ramsey's theory. 5.2 Savage's theory. 5.3 Allais revisited. 5.4 Ellsberg paradox. 5.5 Exercises. 6 State independence. 6.1 Horse lotteries. 6.2 State-dependent utilities. 6.3 State-independent utilities. 6.4 Anscombe-Aumann representation theorem. 6.5 Exercises. Part Two Statistical Decision Theory. 7 Decision functions. 7.1 Basic concepts. 7.2 Data-based decisions. 7.3 The travel insurance example. 7.4 Randomized decision rules. 7.5 Classification and hypothesis tests. 7.6 Estimation. 7.7 Minimax-Bayes connections. 7.8 Exercises. 8 Admissibility. 8.1 Admissibility and completeness. 8.2 Admissibility and minimax. 8.3 Admissibility and Bayes. 8.4 Complete classes. 8.5 Using the same alpha level across studies with different sample sizes is inadmissible. 8.6 Exercises. 9 Shrinkage. 9.1 The Stein effect. 9.2 Geometric and empirical Bayes heuristics. 9.3 General shrinkage functions. 9.4 Shrinkage with different likelihood and losses. 9.5 Exercises. 10 Scoring rules. 10.1 Betting and forecasting. 10.2 Scoring rules. 10.3 Local scoring rules. 10.4 Calibration and refinement. 10.5 Exercises. 11 Choosing models. 11.1 The "true model" perspective. 11.2 Model elaborations. 11.3 Exercises. Part Three Optimal Design. 12 Dynamic programming. 12.1 History. 12.2 The travel insurance example revisited. 12.3 Dynamic programming. 12.4 Trading off immediate gains and information. 12.5 Sequential clinical trials. 12.6 Variable selection in multiple regression. 12.7 Computing. 12.8 Exercises. 13 Changes in utility as information. 13.1 Measuring the value of information. 13.2 Examples. 13.3 Lindley information. 13.4 Minimax and the value of information. 13.5 Exercises. 14 Sample size. 14.1 Decision-theoretic approaches to sample size. 14.2 Computing. 14.3 Examples. 14.4 Exercises. 15 Stopping. 15.1 Historical note. 15.2 A motivating example. 15.3 Bayesian optimal stopping. 15.4 Examples. 15.5 Sequential sampling to reduce uncertainty. 15.6 The stopping rule principle. 15.7 Exercises. Appendix. A.1 Notation. A.2 Relations. A.3 Probability (density) functions of some distributions. A.4 Conjugate updating. References. Index.

/pdf/decision-theory-principles-and-approaches-3w5kivjm8n.pdf

Decision theory : principles and approaches

Common Errors in Statistics (and How to Avoid Them): Good/Common Errors in Statistics (and How to Avoid Them), Second Edition

In his original paper on the subject, Shannon found upper and lower bounds for the entropy of printed English based on the number of trials required for a subject to guess subsequent symbols in a given text. The guessing approach precludes asymptotic consistency of either the upper or lower bounds except for degenerate ergodic processes. Shannon's technique of guessing the next symbol is altered by having the subject place sequential bets on the next symbol of text. If S_{n} denotes the subject's capital after n bets at 27 for 1 odds, and if it is assumed that the subject knows the underlying probability distribution for the process X , then the entropy estimate is \hat{H}_{n}(X)=(1-(1/n) \log_{27}S_{n}) \log_{2} 27 bits/symbol. If the subject does not know the true probability distribution for the stochastic process, then \hat{H}_{n}(X) is an asymptotic upper bound for the true entropy. If X is stationary, E\hat{H}_{n}(X)\rightarrowH(X), H(X) being the true entropy of the process. Moreover, if X is ergodic, then by the Shannon-McMillan-Breiman theorem \hat{H}_{n}(X)\rightarrowH(X) with probability one. Preliminary indications are that English text has an entropy of approximately 1.3 bits/symbol, which agrees well with Shannon's estimate. In his original paper on the subject, Shannon found upper and lower bounds for the entropy of printed English based on the number of trials required for a subject to guess subsequent symbols in a given text. The guessing approach precludes asymptotic consistency of either the upper or lower bounds except for degenerate ergodic processes. Shannon's technique of guessing the next symbol is altered by having the subject place sequential bets on the next symbol of text. If S_{n} denotes the subject's capital after n bets at 27 for 1 odds, and if it is assumed that the subject knows the underlying probability distribution for the process X , then the entropy estimate is \hat{H}_{n}(X)=(1-(1/n) \log_{27}S_{n}) \log_{2} 27 bits/symbol. If the subject does not know the true probability distribution for the stochastic process, then \hat{H}_{n}(X) is an asymptotic upper bound for the true entropy. If X is stationary, E\hat{H}_{n}(X)\rightarrowH(X), H(X) being the true entropy of the process. Moreover, if X is ergodic, then by the Shannon-McMillan-Breiman theorem \hat{H}_{n}(X)\rightarrowH(X) with probability one. Preliminary indications are that English text has an entropy of approximately 1.3 bits/symbol, which agrees well with Shannon's estimate. In his original paper on the subject, Shannon found upper and lower bounds for the entropy of printed English based on the number of trials required for a subject to guess subsequent symbols in a given text. The guessing approach precludes asymptotic consistency of either the upper or lower bounds except for degenerate ergodic processes. Shannon's technique of guessing the next symbol is altered by having the subject place sequential bets on the next symbol of text. If S_{n} denotes the subject's capital after n bets at 27 for 1 odds, and if it is assumed that the subject knows the underlying probability distribution for the process X , then the entropy estimate is \hat{H}_{n}(X)=(1-(1/n) \log_{27}S_{n}) \log_{2} 27 bits/symbol. If the subject does not know the true probability distribution for the stochastic process, then \hat{H}_{n}(X) is an asymptotic upper bound for the true entropy. If X is stationary, E\hat{H}_{n}(X)\rightarrowH(X), H(X) being the true entropy of the process.Moreover, if X is ergodic, then by the Shannon-McMillan-Breiman theorem \hat{H}_{n}(X)\rightarrowH(X) with probability one. Preliminary indications are that English text has an entropy of approximately 1.3 bits/symbol, which agrees well with Shannon's estimate. In his original paper on the subject, Shannon found upper and lower bounds for the entropy of printed English based on the number of trials required for a subject to guess subsequent symbols in a given text. The guessing approach precludes asymptotic consistency of either the upper or lower bounds except for degenerate ergodic processes. Shannon's technique of guessing the next symbol is altered by having the subject place sequential bets on the next symbol of text. If S_{n} denotes the subject's capital after n bets at 27 for 1 odds, and if it is assumed that the subject knows the underlying probability distribution for the process X , then the entropy estimate is \hat{H}_{n}(X)=(1-(1/n) \log_{27}S_{n}) \log_{2} 27 bits/symbol. If the subject does not know the true probability distribution for the stochastic process, then \hat{H}_{n}(X) is an asymptotic upper bound for the true entropy. If X is stationary, E\hat{H}_{n}(X)\rightarrowH(X), H(X) being the true entropy of the process. Moreover, if X is ergodic, then by the Shannon-McMillan-Breiman theorem \hat{H}_{n}(X)\rightarrowH(X) with probability one. Preliminary indications are that English text has an entropy of approximately 1.3 bits/symbol, which agrees well with Shannon's estimate.

/pdf/a-convergent-gambling-estimate-of-the-entropy-of-english-3bctwv2d3b.pdf

A convergent gambling estimate of the entropy of English

This paper is concerned with the problem of making a decision on the basis of a sequence of observations on a random variable. Two loss functions, each depending on the distribution of the random variable, the number of observations taken, and the decision made, are assumed given. Minimax problems can be stated for weighted sums of the two loss functions, or for either one subject to an upper bound on the expectation of the other. Under suitable conditions it is shown that solutions of the first type of problem provide solutions for all problems of the latter types, and that admissibility for a problem of the first type implies admissibility for problems of the latter types. Two examples are given: Estimation of the mean of a random variable which is (1) normal with known variance, (2) rectangular with known range. The resulting minimax estimates are, with a small class of exceptions, proved admissible among the class of all procedures with continuous risk functions. The two loss functions are in each case the number of observations, and an arbitrary nondecreasing function of the absolute error of estimate. Extensions to a function of the number of observations for the first loss function are indicated, and two examples are given for the normal case where the sample size can or must be randomised among more than a consecutive pair of integers.

/pdf/on-minimax-statistical-decision-procedures-and-their-1447iip4gv.pdf

On minimax statistical decision procedures and their admissibility

This note is concerned with estimation of the Shannon-Wiener measure of information. Low bias estimates are obtained, and their bias and variance. These estimates are extended to the case where the number of possible values of the random variable is not known. The estimates are compared asymptotically with the maximum likelihood estimates. They are also compared with the minimax estimates (for squared error loss function) for a few special cases where these are easily found.

/pdf/note-on-estimating-information-1wkkxfn05d.pdf

Note on Estimating Information

The inference and decision models considered here are those described by Neyman in [5] pages 16 and 17. For a random variable $X$ with possible probability distributions indexed by a parameter $\theta$, Neyman distinguishes two inference approaches to the problem of estimating $\theta$: (i) having observed $X = x$, find the most probable values $z(x)$ of $\theta$ (This requires a priori probabilities, which often must be chosen rather arbitrarily, and whose existence may be questioned.); and (ii) having observed $X = x$, find the values $z(x)$ that are most reasonable or in which we have the greatest confidence (This requires the rather arbitrary choice of a real-valued function $L$, with $L(x, \theta)$ measuring the degree of confidence we have in the parameter value $\theta$ given that $X = x$ has been observed.). Of these two approaches, (i) is a special case of (ii) since in particular $L(x, \theta)$ can be taken to be the a posteriori probability of $\theta$ given $X = x$, for a specified a priori distribution of the parameter. Neyman remarks that in his opinion "the inferential theory solves no problem" and proceeds to describe a real world situation for which the decision model is a very good one. For a random variable $X$ with possible probability distributions indexed by a parameter $\theta$ the decision approach to estimating $\theta$ is to associate, with the use of each possible estimator $z(X)$, a random loss $W(z, X, \theta)$ whose possible distributions are indexed by $\theta$; and to determine $z$ so that this loss will, in some specified sense, be as small as possible on the whole over all $\theta$ values. The purpose of the present paper is to examine Neyman's inference model in detail, to describe real situations for which it appears to be a good model, and to compare these with situations for which the decision model is more appropriate. Mathematical models are described, but there is almost no mathematics in the sense of deriving details for the models: concern is mostly with the applied mathematics question of what mathematical model to use for a real situation. Section 2 is a detailed description of Neyman's general inference model. Here $L(x, \theta)$ is interpreted as a measure of agreement between $P_\theta$ probabilities and observed proportions; terms such as "most probable," "most reasonable," "greatest confidence" are avoided as having connotations that are difficult to support. The general inference estimator is just Wolfowitz's minimum distance estimator [12]. Wolfowitz evaluates such procedures purely from the decision viewpoint. Section 3 is a detailed description of the general decision model. This is given in a form closely paralleling the description of Section 2, in order that the two models can be compared easily. In Section 4, the inference and decision models are compared from several viewpoints. The inference problem seeks an estimator $z$ such that the $P_{z(x)}$ probabilities will be (for all $x$) close to the proportions observed in $x$; the decision problem of estimating $\theta$ seeks an estimator $z$ such that $z(X)$ will be (for all $\theta$) close to $\theta$. The decision problem requires the idea of distance or error, measured by loss, in the parameter space; this idea is completely or partially absent in the inference model, where $\theta$ merely indexes the possible probability models. It is this presence or absence of a loss function that distinguishes between the decision and inference models as defined here: in the decision problem the use of an estimator $z$ results in definite losses and we are to determine a $z$ for which they are small; in the inference model the idea of definite losses does not appear. The decision model is a much more specific model for a much more specific real problem. In both problems we want to choose a probability model for the real situation: in the decision problem we know what the model is to be used for; in the inference problem we do not. It can be reasonably argued, following Neyman [5] page 17, that the inference model is sometimes used in the mistaken belief that making $P_{z(x)}$ close to the observed proportions will make $P_{z(X)}$ close to $P_\theta$, and that if so it should be replaced by the decision model with loss $W\lbrack P_{z(X)}, P_\theta\rbrack$. In the inference problem as described here, we do not have this definite aim of making $P_{z(X)}$ close to $P_\theta$: we are uncertain as to whether we want $P_{z(X)}$ close to $P_\theta$ or want $z(X)$ close to $\theta$ or want $\lbrack z(X)\rbrack^2$ close to $\theta^2$ or want something else; vaguely, we have all decision-type aims, but we have no definite one. Both the inference and decision models require the making of somewhat analogous and rather arbitrary choices at two levels. (Here, and throughout this paper "arbitrary" is used in its primary dictionary meaning of "depending on will or discretion; discretionary; can be freely chosen," with none of the secondary dictionary meanings of "unreasoned, despotic." Possible synonyms such as "subjective," "individualistic," "personalistic" are avoided because of their technical meanings.) Neyman's view that the user should be fully entitled to any choices he cares to make and that no attempt should be made to impose particular choices on all, is followed here. In Section 5 the most commonly used inference methods (Likelihood, Least Squares and Moments, Chi-square, Kolmogorov-Smirnov) are examined as special cases of the general inference method.

/pdf/on-the-inference-and-decision-models-of-statistics-151cmd04ef.pdf

On the Inference and Decision Models of Statistics

This note is concerned with possible definitions of relative efficiency for two sequences of tests of the same hypothesis. For two examples of one kind of definition, relative efficiencies of the Student test and sign test against normal alternatives are calculated for fixed sample size and asymptotically.

https://projecteuclid.org/download/pdf_1/euclid.aoms/1177706547

Colin R. Blyth

Papers

On minimax statistical decision procedures and their admissibility

Note on Estimating Information

On the Inference and Decision Models of Statistics

Note on Relative Efficiency of Tests