Setting of the learning problem consistency of learning processes bounds on the rate of convergence of learning processes controlling the generalization ability of learning processes constructing learning algorithms what is important in learning theory?.

The Nature of Statistical Learning Theory

We propose a generative model for text and other collections of discrete data that generalizes or improves on several previous models including naive Bayes/unigram, mixture of unigrams [6], and Hof-mann's aspect model, also known as probabilistic latent semantic indexing (pLSI) [3]. In the context of text modeling, our model posits that each document is generated as a mixture of topics, where the continuous-valued mixture proportions are distributed as a latent Dirichlet random variable. Inference and learning are carried out efficiently via variational algorithms. We present empirical results on applications of this model to problems in text modeling, collaborative filtering, and text classification.

/pdf/latent-dirichlet-allocation-rkwbqihm3p.pdf

Latent Dirichlet Allocation

THE DESIGN AND ANALYSIS OF EXPERIMENTS. By Oscar Kempthorne. New York, John Wiley and Sons, Inc., 1952. 631 pp. $8.50. This book by a teacher of statistics (as well as a consultant for \"experimenters\") is a comprehensive study of the philosophical background for the statistical design of experiment. It is necessary to have some facility with algebraic notation and manipulation to be able to use the volume intelligently. The problems are presented from the theoretical point of view, without such practical examples as would be helpful for those not acquainted with mathematics. The mathematical justification for the techniques is given. As a somewhat advanced treatment of the design and analysis of experiments, this volume will be interesting and helpful for many who approach statistics theoretically as well as practically. With emphasis on the \"why,\" and with description given broadly, the author relates the subject matter to the general theory of statistics and to the general problem of experimental inference. MARGARET J. ROBERTSON

The Design and Analysis of Experiments

The application of maximum likelihood techniques to the estimation of evolutionary trees from nucleic acid sequence data is discussed. A computationally feasible method for finding such maximum likelihood estimates is developed, and a computer program is available. This method has advantages over the traditional parsimony algorithms, which can give misleading results if rates of evolution differ in different lineages. It also allows the testing of hypotheses about the constancy of evolutionary rates by likelihood ratio tests, and gives rough indication of the error of the estimate of the tree.

Evolutionary trees from DNA sequences: A maximum likelihood approach

An essential textbook for any student or researcher in biology needing to design experiments, sample programs or analyse the resulting data The text begins with a revision of estimation and hypothesis testing methods, covering both classical and Bayesian philosophies, before advancing to the analysis of linear and generalized linear models Topics covered include linear and logistic regression, simple and complex ANOVA models (for factorial, nested, block, split-plot and repeated measures and covariance designs), and log-linear models Multivariate techniques, including classification and ordination, are then introduced Special emphasis is placed on checking assumptions, exploratory data analysis and presentation of results The main analyses are illustrated with many examples from published papers and there is an extensive reference list to both the statistical and biological literature The book is supported by a website that provides all data sets, questions for each chapter and links to software

/pdf/experimental-design-and-data-analysis-for-biologists-8ybva8d53x.pdf

Experimental Design and Data Analysis for Biologists

Let $\mathbf{X} = (X_1, \cdots, X_p)^t$ be an observation from a $p$-variate normal distribution with unknown mean $\mathbf{\theta} = (\theta_1, \cdots, \theta_p)^t$ and identity covariance matrix. We consider a control problem which, in canonical form, is the problem of estimating $\mathbf{\theta}$ under the loss $L(\mathbf{\theta, \delta}) = (\mathbf{\theta}^t \mathbf{\delta} - 1)^2$, where $\mathbf{\delta(x)} = (\delta_1(\mathbf{x}), \cdots, \delta_p(\mathbf{x}))^t$ is the estimate of $\mathbf{\theta}$ for a given $\mathbf{x}$. General theorems are given for establishing admissibility or inadmissibility of estimators in this problem. As an application, it is shown that estimators of the form $\mathbf{\delta(x)} = (|\mathbf{x}|^2 + c)^{-1}\mathbf{x} + |\mathbf{x}|^{-4}w(|\mathbf{x}|)\mathbf{x}$, where $w(|\mathbf{x}|)$ tends to zero as $|\mathbf{x}| \rightarrow \infty$, are inadmissible if $c > 5 - p$, but are admissible if $c \leq 5 - p$ and $\mathbf{\delta}$ is generalized Bayes for an appropriate prior measure. Also, an approximation to generalized Bayes estimators for large $|\mathbf{x}|$ is developed.

/pdf/general-admissibility-and-inadmissibility-results-for-437zkc6b7g.pdf

General Admissibility and Inadmissibility Results for Estimation in a Control Problem

Hypothesis testing and model uncertainty

Bayesian analysis for the covariance matrix of a multivariate normal distribution has received a lot of attention in the last two decades. In this paper, we propose a new class of priors for the covariance matrix, including both inverse Wishart and reference priors as special cases. The main motivation for the new class is to have available priors—both subjective and objective—that do not “force eigenvalues apart,” which is a criticism of inverse Wishart and Jeffreys priors. Extensive comparison of these “shrinkage priors” with inverse Wishart and Jeffreys priors is undertaken, with the new priors seeming to have considerably better performance. A number of curious facts about the new priors are also observed, such as that the posterior distribution will be proper with just three vector observations from the multivariate normal distribution—regardless of the dimension of the covariance matrix—and that useful inference about features of the covariance matrix can be possible. Finally, a new MCMC algorithm is developed for this class of priors and is shown to be computationally effective for matrices of up to 100 dimensions.

Bayesian analysis of the covariance matrix of a multivariate normal distribution with a new class of priors

Publisher Summary A version of the Karhunen–Loeve expansion of X(·) has been used to reduce the estimation problem to that of estimating a countable sequence of normal means {θi}. The prior information concerning θ(·) was also transformed into prior information about the θi, but in selecting a minimax estimator using the prior information, the covariances among the θi were ignored. This chapter discusses a more complicated expansion of the process. It allows use of all the prior information in selecting a minimax estimator. This expansion is developed in which the desired minimax estimator is also derived.

Incorporating prior information in minimax estimation of the mean of a gaussian process

We present a new approach to model selection and Bayes factor determination, based on Laplace expansions (as in BIC), which we call Prior-based Bayes Information Criterion (PBIC). In this approach, the Laplace expansion is only done with the likelihood function, and then a suitable prior distribution is chosen to allow exact computation of the (approximate) marginal likelihood arising from the Laplace approximation and the prior. The result is a closed-form expression similar to BIC, but now involves a term arising from the prior distribution (which BIC ignores) and also incorporates the idea that different parameters can have different effective sample sizes (whereas BIC only allows one overall sample size n). We also consider a modification of PBIC which is more favourable to complex models.

/pdf/prior-based-bayesian-information-criterion-5n8y3ts58b.pdf

James O. Berger

Papers

General Admissibility and Inadmissibility Results for Estimation in a Control Problem

Hypothesis testing and model uncertainty

Bayesian analysis of the covariance matrix of a multivariate normal distribution with a new class of priors

Incorporating prior information in minimax estimation of the mean of a gaussian process

Prior-based Bayesian information criterion