Summary. We consider the problem of comparing complex hierarchical models in which the number of parameters is not clearly defined. Using an information theoretic argument we derive a measure pD for the effective number of parameters in a model as the difference between the posterior mean of the deviance and the deviance at the posterior means of the parameters of interest. In general pD approximately corresponds to the trace of the product of Fisher's information and the posterior covariance, which in normal models is the trace of the ‘hat’ matrix projecting observations onto fitted values. Its properties in exponential families are explored. The posterior mean deviance is suggested as a Bayesian measure of fit or adequacy, and the contributions of individual observations to the fit and complexity can give rise to a diagnostic plot of deviance residuals against leverages. Adding pD to the posterior mean deviance gives a deviance information criterion for comparing models, which is related to other information criteria and has an approximate decision theoretic justification. The procedure is illustrated in some examples, and comparisons are drawn with alternative Bayesian and classical proposals. Throughout it is emphasized that the quantities required are trivial to compute in a Markov chain Monte Carlo analysis.

/pdf/bayesian-measures-of-model-complexity-and-fit-1ouzmtfu7x.pdf

Bayesian measures of model complexity and fit

Probability Distributions.- Linear Models for Regression.- Linear Models for Classification.- Neural Networks.- Kernel Methods.- Sparse Kernel Machines.- Graphical Models.- Mixture Models and EM.- Approximate Inference.- Sampling Methods.- Continuous Latent Variables.- Sequential Data.- Combining Models.

Pattern Recognition and Machine Learning

Today's Web-enabled deluge of electronic data calls for automated methods of data analysis. Machine learning provides these, developing methods that can automatically detect patterns in data and then use the uncovered patterns to predict future data. This textbook offers a comprehensive and self-contained introduction to the field of machine learning, based on a unified, probabilistic approach. The coverage combines breadth and depth, offering necessary background material on such topics as probability, optimization, and linear algebra as well as discussion of recent developments in the field, including conditional random fields, L1 regularization, and deep learning. The book is written in an informal, accessible style, complete with pseudo-code for the most important algorithms. All topics are copiously illustrated with color images and worked examples drawn from such application domains as biology, text processing, computer vision, and robotics. Rather than providing a cookbook of different heuristic methods, the book stresses a principled model-based approach, often using the language of graphical models to specify models in a concise and intuitive way. Almost all the models described have been implemented in a MATLAB software package--PMTK (probabilistic modeling toolkit)--that is freely available online. The book is suitable for upper-level undergraduates with an introductory-level college math background and beginning graduate students.

Machine Learning : A Probabilistic Perspective

How should ecologists and evolutionary biologists analyze nonnormal data that involve random effects? Nonnormal data such as counts or proportions often defy classical statistical procedures. Generalized linear mixed models (GLMMs) provide a more flexible approach for analyzing nonnormal data when random effects are present. The explosion of research on GLMMs in the last decade has generated considerable uncertainty for practitioners in ecology and evolution. Despite the availability of accurate techniques for estimating GLMM parameters in simple cases, complex GLMMs are challenging to fit and statistical inference such as hypothesis testing remains difficult. We review the use (and misuse) of GLMMs in ecology and evolution, discuss estimation and inference and summarize 'best-practice' data analysis procedures for scientists facing this challenge.

/pdf/generalized-linear-mixed-models-a-practical-guide-for-3gh61pkir0.pdf

Generalized linear mixed models: a practical guide for ecology and evolution

Using a Bayesian likelihood approach, we estimate a dynamic stochastic general equilibrium model for the US economy using seven macro-economic time series. The model incorporates many types of real and nominal frictions and seven types of structural shocks. We show that this model is able to compete with Bayesian Vector Autoregression models in out-of-sample prediction. We investigate the relative empirical importance of the various frictions. Finally, using the estimated model we address a number of key issues in business cycle analysis: What are the sources of business cycle fluctuations? Can the model explain the cross-correlation between output and inflation? What are the effects of productivity on hours worked? What are the sources of the "Great Moderation"?

/pdf/shocks-and-frictions-in-us-business-cycles-a-bayesian-dsge-d1f6ynd58q.pdf

Shocks and Frictions in US Business Cycles: A Bayesian DSGE Approach

We generalize the method proposed by Gelman and Rubin (1992a) for monitoring the convergence of iterative simulations by comparing between and within variances of multiple chains, in order to obtain a family of tests for convergence. We review methods of inference from simulations in order to develop convergence-monitoring summaries that are relevant for the purposes for which the simulations are used. We recommend applying a battery of tests for mixing based on the comparison of inferences from individual sequences and from the mixture of sequences. Finally, we discuss multivariate analogues, for assessing convergence of several parameters simultaneously.

/pdf/general-methods-for-monitoring-convergence-of-iterative-45g63vyekv.pdf

General methods for monitoring convergence of iterative simulations

The Markov chain Monte Carlo (MCMC) method, as a computer-intensive statistical tool, has enjoyed an enormous upsurge in interest over the last few years. This paper provides a simple, comprehensive and tutorial review of some of the most common areas of research in this field. We begin by discussing how MCMC algorithms can be constructed from standard building-blocks to produce Markov chains with the desired stationary distribution. We also motivate and discuss more complex ideas that have been proposed in the literature, such as continuous time and dimension jumping methods. We discuss some implementational issues associated with MCMC methods. We take a look at the arguments for and against multiple replications, consider how long chains should be run for and how to determine suitable starting points. We also take a look at graphical models and how graphical approaches can be used to simplify MCMC implementation. Finally, we present a couple of examples, which we use as case-studies to highlight some of the points made earlier in the text. In particular, we use a simple changepoint model to illustrate how to tackle a typical Bayesian modelling problem via the MCMC method, before using mixture model problems to provide illustrations of good sampler output and of the implementation of a reversible jump MCMC algorithm.

https://stat.duke.edu/~scs/Courses/Stat376/Papers/Basic/Brooks1998.pdf

Markov chain Monte Carlo method and its application

MCMC methods have effectively revolutionised the field of Bayesian statistics over the past few years. Such methods provide invaluable tools to overcome problems with analytic intractability inherent in adopting the Bayesian approach to statistical modelling.

However, any inference based upon MCMC output relies critically upon the assumption that the Markov chain being simulated has achieved a steady state or ’’converged‘‘. Many techniques have been developed for trying to determine whether or not a particular Markov chain has converged, and this paper aims to review these methods with an emphasis on the mathematics underpinning these techniques, in an attempt to summarise the current ’’state-of-play‘‘ for convergence assessment techniques and to motivate directions for future research in this area.

Convergence assessment techniques for Markov chain Monte Carlo

The major implementational problem for reversible jump Markov chain Monte Carlo methods is that there is commonly no natural way to choose jump proposals since there is no Euclidean structure in the parameter space to guide our choice We consider mechanisms for guiding the choice of proposal The first group of methods is based on an analysis of acceptance probabilities for jumps Essentially, these methods involve a Taylor series expansion of the acceptance probability around certain canonical jumps and turn out to have close connections to Langevin algorithms The second group of methods generalizes the reversible jump algorithm by using the so-called saturated space approach These allow the chain to retain some degree of memory so that, when proposing to move from a smaller to a larger model, information is borrowed from the last time that the reverse move was performed The main motivation for this paper is that, in complex problems, the probability that the Markov chain moves between such spaces may be prohibitively small, as the probability mass can be very thinly spread across the space Therefore, finding reasonable jump proposals becomes extremely important We illustrate the procedure by using several examples of reversible jump Markov chain Monte Carlo applications including the analysis of autoregressive time series, graphical Gaussian modelling and mixture modelling

Efficient construction of reversible jump Markov chain Monte Carlo proposal distributions

Much work has been published on the theoretical aspects of simulated annealing. This paper provides a brief overview of this theory and provides an introduction to the practical aspects of function optimization using this approach. Different implementations of the general simulated annealing algorithm are discussed, and two examples are used to illustrate the behaviour of the algorithm in low dimensions. A third example illustrates a hybrid approach, combining simulated annealing with traditional techniques.

Stephen P. Brooks

Papers

General methods for monitoring convergence of iterative simulations

Markov chain Monte Carlo method and its application

Convergence assessment techniques for Markov chain Monte Carlo

Efficient construction of reversible jump Markov chain Monte Carlo proposal distributions

Optimization Using Simulated Annealing