Summary. We consider the problem of comparing complex hierarchical models in which the number of parameters is not clearly defined. Using an information theoretic argument we derive a measure pD for the effective number of parameters in a model as the difference between the posterior mean of the deviance and the deviance at the posterior means of the parameters of interest. In general pD approximately corresponds to the trace of the product of Fisher's information and the posterior covariance, which in normal models is the trace of the ‘hat’ matrix projecting observations onto fitted values. Its properties in exponential families are explored. The posterior mean deviance is suggested as a Bayesian measure of fit or adequacy, and the contributions of individual observations to the fit and complexity can give rise to a diagnostic plot of deviance residuals against leverages. Adding pD to the posterior mean deviance gives a deviance information criterion for comparing models, which is related to other information criteria and has an approximate decision theoretic justification. The procedure is illustrated in some examples, and comparisons are drawn with alternative Bayesian and classical proposals. Throughout it is emphasized that the quantities required are trivial to compute in a Markov chain Monte Carlo analysis.

/pdf/bayesian-measures-of-model-complexity-and-fit-1ouzmtfu7x.pdf

Bayesian measures of model complexity and fit

Probability Distributions.- Linear Models for Regression.- Linear Models for Classification.- Neural Networks.- Kernel Methods.- Sparse Kernel Machines.- Graphical Models.- Mixture Models and EM.- Approximate Inference.- Sampling Methods.- Continuous Latent Variables.- Sequential Data.- Combining Models.

Pattern Recognition and Machine Learning

Today's Web-enabled deluge of electronic data calls for automated methods of data analysis. Machine learning provides these, developing methods that can automatically detect patterns in data and then use the uncovered patterns to predict future data. This textbook offers a comprehensive and self-contained introduction to the field of machine learning, based on a unified, probabilistic approach. The coverage combines breadth and depth, offering necessary background material on such topics as probability, optimization, and linear algebra as well as discussion of recent developments in the field, including conditional random fields, L1 regularization, and deep learning. The book is written in an informal, accessible style, complete with pseudo-code for the most important algorithms. All topics are copiously illustrated with color images and worked examples drawn from such application domains as biology, text processing, computer vision, and robotics. Rather than providing a cookbook of different heuristic methods, the book stresses a principled model-based approach, often using the language of graphical models to specify models in a concise and intuitive way. Almost all the models described have been implemented in a MATLAB software package--PMTK (probabilistic modeling toolkit)--that is freely available online. The book is suitable for upper-level undergraduates with an introductory-level college math background and beginning graduate students.

Machine Learning : A Probabilistic Perspective

Economics and Information Theory

Introduction to stochastic programming

(2004). Bayesian Methods for Nonlinear Classification and Regression. Technometrics: Vol. 46, No. 2, pp. 251-252.

Bayesian Methods for Nonlinear Classification and Regression

This paper reviews the literature on urban water demand forecasting published from 2000 to 2010 to identify methods and models useful for specific water utility decision making problems. Results show that although a wide variety of methods and models have attracted attention, applications of these models differ, depending on the forecast variable, its periodicity and the forecast horizon. Whereas artificial neural networks are more likely to be used for short-term forecasting, econometric models, coupled with simulation or scenario-based forecasting, tend to be used for long-term strategic decisions. Much more attention needs to be given to probabilistic forecasting methods if utilities are to make decisions that reflect the level of uncertainty in future demand forecasts.

Urban Water Demand Forecasting: Review of Methods and Models

A Bayesian perspective on some replacement strategies

In this paper we motivate a random coefficient autoregressive process of order 1 for describing reliability growth or decay. We introduce several ramifications of this process, some of which reduce it to a Kalman Filter model. We illustrate the usefulness of our approach by applying these processes to some real life data on software failures. Finally, we make a pairwise comparison of the models in terms of the ratio of likelihoods of their predictive distributions, and identify the "best" model.

Assessing (Software) Reliability Growth Using a Random Coefficient Autoregressive Process and Its Ramifications

Information-theoretic methodologies are increasingly being used in various disciplines. Fre- quently an information measure is adapted for a problem, yet the perspective of information as the unifying notion is overlooked. We set forth this perspective through presenting information- theoretic methodologies for a set of problems in probability and statistics. Our focal measures are Shannon entropy and Kullback-Leibler information. The background topics for these mea- sures include notions of uncertainty and information, their axiomaticfoundation, interpretations, properties, and generalizations. Topics with broad methodological applications include discrep- ancy between distributions, derivation of probability models, dependence between variables, and Bayesian analysis. More specific methodological topics include model selection, limiting distributions, optimal prior distribution and design of experiment, modeling duration variables, order statistics, data disclosure, and relative importance of predictors. Illustrations range from very basic to highly technical ones that draw attention to subtle points.

Refik Soyer

Papers

Bayesian Methods for Nonlinear Classification and Regression

Urban Water Demand Forecasting: Review of Methods and Models

A Bayesian perspective on some replacement strategies

Assessing (Software) Reliability Growth Using a Random Coefficient Autoregressive Process and Its Ramifications

Information Measures in Perspective