Probability Distributions.- Linear Models for Regression.- Linear Models for Classification.- Neural Networks.- Kernel Methods.- Sparse Kernel Machines.- Graphical Models.- Mixture Models and EM.- Approximate Inference.- Sampling Methods.- Continuous Latent Variables.- Sequential Data.- Combining Models.

Pattern Recognition and Machine Learning

Today's Web-enabled deluge of electronic data calls for automated methods of data analysis. Machine learning provides these, developing methods that can automatically detect patterns in data and then use the uncovered patterns to predict future data. This textbook offers a comprehensive and self-contained introduction to the field of machine learning, based on a unified, probabilistic approach. The coverage combines breadth and depth, offering necessary background material on such topics as probability, optimization, and linear algebra as well as discussion of recent developments in the field, including conditional random fields, L1 regularization, and deep learning. The book is written in an informal, accessible style, complete with pseudo-code for the most important algorithms. All topics are copiously illustrated with color images and worked examples drawn from such application domains as biology, text processing, computer vision, and robotics. Rather than providing a cookbook of different heuristic methods, the book stresses a principled model-based approach, often using the language of graphical models to specify models in a concise and intuitive way. Almost all the models described have been implemented in a MATLAB software package--PMTK (probabilistic modeling toolkit)--that is freely available online. The book is suitable for upper-level undergraduates with an introductory-level college math background and beginning graduate students.

Machine Learning : A Probabilistic Perspective

Statistics for Spatial Data.

/pdf/convex-analysisnoer-san-nojin-zhan-nituite-5dl2rbmoyr.pdf

Convex Analysisの二,三の進展について

Summary. Recent work by Reiss and Ogden provides a theoretical basis for sometimes preferring restricted maximum likelihood (REML) to generalized cross-validation (GCV) for smoothing parameter selection in semiparametric regression. However, existing REML or marginal likelihood (ML) based methods for semiparametric generalized linear models (GLMs) use iterative REML or ML estimation of the smoothing parameters of working linear approximations to the GLM. Such indirect schemes need not converge and fail to do so in a non-negligible proportion of practical analyses. By contrast, very reliable prediction error criteria smoothing parameter selection methods are available, based on direct optimization of GCV, or related criteria, for the GLM itself. Since such methods directly optimize properly defined functions of the smoothing parameters, they have much more reliable convergence properties. The paper develops the first such method for REML or ML estimation of smoothing parameters. A Laplace approximation is used to obtain an approximate REML or ML for any GLM, which is suitable for efficient direct optimization. This REML or ML criterion requires that Newton–Raphson iteration, rather than Fisher scoring, be used for GLM fitting, and a computationally stable approach to this is proposed. The REML or ML criterion itself is optimized by a Newton method, with the derivatives required obtained by a mixture of implicit differentiation and direct methods. The method will cope with numerical rank deficiency in the fitted model and in fact provides a slight improvement in numerical robustness on the earlier method of Wood for prediction error criteria based smoothness selection. Simulation results suggest that the new REML and ML methods offer some improvement in mean-square error performance relative to GCV or Akaike's information criterion in most cases, without the small number of severe undersmoothing failures to which Akaike's information criterion and GCV are prone. This is achieved at the same computational cost as GCV or Akaike's information criterion. The new approach also eliminates the convergence failures of previous REML- or ML-based approaches for penalized GLMs and usually has lower computational cost than these alternatives. Example applications are presented in adaptive smoothing, scalar on function regression and generalized additive model selection.

/pdf/fast-stable-restricted-maximum-likelihood-and-marginal-4rm5jbnyqs.pdf

Fast stable restricted maximum likelihood and marginal likelihood estimation of semiparametric generalized linear models

Semiparametric Regression: Example Index

Nonparametric regression using locally weighted least squares was first discussed by Stone and by Cleveland. Recently, it was shown by Fan and by Fan and Gijbels that the local linear kernel-weighted least squares regression estimator has asymptotic properties making it superior, in certain senses, to the Nadaraya-Watson and Gasser-Muller kernel estimators. In this paper we extend their results on asymptotic bias and variance to the case of multivariate predictor variables. We are able to derive the leading bias and variance terms for general multivariate kernel weights using weighted least squares matrix theory. This approach is especially convenient when analysing the asymptotic conditional bias and variance of the estimator at points near the boundary of the support of the predictors. We also investigate the asymptotic properties of the multivariate local quadratic least squares regression estimator discussed by Cleveland and Devlin and, in the univariate case, higher-order polynomial fits and derivative estimation.

/pdf/multivariate-locally-weighted-least-squares-regression-4xx3pmwaw6.pdf

Multivariate Locally Weighted Least Squares Regression

Local least squares kernel regression provides an appealing solution to the nonparametric regression, or “scatterplot smoothing,” problem, as demonstrated by Fan, for example. The practical implementation of any scatterplot smoother is greatly enhanced by the availability of a reliable rule for automatic selection of the smoothing parameter. In this article we apply the ideas of plug-in bandwidth selection to develop strategies for choosing the smoothing parameter of local linear squares kernel estimators. Our results are applicable to odd-degree local polynomial fits and can be extended to other settings, such as derivative estimation and multiple nonparametric regression. An implementation in the important case of local linear fits with univariate predictors is shown to perform well in practice. A by-product of our work is the development of a class of nonparametric variance estimators, based on local least squares ideas, and plug-in rules for their implementation.

https://www.jstor.org/stable/pdf/2291516.pdf

An Effective Bandwidth Selector for Local Least Squares Regression

An exact and easily computable expression for the mean integrated squared error (MISE) for the kernel estimator of a general normal mixture density, is given for Gaussian kernels of arbitrary order. This provides a powerful new way of understanding density estimation which complements the usual tools of simulation and asymptotic analysis. The family of normal mixture densities is very flexible and the formulae derived allow simple exact analysis for a wide variety of density shapes. A number of applications of this method giving important new insights into kernel density estimation are presented. Among these is the discovery that the usual asymptotic approximations to the MISE can be quite inaccurate, especially when the underlying density contains substantial fine structure and also strong evidence that the practical importance of higher order kernels is surprisingly small for moderate sample sizes. 1. Introduction. Substantial research has been devoted to kernel density estimation. This is because it provides a simple, yet appealing, context in which to study problems and issues that arise in all types of nonparametric curve estimation. This includes regression, spectral density and hazard estimation, and also a variety of other estimators, including histograms, splines and orthogonal series. Three important and useful tools for understanding the behavior of nonparametric curve estimators are asymptotic analysis, simulation and numerical calculation of error criteria. Each of these methods provides many useful insights into the complicated structure present in the study of curve estimation. However, each has its limitations as well. rhe strength of asymptotic analysis is that it frequently allows simultaneous study of many different specific examples, through general results applying to entire classes of settings. The weakness of asymptotics is that they only describe behavior in the limit. This is still very useful in many situations because the asymptotics describe the actual situation quite well. However, it is less useful when the asymptotics have not yet kicked in (that is, in studying situations where the asymptotically dominant effect has not taken over yet). Perhaps the biggest drawback to asymptotics is that it is very difficult to determine in a given situation which of these possibilities is occurring.

/pdf/exact-mean-integrated-squared-error-pq6wa9mvum.pdf

Exact Mean Integrated Squared Error

The typical generalized linear model for a regression of a response Y on predictors (X, Z) has conditional mean function based on a linear combination of (X, Z). We generalize these models to have a nonparametric component, replacing the linear combination α T 0X + β T 0Z by η0(α T 0X) + β T 0Z, where η0(·) is an unknown function. We call these generalized partially linear single-index models (GPLSIM). The models include the “single-index” models, which have β0 = 0. Using local linear methods, we propose estimates of the unknown parameters (α0, β0) and the unknown function η0(·) and obtain their asymptotic distributions. Examples illustrate the models and the proposed estimation methodology.

Matt P. Wand

Papers

Semiparametric Regression: Example Index

Multivariate Locally Weighted Least Squares Regression

An Effective Bandwidth Selector for Local Least Squares Regression

Exact Mean Integrated Squared Error

Generalized Partially Linear Single-Index Models