There is, I think, something ethereal about i —the square root of minus one. I remember first hearing about it at school. It seemed an odd beast at that time—an intruder hovering on the edge of reality.

Usually familiarity dulls this sense of the bizarre, but in the case of i it was the reverse: over the years the sense of its surreal nature intensified. It seemed that it was impossible to write mathematics that described the real world in …

I and i

Probability Distributions.- Linear Models for Regression.- Linear Models for Classification.- Neural Networks.- Kernel Methods.- Sparse Kernel Machines.- Graphical Models.- Mixture Models and EM.- Approximate Inference.- Sampling Methods.- Continuous Latent Variables.- Sequential Data.- Combining Models.

Pattern Recognition and Machine Learning

System Identification I

The paper proposes Metropolis adjusted Langevin and Hamiltonian Monte Carlo sampling methods defined on the Riemann manifold to resolve the shortcomings of existing Monte Carlo algorithms when sampling from target densities that may be high dimensional and exhibit strong correlations. The methods provide fully automated adaptation mechanisms that circumvent the costly pilot runs that are required to tune proposal densities for Metropolis-Hastings or indeed Hamiltonian Monte Carlo and Metropolis adjusted Langevin algorithms. This allows for highly efficient sampling even in very high dimensions where different scalings may be required for the transient and stationary phases of the Markov chain. The methodology proposed exploits the Riemann geometry of the parameter space of statistical models and thus automatically adapts to the local structure when simulating paths across this manifold, providing highly efficient convergence and exploration of the target density. The performance of these Riemann manifold Monte Carlo methods is rigorously assessed by performing inference on logistic regression models, log-Gaussian Cox point processes, stochastic volatility models and Bayesian estimation of dynamic systems described by non-linear differential equations. Substantial improvements in the time-normalized effective sample size are reported when compared with alternative sampling approaches. MATLAB code that is available from http://www.ucl.ac.uk/statistics/research/rmhmc allows replication of all the results reported.

Riemann manifold Langevin and Hamiltonian Monte Carlo methods

Filtering and smoothing methods are used to produce an accurate estimate of the state of a time-varying system based on multiple observational inputs (data). Interest in these methods has exploded in recent years, with numerous applications emerging in fields such as navigation, aerospace engineering, telecommunications and medicine. This compact, informal introduction for graduate students and advanced undergraduates presents the current state-of-the-art filtering and smoothing methods in a unified Bayesian framework. Readers learn what non-linear Kalman filters and particle filters are, how they are related, and their relative advantages and disadvantages. They also discover how state-of-the-art Bayesian parameter estimation methods can be combined with state-of-the-art filtering and smoothing algorithms. The book's practical and algorithmic approach assumes only modest mathematical prerequisites. Examples include MATLAB computations, and the numerous end-of-chapter exercises include computational assignments. MATLAB/GNU Octave source code is available for download at www.cambridge.org/sarkka, promoting hands-on work with the methods.

/pdf/bayesian-filtering-and-smoothing-4w1c72z7m8.pdf

Bayesian Filtering and Smoothing

The particle filter is one of the most successful methods for state inference and identification of general non-linear and non-Gaussian models. However, standard particle filters suffer from degeneracy of the particle weights for high-dimensional problems. We propose a method for improving the performance of the particle filter for certain challenging state space models, with implications for high-dimensional inference. First we approximate the model by adding artificial process noise in an additional state update, then we design a proposal that combines the standard and the locally optimal proposal. This results in a bias-variance trade-off, where adding more noise reduces the variance of the estimate but increases the model bias. The performance of the proposed method is evaluated on a linear Gaussian state space model and on the non-linear Lorenz'96 model. For both models we observe a significant improvement in performance over the standard particle filter.

Improving the particle filter for high-dimensional problems using artificial process noise

/pdf/likelihood-free-out-of-distribution-detection-with-2g6vet8k05.pdf

Likelihood-free Out-of-Distribution Detection with Invertible Generative Models

The original version of this Article contained an error in the first sentence in the legend of Fig. 1, which incorrectly read 'The first letter of 'Hatteras' should be capitalized, in both Figure 1a and 1b since Hatteras is a proper noun.' The correct version removes this sentence. This has been corrected in both the PDF and HTML versions of the Article.

/pdf/publisher-correction-coherent-modulation-of-the-sea-level-1o5mjedo1m.pdf

Publisher Correction: Coherent modulation of the sea-level annual cycle in the United States by Atlantic Rossby waves

A deep neural classifier trained on an upstream task can be leveraged to boost the performance of another classifier in a related downstream task through the representations learned in hidden layers. However, presence of shortcuts (easy-to-learn features) in the upstream task can considerably impair the versatility of intermediate representations and, in turn, the downstream performance. In this paper, we propose a method to improve the representations learned by deep neural image classifiers in spite of a shortcut in upstream data. In our method, the upstream classification objective is augmented with a type of adversarial training where an auxiliary network, so called lens, fools the classifier by exploiting the shortcut in reconstructing images. Empirical comparisons in self-supervised and transfer learning problems with three shortcut-biased datasets suggest the advantages of our method in terms of downstream performance and/or training time. 

Enhancing Representation Learning with Deep Classifiers in Presence of Shortcut

This supplemental material contains the proof of [1, Proposition 1]. 1 Proof of Proposition 1 With M = T − t and w(k) = w t , the distributions of interest are given by P (k) = w(k) ∏M s=1 hs(k) ∑ l w(l) ∏M s=1 hs(l) and Pp(k) = w(k) ∏p s=1 hs(k) ∑ l w(l) ∏p s=1 hs(l) , respectively. Let es , maxk,l (hs(k)/hs(l)− 1) ≤ A exp(−cs) and consider (∑ l w(l) p ∏ s=1 hs(l) ) M ∏ s=p+1 hs(k) ≤ ∑ l ( w(l) p ∏ s=1 hs(l) M ∏ s=p+1 hs(l)(1 + es) )

/pdf/ancestor-sampling-for-particle-gibbs-supplemental-material-8y56zead65.pdf

Fredrik Lindsten

Papers

Improving the particle filter for high-dimensional problems using artificial process noise

Likelihood-free Out-of-Distribution Detection with Invertible Generative Models

Publisher Correction: Coherent modulation of the sea-level annual cycle in the United States by Atlantic Rossby waves

Enhancing Representation Learning with Deep Classifiers in Presence of Shortcut

Ancestor Sampling for Particle Gibbs: Supplemental material