There is, I think, something ethereal about i —the square root of minus one. I remember first hearing about it at school. It seemed an odd beast at that time—an intruder hovering on the edge of reality.

Usually familiarity dulls this sense of the bizarre, but in the case of i it was the reverse: over the years the sense of its surreal nature intensified. It seemed that it was impossible to write mathematics that described the real world in …

I and i

Probability Distributions.- Linear Models for Regression.- Linear Models for Classification.- Neural Networks.- Kernel Methods.- Sparse Kernel Machines.- Graphical Models.- Mixture Models and EM.- Approximate Inference.- Sampling Methods.- Continuous Latent Variables.- Sequential Data.- Combining Models.

Pattern Recognition and Machine Learning

System Identification I

The paper proposes Metropolis adjusted Langevin and Hamiltonian Monte Carlo sampling methods defined on the Riemann manifold to resolve the shortcomings of existing Monte Carlo algorithms when sampling from target densities that may be high dimensional and exhibit strong correlations. The methods provide fully automated adaptation mechanisms that circumvent the costly pilot runs that are required to tune proposal densities for Metropolis-Hastings or indeed Hamiltonian Monte Carlo and Metropolis adjusted Langevin algorithms. This allows for highly efficient sampling even in very high dimensions where different scalings may be required for the transient and stationary phases of the Markov chain. The methodology proposed exploits the Riemann geometry of the parameter space of statistical models and thus automatically adapts to the local structure when simulating paths across this manifold, providing highly efficient convergence and exploration of the target density. The performance of these Riemann manifold Monte Carlo methods is rigorously assessed by performing inference on logistic regression models, log-Gaussian Cox point processes, stochastic volatility models and Bayesian estimation of dynamic systems described by non-linear differential equations. Substantial improvements in the time-normalized effective sample size are reported when compared with alternative sampling approaches. MATLAB code that is available from http://www.ucl.ac.uk/statistics/research/rmhmc allows replication of all the results reported.

Riemann manifold Langevin and Hamiltonian Monte Carlo methods

Filtering and smoothing methods are used to produce an accurate estimate of the state of a time-varying system based on multiple observational inputs (data). Interest in these methods has exploded in recent years, with numerous applications emerging in fields such as navigation, aerospace engineering, telecommunications and medicine. This compact, informal introduction for graduate students and advanced undergraduates presents the current state-of-the-art filtering and smoothing methods in a unified Bayesian framework. Readers learn what non-linear Kalman filters and particle filters are, how they are related, and their relative advantages and disadvantages. They also discover how state-of-the-art Bayesian parameter estimation methods can be combined with state-of-the-art filtering and smoothing algorithms. The book's practical and algorithmic approach assumes only modest mathematical prerequisites. Examples include MATLAB computations, and the numerous end-of-chapter exercises include computational assignments. MATLAB/GNU Octave source code is available for download at www.cambridge.org/sarkka, promoting hands-on work with the methods.

/pdf/bayesian-filtering-and-smoothing-4w1c72z7m8.pdf

Bayesian Filtering and Smoothing

In state space models, smoothing refers to the task of estimating a latent stochastic process given noisy measurements related to the process. We propose an unbiased estimator of smoothing expectat ...

Retracted article: Smoothing with Couplings of Conditional Particle Filters

This thesis investigates the crucial problem of collision avoidance for autonomous vehicles. An anti-collision system for an unmanned aerial vehicle (UAV) is studied in particular. The purpose of this system is to make sure that the own vehicle avoids collision with other aircraft in mid-air. The sensor used to track any possible threat is for a UAV limited basically to a digital video camera. This sensor can only measure the direction to an intruding vehicle, not the range, and is therefore denoted an angle-only sensor. To estimate the position and velocity of the intruder a tracking system, based on an extended Kalman filter, is used. State estimates supplied by this system are very uncertain due to the difficulties of angle-only tracking. Probabilistic methods are therefore required for risk calculation. The risk assessment module is one of the essential parts of the collision avoidance system and has the purpose of continuously evaluating the risk for collision. To do this in a probabilistic way, it is necessary to assume a probability distribution for the tracking system output. A common approach is to assume normality, more out of habit than on actual grounds. This thesis investigates the normality assumption, and it is found that the tracking output rapidly converge towards a good normal distribution approximation. The thesis furthermore investigates the actual risk assessment module to find out how the collision risk should be determined. The traditional way to do this is to focus on a critical time point (time of closest point of approach, time of maximum collision risk etc.). A recently proposed alternative is to evaluate the risk over a horizon of time. The difference between these two concepts is evaluated. An approximate computational method for integrated risk, suitable for real-time implementations, is also validated. It is shown that the risk seen over a horizon of time is much more robust to estimation accuracy than the risk from a critical time point. The integrated risk also gives a more intuitively correct result, which makes it possible to implement the risk assessment module with a direct connection to specified aviation safety rules.

Angle-only based collision risk assessment for unmanned aerial vehicles

Distributed algorithms for solving additive or consensus optimization problems commonly rely on first-order or proximal splitting methods. These algorithms generally come with restrictive assumptions and at best enjoy a linear convergence rate. Hence, they can require many iterations or communications among agents to converge. In many cases, however, we do not seek a highly accurate solution for consensus problems. Based on this we propose a controlled relaxation of the coupling in the problem which allows us to compute an approximate solution, where the accuracy of the approximation can be controlled by the level of relaxation. The relaxed problem can be efficiently solved in a distributed way using a combination of primal-dual interior-point methods (PDIPMs) and message-passing. This algorithm purely relies on second-order methods and thus requires far fewer iterations and communications to converge. This is illustrated in numerical experiments, showing its superior performance compared to existing methods.

Distributed, scalable and gossip-free consensus optimization with application to data analysis

Ensembles of neural networks have shown to give better predictive performance and more reliable uncertainty estimates than individual networks. Additionally, ensembles allow the uncertainty to be decomposed into aleatoric (data) and epistemic (model) components, giving a more complete picture of the predictive uncertainty. Ensemble distillation is the process of compressing an ensemble into a single model, often resulting in a leaner model that still outperforms the individual ensemble members. Unfortunately, standard distillation erases the natural uncertainty decomposition of the ensemble. We present a general framework for distilling both regression and classification ensembles in a way that preserves the decomposition. We demonstrate the desired behaviour of our framework and show that its predictive performance is on par with standard distillation.

/pdf/a-general-framework-for-ensemble-distribution-distillation-3qj8undryu.pdf

A General Framework for Ensemble Distribution Distillation

The primary contribution of this paper is an algorithm capable of identifying parameters in certain mixed linear/nonlinear state-space models, containing conditionally linear Gaussian substructures. More specifically, we employ the standard maximum likelihood framework and derive an expectation maximization type algorithm. This involves a nonlinear smoothing problem for the state variables, which for the conditionally linear Gaussian system can be efficiently solved using so called Rao-Blackwellized particle smoother (RBPS). As a secondary contribution of this paper we extend an existing RBPS to be able to handle the fully interconnected model under study.

Fredrik Lindsten

Papers

Retracted article: Smoothing with Couplings of Conditional Particle Filters

Angle-only based collision risk assessment for unmanned aerial vehicles

Distributed, scalable and gossip-free consensus optimization with application to data analysis

A General Framework for Ensemble Distribution Distillation

Maximum Likelihood Estimation in Mixed Linear/Nonlinear State-Space Models