https://users.fmrib.ox.ac.uk/~jesper/papers/readgroup_071009/Ashburner07.pdf

A fast diffeomorphic image registration algorithm

We address the image denoising problem, where zero-mean white and homogeneous Gaussian additive noise is to be removed from a given image. The approach taken is based on sparse and redundant representations over trained dictionaries. Using the K-SVD algorithm, we obtain a dictionary that describes the image content effectively. Two training options are considered: using the corrupted image itself, or training on a corpus of high-quality image database. Since the K-SVD is limited in handling small image patches, we extend its deployment to arbitrary image sizes by defining a global image prior that forces sparsity over patches in every location in the image. We show how such Bayesian treatment leads to a simple and effective denoising algorithm. This leads to a state-of-the-art denoising performance, equivalent and sometimes surpassing recently published leading alternative denoising methods

Image Denoising Via Sparse and Redundant Representations Over Learned Dictionaries

On robust estimation of the location parameter

The purpose of this paper is to provide a comprehensive presentation and interpretation of the Ensemble Kalman Filter (EnKF) and its numerical implementation. The EnKF has a large user group, and numerous publications have discussed applications and theoretical aspects of it. This paper reviews the important results from these studies and also presents new ideas and alternative interpretations which further explain the success of the EnKF. In addition to providing the theoretical framework needed for using the EnKF, there is also a focus on the algorithmic formulation and optimal numerical implementation. A program listing is given for some of the key subroutines. The paper also touches upon specific issues such as the use of nonlinear measurements, in situ profiles of temperature and salinity, and data which are available with high frequency in time. An ensemble based optimal interpolation (EnOI) scheme is presented as a cost-effective approach which may serve as an alternative to the EnKF in some applications. A fairly extensive discussion is devoted to the use of time correlated model errors and the estimation of model bias.

The Ensemble Kalman Filter: Theoretical formulation and practical implementation

Large linear systems of saddle point type arise in a wide variety of applications throughout computational science and engineering. Due to their indefiniteness and often poor spectral properties, such linear systems represent a significant challenge for solver developers. In recent years there has been a surge of interest in saddle point problems, and numerous solution techniques have been proposed for this type of system. The aim of this paper is to present and discuss a large selection of solution methods for linear systems in saddle point form, with an emphasis on iterative methods for large and sparse problems.

/pdf/numerical-solution-of-saddle-point-problems-l583iob5c2.pdf

Numerical solution of saddle point problems

Deep neural networks have become invaluable tools for supervised machine learning, e.g. classification of text or images. While often offering superior results over traditional techniques and successfully expressing complicated patterns in data, deep architectures are known to be challenging to design and train such that they generalize well to new data. Critical issues with deep architectures are numerical instabilities in derivative-based learning algorithms commonly called exploding or vanishing gradients. In this paper, we propose new forward propagation techniques inspired by systems of ordinary differential equations (ODE) that overcome this challenge and lead to well-posed learning problems for arbitrarily deep networks. The backbone of our approach is our interpretation of deep learning as a parameter estimation problem of nonlinear dynamical systems. Given this formulation, we analyze stability and well-posedness of deep learning and use this new understanding to develop new network architectures. We relate the exploding and vanishing gradient phenomenon to the stability of the discrete ODE and present several strategies for stabilizing deep learning for very deep networks. While our new architectures restrict the solution space, several numerical experiments show their competitiveness with state-of-the-art networks.

Stable Architectures for Deep Neural Networks

Partial differential equations (PDEs) are indispensable for modeling many physical phenomena and also commonly used for solving image processing tasks. In the latter area, PDE-based approaches interpret image data as discretizations of multivariate functions and the output of image processing algorithms as solutions to certain PDEs. Posing image processing problems in the infinite-dimensional setting provides powerful tools for their analysis and solution. For the last few decades, the reinterpretation of classical image processing problems through the PDE lens has been creating multiple celebrated approaches that benefit a vast area of tasks including image segmentation, denoising, registration, and reconstruction. In this paper, we establish a new PDE interpretation of a class of deep convolutional neural networks (CNN) that are commonly used to learn from speech, image, and video data. Our interpretation includes convolution residual neural networks (ResNet), which are among the most promising approaches for tasks such as image classification having improved the state-of-the-art performance in prestigious benchmark challenges. Despite their recent successes, deep ResNets still face some critical challenges associated with their design, immense computational costs and memory requirements, and lack of understanding of their reasoning. Guided by well-established PDE theory, we derive three new ResNet architectures that fall into two new classes: parabolic and hyperbolic CNNs. We demonstrate how PDE theory can provide new insights and algorithms for deep learning and demonstrate the competitiveness of three new CNN architectures using numerical experiments.

/pdf/deep-neural-networks-motivated-by-partial-differential-52td7cd0in.pdf

Deep Neural Networks Motivated by Partial Differential Equations

This paper considers optimization techniques for the solution of nonlinear inverse problems where the forward problems, like those encountered in electromagnetics, are modelled by differential equations. Such problems are often solved by utilizing a Gauss-Newton method in which the forward model constraints are implicitly incorporated. Variants of Newton's method which use second-derivative information are rarely employed because their perceived disadvantage in computational cost per step offsets their potential benefits of faster convergence. In this paper we show that, by formulating the inversion as a constrained or unconstrained optimization problem, and by employing sparse matrix techniques, we can carry out variants of sequential quadratic programming and the full Newton iteration with only a modest additional cost. By working with the differential equation explicitly we are able to relate the constrained and the unconstrained formulations and discuss the advantages of each. To make the comparisons meaningful we adopt the same global optimization strategy for all inversions. As an illustration, we focus upon a 1D electromagnetic (EM) example simulating a magnetotelluric survey. This problem is sufficiently rich that it illuminates most of the computational complexities that are prevalent in multi-source inverse problems and we therefore describe its solution process in detail. The numerical results illustrate that variants of Newton's method which utilize second-derivative information can produce a solution in fewer iterations and, in some cases where the data contain significant noise, requiring fewer floating point operations than Gauss-Newton techniques. Although further research is required, we believe that the variants proposed here will have a significant impact on developing practical solutions to large-scale 3D EM inverse problems.

https://iopscience.iop.org/article/10.1088/0266-5611/16/5/309/pdf

On optimization techniques for solving nonlinear inverse problems

We develop a methodology to invert two different data sets with the assumption that the underlying models have a common structure. Structure is defined in terms of absolute value of curvature of the model and two models are said to have common structure if the changes occur at the same physical locations. The joint inversion is solved by defining an objective function which quantifies the difference in structure between two models, and then minimizing this objective function subject to satisfying the data constraints. The problem is nonlinear and is solved iteratively using Krylov space techniques. Testing the algorithm on synthetic data sets shows that the joint inversion is superior to individual inversions. In an application to field data we show that the data sets are consistent with models that are quite similar.

/pdf/joint-inversion-a-structural-approach-58wztdngzg.pdf

Joint inversion: a structural approach

A particular problem in image registration arises for multi-modal images taken from different imaging devices and/or modalities. Starting in 1995, mutual information has shown to be a very successful distance measure for multi-modal image registration. However, mutual information has also a number of well-known drawbacks. Its main disadvantage is that it is known to be highly non-convex and has typically many local maxima.

This observation motivate us to seek a different image similarity measure which is better suited for optimization but as well capable to handle multi-modal images. In this work we investigate an alternative distance measure which is based on normalized gradients and compare its performance to Mutual Information. We call the new distance measure Normalized Gradient Fields (NGF).

/pdf/intensity-gradient-based-registration-and-fusion-of-multi-53x88w6i65.pdf

Eldad Haber

Papers

Stable Architectures for Deep Neural Networks

Deep Neural Networks Motivated by Partial Differential Equations

On optimization techniques for solving nonlinear inverse problems

Joint inversion: a structural approach

Intensity gradient based registration and fusion of multi-modal images