We consider linear inverse problems where the solution is assumed to have a sparse expansion on an arbitrary preassigned orthonormal basis. We prove that replacing the usual quadratic regularizing penalties by weighted p-penalties on the coefficients of such expansions, with 1 ≤ p ≤ 2, still regularizes the problem. Use of such p-penalized problems with p < 2 is often advocated when one expects the underlying ideal noiseless solution to have a sparse expansion with respect to the basis under consideration. To compute the corresponding regularized solutions, we analyze an iterative algorithm that amounts to a Landweber iteration with thresholding (or nonlinear shrinkage) applied at each iteration step. We prove that this algorithm converges in norm. © 2004 Wiley Periodicals, Inc.

/pdf/an-iterative-thresholding-algorithm-for-linear-inverse-565i4q2jj2.pdf

An Iterative Thresholding Algorithm for Linear Inverse Problems with a Sparsity Constraint

We consider linear inverse problems where the solution is assumed to have a sparse expansion on an arbitrary pre-assigned orthonormal basis. We prove that replacing the usual quadratic regularizing penalties by weighted l^p-penalties on the coefficients of such expansions, with 1 < or = p < or =2, still regularizes the problem. If p < 2, regularized solutions of such l^p-penalized problems will have sparser expansions, with respect to the basis under consideration. To compute the corresponding regularized solutions we propose an iterative algorithm that amounts to a Landweber iteration with thresholding (or nonlinear shrinkage) applied at each iteration step. We prove that this algorithm converges in norm. We also review some potential applications of this method.

An iterative thresholding algorithm for linear inverse problems with a sparsity constraint

This book provides a broad survey of models and efficient algorithms for Nonnegative Matrix Factorization (NMF) This includes NMFs various extensions and modifications, especially Nonnegative Tensor Factorizations (NTF) and Nonnegative Tucker Decompositions (NTD) NMF/NTF and their extensions are increasingly used as tools in signal and image processing, and data analysis, having garnered interest due to their capability to provide new insights and relevant information about the complex latent relationships in experimental data sets It is suggested that NMF can provide meaningful components with physical interpretations; for example, in bioinformatics, NMF and its extensions have been successfully applied to gene expression, sequence analysis, the functional characterization of genes, clustering and text mining As such, the authors focus on the algorithms that are most useful in practice, looking at the fastest, most robust, and suitable for large-scale models Key features: Acts as a single source reference guide to NMF, collating information that is widely dispersed in current literature, including the authors own recently developed techniques in the subject area Uses generalized cost functions such as Bregman, Alpha and Beta divergences, to present practical implementations of several types of robust algorithms, in particular Multiplicative, Alternating Least Squares, Projected Gradient and Quasi Newton algorithms Provides a comparative analysis of the different methods in order to identify approximation error and complexity Includes pseudo codes and optimized MATLAB source codes for almost all algorithms presented in the book The increasing interest in nonnegative matrix and tensor factorizations, as well as decompositions and sparse representation of data, will ensure that this book is essential reading for engineers, scientists, researchers, industry practitioners and graduate students across signal and image processing; neuroscience; data mining and data analysis; computer science; bioinformatics; speech processing; biomedical engineering; and multimedia

Nonnegative Matrix and Tensor Factorizations: Applications to Exploratory Multi-way Data Analysis and Blind Source Separation

Most problems in frequentist statistics involve optimization of a function such as a likelihood or a sum of squares. EM algorithms are among the most effective algorithms for maximum likelihood estimation because they consistently drive the likelihood uphill by maximizing a simple surrogate function for the log-likelihood. Iterative optimization of a surrogate function as exemplified by an EM algorithm does not necessarily require missing data. Indeed, every EM algorithm is a special case of the more general class of MM optimization algorithms, which typically exploit convexity rather than missing data in majorizing or minorizing an objective function. In our opinion, MM algorithms deserve to be part of the standard toolkit of professional statisticians. This article explains the principle behind MM algorithms, suggests some methods for constructing them, and discusses some of their attractive features. We include numerous examples throughout the article to illustrate the concepts described. In addition t...

/pdf/a-tutorial-on-mm-algorithms-4yt18j1td5.pdf

A Tutorial on MM Algorithms

This book provides a broad survey of models and efficient algorithms for Nonnegative Matrix Factorization (NMF). This includes NMFs various extensions and modifications, especially Nonnegative Tensor Factorizations (NTF) and Nonnegative Tucker Decompositions (NTD). NMF/NTF and their extensions are increasingly used as tools in signal and image processing, and data analysis, having garnered interest due to their capability to provide new insights and relevant information about the complex latent relationships in experimental data sets. It is suggested that NMF can provide meaningful components with physical interpretations; for example, in bioinformatics, NMF and its extensions have been successfully applied to gene expression, sequence analysis, the functional characterization of genes, clustering and text mining. As such, the authors focus on the algorithms that are most useful in practice, looking at the fastest, most robust, and suitable for large-scale models. Key features: Acts as a single source reference guide to NMF, collating information that is widely dispersed in current literature, including the authors own recently developed techniques in the subject area. Uses generalized cost functions such as Bregman, Alpha and Beta divergences, to present practical implementations of several types of robust algorithms, in particular Multiplicative, Alternating Least Squares, Projected Gradient and Quasi Newton algorithms. Provides a comparative analysis of the different methods in order to identify approximation error and complexity. Includes pseudo codes and optimized MATLAB source codes for almost all algorithms presented in the book. The increasing interest in nonnegative matrix and tensor factorizations, as well as decompositions and sparse representation of data, will ensure that this book is essential reading for engineers, scientists, researchers, industry practitioners and graduate students across signal and image processing; neuroscience; data mining and data analysis; computer science; bioinformatics; speech processing; biomedical engineering; and multimedia.

Nonnegative Matrix and Tensor Factorizations

The maximum likelihood (ML) expectation maximization (EM) approach in emission tomography has been very popular in medical imaging for several years. In spite of this, no satisfactory convergent modifications have been proposed for the regularized approach. Here, a modification of the EM algorithm is presented. The new method is a natural extension of the EM for maximizing likelihood with concave priors. Convergence proofs are given. >

A modified expectation maximization algorithm for penalized likelihood estimation in emission tomography

The image space reconstruction algorithm (ISRA) was proposed as a modification of the expectation maximization (EM) algorithm based on physical considerations for application in volume emission computered tomography. As a consequence of this modification, ISRA searches for least squares solutions instead of maximizing Poisson likelihoods as the EM algorithm. It is shown that both algorithms may be obtained from a common mathematical framework. This fact is used to extend ISRA for penalized likelihood estimates. >

On the relation between the ISRA and the EM algorithm for positron emission tomography

The maximum-likelihood (ML) approach in emission tomography provides images with superior noise characteristics compared to conventional filtered backprojection (FBP) algorithms The expectation-maximization (EM) algorithm is an iterative algorithm for maximizing the Poisson likelihood in emission computed tomography that became very popular for solving the ML problem because of its attractive theoretical and practical properties Recently, (Browne and DePierro, 1996 and Hudson and Larkin, 1991) block sequential versions of the EM algorithm that take advantage of the scanner's geometry have been proposed in order to accelerate its convergence In Hudson and Larkin, 1991, the ordered subsets EM (OS-EM) method was applied to the hit problem and a modification (OS-GP) to the maximum a posteriori (MAP) regularized approach without showing convergence In Browne and DePierro, 1996, we presented a relaxed version of OS-EM (RAMLA) that converges to an ML solution In this paper, we present an extension of RAMLA for MAP reconstruction We show that, if the sequence generated by this method converges, then it must converge to the true MAP solution Experimental evidence of this convergence is also shown To illustrate this behavior we apply the algorithm to positron emission tomography simulated data comparing its performance to OS-GP

Fast EM-like methods for maximum "a posteriori" estimates in emission tomography

We describe a new approach for the inversion of the generalized attenuated radon transform in X-ray fluorescence computed tomography (XFCT). The approach consists of using the radon inverse as an approximation for the actual one, followed by an iterative refinement. Also, we analyze the problem of retrieving the attenuation map directly from the emission data, giving rise to a novel alternating method for the solution. We applied our approach to real and simulated XFCT data and compared its performance to previous inversion algorithms for the problem, showing its main advantages: better images than those obtained by other analytic methods and much faster than iterative methods in the discrete setting.

Iterative Reconstruction in X-ray Fluorescence Tomography Based on Radon Inversion

Emission computed tomography reconstruction requires compensation for photon attenuation. The usual way to do this is by performing a transmission scan to reconstruct the attenuation map. An important improvement could be achieved if it is possible to retrieve the attenuation map directly from the activity data. Several approaches have been suggested before to do this by using iterative methods for solving Maximum Likelihood (ML) problems (or Penalized Maximum Likelihood, MAP) that take into account the Poisson nature of the noise. One of the main drawbacks has been that these methods tend to retrieve solutions that generate an undesired `crosstalk' between the attenuation and the activity maps. In this paper, we present a new approach that consists of a combination of a minorizing function algorithm applied to the likelihood function plus the application of an appropriate decreasing multiplicative factor and iterative data refinement. We compare this new approach with previous ones and our simulations show very encouraging results as far as solving the `crosstalk' problem is concerned

A.R. De Pierro

Papers

A modified expectation maximization algorithm for penalized likelihood estimation in emission tomography

On the relation between the ISRA and the EM algorithm for positron emission tomography

Fast EM-like methods for maximum "a posteriori" estimates in emission tomography

Iterative Reconstruction in X-ray Fluorescence Tomography Based on Radon Inversion

Activity and Attenuation Reconstruction for Positron Emission Tomography Using Emission Data Only Via Maximum Likelihood and Iterative Data Refinement