This article is about a curious phenomenon. Suppose we have a data matrix, which is the superposition of a low-rank component and a sparse component. Can we recover each component individuallyq We prove that under some suitable assumptions, it is possible to recover both the low-rank and the sparse components exactly by solving a very convenient convex program called Principal Component Pursuit; among all feasible decompositions, simply minimize a weighted combination of the nuclear norm and of the e1 norm. This suggests the possibility of a principled approach to robust principal component analysis since our methodology and results assert that one can recover the principal components of a data matrix even though a positive fraction of its entries are arbitrarily corrupted. This extends to the situation where a fraction of the entries are missing as well. We discuss an algorithm for solving this optimization problem, and present applications in the area of video surveillance, where our methodology allows for the detection of objects in a cluttered background, and in the area of face recognition, where it offers a principled way of removing shadows and specularities in images of faces.

/pdf/robust-principal-component-analysis-59kf1xh72d.pdf

Robust principal component analysis

This paper introduces a novel algorithm to approximate the matrix with minimum nuclear norm among all matrices obeying a set of convex constraints. This problem may be understood as the convex relaxation of a rank minimization problem and arises in many important applications as in the task of recovering a large matrix from a small subset of its entries (the famous Netflix problem). Off-the-shelf algorithms such as interior point methods are not directly amenable to large problems of this kind with over a million unknown entries. This paper develops a simple first-order and easy-to-implement algorithm that is extremely efficient at addressing problems in which the optimal solution has low rank. The algorithm is iterative, produces a sequence of matrices $\{\boldsymbol{X}^k,\boldsymbol{Y}^k\}$, and at each step mainly performs a soft-thresholding operation on the singular values of the matrix $\boldsymbol{Y}^k$. There are two remarkable features making this attractive for low-rank matrix completion problems. The first is that the soft-thresholding operation is applied to a sparse matrix; the second is that the rank of the iterates $\{\boldsymbol{X}^k\}$ is empirically nondecreasing. Both these facts allow the algorithm to make use of very minimal storage space and keep the computational cost of each iteration low. On the theoretical side, we provide a convergence analysis showing that the sequence of iterates converges. On the practical side, we provide numerical examples in which $1,000\times1,000$ matrices are recovered in less than a minute on a modest desktop computer. We also demonstrate that our approach is amenable to very large scale problems by recovering matrices of rank about 10 with nearly a billion unknowns from just about 0.4% of their sampled entries. Our methods are connected with the recent literature on linearized Bregman iterations for $\ell_1$ minimization, and we develop a framework in which one can understand these algorithms in terms of well-known Lagrange multiplier algorithms.

/pdf/a-singular-value-thresholding-algorithm-for-matrix-3rmbvujnls.pdf

A Singular Value Thresholding Algorithm for Matrix Completion

It is now well understood that (1) it is possible to reconstruct sparse signals exactly from what appear to be highly incomplete sets of linear measurements and (2) that this can be done by constrained l1 minimization. In this paper, we study a novel method for sparse signal recovery that in many situations outperforms l1 minimization in the sense that substantially fewer measurements are needed for exact recovery. The algorithm consists of solving a sequence of weighted l1-minimization problems where the weights used for the next iteration are computed from the value of the current solution. We present a series of experiments demonstrating the remarkable performance and broad applicability of this algorithm in the areas of sparse signal recovery, statistical estimation, error correction and image processing. Interestingly, superior gains are also achieved when our method is applied to recover signals with assumed near-sparsity in overcomplete representations—not by reweighting the l1 norm of the coefficient sequence as is common, but by reweighting the l1 norm of the transformed object. An immediate consequence is the possibility of highly efficient data acquisition protocols by improving on a technique known as Compressive Sensing.

/pdf/enhancing-sparsity-by-reweighted-l-1-minimization-znwoz4koxr.pdf

Enhancing Sparsity by Reweighted ℓ 1 Minimization

The affine rank minimization problem consists of finding a matrix of minimum rank that satisfies a given system of linear equality constraints. Such problems have appeared in the literature of a diverse set of fields including system identification and control, Euclidean embedding, and collaborative filtering. Although specific instances can often be solved with specialized algorithms, the general affine rank minimization problem is NP-hard because it contains vector cardinality minimization as a special case.

In this paper, we show that if a certain restricted isometry property holds for the linear transformation defining the constraints, the minimum-rank solution can be recovered by solving a convex optimization problem, namely, the minimization of the nuclear norm over the given affine space. We present several random ensembles of equations where the restricted isometry property holds with overwhelming probability, provided the codimension of the subspace is sufficiently large.

The techniques used in our analysis have strong parallels in the compressed sensing framework. We discuss how affine rank minimization generalizes this preexisting concept and outline a dictionary relating concepts from cardinality minimization to those of rank minimization. We also discuss several algorithmic approaches to minimizing the nuclear norm and illustrate our results with numerical examples.

https://faculty.washington.edu/mfazel/low-rank-v2.pdf

Guaranteed Minimum-Rank Solutions of Linear Matrix Equations via Nuclear Norm Minimization

We describe a generalization of the trace heuristic that applies to general nonsymmetric, even non-square, matrices, and reduces to the trace heuristic when the matrix is positive semidefinite The heuristic is to replace the (nonconvex) rank objective with the sum of the singular values of the matrix, which is the dual of the spectral norm We show that this problem can be reduced to a semidefinite program, hence efficiently solved To motivate the heuristic, we, show that the dual spectral norm is the convex envelope of the rank on the set of matrices with norm less than one We demonstrate the method on the problem of minimum-order system approximation

/pdf/a-rank-minimization-heuristic-with-application-to-minimum-301i0r0izy.pdf

A rank minimization heuristic with application to minimum order system approximation

We present a heuristic for minimizing the rank of a positive semidefinite matrix over a convex set. We use the logarithm of the determinant as a smooth approximation for rank, and locally minimize this function to obtain a sequence of trace minimization problems. We then present a lemma that relates the rank of any general matrix to that of a corresponding positive semidefinite one. Using this, we readily extend the proposed heuristic to handle general matrices. We examine the vector case as a special case, where the heuristic reduces to an iterative l/sub 1/-norm minimization technique. As practical applications of the rank minimization problem and our heuristic, we consider two examples: minimum-order system realization with time-domain constraints, and finding lowest-dimension embedding of points in a Euclidean space from noisy distance data.

Log-det heuristic for matrix rank minimization with applications to Hankel and Euclidean distance matrices

We show how linear matrix inequalities (LMI) can be used to perform local stability and performance analysis of linear systems with saturating elements. This leads to less conservative information on stability regions, disturbance rejection, and L/sub 2/-gain than standard global stability and performance analysis. The circle and Popov criteria are used to obtain Lyapunov functions whose sublevel sets provide regions of guaranteed stability and performance within a restricted state space region. Our LMI formulation leads directly to simple convex optimization problems that can be solved efficiently as semidefinite programs. The results cover both single and multiple saturation elements and can be immediately extended to discrete time systems. An obvious application of these techniques is in the analysis of control systems with saturating control inputs.

Analysis of linear systems with saturation using convex optimization

In this tutorial paper, we consider the problem of minimizing the rank of a matrix over a convex set. The rank minimization problem (RMP) arises in diverse areas such as control, system identification, statistics and signal processing, and is known to be computationally NP-hard. We give an overview of the problem, its interpretations, applications, and solution methods. In particular, we focus on how convex optimization can be used to develop heuristic methods for this problem.

/pdf/rank-minimization-and-applications-in-system-theory-2sslyapjgv.pdf

Rank minimization and applications in system theory

We consider a linear control system in which several signals are transmitted over communication channels with bit rate limitations. With the coding and medium access schemes of the communication system fixed, the achievable bit rates are determined by the allocation of communications resources such as transmit powers and bandwidths, to different communication channels. We model the effect of bit rate limited communication channels by uniform quantization and the quantization errors are modeled by additive white noises whose variances depend on the achievable bit rates. We optimize the stationary performance of the linear system by jointly allocating resources in the communication system and tuning parameters of the controller.

Haitham Hindi

Papers

A rank minimization heuristic with application to minimum order system approximation

Log-det heuristic for matrix rank minimization with applications to Hankel and Euclidean distance matrices

Analysis of linear systems with saturation using convex optimization

Rank minimization and applications in system theory

Joint optimization of communication rates and linear systems