Showing papers on "Singular value decomposition published in 2011"

Finding Structure with Randomness: Probabilistic Algorithms for Constructing Approximate Matrix Decompositions

Abstract: Low-rank matrix approximations, such as the truncated singular value decomposition and the rank-revealing QR decomposition, play a central role in data analysis and scientific computing. This work surveys and extends recent research which demonstrates that randomization offers a powerful tool for performing low-rank matrix approximation. These techniques exploit modern computational architectures more fully than classical methods and open the possibility of dealing with truly massive data sets. This paper presents a modular framework for constructing randomized algorithms that compute partial matrix decompositions. These methods use random sampling to identify a subspace that captures most of the action of a matrix. The input matrix is then compressed—either explicitly or implicitly—to this subspace, and the reduced matrix is manipulated deterministically to obtain the desired low-rank factorization. In many cases, this approach beats its classical competitors in terms of accuracy, robustness, and/or speed. These claims are supported by extensive numerical experiments and a detailed error analysis. The specific benefits of randomized techniques depend on the computational environment. Consider the model problem of finding the $k$ dominant components of the singular value decomposition of an $m \times n$ matrix. (i) For a dense input matrix, randomized algorithms require $\bigO(mn \log(k))$ floating-point operations (flops) in contrast to $ \bigO(mnk)$ for classical algorithms. (ii) For a sparse input matrix, the flop count matches classical Krylov subspace methods, but the randomized approach is more robust and can easily be reorganized to exploit multiprocessor architectures. (iii) For a matrix that is too large to fit in fast memory, the randomized techniques require only a constant number of passes over the data, as opposed to $\bigO(k)$ passes for classical algorithms. In fact, it is sometimes possible to perform matrix approximation with a single pass over the data.

Tensor-Train Decomposition

Abstract: A simple nonrecursive form of the tensor decomposition in $d$ dimensions is presented. It does not inherently suffer from the curse of dimensionality, it has asymptotically the same number of parameters as the canonical decomposition, but it is stable and its computation is based on low-rank approximation of auxiliary unfolding matrices. The new form gives a clear and convenient way to implement all basic operations efficiently. A fast rounding procedure is presented, as well as basic linear algebra operations. Examples showing the benefits of the decomposition are given, and the efficiency is demonstrated by the computation of the smallest eigenvalue of a 19-dimensional operator.

Fixed point and Bregman iterative methods for matrix rank minimization

Abstract: The linearly constrained matrix rank minimization problem is widely applicable in many fields such as control, signal processing and system identification. The tightest convex relaxation of this problem is the linearly constrained nuclear norm minimization. Although the latter can be cast as a semidefinite programming problem, such an approach is computationally expensive to solve when the matrices are large. In this paper, we propose fixed point and Bregman iterative algorithms for solving the nuclear norm minimization problem and prove convergence of the first of these algorithms. By using a homotopy approach together with an approximate singular value decomposition procedure, we get a very fast, robust and powerful algorithm, which we call FPCA (Fixed Point Continuation with Approximate SVD), that can solve very large matrix rank minimization problems (the code can be downloaded from http://www.columbia.edu/~sm2756/FPCA.htmfor non-commercial use). Our numerical results on randomly generated and real matrix completion problems demonstrate that this algorithm is much faster and provides much better recoverability than semidefinite programming solvers such as SDPT3. For example, our algorithm can recover 1000 × 1000 matrices of rank 50 with a relative error of 10−5 in about 3 min by sampling only 20% of the elements. We know of no other method that achieves as good recoverability. Numerical experiments on online recommendation, DNA microarray data set and image inpainting problems demonstrate the effectiveness of our algorithms.

Linearized Alternating Direction Method with Adaptive Penalty for Low-Rank Representation

Abstract: Many machine learning and signal processing problems can be formulated as linearly constrained convex programs, which could be efficiently solved by the alternating direction method (ADM). However, usually the subproblems in ADM are easily solvable only when the linear mappings in the constraints are identities. To address this issue, we propose a linearized ADM (LADM) method by linearizing the quadratic penalty term and adding a proximal term when solving the sub-problems. For fast convergence, we also allow the penalty to change adaptively according a novel update rule. We prove the global convergence of LADM with adaptive penalty (LADMAP). As an example, we apply LADMAP to solve low-rank representation (LRR), which is an important subspace clustering technique yet suffers from high computation cost. By combining LADMAP with a skinny SVD representation technique, we are able to reduce the complexity O(n3) of the original ADM based method to O(rn2), where r and n are the rank and size of the representation matrix, respectively, hence making LRR possible for large scale applications. Numerical experiments verify that for LRR our LADMAP based methods are much faster than state-of-the-art algorithms.

A Simpler Approach to Matrix Completion

Abstract: This paper provides the best bounds to date on the number of randomly sampled entries required to reconstruct an unknown low-rank matrix. These results improve on prior work by Candes and Recht (2009), Candes and Tao (2009), and Keshavan et al. (2009). The reconstruction is accomplished by minimizing the nuclear norm, or sum of the singular values, of the hidden matrix subject to agreement with the provided entries. If the underlying matrix satisfies a certain incoherence condition, then the number of entries required is equal to a quadratic logarithmic factor times the number of parameters in the singular value decomposition. The proof of this assertion is short, self contained, and uses very elementary analysis. The novel techniques herein are based on recent work in quantum information theory.

Tight Oracle Inequalities for Low-Rank Matrix Recovery From a Minimal Number of Noisy Random Measurements

Abstract: This paper presents several novel theoretical results regarding the recovery of a low-rank matrix from just a few measurements consisting of linear combinations of the matrix entries. We show that properly constrained nuclear-norm minimization stably recovers a low-rank matrix from a constant number of noisy measurements per degree of freedom; this seems to be the first result of this nature. Further, with high probability, the recovery error from noisy data is within a constant of three targets: (1) the minimax risk, (2) an “oracle” error that would be available if the column space of the matrix were known, and (3) a more adaptive “oracle” error which would be available with the knowledge of the column space corresponding to the part of the matrix that stands above the noise. Lastly, the error bounds regarding low-rank matrices are extended to provide an error bound when the matrix has full rank with decaying singular values. The analysis in this paper is based on the restricted isometry property (RIP).

Simultaneous seismic data denoising and reconstruction via multichannel singular spectrum analysis

Abstract: We present a rank reduction algorithm that permits simultaneous reconstruction and random noise attenuation of seismic records. We based our technique on multichannel singular spectrum analysis (MSSA). The technique entails organizing spatial data at a given temporal frequency into a block Hankel matrix that in ideal conditions is a matrix of rank k , where k is the number of plane waves in the window of analysis. Additive noise and missing samples will increase the rank of the block Hankel matrix of the data. Consequently, rank reduction is proposed as a means to attenuate noise and recover missing traces. We present an iterative algorithm that resembles seismic data reconstruction with the method of projection onto convex sets. In addition, we propose to adopt a randomized singular value decomposition to accelerate the rank reduction stage of the algorithm. We apply MSSA reconstruction to synthetic examples and a field data set. Synthetic examples were used to assess the performance of the method in two...

Finding Structure with Randomness: Probabilistic Algorithms for Constructing Approximate

Abstract: Low-rank matrix approximations, such as the truncated singular value decomposition and the rank-revealing QR decomposition, play a central role in data analysis and scientific computing. This work surveys and extends recent research which demonstrates that ran- domization offers a powerful tool for performing low-rank matrix approximation. These techniques exploit modern computational architectures more fully than classical methods and open the possibility of dealing with truly massive data sets. This paper presents a modular framework for constructing randomized algorithms that compute partial matrix decompositions. These methods use random sampling to identify a subspace that captures most of the action of a matrix. The input matrix is then compressed—either explicitly or implicitly—to this subspace, and the reduced matrix is manipulated deterministically to obtain the desired low-rank factorization. In many cases, this approach beats its classical competitors in terms of accuracy, robustness, and/or speed. These claims are supported by extensive numerical experiments and a detailed error analysis. The specific benefits of randomized techniques depend on the computational environment. Consider the model problem of finding the k dominant components of the singular value decomposition of an m × n matrix. (i) For a dense input matrix, randomized algorithms require O(mn log(k)) floating-point operations (flops) in contrast to O(mnk) for classical algorithms. (ii) For a sparse input matrix, the flop count matches classical Krylov subspace methods, but the randomized approach is more robust and can easily be reorganized to exploit multi- processor architectures. (iii) For a matrix that is too large to fit in fast memory, the randomized techniques require only a constant number of passes over the data, as opposed to O(k) passes for classical algorithms. In fact, it is sometimes possible to perform matrix approximation with a single pass over the data.

Temporal Link Prediction Using Matrix and Tensor Factorizations

Abstract: The data in many disciplines such as social networks, Web analysis, etc. is link-based, and the link structure can be exploited for many different data mining tasks. In this article, we consider the problem of temporal link prediction: Given link data for times 1 through T, can we predict the links at time T + 1? If our data has underlying periodic structure, can we predict out even further in time, i.e., links at time T + 2, T + 3, etc.? In this article, we consider bipartite graphs that evolve over time and consider matrix- and tensor-based methods for predicting future links. We present a weight-based method for collapsing multiyear data into a single matrix. We show how the well-known Katz method for link prediction can be extended to bipartite graphs and, moreover, approximated in a scalable way using a truncated singular value decomposition. Using a CANDECOMP/PARAFAC tensor decomposition of the data, we illustrate the usefulness of exploiting the natural three-dimensional structure of temporal link data. Through several numerical experiments, we demonstrate that both matrix- and tensor-based techniques are effective for temporal link prediction despite the inherent difficulty of the problem. Additionally, we show that tensor-based techniques are particularly effective for temporal data with varying periodic patterns.

A closed form solution to robust subspace estimation and clustering

Abstract: We consider the problem of fitting one or more subspaces to a collection of data points drawn from the subspaces and corrupted by noise/outliers. We pose this problem as a rank minimization problem, where the goal is to decompose the corrupted data matrix as the sum of a clean, self-expressive, low-rank dictionary plus a matrix of noise/outliers. Our key contribution is to show that, for noisy data, this non-convex problem can be solved very efficiently and in closed form from the SVD of the noisy data matrix. Remarkably, this is true for both one or more subspaces. An important difference with respect to existing methods is that our framework results in a polynomial thresholding of the singular values with minimal shrinkage. Indeed, a particular case of our framework in the case of a single subspace leads to classical PCA, which requires no shrinkage. In the case of multiple subspaces, our framework provides an affinity matrix that can be used to cluster the data according to the sub-spaces. In the case of data corrupted by outliers, a closed-form solution appears elusive. We thus use an augmented Lagrangian optimization framework, which requires a combination of our proposed polynomial thresholding operator with the more traditional shrinkage-thresholding operator.

Jacobi's Method Is More Accurate Than Qr

Abstract: It is shown that Jacobi’s method (with a proper stopping criterion) computes small eigenvalues of symmetric positive definite matrices with a uniformly better relative accuracy bound than QR, divide and conquer, traditional bisection, or any algorithm which first involves tridiagonalizing the matrix. Modulo an assumption based on extensive numerical tests, Jacobi’s method is optimally accurate in the following sense: if the matrix is such that small relative errors in its entries cause small relative errors in its eigenvalues, Jacobi will compute them with nearly this accuracy. In other words, as long as the initial matrix has small relative errors in each component, even using infinite precision will not improve on Jacobi (modulo factors of dimensionality). It is also shown that the eigenvectors are computed more accurately by Jacobi than previously thought possible. Similar results are proved for using one-sided Jacobi for the singular value decomposition of a general matrix.

Image Fusion Technique using Multi-resolution Singular Value Decomposition

Abstract: A novel image fusion technique based on multi-resolution singular value decomposition (MSVD) has been presented and evaluated. The performance of this algorithm is compared with that of well known image fusion technique using wavelets. It is observed that image fusion by MSVD perform almost similar to that of wavelets. It is computationally very simple and it could be well suited for real time applications. Moreover, MSVD does not have a fixed set of basis vectors like FFT, DCT and wavelet etc. and its basis vectors depend on the data set.Defence Science Journal, 2011, 61(5), pp.479-484, DOI:http://dx.doi.org/10.14429/dsj.61.705

Linearized Alternating Direction Method with Adaptive Penalty for Low-Rank Representation

Abstract: Low-rank representation (LRR) is an effective method for subspace clustering and has found wide applications in computer vision and machine learning. The existing LRR solver is based on the alternating direction method (ADM). It suffers from $O(n^3)$ computation complexity due to the matrix-matrix multiplications and matrix inversions, even if partial SVD is used. Moreover, introducing auxiliary variables also slows down the convergence. Such a heavy computation load prevents LRR from large scale applications. In this paper, we generalize ADM by linearizing the quadratic penalty term and allowing the penalty to change adaptively. We also propose a novel rule to update the penalty such that the convergence is fast. With our linearized ADM with adaptive penalty (LADMAP) method, it is unnecessary to introduce auxiliary variables and invert matrices. The matrix-matrix multiplications are further alleviated by using the skinny SVD representation technique. As a result, we arrive at an algorithm for LRR with complexity $O(rn^2)$, where $r$ is the rank of the representation matrix. Numerical experiments verify that for LRR our LADMAP method is much faster than state-of-the-art algorithms. Although we only present the results on LRR, LADMAP actually can be applied to solving more general convex programs.

Convergence of Fixed-Point Continuation Algorithms for Matrix Rank Minimization

Abstract: The matrix rank minimization problem has applications in many fields, such as system identification, optimal control, low-dimensional embedding, etc. As this problem is NP-hard in general, its convex relaxation, the nuclear norm minimization problem, is often solved instead. Recently, Ma, Goldfarb and Chen proposed a fixed-point continuation algorithm for solving the nuclear norm minimization problem (Math. Program., doi: 10.1007/s10107-009-0306-5, 2009). By incorporating an approximate singular value decomposition technique in this algorithm, the solution to the matrix rank minimization problem is usually obtained. In this paper, we study the convergence/recoverability properties of the fixed-point continuation algorithm and its variants for matrix rank minimization. Heuristics for determining the rank of the matrix when its true rank is not known are also proposed. Some of these algorithms are closely related to greedy algorithms in compressed sensing. Numerical results for these algorithms for solving affinely constrained matrix rank minimization problems are reported.

A robust audio watermarking scheme based on lifting wavelet transform and singular value decomposition

Abstract: In this paper, a new and robust audio watermarking scheme based on lifting wavelet transform (LWT) and singular value decomposition (SVD) is proposed. Specifically, the watermark data is inserted in the LWT coefficients of the low frequency subband taking advantage of SVD and quantization index modulation (QIM). The use of QIM renders our scheme blind in nature. Furthermore, the synchronization code technique is also integrated with hybrid LWT-SVD audio watermarking. Experimental results demonstrate that the proposed LWT-SVD method is not only robust to both general signal processing and desynchronization attacks but also outperform the selected previous studies.

A Generalized Technique of Modeling, Analysis, and Control of a Matrix Converter Using SVD

Abstract: In this paper, a new simple and complete technique of modeling and analysis of a matrix converter is presented based on the singular value decomposition (SVD) of modulation matrix. The proposed modeling method yields a new limitation between the matrix converter gain and its input power factor, which is more relaxed as compared to the limits reported so far in the literature. The SVD of the modulation matrix leads to a unified modulation technique which achieves the full capability of a matrix converter. It is shown that this approach is general and all other modulation methods established for a matrix converter are specific cases of this technique. The proposed modulation method can be used to obtain the maximum reactive power in the input of a matrix converter in applications such as wind turbine and microturbine generators, where the input reactive power control is necessary.

Denoising by Singular Value Decomposition and Its Application to Electronic Nose Data Processing

Abstract: This paper analyzes the role of singular value decomposition (SVD) in denoising sensor array data of electronic nose systems. It is argued that the SVD decomposition of raw data matrix distributes additive noise over orthogonal singular directions representing both the sensor and the odor variables. The noise removal is done by truncating the SVD matrices up to a few largest singular value components, and then reconstructing a denoised data matrix by using the remaining singular vectors. In electronic nose systems this method seems to be very effective in reducing noise components arising from both the odor sampling and delivery system and the sensors electronics. The feature extraction by principal component analysis based on the SVD denoised data matrix is seen to reduce separation between samples of the same class and increase separation between samples of different classes. This is beneficial for improving classification efficiency of electronic noses by reducing overlap between classes in feature space. The efficacy of SVD denoising method in electronic nose data analysis is demonstrated by analyzing five data sets available in public domain which are based on surface acoustic wave (SAW) sensors, conducting composite polymer sensors and the tin-oxide sensors arrays.

Blind Separation of Exponential Polynomials and the Decomposition of a Tensor in Rank-$(L_r,L_r,1)$ Terms

Abstract: We present a new necessary and sufficient condition for essential uniqueness of the decomposition of a third-order tensor in rank-$(L_r,L_r,1)$ terms We derive a new deterministic technique for blind signal separation that relies on this decomposition The method assumes that the signals can be modeled as linear combinations of exponentials or, more generally, as exponential polynomials The results are illustrated by means of numerical experiments

Optimal Orthogonal Precoding for Power Leakage Suppression in DFT-Based Systems

Abstract: A solution to the power leakage minimization problem in discrete Fourier transform (DFT) based communication systems is presented. In a conventional DFT based system, modulated subcarriers exhibit high sidelobe levels, which leads to significant out-of-band power leakage. Existing techniques found in the literature either do not achieve sufficient sidelobe suppression or suffer from significant spectral efficiency loss. Precoding can be seen as a general linear processing method for power leakage reduction, however, how to design the optimal linear precoder is still an open problem. In this paper, the power leakage suppression is first treated as a matrix Frobenius norm minimization problem, and then the optimal orthogonal precoding matrix design for the power leakage suppression is proposed based on singular value decomposition (SVD). By further exploiting the extra degrees of freedom in the precoding matrix, two kinds of optimized precoding matrices, one with multi-carrier property and the other with single-carrier property, are developed to take the advantages of orthogonal frequency division multiplexing (OFDM) and single carrier frequency division multiple access (SC-FDMA), respectively. Simulation results show that both the multi-carrier and the single-carrier precoding schemes achieve significant power leakage suppression, and have similar peak-to-average power ratio (PAPR) and bit-error-rate (BER) to those of OFDM and SC-FDMA systems, respectively.

Near Optimal Column-Based Matrix Reconstruction

Abstract: We consider low-rank reconstruction of a matrix using a subset of its columns and we present asymptotically optimal algorithms for both spectral norm and Frobenius norm reconstruction. The main tools we introduce to obtain our results are: (i) the use of fast approximate SVD-like decompositions for column-based matrix reconstruction, and (ii) two deterministic algorithms for selecting rows from matrices with orthonormal columns, building upon the sparse representation theorem for decompositions of the identity that appeared in [1].

Numerically stable and accurate stochastic simulation approaches for solving dynamic economic models

Abstract: We develop numerically stable and accurate stochastic simulation approaches for solving dynamic economic models. First, instead of standard least-squares approximation methods, we examine a variety of alternatives, including least- squares methods using singular value decomposition and Tikhonov regulariza- tion, least-absolute deviations methods, and principal component regression method, all of which are numerically stable and can handle ill-conditioned prob- lems. Second, instead of conventional Monte Carlo integration, we use accurate quadrature and monomial integration. We test our generalized stochastic simu- lation algorithm (GSSA) in three applications: the standard representative–agent neoclassical growth model, a model with rare disasters, and a multicountry model with hundreds of state variables. GSSA is simple to program, and MATLAB codes are provided. Keywords. Stochastic simulation, generalized stochastic simulation algorithm, parameterized expectations algorithm, least absolute deviations, linear program- ming, regularization. JEL classification. C63, C68.

Communication-Avoiding QR Decomposition for GPUs

Abstract: We describe an implementation of the Communication-Avoiding QR (CAQR) factorization that runs entirely on a single graphics processor (GPU). We show that the reduction in memory traffic provided by CAQR allows us to outperform existing parallel GPU implementations of QR for a large class of tall-skinny matrices. Other GPU implementations of QR handle panel factorizations by either sending the work to a general-purpose processor or using entirely bandwidth-bound operations, incurring data transfer overheads. In contrast, our QR is done entirely on the GPU using compute-bound kernels, meaning performance is good regardless of the width of the matrix. As a result, we outperform CULA, a parallel linear algebra library for GPUs by up to 17x for tall-skinny matrices and Intel's Math Kernel Library (MKL) by up to 12x. We also discuss stationary video background subtraction as a motivating application. We apply a recent statistical approach, which requires many iterations of computing the singular value decomposition of a tall-skinny matrix. Using CAQR as a first step to getting the singular value decomposition, we are able to get the answer 3x faster than if we use a traditional bandwidth-bound GPU QR factorization tuned specifically for that matrix size, and 30x faster than if we use Intel's Math Kernel Library (MKL) singular value decomposition routine on a multicore CPU.

Optimal Image Watermarking Algorithm Based on LWT-SVD via Multi-objective Ant Colony Optimization

Abstract: In this paper, a new optimal watermarking scheme based on lifting wavelet transform (LWT) and singular value decomposition (SVD) using multi-objective ant colony optimization (MOACO) is presented. The singular values of the binary water- mark are embedded in a detail subband of host image. To achieve the highest possible robustness without losing watermark transparency, multiple scaling factors (MSF) are used instead of a single scaling factor (SSF). Determining the optimal values of the mul- tiple scaling factors (MSF) is a dicult problem. However, to determine these values, a multi-objective ant colony-based optimization method is used. Experimental results show much improved performances in terms of transparency and robustness for the proposed method compared to other watermarking schemes. Furthermore, the proposed scheme does not suer from the problem of high probability of false positive detections of the watermarks.