Showing papers on "Singular value decomposition published in 2014"

Neural Word Embedding as Implicit Matrix Factorization

Abstract: We analyze skip-gram with negative-sampling (SGNS), a word embedding method introduced by Mikolov et al., and show that it is implicitly factorizing a word-context matrix, whose cells are the pointwise mutual information (PMI) of the respective word and context pairs, shifted by a global constant. We find that another embedding method, NCE, is implicitly factorizing a similar matrix, where each cell is the (shifted) log conditional probability of a word given its context. We show that using a sparse Shifted Positive PMI word-context matrix to represent words improves results on two word similarity tasks and one of two analogy tasks. When dense low-dimensional vectors are preferred, exact factorization with SVD can achieve solutions that are at least as good as SGNS's solutions for word similarity tasks. On analogy questions SGNS remains superior to SVD. We conjecture that this stems from the weighted nature of SGNS's factorization.

Tensor decompositions for learning latent variable models

Abstract: This work considers a computationally and statistically efficient parameter estimation method for a wide class of latent variable models--including Gaussian mixture models, hidden Markov models, and latent Dirichlet allocation--which exploits a certain tensor structure in their low-order observable moments (typically, of second- and third-order). Specifically, parameter estimation is reduced to the problem of extracting a certain (orthogonal) decomposition of a symmetric tensor derived from the moments; this decomposition can be viewed as a natural generalization of the singular value decomposition for matrices. Although tensor decompositions are generally intractable to compute, the decomposition of these specially structured tensors can be efficiently obtained by a variety of approaches, including power iterations and maximization approaches (similar to the case of matrices). A detailed analysis of a robust tensor power method is provided, establishing an analogue of Wedin's perturbation theorem for the singular vectors of matrices. This implies a robust and computationally tractable estimation approach for several popular latent variable models.

Novel Methods for Multilinear Data Completion and De-noising Based on Tensor-SVD

Abstract: In this paper we propose novel methods for completion (from limited samples) and de-noising of multilinear (tensor) data and as an application consider 3-D and 4- D (color) video data completion and de-noising. We exploit the recently proposed tensor-Singular Value Decomposition (t-SVD)[11]. Based on t-SVD, the notion of multilinear rank and a related tensor nuclear norm was proposed in [11] to characterize informational and structural complexity of multilinear data. We first show that videos with linear camera motion can be represented more efficiently using t-SVD compared to the approaches based on vectorizing or flattening of the tensors. Since efficiency in representation implies efficiency in recovery, we outline a tensor nuclear norm penalized algorithm for video completion from missing entries. Application of the proposed algorithm for video recovery from missing entries is shown to yield a superior performance over existing methods. We also consider the problem of tensor robust Principal Component Analysis (PCA) for de-noising 3-D video data from sparse random corruptions. We show superior performance of our method compared to the matrix robust PCA adapted to this setting as proposed in [4].

The Optimal Hard Threshold for Singular Values is $4/\sqrt {3}$

Abstract: We consider recovery of low-rank matrices from noisy data by hard thresholding of singular values, in which empirical singular values below a threshold λ are set to 0. We study the asymptotic mean squared error (AMSE) in a framework, where the matrix size is large compared with the rank of the matrix to be recovered, and the signal-to-noise ratio of the low-rank piece stays constant. The AMSE-optimal choice of hard threshold, in the case of n-by-n matrix in white noise of level σ, is simply (4/√3)√nσ ≈ 2.309√nσ when σ is known, or simply 2.858 · y med when σ is unknown, where y med is the median empirical singular value. For nonsquare, m by n matrices with m ≠ n the thresholding coefficients 4/√3 and 2.858 are replaced with different provided constants that depend on m/n. Asymptotically, this thresholding rule adapts to unknown rank and unknown noise level in an optimal manner: it is always better than hard thresholding at any other value, and is always better than ideal truncated singular value decomposition (TSVD), which truncates at the true rank of the low-rank matrix we are trying to recover. Hard thresholding at the recommended value to recover an n-by-n matrix of rank r guarantees an AMSE at most 3 nrσ 2 . In comparison, the guarantees provided by TSVD, optimally tuned singular value soft thresholding and the best guarantee achievable by any shrinkage of the data singular values are 5 nrσ 2 , 6 nrσ 2 , and 2 nrσ 2 , respectively. The recommended value for hard threshold also offers, among hard thresholds, the best possible AMSE guarantees for recovering matrices with bounded nuclear norm. Empirical evidence suggests that performance improvement over TSVD and other popular shrinkage rules can be substantial, for different noise distributions, even in relatively small n.

Novel methods for multilinear data completion and de-noising based on tensor-SVD

Abstract: In this paper we propose novel methods for completion (from limited samples) and de-noising of multilinear (tensor) data and as an application consider 3-D and 4- D (color) video data completion and de-noising. We exploit the recently proposed tensor-Singular Value Decomposition (t-SVD)[11]. Based on t-SVD, the notion of multilinear rank and a related tensor nuclear norm was proposed in [11] to characterize informational and structural complexity of multilinear data. We first show that videos with linear camera motion can be represented more efficiently using t-SVD compared to the approaches based on vectorizing or flattening of the tensors. Since efficiency in representation implies efficiency in recovery, we outline a tensor nuclear norm penalized algorithm for video completion from missing entries. Application of the proposed algorithm for video recovery from missing entries is shown to yield a superior performance over existing methods. We also consider the problem of tensor robust Principal Component Analysis (PCA) for de-noising 3-D video data from sparse random corruptions. We show superior performance of our method compared to the matrix robust PCA adapted to this setting as proposed in [4].

Tensors : A brief introduction

Abstract: Tensor decompositions are at the core of many blind source separation (BSS) algorithms, either explicitly or implicitly. In particular, the canonical polyadic (CP) tensor decomposition plays a central role in the identification of underdetermined mixtures. Despite some similarities, CP and singular value decomposition (SVD) are quite different. More generally, tensors and matrices enjoy different properties, as pointed out in this brief introduction.

Optimized gray-scale image watermarking using DWT-SVD and Firefly Algorithm

Abstract: This paper presents an optimized watermarking scheme based on the discrete wavelet transform (DWT) and singular value decomposition (SVD). The singular values of a binary watermark are embedded in singular values of the LL3 sub-band coefficients of the host image by making use of multiple scaling factors (MSFs). The MSFs are optimized using a newly proposed Firefly Algorithm having an objective function which is a linear combination of imperceptibility and robustness. The PSNR values indicate that the visual quality of the signed and attacked images is good. The embedding algorithm is robust against common image processing operations. It is concluded that the embedding and extraction of the proposed algorithm is well optimized, robust and show an improvement over other similar reported methods.

SVD Compression for Magnetic Resonance Fingerprinting in the Time Domain

Abstract: Magnetic resonance (MR) fingerprinting is a technique for acquiring and processing MR data that simultaneously provides quantitative maps of different tissue parameters through a pattern recognition algorithm. A predefined dictionary models the possible signal evolutions simulated using the Bloch equations with different combinations of various MR parameters and pattern recognition is completed by computing the inner product between the observed signal and each of the predicted signals within the dictionary. Though this matching algorithm has been shown to accurately predict the MR parameters of interest, one desires a more efficient method to obtain the quantitative images. We propose to compress the dictionary using the singular value decomposition, which will provide a low-rank approximation. By compressing the size of the dictionary in the time domain, we are able to speed up the pattern recognition algorithm, by a factor of between 3.4-4.8, without sacrificing the high signal-to-noise ratio of the original scheme presented previously.

Matrix Completion and Low-Rank SVD via Fast Alternating Least Squares

Abstract: The matrix-completion problem has attracted a lot of attention, largely as a result of the celebrated Netflix competition. Two popular approaches for solving the problem are nuclear-norm-regularized matrix approximation (Candes and Tao, 2009, Mazumder, Hastie and Tibshirani, 2010), and maximum-margin matrix factorization (Srebro, Rennie and Jaakkola, 2005). These two procedures are in some cases solving equivalent problems, but with quite different algorithms. In this article we bring the two approaches together, leading to an efficient algorithm for large matrix factorization and completion that outperforms both of these. We develop a software package "softImpute" in R for implementing our approaches, and a distributed version for very large matrices using the "Spark" cluster programming environment.

Learning with tensors: a framework based on convex optimization and spectral regularization

Abstract: We present a framework based on convex optimization and spectral regularization to perform learning when feature observations are multidimensional arrays (tensors). We give a mathematical characterization of spectral penalties for tensors and analyze a unifying class of convex optimization problems for which we present a provably convergent and scalable template algorithm. We then specialize this class of problems to perform learning both in a transductive as well as in an inductive setting. In the transductive case one has an input data tensor with missing features and, possibly, a partially observed matrix of labels. The goal is to both infer the missing input features as well as predict the missing labels. For induction, the goal is to determine a model for each learning task to be used for out of sample prediction. Each training pair consists of a multidimensional array and a set of labels each of which corresponding to related but distinct tasks. In either case the proposed technique exploits precise low multilinear rank assumptions over unknown multidimensional arrays; regularization is based on composite spectral penalties and connects to the concept of Multilinear Singular Value Decomposition. As a by-product of using a tensor-based formalism, our approach allows one to tackle the multi-task case in a natural way. Empirical studies demonstrate the merits of the proposed methods.

Singular value decomposition based low-footprint speaker adaptation and personalization for deep neural network

Abstract: The large number of parameters in deep neural networks (DNN) for automatic speech recognition (ASR) makes speaker adaptation very challenging. It also limits the use of speaker personalization due to the huge storage cost in large-scale deployments. In this paper we address DNN adaptation and personalization issues by presenting two methods based on the singular value decomposition (SVD). The first method uses an SVD to replace the weight matrix of a speaker independent DNN by the product of two low rank matrices. Adaptation is then performed by updating a square matrix inserted between the two low-rank matrices. In the second method, we adapt the full weight matrix but only store the delta matrix - the difference between the original and adapted weight matrices. We decrease the footprint of the adapted model by storing a reduced rank version of the delta matrix via an SVD. The proposed methods were evaluated on short message dictation task. Experimental results show that we can obtain similar accuracy improvements as the previously proposed Kullback-Leibler divergence (KLD) regularized method with far fewer parameters, which only requires 0.89% of the original model storage.

A Tensor-Based Approach for Big Data Representation and Dimensionality Reduction

Abstract: Variety and veracity are two distinct characteristics of large-scale and heterogeneous data It has been a great challenge to efficiently represent and process big data with a unified scheme In this paper, a unified tensor model is proposed to represent the unstructured, semistructured, and structured data With tensor extension operator, various types of data are represented as subtensors and then are merged to a unified tensor In order to extract the core tensor which is small but contains valuable information, an incremental high order singular value decomposition (IHOSVD) method is presented By recursively applying the incremental matrix decomposition algorithm, IHOSVD is able to update the orthogonal bases and compute the new core tensor Analyzes in terms of time complexity, memory usage, and approximation accuracy of the proposed method are provided in this paper A case study illustrates that approximate data reconstructed from the core set containing 18% elements can guarantee 93% accuracy in general Theoretical analyzes and experimental results demonstrate that the proposed unified tensor model and IHOSVD method are efficient for big data representation and dimensionality reduction

A tensor approach to learning mixed membership community models

Abstract: Community detection is the task of detecting hidden communities from observed interactions. Guaranteed community detection has so far been mostly limited to models with non-overlapping communities such as the stochastic block model. In this paper, we remove this restriction, and provide guaranteed community detection for a family of probabilistic network models with overlapping communities, termed as the mixed membership Dirichlet model, first introduced by Airoldi et al. (2008). This model allows for nodes to have fractional memberships in multiple communities and assumes that the community memberships are drawn from a Dirichlet distribution. Moreover, it contains the stochastic block model as a special case. We propose a unified approach to learning these models via a tensor spectral decomposition method. Our estimator is based on low-order moment tensor of the observed network, consisting of 3-star counts. Our learning method is fast and is based on simple linear algebraic operations, e.g., singular value decomposition and tensor power iterations. We provide guaranteed recovery of community memberships and model parameters and present a careful finite sample analysis of our learning method. As an important special case, our results match the best known scaling requirements for the (homogeneous) stochastic block model.

A Generalized Least-Square Matrix Decomposition

Abstract: Variables in many big-data settings are structured, arising, for example, from measurements on a regular grid as in imaging and time series or from spatial-temporal measurements as in climate studies. Classical multivariate techniques ignore these structural relationships often resulting in poor performance. We propose a generalization of principal components analysis (PCA) that is appropriate for massive datasets with structured variables or known two-way dependencies. By finding the best low-rank approximation of the data with respect to a transposable quadratic norm, our decomposition, entitled the generalized least-square matrix decomposition (GMD), directly accounts for structural relationships. As many variables in high-dimensional settings are often irrelevant, we also regularize our matrix decomposition by adding two-way penalties to encourage sparsity or smoothness. We develop fast computational algorithms using our methods to perform generalized PCA (GPCA), sparse GPCA, and functional GPCA on ma...

Fast Single Image Super-Resolution via Self-Example Learning and Sparse Representation

Abstract: In this paper, we propose a novel algorithm for fast single image super-resolution based on self-example learning and sparse representation. We propose an efficient implementation based on the K-singular value decomposition (SVD) algorithm, where we replace the exact SVD computation with a much faster approximation, and we employ the straightforward orthogonal matching pursuit algorithm, which is more suitable for our proposed self-example-learning-based sparse reconstruction with far fewer signals. The patches used for dictionary learning are efficiently sampled from the low-resolution input image itself using our proposed sample mean square error strategy, without an external training set containing a large collection of high- resolution images. Moreover, the l 0 -optimization-based criterion, which is much faster than l 1 -optimization-based relaxation, is applied to both the dictionary learning and reconstruction phases. Compared with other super-resolution reconstruction methods, our low- dimensional dictionary is a more compact representation of patch pairs and it is capable of learning global and local information jointly, thereby reducing the computational cost substantially. Our algorithm can generate high-resolution images that have similar quality to other methods but with an increase in the computational efficiency greater than hundredfold.

OptShrink: An Algorithm for Improved Low-Rank Signal Matrix Denoising by Optimal, Data-Driven Singular Value Shrinkage

Abstract: The truncated singular value decomposition of the measurement matrix is the optimal solution to the representation problem of how to best approximate a noisy measurement matrix using a low-rank matrix. Here, we consider the (unobservable) denoising problem of how to best approximate a low-rank signal matrix buried in noise by optimal (re)weighting of the singular vectors of the measurement matrix. We exploit recent results from random matrix theory to exactly characterize the large matrix limit of the optimal weighting coefficients and show that they can be computed directly from data for a large class of noise models that includes the independent identically distributed Gaussian noise case. Our analysis brings into sharp focus the shrinkage-and-thresholding form of the optimal weights, the nonconvex nature of the associated shrinkage function (on the singular values), and explains why matrix regularization via singular value thresholding with convex penalty functions (such as the nuclear norm) will always be suboptimal. We validate our theoretical predictions with numerical simulations, develop an implementable algorithm (OptShrink) that realizes the predicted performance gains and show how our methods can be used to improve estimation in the setting where the measured matrix has missing entries.

Hybrid Technique for Robust and Imperceptible Image Watermarking in DWT–DCT–SVD Domain

Abstract: In this paper an algorithm for digital watermarking based on discrete wavelet transforms (DWT), discrete cosine transforms (DCT), and singular value decomposition (SVD) has been proposed. In the embedding process, the host image is decomposed into first level DWTs. Low frequency band (LL) is transformed by DCT and SVD. The watermark image is also transformed by DCT and SVD. The S vector of watermark information is embedded in the S component of the host image. Watermarked image is generated by inverse SVD on modified S vector and original U, V vectors followed by inverse DCT and inverse DWT. Watermark is extracted using an extraction algorithm. The proposed method has been extensively tested against numerous known attacks and has been found to be giving superior performance for robustness and imperceptibility compared to existing methods suggested by other authors.

Joint Matrices Decompositions and Blind Source Separation: A survey of methods, identification, and applications

Abstract: Matrix decompositions such as the eigenvalue decomposition (EVD) or the singular value decomposition (SVD) have a long history in ?signal processing. They have been used in spectral analysis, signal/noise subspace estimation, principal component analysis (PCA), dimensionality reduction, and whitening in independent component analysis (ICA). Very often, the matrix under consideration is the covariance matrix of some observation signals. However, many other kinds of matrices can be encountered in signal processing problems, such as time-lagged covariance matrices, quadratic spatial time-frequency matrices [21], and matrices of higher-order statistics.

Cuckoo search algorithm based satellite image contrast and brightness enhancement using DWT-SVD.

Abstract: This paper presents a new contrast enhancement approach which is based on Cuckoo Search (CS) algorithm and DWT-SVD for quality improvement of the low contrast satellite images. The input image is decomposed into the four frequency subbands through Discrete Wavelet Transform (DWT), and CS algorithm used to optimize each subband of DWT and then obtains the singular value matrix of the low-low thresholded subband image and finally, it reconstructs the enhanced image by applying IDWT. The singular value matrix employed intensity information of the particular image, and any modification in the singular values changes the intensity of the given image. The experimental results show superiority of the proposed method performance in terms of PSNR, MSE, Mean and Standard Deviation over conventional and state-of-the-art techniques.

A Computationally Efficient Subspace Algorithm for 2-D DOA Estimation with L-shaped Array

Abstract: In this letter, a computationally efficient subspace algorithm is developed for two-dimensional (2-D) direction-of-arrival (DOA) estimation with L-shaped array structured by two uniform linear arrays (ULAs). The proposed method requires neither constructing the correlation matrix of the received data nor performing the singular value decomposition (SVD) of the correlation matrix. The problem is solved by dealing with three vectors composed of the first column, the first row and diagonal entries of the correlation matrix, which reduces the computational burden. Simultaneously, the proposed method utilizes the conjugate symmetry to enlarge the effective array aperture, which improves the estimation precision. The simulation results are presented to validate the effectiveness of the proposed algorithm.

Analytical Performance Assessment of Multi-Dimensional Matrix- and Tensor-Based ESPRIT-Type Algorithms

Abstract: In this paper we present a generic framework for the asymptotic performance analysis of subspace-based parameter estimation schemes. It is based on earlier results on an explicit first-order expansion of the estimation error in the signal subspace obtained via an SVD of the noisy observation matrix. We extend these results in a number of aspects. Firstly, we demonstrate that an explicit first-order expansion of the Higher-Order SVD (HOSVD)-based subspace estimate can be derived. Secondly, we show how to obtain explicit first-order expansions of the estimation error of arbitrary ESPRIT-type algorithms and provide the expressions for R-D Standard ESPRIT, R-D Unitary ESPRIT, R-D Standard Tensor-ESPRIT, as well as R-D Unitary Tensor-ESPRIT. Thirdly, we derive closed-form expressions for the mean square error (MSE) and show that they only depend on the second-order moments of the noise. Hence, to apply this framework we only need the noise to be zero mean and possess finite second order moments. Additional assumptions such as Gaussianity or circular symmetry are not needed.

Radar micro-Doppler feature extraction using the Singular Value Decomposition

Abstract: The micro-Doppler spectrogram depends on parts of a target moving and rotating in addition to the main body motion (e.g., spinning rotor blades) and is thus characteristic for the type of target. In this study, the micro-Doppler spectrogram is exploited to distinguish between birds and small unmanned aerial vehicles (UAVs). The focus hereby is on micro-Doppler features enabling fast classification of birds and mini-UAVs. In a second classification step, it is desired to exploit micro-Doppler features to further characterize the type of UAV, e.g., fixed-wing vs. rotary-wing. In this paper, potentially robust features are discussed supporting the first classification step, i.e., separation of birds and UAVs. The Singular Value Decomposition seems a powerful tool to extract such features, since the information content of the micro-Doppler spectrogram is preserved in the singular vectors. In the paper, some examples of micro-Doppler feature extraction via Singular Value Decomposition are given.

Recognition of Complex Events: Exploiting Temporal Dynamics between Underlying Concepts

Abstract: While approaches based on bags of features excel at low-level action classification, they are ill-suited for recognizing complex events in video, where concept-based temporal representations currently dominate. This paper proposes a novel representation that captures the temporal dynamics of windowed mid-level concept detectors in order to improve complex event recognition. We first express each video as an ordered vector time series, where each time step consists of the vector formed from the concatenated confidences of the pre-trained concept detectors. We hypothesize that the dynamics of time series for different instances from the same event class, as captured by simple linear dynamical system (LDS) models, are likely to be similar even if the instances differ in terms of low-level visual features. We propose a two-part representation composed of fusing: (1) a singular value decomposition of block Hankel matrices (SSID-S) and (2) a harmonic signature (HS) computed from the corresponding eigen-dynamics matrix. The proposed method offers several benefits over alternate approaches: our approach is straightforward to implement, directly employs existing concept detectors and can be plugged into linear classification frameworks. Results on standard datasets such as NIST's TRECVID Multimedia Event Detection task demonstrate the improved accuracy of the proposed method.