Showing papers on "Singular value decomposition published in 2012"

Tensor decompositions for learning latent variable models

Abstract: This work considers a computationally and statistically efficient parameter estimation method for a wide class of latent variable models---including Gaussian mixture models, hidden Markov models, and latent Dirichlet allocation---which exploits a certain tensor structure in their low-order observable moments (typically, of second- and third-order). Specifically, parameter estimation is reduced to the problem of extracting a certain (orthogonal) decomposition of a symmetric tensor derived from the moments; this decomposition can be viewed as a natural generalization of the singular value decomposition for matrices. Although tensor decompositions are generally intractable to compute, the decomposition of these specially structured tensors can be efficiently obtained by a variety of approaches, including power iterations and maximization approaches (similar to the case of matrices). A detailed analysis of a robust tensor power method is provided, establishing an analogue of Wedin's perturbation theorem for the singular vectors of matrices. This implies a robust and computationally tractable estimation approach for several popular latent variable models.

Robust PCA via Outlier Pursuit

Abstract: Singular-value decomposition (SVD) [and principal component analysis (PCA)] is one of the most widely used techniques for dimensionality reduction: successful and efficiently computable, it is nevertheless plagued by a well-known, well-documented sensitivity to outliers. Recent work has considered the setting where each point has a few arbitrarily corrupted components. Yet, in applications of SVD or PCA, such as robust collaborative filtering or bioinformatics, malicious agents, defective genes, or simply corrupted or contaminated experiments may effectively yield entire points that are completely corrupted. We present an efficient convex optimization-based algorithm that we call outlier pursuit, which under some mild assumptions on the uncorrupted points (satisfied, e.g., by the standard generative assumption in PCA problems) recovers the exact optimal low-dimensional subspace and identifies the corrupted points. Such identification of corrupted points that do not conform to the low-dimensional approximation is of paramount interest in bioinformatics, financial applications, and beyond. Our techniques involve matrix decomposition using nuclear norm minimization; however, our results, setup, and approach necessarily differ considerably from the existing line of work in matrix completion and matrix decomposition, since we develop an approach to recover the correct column space of the uncorrupted matrix, rather than the exact matrix itself. In any problem where one seeks to recover a structure rather than the exact initial matrices, techniques developed thus far relying on certificates of optimality will fail. We present an important extension of these methods, which allows the treatment of such problems.

Coarse-graining renormalization by higher-order singular value decomposition

Abstract: We propose a novel coarse-graining tensor renormalization group method based on the higher-order singular value decomposition. This method provides an accurate but low computational cost technique for studying both classical and quantum lattice models in two or three dimensions. We have demonstrated this method using the Ising model on the square and cubic lattices. By keeping up to 16 bond basis states, we obtain by far the most accurate numerical renormalization group results for the three-dimensional Ising model. We have also applied the method to study the ground state as well as finite temperature properties for the two-dimensional quantum transverse Ising model and obtain the results which are consistent with published data.

A New Truncation Strategy for the Higher-Order Singular Value Decomposition

Abstract: We present an alternative strategy for truncating the higher-order singular value decomposition (T-HOSVD). An error expression for an approximate Tucker decomposition with orthogonal factor matrices is presented, leading us to propose a novel truncation strategy for the HOSVD, which we refer to as the sequentially truncated higher-order singular value decomposition (ST-HOSVD). This decomposition retains several favorable properties of the T-HOSVD, while reducing the number of operations required to compute the decomposition and practically always improving the approximation error. Three applications are presented, demonstrating the effectiveness of ST-HOSVD. In the first application, ST-HOSVD, T-HOSVD, and higher-order orthogonal iteration (HOOI) are employed to compress a database of images of faces. On average, the ST-HOSVD approximation was only $0.1\%$ worse than the optimum computed by HOOI, while cutting the execution time by a factor of $20$. In the second application, classification of handwritten digits, ST-HOSVD achieved a speedup factor of $50$ over T-HOSVD during the training phase, and reduced the classification time and storage costs, while not significantly affecting the classification error. The third application demonstrates the effectiveness of ST-HOSVD in compressing results from a numerical simulation of a partial differential equation. In such problems, ST-HOSVD inevitably can greatly improve the running time. We present an example wherein the $2$ hour $45$ minute calculation of T-HOSVD was reduced to just over one minute by ST-HOSVD, representing a speedup factor of $133$, while even improving the memory consumption.

A Spectral Algorithm for Latent Dirichlet Allocation

Abstract: The problem of topic modeling can be seen as a generalization of the clustering problem, in that it posits that observations are generated due to multiple latent factors (e.g., the words in each document are generated as a mixture of several active topics, as opposed to just one). This increased representational power comes at the cost of a more challenging unsupervised learning problem of estimating the topic probability vectors (the distributions over words for each topic), when only the words are observed and the corresponding topics are hidden. We provide a simple and efficient learning procedure that is guaranteed to recover the parameters for a wide class of mixture models, including the popular latent Dirichlet allocation (LDA) model. For LDA, the procedure correctly recovers both the topic probability vectors and the prior over the topics, using only trigram statistics (i.e., third order moments, which may be estimated with documents containing just three words). The method, termed Excess Correlation Analysis (ECA), is based on a spectral decomposition of low order moments (third and fourth order) via two singular value decompositions (SVDs). Moreover, the algorithm is scalable since the SVD operations are carried out on $k\times k$ matrices, where $k$ is the number of latent factors (e.g. the number of topics), rather than in the $d$-dimensional observed space (typically $d \gg k$).

Image Fusion Using Higher Order Singular Value Decomposition

Abstract: A novel higher order singular value decomposition (HOSVD)-based image fusion algorithm is proposed. The key points are given as follows: 1) Since image fusion depends on local information of source images, the proposed algorithm picks out informative image patches of source images to constitute the fused image by processing the divided subtensors rather than the whole tensor; 2) the sum of absolute values of the coefficients (SAVC) from HOSVD of subtensors is employed for activity-level measurement to evaluate the quality of the related image patch; and 3) a novel sigmoid-function-like coefficient-combining scheme is applied to construct the fused result. Experimental results show that the proposed algorithm is an alternative image fusion approach.

Reduced rank stochastic regression with a sparse singular value decomposition

Abstract: Summary. For a reduced rank multivariate stochastic regression model of rank r*, the regression coefficient matrix can be expressed as a sum of r* unit rank matrices each of which is proportional to the outer product of the left and right singular vectors. For improving predictive accuracy and facilitating interpretation, it is often desirable that these left and right singular vectors be sparse or enjoy some smoothness property. We propose a regularized reduced rank regression approach for solving this problem. Computation algorithms and regularization parameter selection methods are developed, and the properties of the new method are explored both theoretically and by simulation. In particular, the regularization method proposed is shown to be selection consistent and asymptotically normal and to enjoy the oracle property. We apply the proposed model to perform biclustering analysis with microarray gene expression data.

Comparison of orthogonal matching pursuit implementations

Abstract: We study the numerical and computational performance of three implementations of orthogonal matching pursuit: one using the QR matrix decomposition, one using the Cholesky matrix decomposition, and one using the matrix inversion lemma. We find that none of these implementations suffer from numerical error accumulation in the inner products or the solution. Furthermore, we empirically compare the computational times of each algorithm over the phase plane.

Sparse Higher-Order Principal Components Analysis

Abstract: Traditional tensor decompositions such as the CANDECOMP / PARAFAC (CP) and Tucker decompositions yield higher-order principal components that have been used to understand tensor data in areas such as neuroimaging, microscopy, chemometrics, and remote sensing. Sparsity in high-dimensional matrix factorizations and principal components has been well-studied exhibiting many benefits; less attention has been given to sparsity in tensor decompositions. We propose two novel tensor decompositions that incorporate sparsity: the Sparse Higher-Order SVD and the Sparse CP Decomposition. The latter solves an `1-norm penalized relaxation of the single-factor CP optimization problem, thereby automatically selecting relevant features for each tensor factor. Through experiments and a scientific data analysis example, we demonstrate the utility of our methods for dimension reduction, feature selection, signal recovery, and exploratory data analysis of high-dimensional tensors.

Observer-based robust control of a (1⩽ a ≪ 2) fractional-order uncertain systems: a linear matrix inequality approach

Abstract: The study is concerned with a method of observer-based control for fractional-order uncertain linear systems with the fractional commensurate order a (1≤a<2) based on linear matrix inequality (LMI) approach. First, a sufficient condition for robust asymptotic stability of the observer-based fractional-order control systems is presented. Next, by using matrix's singular value decomposition and LMI techniques, the existence condition and method of designing a robust observer-based controller for such fractional-order control systems are derived. Unlike previous methods, the results are obtained in terms of LMIs, which can be easily obtained by Matlab's LMI toolbox. Finally, a numerical example demonstrates the validity of this approach.

Low-Complexity Lattice Reduction-Aided Regularized Block Diagonalization for MU-MIMO Systems

Abstract: By employing the regularized block diagonalization (RBD) preprocessing technique, the MU-MIMO broadcast channel is decomposed into multiple parallel independent SU-MIMO channels and achieves the maximum diversity order at high data rates. The computational complexity of RBD, however, is relatively high due to two singular value decomposition (SVD) operations. In this letter, a low-complexity lattice reduction-aided RBD is proposed. The first SVD is replaced by a QR decomposition, and the orthogonalization procedure provided by the second SVD is substituted by a lattice-reduction whose complexity is mainly contributed by a QR decomposition. Simulation results show that the proposed algorithm can achieve almost the same sum-rate as RBD, substantial bit error rate (BER) performance gains and a simplified receiver structure, while requiring a lower complexity.

Singular Value Decomposition

Abstract: Digital information transmission is a growing field Emails, videos and so on are transmitting around the world on a daily basis Along the growth of using digital devises there is in some cases a great interest of keeping this information secureIn the field of signal processing a general concept is antenna transmission Free space between an antenna transmitter and a receiver is an example of a system In a rough environment such as a room with reflections and independent electrical devices there will be a lot of distortion in the system and the signal that is transmitted might, due to the system characteristics and noise be distortedSystem identification is another well-known concept in signal processing This thesis will focus on system identification in a rough environment and unknown systems It will introduce mathematical tools from the field of linear algebra and applying them in signal processing Mainly this thesis focus on a specific matrix factorization called Singular Value Decomposition (SVD) This is used to solve complicated inverses and identifying systemsThis thesis is formed and accomplished in collaboration with Combitech AB Their expertise in the field of signal processing was of great help when putting the algorithm in practice Using a well-known programming script called LabView the mathematical tools were synchronized with the instruments that were used to generate the systems and signals

Beating randomized response on incoherent matrices

Abstract: Computing accurate low rank approximations of large matrices is a fundamental data mining task. In many applications however the matrix contains sensitive information about individuals. In such case we would like to release a low rank approximation that satisfies a strong privacy guarantee such as differential privacy. Unfortunately, to date the best known algorithm for this task that satisfies differential privacy is based on naive input perturbation or randomized response: Each entry of the matrix is perturbed independently by a sufficiently large random noise variable, a low rank approximation is then computed on the resulting matrix. We give (the first) significant improvements in accuracy over randomized response under the natural and necessary assumption that the matrix has low coherence. Our algorithm is also very efficient and finds a constant rank approximation of an m x n matrix in time O(mn). Note that even generating the noise matrix required for randomized response already requires time O(mn).

Copyright Protection for E-Government Document Images

Abstract: The proposed copyright protection scheme combines the discrete cosine transform (DCT) and singular value decomposition (SVD) using a control parameter to avoid the false-positive problem. In this article, we propose an efficient copyright protection scheme for e-government document images. First, we apply the discrete cosine transform (DCT) to the host image and use the zigzag space-filling curve (SFC) for the DCT coefficients. The DCT coefficients in the zigzag manner are then mapped into four areas with different frequencies in a rectangular shape. Then, we apply the singular value decomposition (SVD) operation to each area, and the host image is modified by the left singular vectors and the singular values of the DCT-transformed watermark to embed the watermark image. The left singular vectors and singular values are used as a control parameter to avoid the false-positive problem. Each area decides the scaling factor's optimal value using a genetic algorithm (GA) with the mean of the watermark's SVs. A scaling factor is simulated by chromosomes, and several optimization GA operators are used. After remapping each modified coefficient DCT back to the original position, the proposed inverse DCT produces the watermarked image. Our experimental results show that we can improve the image quality GA-based evolution and that this approach is robust under several kinds of attacks.

Two SVDs Suffice: Spectral decompositions for probabilistic topic modeling and latent Dirichlet allocation

Abstract: The problem of topic modeling can be seen as a generalization of the clustering problem, in that it posits that observations are generated due to multiple latent factors (e.g., the words in each document are generated as a mixture of several active topics, as opposed to just one). This increased representational power comes at the cost of a more challenging unsupervised learning problem of estimating the topic probability vectors (the distributions over words for each topic), when only the words are observed and the corresponding topics are hidden. We provide a simple and efficient learning procedure that is guaranteed to recover the parameters for a wide class of mixture models, including the popular latent Dirichlet allocation (LDA) model. For LDA, the procedure correctly recovers both the topic probability vectors and the prior over the topics, using only trigram statistics (i.e., third order moments, which may be estimated with documents containing just three words). The method, termed Excess Correlation Analysis (ECA), is based on a spectral decomposition of low order moments (third and fourth order) via two singular value decompositions (SVDs). Moreover, the algorithm is scalable since the SVD operations are carried out on $k\times k$ matrices, where $k$ is the number of latent factors (e.g. the number of topics), rather than in the $d$-dimensional observed space (typically $d \gg k$).

An SVD-based image watermarking in wavelet domain using SVR and PSO

Abstract: The paper presents a novel blind watermarking scheme for image copyright protection, which is developed in the discrete wavelet transform (DWT) and is based on the singular value decomposition (SVD) and the support vector regression (SVR). Its embedding algorithm hides a watermark bit in the low-low (LL) subband of a target non-overlap block of the host image by modifying a coefficient of U component on SVD version of the block. A blind watermark-extraction is designed using a trained SVR to estimate original coefficients. Subsequently, the watermark bit can be computed using the watermarked coefficient and its corresponding estimate coefficient. Additionally, the particle swarm optimization (PSO) is further utilized to optimize the proposed scheme. Experimental results show the proposed scheme possesses significant improvements in both transparency and robustness, and is superior to existing methods under consideration here.

Online learning in the embedded manifold of low-rank matrices

Abstract: When learning models that are represented in matrix forms, enforcing a low-rank constraint can dramatically improve the memory and run time complexity, while providing a natural regularization of the model. However, naive approaches to minimizing functions over the set of low-rank matrices are either prohibitively time consuming (repeated singular value decomposition of the matrix) or numerically unstable (optimizing a factored representation of the low-rank matrix). We build on recent advances in optimization over manifolds, and describe an iterative online learning procedure, consisting of a gradient step, followed by a second-order retraction back to the manifold. While the ideal retraction is costly to compute, and so is the projection operator that approximates it, we describe another retraction that can be computed efficiently. It has run time and memory complexity of O((n+m)k) for a rank-k matrix of dimension m×n, when using an online procedure with rank-one gradients. We use this algorithm, LORETA, to learn a matrix-form similarity measure over pairs of documents represented as high dimensional vectors. LORETA improves the mean average precision over a passive-aggressive approach in a factorized model, and also improves over a full model trained on pre-selected features using the same memory requirements. We further adapt LORETA to learn positive semi-definite low-rank matrices, providing an online algorithm for low-rank metric learning. LORETA also shows consistent improvement over standard weakly supervised methods in a large (1600 classes and 1 million images, using ImageNet) multilabel image classification task.

A new Tikhonov regularization method

Abstract: The numerical solution of linear discrete ill-posed problems typically requires regularization, i.e., replacement of the available ill-conditioned problem by a nearby better conditioned one. The most popular regularization methods for problems of small to moderate size, which allow evaluation of the singular value decomposition of the matrix defining the problem, are the truncated singular value decomposition and Tikhonov regularization. The present paper proposes a novel choice of regularization matrix for Tikhonov regularization that bridges the gap between Tikhonov regularization and truncated singular value decomposition. Computed examples illustrate the benefit of the proposed method.

A Spatial-Domain Joint-Nulling Method of Self-Interference in Full-Duplex Relays

Abstract: In this letter, a spatial-domain joint-nulling method is proposed as a new means of suppressing the self-interference (or echo) occurring in full-duplex relays. Both amplify-and-forward (AF) and decode-and-forward (DF) type relays are considered. While the conventional method searches for the relay processing matrix from a discrete set of basis vectors obtained from the singular value decomposition (SVD) of the echo channel, the proposed method directly solves the optimal relay processing matrix over a continuous domain. As a result, it is shown that the proposed approach gives better performance than the conventional one in terms of the achievable rate.

H∞ control for discrete-time singular Markov jump systems subject to actuator saturation

Abstract: In this paper, the H ∞ control problem for discrete-time singular Markov jump systems with actuator saturation is considered A sufficient condition is obtained which guarantees that the discrete-time singular Markov jump system with actuator saturation is regular, causal, bounded state stable, and satisfies the H ∞ performance With this condition, and based on singular value decomposition approach, the design method of H ∞ state feedback controller is developed by using linear matrix inequalities (LMIs) optimization problem Numerical examples are given to illustrate the effectiveness of the proposed methods

Four Decades of Studying Global Linear Instability: Progress and Challenges

Abstract: Global linear instability theory is concerned with the temporal or spatial development of small-amplitude perturbations superposed upon laminar steady or time-periodic three-dimensional flows, which are inhomogeneous in two (and periodic in one) or all three spatial directions After a brief exposition of the theory, some recent advances are reported First, results are presented on the implementation of a Jacobian-free Newton-Krylov time-stepping method into a standard finite-volume aerodynamic code to obtain global linear instability results in flows of industrial interest Second, connections are sought between established and more-modern approaches for structure identification in flows, such as proper orthogonal decomposition and Koopman modes analysis (dynamic mode decomposition), and the possibility to connect solutions of the eigenvalue problem obtained by matrix formation or time-stepping with those delivered by dynamic mode decomposition, residual algorithm, and proper orthogonal decomposition analysis is highlighted in the laminar regime; turbulent and three-dimensional flows are identified as open areas for future research Finally, a new stable very-high-order finite-difference method is implemented for the spatial discretization of the operators describing the spatial biglobal eigenvalue problem, parabolized stability equation three-dimensional analysis, and the triglobal eigenvalue problem; it is shown that, combined with sparse matrix treatment, all these problems may now be solved on standard desktop computers

Limited Memory Block Krylov Subspace Optimization for Computing Dominant Singular Value Decompositions

Abstract: In many data-intensive applications, the use of principal component analysis and other related techniques is ubiquitous for dimension reduction, data mining, or other transformational purposes. Such transformations often require efficiently, reliably, and accurately computing dominant singular value decompositions (SVDs) of large and dense matrices. In this paper, we propose and study a subspace optimization technique for significantly accelerating the classic simultaneous iteration method. We analyze the convergence of the proposed algorithm and numerically compare it with several state-of-the-art SVD solvers under the MATLAB environment. Extensive computational results show that on a wide range of large unstructured dense matrices, the proposed algorithm can often provide improved efficiency or robustness over existing algorithms.

Spectral Learning of General Weighted Automata via Constrained Matrix Completion

Abstract: Many tasks in text and speech processing and computational biology require estimating functions mapping strings to real numbers. A broad class of such functions can be defined by weighted automata. Spectral methods based on the singular value decomposition of a Hankel matrix have been recently proposed for learning a probability distribution represented by a weighted automaton from a training sample drawn according to this same target distribution. In this paper, we show how spectral methods can be extended to the problem of learning a general weighted automaton from a sample generated by an arbitrary distribution. The main obstruction to this approach is that, in general, some entries of the Hankel matrix may be missing. We present a solution to this problem based on solving a constrained matrix completion problem. Combining these two ingredients, matrix completion and spectral method, a whole new family of algorithms for learning general weighted automata is obtained. We present generalization bounds for a particular algorithm in this family. The proofs rely on a joint stability analysis of matrix completion and spectral learning.