Showing papers on "Singular value decomposition published in 2016"

Deep neural networks for learning graph representations

Abstract: In this paper, we propose a novel model for learning graph representations, which generates a low-dimensional vector representation for each vertex by capturing the graph structural information. Different from other previous research efforts, we adopt a random surfing model to capture graph structural information directly, instead of using the sampling-based method for generating linear sequences proposed by Perozzi et al. (2014). The advantages of our approach will be illustrated from both theorical and empirical perspectives. We also give a new perspective for the matrix factorization method proposed by Levy and Goldberg (2014), in which the pointwise mutual information (PMI) matrix is considered as an analytical solution to the objective function of the skip-gram model with negative sampling proposed by Mikolov et al. (2013). Unlike their approach which involves the use of the SVD for finding the low-dimensitonal projections from the PMI matrix, however, the stacked denoising autoencoder is introduced in our model to extract complex features and model non-linearities. To demonstrate the effectiveness of our model, we conduct experiments on clustering and visualization tasks, employing the learned vertex representations as features. Empirical results on datasets of varying sizes show that our model outperforms other stat-of-the-art models in such tasks.

Diffusion MRI noise mapping using random matrix theory.

Abstract: PURPOSE To estimate the spatially varying noise map using a redundant series of magnitude MR images. METHODS We exploit redundancy in non-Gaussian distributed multidirectional diffusion MRI data by identifying its noise-only principal components, based on the theory of noisy covariance matrices. The bulk of principal component analysis eigenvalues, arising due to noise, is described by the universal Marchenko-Pastur distribution, parameterized by the noise level. This allows us to estimate noise level in a local neighborhood based on the singular value decomposition of a matrix combining neighborhood voxels and diffusion directions. RESULTS We present a model-independent local noise mapping method capable of estimating the noise level down to about 1% error. In contrast to current state-of-the-art techniques, the resultant noise maps do not show artifactual anatomical features that often reflect physiological noise, the presence of sharp edges, or a lack of adequate a priori knowledge of the expected form of MR signal. CONCLUSIONS Simulations and experiments show that typical diffusion MRI data exhibit sufficient redundancy that enables accurate, precise, and robust estimation of the local noise level by interpreting the principal component analysis eigenspectrum in terms of the Marchenko-Pastur distribution. Magn Reson Med 76:1582-1593, 2016. © 2015 International Society for Magnetic Resonance in Medicine.

Tensor Robust Principal Component Analysis: Exact Recovery of Corrupted Low-Rank Tensors via Convex Optimization

Abstract: This paper studies the Tensor Robust Principal Component (TRPCA) problem which extends the known Robust PCA [4] to the tensor case. Our model is based on a new tensor Singular Value Decomposition (t-SVD) [14] and its induced tensor tubal rank and tensor nuclear norm. Consider that we have a 3-way tensor X e Rn1×n2×n3 such that X = L0 + S0, where L0 has low tubal rank and S0 is sparse. Is that possible to recover both components? In this work, we prove that under certain suitable assumptions, we can recover both the low-rank and the sparse components exactly by simply solving a convex program whose objective is a weighted combination of the tensor nuclear norm and the l1-norm, i.e., min L, E ||L||* + λ||e||1, s.t. X = L + e, where λ = 1/√max(n1, n2)n3. Interestingly, TRPCA involves RPCA as a special case when n3 = 1 and thus it is a simple and elegant tensor extension of RPCA. Also numerical experiments verify our theory and the application for the image denoising demonstrates the effectiveness of our method.

Low-Rank Tensor Networks for Dimensionality Reduction and Large-Scale Optimization Problems: Perspectives and Challenges PART 1.

Abstract: Machine learning and data mining algorithms are becoming increasingly important in analyzing large volume, multi-relational and multi--modal datasets, which are often conveniently represented as multiway arrays or tensors. It is therefore timely and valuable for the multidisciplinary research community to review tensor decompositions and tensor networks as emerging tools for large-scale data analysis and data mining. We provide the mathematical and graphical representations and interpretation of tensor networks, with the main focus on the Tucker and Tensor Train (TT) decompositions and their extensions or generalizations. Keywords: Tensor networks, Function-related tensors, CP decomposition, Tucker models, tensor train (TT) decompositions, matrix product states (MPS), matrix product operators (MPO), basic tensor operations, multiway component analysis, multilinear blind source separation, tensor completion, linear/multilinear dimensionality reduction, large-scale optimization problems, symmetric eigenvalue decomposition (EVD), PCA/SVD, huge systems of linear equations, pseudo-inverse of very large matrices, Lasso and Canonical Correlation Analysis (CCA) (This is Part 1)

Ergodic theory, Dynamic Mode Decomposition and Computation of Spectral Properties of the Koopman operator

Abstract: We establish the convergence of a class of numerical algorithms, known as Dynamic Mode Decomposition (DMD), for computation of the eigenvalues and eigenfunctions of the infinite-dimensional Koopman operator. The algorithms act on data coming from observables on a state space, arranged in Hankel-type matrices. The proofs utilize the assumption that the underlying dynamical system is ergodic. This includes the classical measure-preserving systems, as well as systems whose attractors support a physical measure. Our approach relies on the observation that vector projections in DMD can be used to approximate the function projections by the virtue of Birkhoff's ergodic theorem. Using this fact, we show that applying DMD to Hankel data matrices in the limit of infinite-time observations yields the true Koopman eigenfunctions and eigenvalues. We also show that the Singular Value Decomposition, which is the central part of most DMD algorithms, converges to the Proper Orthogonal Decomposition of observables. We use this result to obtain a representation of the dynamics of systems with continuous spectrum based on the lifting of the coordinates to the space of observables. The numerical application of these methods is demonstrated using well-known dynamical systems and examples from computational fluid dynamics.

An Efficient SVD-Based Method for Image Denoising

Abstract: Nonlocal self-similarity of images has attracted considerable interest in the field of image processing and has led to several state-of-the-art image denoising algorithms, such as block matching and 3-D, principal component analysis with local pixel grouping, patch-based locally optimal wiener, and spatially adaptive iterative singular-value thresholding. In this paper, we propose a computationally simple denoising algorithm using the nonlocal self-similarity and the low-rank approximation (LRA). The proposed method consists of three basic steps. First, our method classifies similar image patches by the block-matching technique to form the similar patch groups, which results in the similar patch groups to be low rank. Next, each group of similar patches is factorized by singular value decomposition (SVD) and estimated by taking only a few largest singular values and corresponding singular vectors. Finally, an initial denoised image is generated by aggregating all processed patches. For low-rank matrices, SVD can provide the optimal energy compaction in the least square sense. The proposed method exploits the optimal energy compaction property of SVD to lead an LRA of similar patch groups. Unlike other SVD-based methods, the LRA in SVD domain avoids learning the local basis for representing image patches, which usually is computationally expensive. The experimental results demonstrate that the proposed method can effectively reduce noise and be competitive with the current state-of-the-art denoising algorithms in terms of both quantitative metrics and subjective visual quality.

Damped multichannel singular spectrum analysis for 3D random noise attenuation

Abstract: Multichannel singular spectrum analysis (MSSA) is an effective algorithm for random noise attenuation in seismic data, which decomposes the vector space of the Hankel matrix of the noisy signal into a signal subspace and a noise subspace by truncated singular value decomposition (TSVD). However, this signal subspace actually still contains residual noise. We have derived a new formula of low-rank reduction, which is more powerful in distinguishing between signal and noise compared with the traditional TSVD. By introducing a damping factor into traditional MSSA to dampen the singular values, we have developed a new algorithm for random noise attenuation. We have named our modified MSSA as damped MSSA. The denoising performance is controlled by the damping factor, and our approach reverts to the traditional MSSA approach when the damping factor is sufficiently large. Application of the damped MSSA algorithm on synthetic and field seismic data demonstrates superior performance compared with the conve...

Block-based discrete wavelet transform-singular value decomposition image watermarking scheme using human visual system characteristics

Abstract: Digital watermarking has been suggested as a way to achieve digital protection. The aim of digital watermarking is to insert the secret data into the image without significantly affecting the visual quality. This study presents a robust block-based image watermarking scheme based on the singular value decomposition (SVD) and human visual system in the discrete wavelet transform (DWT) domain. The proposed method is considered to be a block-based scheme that utilises the entropy and edge entropy as HVS characteristics for the selection of significant blocks to embed the watermark, which is a binary watermark logo. The blocks of the lowest entropy values and edge entropy values are selected as the best regions to insert the watermark. After the first level of DWT decomposition, the SVD is performed on the low-low sub-band to modify several elements in its U matrix according to predefined conditions. The experimental results of the proposed scheme showed high imperceptibility and high robustness against all image processing attacks and several geometrical attacks using examples of standard and real images. Furthermore, the proposed scheme outperformed several previous schemes in terms of imperceptibility and robustness. The security issue is improved by encrypting a portion of the important information using Advanced Standard Encryption a key size of 192-bits (AES-192).

Tensor Networks and Hierarchical Tensors for the Solution of High-Dimensional Partial Differential Equations

Abstract: Hierarchical tensors can be regarded as a generalisation, preserving many crucial features, of the singular value decomposition to higher-order tensors. For a given tensor product space, a recursive decomposition of the set of coordinates into a dimension tree gives a hierarchy of nested subspaces and corresponding nested bases. The dimensions of these subspaces yield a notion of multilinear rank. This rank tuple, as well as quasi-optimal low-rank approximations by rank truncation, can be obtained by a hierarchical singular value decomposition. For fixed multilinear ranks, the storage and operation complexity of these hierarchical representations scale only linearly in the order of the tensor. As in the matrix case, the set of hierarchical tensors of a given multilinear rank is not a convex set, but forms an open smooth manifold. A number of techniques for the computation of hierarchical low-rank approximations have been developed, including local optimisation techniques on Riemannian manifolds as well as truncated iteration methods, which can be applied for solving high-dimensional partial differential equations. This article gives a survey of these developments. We also discuss applications to problems in uncertainty quantification, to the solution of the electronic Schrodinger equation in the strongly correlated regime, and to the computation of metastable states in molecular dynamics.

Principal component analysis - a tutorial

Abstract: Dimensionality reduction is one of the preprocessing steps in many machine learning applications and it is used to transform the features into a lower dimension space. Principal component analysis (PCA) technique is one of the most famous unsupervised dimensionality reduction techniques. The goal of the technique is to find the PCA space, which represents the direction of the maximum variance of the given data. This paper highlights the basic background needed to understand and implement the PCA technique. This paper starts with basic definitions of the PCA technique and the algorithms of two methods of calculating PCA, namely, the covariance matrix and singular value decomposition (SVD) methods. Moreover, a number of numerical examples are illustrated to show how the PCA space is calculated in easy steps. Three experiments are conducted to show how to apply PCA in the real applications including biometrics, image compression, and visualisation of high-dimensional datasets.

Matrix and Tensor Based Methods for Missing Data Estimation in Large Traffic Networks

Abstract: Intelligent transportation systems (ITSs) gather information about traffic conditions by collecting data from a wide range of on-ground sensors. The collected data usually suffer from irregular spatial and temporal resolution. Consequently, missing data is a common problem faced by ITSs. In this paper, we consider the problem of missing data in large and diverse road networks. We propose various matrix and tensor based methods to estimate these missing values by extracting common traffic patterns in large road networks. To obtain these traffic patterns in the presence of missing data, we apply fixed-point continuation with approximate singular value decomposition, canonical polyadic decomposition, least squares, and variational Bayesian principal component analysis. For analysis, we consider different road networks, each of which is composed of around 1500 road segments. We evaluate the performance of these methods in terms of estimation accuracy, variance of the data set, and the bias imparted by these methods.

Low Rank Alternating Direction Method of Multipliers Reconstruction for MR Fingerprinting

Abstract: Purpose The proposed reconstruction framework addresses the reconstruction accuracy, noise propagation and computation time for Magnetic Resonance Fingerprinting (MRF). Methods Based on a singular value decomposition (SVD) of the signal evolution, MRF is formulated as a low rank inverse problem in which one image is reconstructed for each singular value under consideration. This low rank approximation of the signal evolution reduces the computational burden by reducing the number of Fourier transformations. Also, the low rank approximation improves the conditioning of the problem, which is further improved by extending the low rank inverse problem to an augmented Lagrangian that is solved by the alternating direction method of multipliers (ADMM). The root mean square error and the noise propagation are analyzed in simulations. For verification, in vivo examples are provided. Results The proposed low rank ADMM approach shows a reduced root mean square error compared to the original fingerprinting reconstruction, to a low rank approximation alone and to an ADMM approach without a low rank approximation. Incorporating sensitivity encoding allows for further artifact reduction. Conclusion The proposed reconstruction provides robust convergence, reduced computational burden and improved image quality compared to other MRF reconstruction approaches evaluated in this study.

Optimal Communication Network-Based $H_\infty $ Quantized Control With Packet Dropouts for a Class of Discrete-Time Neural Networks With Distributed Time Delay

Abstract: This paper is concerned with optimal communication network-based $H_\infty $ quantized control for a discrete-time neural network with distributed time delay. Control of the neural network (plant) is implemented via a communication network. Both quantization and communication network-induced data packet dropouts are considered simultaneously. It is assumed that the plant state signal is quantized by a logarithmic quantizer before transmission, and communication network-induced packet dropouts can be described by a Bernoulli distributed white sequence. A new approach is developed such that controller design can be reduced to the feasibility of linear matrix inequalities, and a desired optimal control gain can be derived in an explicit expression. It is worth pointing out that some new techniques based on a new sector-like expression of quantization errors, and the singular value decomposition of a matrix are developed and employed in the derivation of main results. An illustrative example is presented to show the effectiveness of the obtained results.

A Randomized Blocked Algorithm for Efficiently Computing Rank-revealing Factorizations of Matrices

Abstract: This manuscript describes a technique for computing partial rank-revealing factorizations, such as a partial QR factorization or a partial singular value decomposition. The method takes as input a tolerance $\varepsilon$ and an $m\times n$ matrix $\boldsymbol{\mathsf{A}}$ and returns an approximate low-rank factorization of $\boldsymbol{\mathsf{A}}$ that is accurate to within precision $\varepsilon$ in the Frobenius norm (or some other easily computed norm). The rank $k$ of the computed factorization (which is an output of the algorithm) is in all examples we examined very close to the theoretically optimal $\varepsilon$-rank. The proposed method is inspired by the Gram--Schmidt algorithm and has the same $O(mnk)$ asymptotic flop count. However, the method relies on randomized sampling to avoid column pivoting, which allows it to be blocked, and hence accelerates practical computations by reducing communication. Numerical experiments demonstrate that the accuracy of the scheme is for every matrix that was...

Fast covariance estimation for high-dimensional functional data

Abstract: We propose two fast covariance smoothing methods and associated software that scale up linearly with the number of observations per function. Most available methods and software cannot smooth covariance matrices of dimension $$J>500$$J>500; a recently introduced sandwich smoother is an exception but is not adapted to smooth covariance matrices of large dimensions, such as $$J= 10{,}000$$J=10,000. We introduce two new methods that circumvent those problems: (1) a fast implementation of the sandwich smoother for covariance smoothing; and (2) a two-step procedure that first obtains the singular value decomposition of the data matrix and then smoothes the eigenvectors. These new approaches are at least an order of magnitude faster in high dimensions and drastically reduce computer memory requirements. The new approaches provide instantaneous (a few seconds) smoothing for matrices of dimension $$J=10{,}000$$J=10,000 and very fast ($$<$$<10 min) smoothing for $$J=100{,}000$$J=100,000. R functions, simulations, and data analysis provide ready to use, reproducible, and scalable tools for practical data analysis of noisy high-dimensional functional data.

Weighted low rank approximations with provable guarantees

Abstract: The classical low rank approximation problem is: given a matrix A, find a rank-k matrix B such that the Frobenius norm of A − B is minimized It can be solved efficiently using, for instance, the Singular Value Decomposition (SVD) If one allows randomization and approximation, it can be solved in time proportional to the number of non-zero entries of A with high probability Inspired by practical applications, we consider a weighted version of low rank approximation: for a non-negative weight matrix W we seek to minimize ∑i, j (Wi, j · (Ai,j − Bi,j))2 The classical problem is a special case of this problem when all weights are 1 Weighted low rank approximation is known to be NP-hard, so we are interested in a meaningful parametrization that would allow efficient algorithms In this paper we present several efficient algorithms for the case of small k and under the assumption that the weight matrix W is of low rank, or has a small number of distinct columns An important feature of our algorithms is that they do not assume anything about the matrix A We also obtain lower bounds that show that our algorithms are nearly optimal in these parameters We give several applications in which these parameters are small To the best of our knowledge, the present paper is the first to provide algorithms for the weighted low rank approximation problem with provable guarantees Perhaps even more importantly, our algorithms proceed via a new technique, which we call “guess the sketch” The technique turns out to be general enough to give solutions to several other fundamental problems: adversarial matrix completion, weighted non-negative matrix factorization and tensor completion

UAV State Estimation Using Adaptive Complementary Filters

Abstract: We address the complete state estimation problem of unmanned aerial vehicles, even under high-dynamic 3-D aerobatic maneuvers, while using low-cost sensors with bias variations and higher levels of noise. In such conditions, the control demand, for a robust real-time data fusion filter with minimal lag and noise, is addressed with the efficiency of a complementary filter scheme. First, the attitude is directly estimated in Special Orthogonal Group (SO(3)) by complementing the noisy accelerometer/magnetometer vector basis with a gyro propagated vector basis. Data fusion follows a least square minimization in SO(3) (Wahba’s problem) solved in an analytic nonrecursive manner. Stability of the proposed filter is shown and performance metrics are extracted, whereas the computational complexity has been minimized with an appropriate reference frame and a custom singular value decomposition algorithm. An adaptation scheme is proposed to allow unhindered operation of the filter to erroneous inputs introduced by the high dynamics of a 3-D flight. Finally, the velocity/position estimation is mainly constructed by complementary filters combining multiple sensors. In addition to the low complexity and the filtering of the noise, the proposed observer is aided through a developed vision algorithm, enabling the use of the filter in Global-Positioning-System-denied environments. Extensive experimental results and comparative studies with state-of-the-art filters, either in the laboratory or in the field using high-performance autonomous helicopters, demonstrate the efficacy of the proposed scheme in demanding conditions.

Adaptive neuro-fuzzy inference system multi-objective optimization using the genetic algorithm/singular value decomposition method for modelling the discharge coefficient in rectangular sharp-crested side weirs

Abstract: In the present article, the adaptive neuro-fuzzy inference system (ANFIS) is employed to model the discharge coefficient in rectangular sharp-crested side weirs. The genetic algorithm (GA) is used for the optimum selection of membership functions, while the singular value decomposition (SVD) method helps in computing the linear parameters of the ANFIS results section (GA/SVD-ANFIS). The effect of each dimensionless parameter on discharge coefficient prediction is examined in five different models to conduct sensitivity analysis by applying the above-mentioned dimensionless parameters. Two different sets of experimental data are utilized to examine the models and obtain the best model. The study results indicate that the model designed through GA/SVD-ANFIS predicts the discharge coefficient with a good level of accuracy (mean absolute percentage error = 3.362 and root mean square error = 0.027). Moreover, comparing this method with existing equations and the multi-layer perceptron–artificial neural network...

Identification of Protein–Protein Interactions via a Novel Matrix-Based Sequence Representation Model with Amino Acid Contact Information

Abstract: Identification of protein-protein interactions (PPIs) is a difficult and important problem in biology. Since experimental methods for predicting PPIs are both expensive and time-consuming, many computational methods have been developed to predict PPIs and interaction networks, which can be used to complement experimental approaches. However, these methods have limitations to overcome. They need a large number of homology proteins or literature to be applied in their method. In this paper, we propose a novel matrix-based protein sequence representation approach to predict PPIs, using an ensemble learning method for classification. We construct the matrix of Amino Acid Contact (AAC), based on the statistical analysis of residue-pairing frequencies in a database of 6323 protein-protein complexes. We first represent the protein sequence as a Substitution Matrix Representation (SMR) matrix. Then, the feature vector is extracted by applying algorithms of Histogram of Oriented Gradient (HOG) and Singular Value Decomposition (SVD) on the SMR matrix. Finally, we feed the feature vector into a Random Forest (RF) for judging interaction pairs and non-interaction pairs. Our method is applied to several PPI datasets to evaluate its performance. On the S . c e r e v i s i a e dataset, our method achieves 94 . 83 % accuracy and 92 . 40 % sensitivity. Compared with existing methods, and the accuracy of our method is increased by 0 . 11 percentage points. On the H . p y l o r i dataset, our method achieves 89 . 06 % accuracy and 88 . 15 % sensitivity, the accuracy of our method is increased by 0 . 76 % . On the H u m a n PPI dataset, our method achieves 97 . 60 % accuracy and 96 . 37 % sensitivity, and the accuracy of our method is increased by 1 . 30 % . In addition, we test our method on a very important PPI network, and it achieves 92 . 71 % accuracy. In the Wnt-related network, the accuracy of our method is increased by 16 . 67 % . The source code and all datasets are available at https://figshare.com/s/580c11dce13e63cb9a53.

Structured Matrix Completion with Applications to Genomic Data Integration

Abstract: Matrix completion has attracted significant recent attention in many fields including statistics, applied mathematics, and electrical engineering. Current literature on matrix completion focuses primarily on independent sampling models under which the individual observed entries are sampled independently. Motivated by applications in genomic data integration, we propose a new framework of structured matrix completion (SMC) to treat structured missingness by design. Specifically, our proposed method aims at efficient matrix recovery when a subset of the rows and columns of an approximately low-rank matrix are observed. We provide theoretical justification for the proposed SMC method and derive lower bound for the estimation errors, which together establish the optimal rate of recovery over certain classes of approximately low-rank matrices. Simulation studies show that the method performs well in finite sample under a variety of configurations. The method is applied to integrate several ovarian can...

On Unifying Multi-View Self-Representations for Clustering by Tensor Multi-Rank Minimization

Abstract: In this paper, we address the multi-view subspace clustering problem. Our method utilizes the circulant algebra for tensor, which is constructed by stacking the subspace representation matrices of different views and then rotating, to capture the low rank tensor subspace so that the refinement of the view-specific subspaces can be achieved, as well as the high order correlations underlying multi-view data can be explored.} By introducing a recently proposed tensor factorization, namely tensor-Singular Value Decomposition (t-SVD) \cite{kilmer13}, we can impose a new type of low-rank tensor constraint on the rotated tensor to capture the complementary information from multiple views. Different from traditional unfolding based tensor norm, this low-rank tensor constraint has optimality properties similar to that of matrix rank derived from SVD, so the complementary information among views can be explored more efficiently and thoroughly. The established model, called t-SVD based Multi-view Subspace Clustering (t-SVD-MSC), falls into the applicable scope of augmented Lagrangian method, and its minimization problem can be efficiently solved with theoretical convergence guarantee and relatively low computational complexity. Extensive experimental testing on eight challenging image dataset shows that the proposed method has achieved highly competent objective performance compared to several state-of-the-art multi-view clustering methods.

The Bidiagonal Singular Value Decomposition and Hamiltonian Mechanics

Abstract: Computing the singular value decomposition of a bidiagonal matrix B is considered. This problem arises in the singular value decomposition of a general matrix, and in the eigenproblem for a symmetric positive-definite tridiagonal matrix. It is shown that if the entries of B are known with high relative accuracy, the singular values and singular vectors of B will be determined to much higher accuracy than the standard perturbation theory suggests. It is also shown that the algorithm in [Demmel and Kahan, SIAM I. Sci. Statist. Comput., 11 (1990), pp. 873–912] computes the singular vectors as well as the singular values to this accuracy. A Hamiltonian interpretation of the algorithm is also given, and differential equation methods are used to prove many of the basic facts. The Hamiltonian approach suggests a way to use flows to predict the accumulation of error in other eigenvalue algorithms as well.

Randomized Matrix Decompositions using R

Abstract: Matrix decompositions are fundamental tools in the area of applied mathematics, statistical computing, and machine learning. In particular, low-rank matrix decompositions are vital, and widely used for data analysis, dimensionality reduction, and data compression. Massive datasets, however, pose a computational challenge for traditional algorithms, placing significant constraints on both memory and processing power. Recently, the powerful concept of randomness has been introduced as a strategy to ease the computational load. The essential idea of probabilistic algorithms is to employ some amount of randomness in order to derive a smaller matrix from a high-dimensional data matrix. The smaller matrix is then used to compute the desired low-rank approximation. Such algorithms are shown to be computationally efficient for approximating matrices with low-rank structure. We present the \proglang{R} package rsvd, and provide a tutorial introduction to randomized matrix decompositions. Specifically, randomized routines for the singular value decomposition, (robust) principal component analysis, interpolative decomposition, and CUR decomposition are discussed. Several examples demonstrate the routines, and show the computational advantage over other methods implemented in R.