Showing papers on "Singular value decomposition published in 2019"

Infrared Small Target Detection Based on Partial Sum of the Tensor Nuclear Norm

Abstract: Excellent performance, real time and strong robustness are three vital requirements for infrared small target detection. Unfortunately, many current state-of-the-art methods merely achieve one of the expectations when coping with highly complex scenes. In fact, a common problem is that real-time processing and great detection ability are difficult to coordinate. Therefore, to address this issue, a robust infrared patch-tensor model for detecting an infrared small target is proposed in this paper. On the basis of infrared patch-tensor (IPT) model, a novel nonconvex low-rank constraint named partial sum of tensor nuclear norm (PSTNN) joint weighted l1 norm was employed to efficiently suppress the background and preserve the target. Due to the deficiency of RIPT which would over-shrink the target with the possibility of disappearing, an improved local prior map simultaneously encoded with target-related and background-related information was introduced into the model. With the help of a reweighted scheme for enhancing the sparsity and high-efficiency version of tensor singular value decomposition (t-SVD), the total algorithm complexity and computation time can be reduced dramatically. Then, the decomposition of the target and background is transformed into a tensor robust principle component analysis problem (TRPCA), which can be efficiently solved by alternating direction method of multipliers (ADMM). A series of experiments substantiate the superiority of the proposed method beyond state-of-the-art baselines.

Defining a bulk-edge correspondence for non-Hermitian Hamiltonians via singular-value decomposition

Abstract: We address the breakdown of the bulk-boundary correspondence observed in non-Hermitian systems, where open and periodic systems can have distinct phase diagrams. The correspondence can be completely restored by considering the Hamiltonian's singular-value decomposition instead of its eigendecomposition. This leads to a natural topological description in terms of a flattened singular decomposition. This description is equivalent to the usual approach for Hermitian systems and coincides with a recent proposal for the classification of non-Hermitian systems. We generalize the notion of the entanglement spectrum to non-Hermitian systems, and show that the edge physics is indeed completely captured by the periodic bulk Hamiltonian. We exemplify our approach by considering the chiral non-Hermitian Su-Schrieffer-Heger and Chern insulator models. Our work advocates a different perspective on topological non-Hermitian Hamiltonians, paving the way to a better understanding of their entanglement structure.

Hyperspectral Image Super-Resolution via Subspace-Based Low Tensor Multi-Rank Regularization

Abstract: Recently, combining a low spatial resolution hyperspectral image (LR-HSI) with a high spatial resolution multispectral image (HR-MSI) into an HR-HSI has become a popular scheme to enhance the spatial resolution of HSI. We propose a novel subspace-based low tensor multi-rank regularization method for the fusion, which fully exploits the spectral correlations and non-local similarities in the HR-HSI. To make use of high spectral correlations, the HR-HSI is approximated by spectral subspace and coefficients. We first learn the spectral subspace from the LR-HSI via singular value decomposition, and then estimate the coefficients via the low tensor multi-rank prior. More specifically, based on the learned cluster structure in the HR-MSI, the patches in coefficients are grouped. We collect the coefficients in the same cluster into a three-dimensional tensor and impose the low tensor multi-rank prior on these collected tensors, which fully model the non-local self-similarities in the HR-HSI. The coefficients optimization is solved by the alternating direction method of multipliers. Experiments on two public HSI datasets demonstrate the advantages of our method.

Essential Tensor Learning for Multi-View Spectral Clustering

Abstract: Recently, multi-view clustering attracts much attention, which aims to take advantage of multi-view information to improve the performance of clustering. However, most recent work mainly focuses on the self-representation-based subspace clustering, which is of high computation complexity. In this paper, we focus on the Markov chain-based spectral clustering method and propose a novel essential tensor learning method to explore the high-order correlations for multi-view representation. We first construct a tensor based on multi-view transition probability matrices of the Markov chain. By incorporating the idea from the robust principle component analysis, tensor singular value decomposition (t-SVD)-based tensor nuclear norm is imposed to preserve the low-rank property of the essential tensor, which can well capture the principle information from multiple views. We also employ the tensor rotation operator for this task to better investigate the relationship among views as well as reduce the computation complexity. The proposed method can be efficiently optimized by the alternating direction method of multipliers (ADMM). Extensive experiments on seven real-world datasets corresponding to five different applications show that our method achieves superior performance over other state-of-the-art methods.

Nonlinear Model Order Reduction via Lifting Transformations and Proper Orthogonal Decomposition

Abstract: This paper presents a structure-exploiting nonlinear model reduction method for systems with general nonlinearities. First, the nonlinear model is lifted to a model with more structure via variable...

Finite Sample Analysis of Stochastic System Identification

Abstract: In this paper, we analyze the finite sample complexity of stochastic system identification using modern tools from machine learning and statistics. An unknown discrete-time linear system evolves over time under Gaussian noise without external inputs. The objective is to recover the system parameters as well as the Kalman filter gain, given a single trajectory of output measurements over a finite horizon of length N. Based on a subspace identification algorithm and a finite number of N output samples, we provide non-asymptotic high-probability upper bounds for the system parameter estimation errors. Our analysis uses recent results from random matrix theory, self-normalized martingales and SVD robustness, in order to show that with high probability the estimation errors decrease with a rate of $1/\sqrt N$ up to logarithmic terms. Our non-asymptotic bounds not only agree with classical asymptotic results, but are also valid even when the system is marginally stable.

Compressed dynamic mode decomposition for background modeling

Abstract: We introduce the method of compressed dynamic mode decomposition (cDMD) for background modeling. The dynamic mode decomposition is a regression technique that integrates two of the leading data analysis methods in use today: Fourier transforms and singular value decomposition. Borrowing ideas from compressed sensing and matrix sketching, cDMD eases the computational workload of high-resolution video processing. The key principal of cDMD is to obtain the decomposition on a (small) compressed matrix representation of the video feed. Hence, the cDMD algorithm scales with the intrinsic rank of the matrix, rather than the size of the actual video (data) matrix. Selection of the optimal modes characterizing the background is formulated as a sparsity-constrained sparse coding problem. Our results show that the quality of the resulting background model is competitive, quantified by the F-measure, recall and precision. A graphics processing unit accelerated implementation is also presented which further boosts the computational performance of the algorithm.

Efficient Neural Network Compression

Abstract: Network compression reduces the computational complexity and memory consumption of deep neural networks by reducing the number of parameters. In SVD-based network compression the right rank needs to be decided for every layer of the network. In this paper we propose an efficient method for obtaining the rank configuration of the whole network. Unlike previous methods which consider each layer separately, our method considers the whole network to choose the right rank configuration. We propose novel accuracy metrics to represent the accuracy and complexity relationship for a given neural network. We use these metrics in a non-iterative fashion to obtain the right rank configuration which satisfies the constraints on FLOPs and memory while maintaining sufficient accuracy. Experiments show that our method provides better compromise between accuracy and computational complexity/memory consumption while performing compression at much higher speed. For VGG-16 our network can reduce the FLOPs by 25% and improve accuracy by 0.7% compared to the baseline, while requiring only 3 minutes on a CPU to search for the right rank configuration. Previously, similar results were achieved in 4 hours with 8 GPUs. The proposed method can be used for lossless compression of a neural network as well. The better accuracy and complexity compromise, as well as the extremely fast speed of our method make it suitable for neural network compression.

An Optimized Image Watermarking Method Based on HD and SVD in DWT Domain

Abstract: In this paper, a novel image watermarking method is proposed which is based on discrete wave transformation (DWT), Hessenberg decomposition (HD), and singular value decomposition (SVD). First, in the embedding process, the host image is decomposed into a number of sub-bands through multi-level DWT, and the resulting coefficients of which are then used as the input for HD. The watermark is operated on the SVD at the same time. The watermark is finally embedded into the host image by the scaling factor. Fruit fly optimization algorithm, one of the natural-inspired optimization algorithms is devoted to find the scaling factor through the proposed objective evaluation function. The proposed method is compared to other research works under various spoof attacks, such as the filter, noise, JPEG compression, JPEG2000 compression, and sharpening attacks. The experimental results show that the proposed image watermarking method has a good trade-off between robustness and invisibility even for the watermarks with multiple sizes.

Large-Scale Low-Rank Matrix Learning with Nonconvex Regularizers

Abstract: Low-rank modeling has many important applications in computer vision and machine learning. While the matrix rank is often approximated by the convex nuclear norm, the use of nonconvex low-rank regularizers has demonstrated better empirical performance. However, the resulting optimization problem is much more challenging. Recent state-of-the-art requires an expensive full SVD in each iteration. In this paper, we show that for many commonly-used nonconvex low-rank regularizers, the singular values obtained from the proximal operator can be automatically threshold. This allows the proximal operator to be efficiently approximated by the power method. We then develop a fast proximal algorithm and its accelerated variant with inexact proximal step. It can be guaranteed that the squared distance between consecutive iterates converges at a rate of $O(1/T)$O(1/T), where $T$T is the number of iterations. Furthermore, we show the proposed algorithm can be parallelized, and the resultant algorithm achieves nearly linear speedup w.r.t. the number of threads. Extensive experiments are performed on matrix completion and robust principal component analysis. Significant speedup over the state-of-the-art is observed.

Streaming Low-Rank Matrix Approximation With An Application To Scientific Simulation

Abstract: This paper argues that randomized linear sketching is a natural tool for on-the-fly compression of data matrices that arise from large-scale scientific simulations and data collection. The technica...

Tensor Ring Decomposition with Rank Minimization on Latent Space: An Efficient Approach for Tensor Completion

Abstract: In tensor completion tasks, the traditional low-rank tensor decomposition models suffer from the laborious model selection problem due to their high model sensitivity. In particular, for tensor ring (TR) decomposition, the number of model possibilities grows exponentially with the tensor order, which makes it rather challenging to find the optimal TR decomposition. In this paper, by exploiting the low-rank structure of the TR latent space, we propose a novel tensor completion method which is robust to model selection. In contrast to imposing the low-rank constraint on the data space, we introduce nuclear norm regularization on the latent TR factors, resulting in the optimization step using singular value decomposition (SVD) being performed at a much smaller scale. By leveraging the alternating direction method of multipliers (ADMM) scheme, the latent TR factors with optimal rank and the recovered tensor can be obtained simultaneously. Our proposed algorithm is shown to effectively alleviate the burden of TR-rank selection, thereby greatly reducing the computational cost. The extensive experimental results on both synthetic and real-world data demonstrate the superior performance and efficiency of the proposed approach against the state-of-the-art algorithms.

The two-to-infinity norm and singular subspace geometry with applications to high-dimensional statistics

Abstract: The singular value matrix decomposition plays a ubiquitous role throughout statistics and related fields. Myriad applications including clustering, classification, and dimensionality reduction involve studying and exploiting the geometric structure of singular values and singular vectors. This paper provides a novel collection of technical and theoretical tools for studying the geometry of singular subspaces using the two-to-infinity norm. Motivated by preliminary deterministic Procrustes analysis, we consider a general matrix perturbation setting in which we derive a new Procrustean matrix decomposition. Together with flexible machinery developed for the two-to-infinity norm, this allows us to conduct a refined analysis of the induced perturbation geometry with respect to the underlying singular vectors even in the presence of singular value multiplicity. Our analysis yields singular vector entrywise perturbation bounds for a range of popular matrix noise models, each of which has a meaningful associated statistical inference task. In addition, we demonstrate how the two-to-infinity norm is the preferred norm in certain statistical settings. Specific applications discussed in this paper include covariance estimation, singular subspace recovery, and multiple graph inference. Both our Procrustean matrix decomposition and the technical machinery developed for the two-to-infinity norm may be of independent interest.

Frames and Numerical Approximation

Abstract: Functions of one or more variables are usually approximated with a basis: a complete, linearly independent system of functions that spans a suitable function space. The topic of this paper is the n...

Hyperspectral Image Denoising via Subspace-Based Nonlocal Low-Rank and Sparse Factorization

Abstract: Hyperspectral images (HSIs) are unavoidably contaminated by different types of noise during data acquisition and transmission, e.g., Gaussian noise, impulse noise, stripes, and deadlines. A variety of mixed noise reduction approaches are developed for HSI, in which the subspace-based methods have achieved comparable performance. In this paper, a novel subspace-based nonlocal low-rank and sparse factorization (SNLRSF) method is proposed to remove the mixture of several types of noise. The SNLRSF method explores spectral low rank based on the fact that spectral signatures of pixels lie in a low-dimensional subspace and employs the nonlocal low-rank factorization to take the spatial nonlocal self-similarity into consideration. At the same time, the successive singular value decomposition (SVD) low-rank factorization algorithm is used to estimate three-dimensional (3-D) tensor generated by nonlocal similar 3-D patches. Moreover, the well-known augmented Lagrangian method is adopted to solve final denoising model efficiently. The experimental results over simulated and real datasets demonstrate that the proposed approach outperforms the related state-of-the-art methods in terms of visual quality and quantitative evaluation.

Learning Efficient Tensor Representations with Ring-structured Networks

Abstract: Tensor train decomposition is a powerful representation for high-order tensors, which has been successfully applied to various machine learning tasks in recent years. In this paper, we study a more generalized tensor decomposition with a ring-structured network by employing circular multilinear products over a sequence of lower-order core tensors. We refer to such tensor decomposition as tensor ring (TR) representation. Our goal is to introduce learning algorithms including sequential singular value decompositions and blockwise alternating least squares with adaptive tensor ranks. Experimental results demonstrate the effectiveness of the TR model and the learning algorithms. In particular, we show that the structure information and high-order correlations within a 2D image can be captured efficiently by employing an appropriate tensorization and TR decomposition.

Extended-Kalman-filter-based dynamic mode decomposition for simultaneous system identification and denoising.

Abstract: A new dynamic mode decomposition (DMD) method is introduced for simultaneous system identification and denoising in conjunction with the adoption of an extended Kalman filter algorithm. The present paper explains the extended-Kalman-filter-based DMD (EKFDMD) algorithm which is an online algorithm for dataset for a small number of degree of freedom (DoF). It also illustrates that EKFDMD requires significant numerical resources for many-degree-of-freedom (many-DoF) problems and that the combination with truncated proper orthogonal decomposition (trPOD) helps us to apply the EKFDMD algorithm to many-DoF problems, though it prevents the algorithm from being fully online. The numerical experiments of a noisy dataset with a small number of DoFs illustrate that EKFDMD can estimate eigenvalues better than or as well as the existing algorithms, whereas EKFDMD can also denoise the original dataset online. In particular, EKFDMD performs better than existing algorithms for the case in which system noise is present. The EKFDMD with trPOD, which unfortunately is not fully online, can be successfully applied to many-DoF problems, including a fluid-problem example, and the results reveal the superior performance of system identification and denoising.

Lanczos method for large-scale quaternion singular value decomposition

Abstract: In many color image processing and recognition applications, one of the most important targets is to compute the optimal low-rank approximations to color images, which can be reconstructed with a small number of dominant singular value decomposition (SVD) triplets of quaternion matrices. All existing methods are designed to compute all SVD triplets of quaternion matrices at first and then to select the necessary dominant ones for reconstruction. This way costs quite a lot of operational flops and CPU times to compute many superfluous SVD triplets. In this paper, we propose a Lanczos-based method of computing partial (several dominant) SVD triplets of the large-scale quaternion matrices. The partial bidiagonalization of large-scale quaternion matrices is derived by using the Lanczos iteration, and the reorthogonalization and thick-restart techniques are also utilized in the implementation. An algorithm is presented to compute the partial quaternion singular value decomposition. Numerical examples, including principal component analysis, color face recognition, video compression and color image completion, illustrate that the performance of the developed Lanczos-based method for low-rank quaternion approximation is better than that of the state-of-the-art methods.

Optimal Sparse Singular Value Decomposition for High-Dimensional High-Order Data

Abstract: In this article, we consider the sparse tensor singular value decomposition, which aims for dimension reduction on high-dimensional high-order data with certain sparsity structure. A method named s...

Early Fault Diagnosis for Planetary Gearbox Based on Adaptive Parameter Optimized VMD and Singular Kurtosis Difference Spectrum

Abstract: Variational mode decomposition (VMD) is widely used in the condition monitoring and fault diagnosis of rotary machinery for its unique advantages. An adaptive parameter optimized VMD (APOVMD) is proposed in order to adaptively determine the suitable decomposed parameters and further enhance its performance. The traditional singular value decomposition (SVD) method cannot effectively select the reconstructed order, which often leads to unsatisfactory results for signal reconstruction. Thus, a singular kurtosis difference spectrum method is proposed to accurately determine the effective reconstructed order for signal noise reduction. In addition, because the fault signal of the planetary gearbox at the early fault stage is weak and susceptible to ambient noise and other signal interference, the fault feature information is difficult to extract. To address this issue, a novel method for early fault feature extraction of planetary gearbox based on APOVMD and singular kurtosis difference spectrum is proposed in this paper. First, the APOVMD is applied to decompose the planetary gearbox vibration signal into a series of band-limited intrinsic mode functions adaptively and non-recursively. Second, the sensitive component is selected from the IMFS according to the cosine similarity index. Third, the Hankel matrix is constructed for the sensitive component in the phase space and decomposed by SVD. Here, the effective reconstructed order is automatically selected by the singular kurtosis difference spectrum method for noise reduction. Finally, the Hilbert envelope spectrum analysis is carried out on the reconstructed signal, and the fault characteristic frequency information of planetary gearbox can be accurately extracted from the envelope spectrum to realize the fault identification and location. The results of simulation studies and actual experimental data analysis demonstrate that the proposed method has superior ability to extract the early weak fault characteristics of the planetary gearbox compared with the VMD-SVD and EEMD-SVD methods, and the validity and feasibility of the presented method are proved.

Hybrid Precoding for Massive mmWave MIMO Systems

Abstract: Due to high costs and power consumptions, fully digital baseband precoding schemes are usually prohibitive in millimeter-wave massive MIMO systems. Therefore, hybrid precoding strategies become promising solutions. In this paper, we present a novel real-time yet high-performance precoding strategy. Specifically, the eigenvectors corresponding to the larger eigenvalues of the right unitary matrix after singular value decomposition on an array response matrix are used to abstract the angle information of an analog precoding matrix. As the obtained eigenvectors correspond to the larger singular values, the major phase information of channels is captured. In this way, the iterative search process for obtaining the analog precoding vectors is avoided, and thus the hybrid precoding can be realized in parallel. To further improve its spectral-efficiency, we enlarge the resultant vector set by involving more relevant vectors in terms of their correlation values with the unconstrained optimal precoder, and a hybrid precoder is thus produced by using the vector set. The simulation results show that our proposed scheme achieves near the same performance as the orthogonal matching pursuit does, whereas it costs much fewer complexities than the OMP, and thus can be realized in parallel.

Trajectory Space Factorization for Deep Video-Based 3D Human Pose Estimation.

Abstract: Existing deep learning approaches on 3d human pose estimation for videos are either based on Recurrent or Convolutional Neural Networks (RNNs or CNNs). However, RNN-based frameworks can only tackle sequences with limited frames because sequential models are sensitive to bad frames and tend to drift over long sequences. Although existing CNN-based temporal frameworks attempt to address the sensitivity and drift problems by concurrently processing all input frames in the sequence, the existing state-of-the-art CNN-based framework is limited to 3d pose estimation of a single frame from a sequential input. In this paper, we propose a deep learning-based framework that utilizes matrix factorization for sequential 3d human poses estimation. Our approach processes all input frames concurrently to avoid the sensitivity and drift problems, and yet outputs the 3d pose estimates for every frame in the input sequence. More specifically, the 3d poses in all frames are represented as a motion matrix factorized into a trajectory bases matrix and a trajectory coefficient matrix. The trajectory bases matrix is precomputed from matrix factorization approaches such as Singular Value Decomposition (SVD) or Discrete Cosine Transform (DCT), and the problem of sequential 3d pose estimation is reduced to training a deep network to regress the trajectory coefficient matrix. We demonstrate the effectiveness of our framework on long sequences by achieving state-of-the-art performances on multiple benchmark datasets. Our source code is available at: this https URL.

Attribute mapping and autoencoder neural network based matrix factorization initialization for recommendation systems

Abstract: The recommendation algorithm is attracting increasing attention in analyzing big data. Matrix factorization (MF) is one of the recommendation methods and Singular Value Decomposition (SVD) is the most popular matrix factorization method. However, the existing SVD methods usually initialize user and item feature randomly, not fully utilize the information of the data, so require plenty of experiments to determine feature matrix dimension, with low convergence efficiency and low accuracy. This paper presents a hybrid initialization method based on attribute mapping and autoencoder neural network to solve these problems, which consists of three parts: (1) use the number of item attribute types to determine feature matrix dimension in order to avoid multiple experiments to select the optimal dimension value; (2) use items’ attributes to initialize the item feature matrix in SVD＋＋, and use an attribute mapping mechanism to get an item feature vector by fitting the rating matrix to accelerate the convergence; (3) adopt the autoencoder neural network to reduce feature dimension and obtain item latent features for initializing SVD＋＋. The experimental results show that our methods achieve better performance than SVD＋＋ random initialization and also be adopted to other matrix factorization methods.

Generalized Tensor Function via the Tensor Singular Value Decomposition based on the T-Product

Abstract: In this paper, we present the definition of generalized tensor function according to the tensor singular value decomposition (T-SVD) via the tensor T-product. Also, we introduce the compact singular value decomposition (T-CSVD) of tensors via the T-product, from which the projection operators and Moore Penrose inverse of tensors are also obtained. We also establish the Cauchy integral formula for tensors by using the partial isometry tensors and applied it into the solution of tensor equations. Then we establish the generalized tensor power and the Taylor expansion of tensors. Explicit generalized tensor functions are also listed. We define the tensor bilinear and sesquilinear forms and proposed theorems on structures preserved by generalized tensor functions. For complex tensors, we established an isomorphism between complex tensors and real tensors. In the last part of our paper, we find that the block circulant operator established an isomorphism between tensors and matrices. This isomorphism is used to prove the F-stochastic structure is invariant under generalized tensor functions. The concept of invariant tensor cones is also raised.

On Universal Features for High-Dimensional Learning and Inference.

Abstract: We consider the problem of identifying universal low-dimensional features from high-dimensional data for inference tasks in settings involving learning. For such problems, we introduce natural notions of universality and we show a local equivalence among them. Our analysis is naturally expressed via information geometry, and represents a conceptually and computationally useful analysis. The development reveals the complementary roles of the singular value decomposition, Hirschfeld-Gebelein-Renyi maximal correlation, the canonical correlation and principle component analyses of Hotelling and Pearson, Tishby's information bottleneck, Wyner's common information, Ky Fan $k$-norms, and Brieman and Friedman's alternating conditional expectations algorithm. We further illustrate how this framework facilitates understanding and optimizing aspects of learning systems, including multinomial logistic (softmax) regression and the associated neural network architecture, matrix factorization methods for collaborative filtering and other applications, rank-constrained multivariate linear regression, and forms of semi-supervised learning.