Showing papers on "Singular value decomposition published in 2018"

Robust Subspace Learning: Robust PCA, Robust Subspace Tracking, and Robust Subspace Recovery

Abstract: Principal component analysis (PCA) is one of the most widely used dimension reduction techniques. A related easier problem is termed subspace learning or subspace estimation. Given relatively clean data, both are easily solved via singular value decomposition (SVD). The problem of subspace learning or PCA in the presence of outliers is called robust subspace learning (RSL) or robust PCA (RPCA). For long data sequences, if one tries to use a single lower-dimensional subspace to represent the data, the required subspace dimension may end up being quite large. For such data, a better model is to assume that it lies in a low-dimensional subspace that can change over time, albeit gradually. The problem of tracking such data (and the subspaces) while being robust to outliers is called robust subspace tracking (RST). This article provides a magazine-style overview of the entire field of RSL and tracking.

On Unifying Multi-view Self-Representations for Clustering by Tensor Multi-rank Minimization

Abstract: In this paper, we address the multi-view subspace clustering problem. Our method utilizes the circulant algebra for tensor, which is constructed by stacking the subspace representation matrices of different views and then rotating, to capture the low rank tensor subspace so that the refinement of the view-specific subspaces can be achieved, as well as the high order correlations underlying multi-view data can be explored. By introducing a recently proposed tensor factorization, namely tensor-Singular Value Decomposition (t-SVD) (Kilmer et al. in SIAM J Matrix Anal Appl 34(1):148–172, 2013), we can impose a new type of low-rank tensor constraint on the rotated tensor to ensure the consensus among multiple views. Different from traditional unfolding based tensor norm, this low-rank tensor constraint has optimality properties similar to that of matrix rank derived from SVD, so the complementary information can be explored and propagated among all the views more thoroughly and effectively. The established model, called t-SVD based Multi-view Subspace Clustering (t-SVD-MSC), falls into the applicable scope of augmented Lagrangian method, and its minimization problem can be efficiently solved with theoretical convergence guarantee and relatively low computational complexity. Extensive experimental testing on eight challenging image datasets shows that the proposed method has achieved highly competent objective performance compared to several state-of-the-art multi-view clustering methods.

Kronecker-Basis-Representation Based Tensor Sparsity and Its Applications to Tensor Recovery

Abstract: As a promising way for analyzing data, sparse modeling has achieved great success throughout science and engineering. It is well known that the sparsity/low-rank of a vector/matrix can be rationally measured by nonzero-entries-number ( $l_0$ norm)/nonzero- singular-values-number (rank), respectively. However, data from real applications are often generated by the interaction of multiple factors, which obviously cannot be sufficiently represented by a vector/matrix, while a high order tensor is expected to provide more faithful representation to deliver the intrinsic structure underlying such data ensembles. Unlike the vector/matrix case, constructing a rational high order sparsity measure for tensor is a relatively harder task. To this aim, in this paper we propose a measure for tensor sparsity, called Kronecker-basis-representation based tensor sparsity measure (KBR briefly), which encodes both sparsity insights delivered by Tucker and CANDECOMP/PARAFAC (CP) low-rank decompositions for a general tensor. Then we study the KBR regularization minimization (KBRM) problem, and design an effective ADMM algorithm for solving it, where each involved parameter can be updated with closed-form equations. Such an efficient solver makes it possible to extend KBR to various tasks like tensor completion and tensor robust principal component analysis. A series of experiments, including multispectral image (MSI) denoising, MSI completion and background subtraction, substantiate the superiority of the proposed methods beyond state-of-the-arts.

Tensor Factorization for Low-Rank Tensor Completion

Abstract: Recently, a tensor nuclear norm (TNN) based method was proposed to solve the tensor completion problem, which has achieved state-of-the-art performance on image and video inpainting tasks. However, it requires computing tensor singular value decomposition (t-SVD), which costs much computation and thus cannot efficiently handle tensor data, due to its natural large scale. Motivated by TNN, we propose a novel low-rank tensor factorization method for efficiently solving the 3-way tensor completion problem. Our method preserves the low-rank structure of a tensor by factorizing it into the product of two tensors of smaller sizes. In the optimization process, our method only needs to update two smaller tensors, which can be more efficiently conducted than computing t-SVD. Furthermore, we prove that the proposed alternating minimization algorithm can converge to a Karush–Kuhn–Tucker point. Experimental results on the synthetic data recovery, image and video inpainting tasks clearly demonstrate the superior performance and efficiency of our developed method over state-of-the-arts including the TNN and matricization methods.

ATOMO: Communication-efficient Learning via Atomic Sparsification

Abstract: Distributed model training suffers from communication overheads due to frequent gradient updates transmitted between compute nodes. To mitigate these overheads, several studies propose the use of sparsified stochastic gradients. We argue that these are facets of a general sparsification method that can operate on any possible atomic decomposition. Notable examples include element-wise, singular value, and Fourier decompositions. We present ATOMO, a general framework for atomic sparsification of stochastic gradients. Given a gradient, an atomic decomposition, and a sparsity budget, ATOMO gives a random unbiased sparsification of the atoms minimizing variance. We show that recent methods such as QSGD and TernGrad are special cases of ATOMO, and that sparsifiying the singular value decomposition of neural networks gradients, rather than their coordinates, can lead to significantly faster distributed training.

Towards Faster Training of Global Covariance Pooling Networks by Iterative Matrix Square Root Normalization

Abstract: Global covariance pooling in convolutional neural networks has achieved impressive improvement over the classical first-order pooling. Recent works have shown matrix square root normalization plays a central role in achieving state-of-the-art performance. However, existing methods depend heavily on eigendecomposition (EIG) or singular value decomposition (SVD), suffering from inefficient training due to limited support of EIG and SVD on GPU. Towards addressing this problem, we propose an iterative matrix square root normalization method for fast end-to-end training of global covariance pooling networks. At the core of our method is a meta-layer designed with loop-embedded directed graph structure. The meta-layer consists of three consecutive nonlinear structured layers, which perform pre-normalization, coupled matrix iteration and post-compensation, respectively. Our method is much faster than EIG or SVD based ones, since it involves only matrix multiplications, suitable for parallel implementation on GPU. Moreover, the proposed network with ResNet architecture can converge in much less epochs, further accelerating network training. On large-scale ImageNet, we achieve competitive performance superior to existing counterparts. By finetuning our models pre-trained on ImageNet, we establish state-of-the-art results on three challenging fine-grained benchmarks. The source code and network models will be available at http://www.peihuali.org/iSQRT-COV.

Adaptive Spatiotemporal SVD Clutter Filtering for Ultrafast Doppler Imaging Using Similarity of Spatial Singular Vectors

Abstract: Singular value decomposition of ultrafast imaging ultrasonic data sets has recently been shown to build a vector basis far more adapted to the discrimination of tissue and blood flow than the classical Fourier basis, improving by large factor clutter filtering and blood flow estimation. However, the question of optimally estimating the boundary between the tissue subspace and the blood flow subspace remained unanswered. Here, we introduce an efficient estimator for automatic thresholding of subspaces and compare it to an exhaustive list of thirteen estimators that could achieve this task based on the main characteristics of the singular components, namely the singular values, the temporal singular vectors, and the spatial singular vectors. The performance of those fourteen estimators was tested in vitro in a large set of controlled experimental conditions with different tissue motion and flow speeds on a phantom. The estimator based on the degree of resemblance of spatial singular vectors outperformed all others. Apart from solving the thresholding problem, the additional benefit with this estimator was its denoising capabilities, strongly increasing the contrast to noise ratio and lowering the noise floor by at least 5 dB. This confirms that, contrary to conventional clutter filtering techniques that are almost exclusively based on temporal characteristics, efficient clutter filtering of ultrafast Doppler imaging cannot overlook space. Finally, this estimator was applied in vivo on various organs (human brain, kidney, carotid, and thyroid) and showed efficient clutter filtering and noise suppression, improving largely the dynamic range of the obtained ultrafast power Doppler images.

Efficient non-Markovian quantum dynamics using time-evolving matrix product operators.

Abstract: In order to model realistic quantum devices it is necessary to simulate quantum systems strongly coupled to their environment. To date, most understanding of open quantum systems is restricted either to weak system–bath couplings or to special cases where specific numerical techniques become effective. Here we present a general and yet exact numerical approach that efficiently describes the time evolution of a quantum system coupled to a non-Markovian harmonic environment. Our method relies on expressing the system state and its propagator as a matrix product state and operator, respectively, and using a singular value decomposition to compress the description of the state as time evolves. We demonstrate the power and flexibility of our approach by numerically identifying the localisation transition of the Ohmic spin-boson model, and considering a model with widely separated environmental timescales arising for a pair of spins embedded in a common environment.

Tensor SVD: Statistical and Computational Limits

Abstract: In this paper, we propose a general framework for tensor singular value decomposition (tensor singular value decomposition (SVD)), which focuses on the methodology and theory for extracting the hidden low-rank structure from high-dimensional tensor data. Comprehensive results are developed on both the statistical and computational limits for tensor SVD. This problem exhibits three different phases according to the signal-to-noise ratio (SNR). In particular, with strong SNR, we show that the classical higher-order orthogonal iteration achieves the minimax optimal rate of convergence in estimation; with weak SNR, the information-theoretical lower bound implies that it is impossible to have consistent estimation in general; with moderate SNR, we show that the non-convex maximum likelihood estimation provides optimal solution, but with NP-hard computational cost; moreover, under the hardness hypothesis of hypergraphic planted clique detection, there are no polynomial-time algorithms performing consistently in general.

Rate-optimal perturbation bounds for singular subspaces with applications to high-dimensional statistics

Abstract: Perturbation bounds for singular spaces, in particular Wedin’s $\mathop{\mathrm{sin}} olimits \Theta$ theorem, are a fundamental tool in many fields including high-dimensional statistics, machine learning and applied mathematics. In this paper, we establish separate perturbation bounds, measured in both spectral and Frobenius $\mathop{\mathrm{sin}} olimits \Theta$ distances, for the left and right singular subspaces. Lower bounds, which show that the individual perturbation bounds are rate-optimal, are also given. The new perturbation bounds are applicable to a wide range of problems. In this paper, we consider in detail applications to low-rank matrix denoising and singular space estimation, high-dimensional clustering and canonical correlation analysis (CCA). In particular, separate matching upper and lower bounds are obtained for estimating the left and right singular spaces. To the best of our knowledge, this is the first result that gives different optimal rates for the left and right singular spaces under the same perturbation.

ATOMO: Communication-efficient Learning via Atomic Sparsification

Abstract: Distributed model training suffers from communication overheads due to frequent gradient updates transmitted between compute nodes. To mitigate these overheads, several studies propose the use of sparsified stochastic gradients. We argue that these are facets of a general sparsification method that can operate on any possible atomic decomposition. Notable examples include element-wise, singular value, and Fourier decompositions. We present ATOMO, a general framework for atomic sparsification of stochastic gradients. Given a gradient, an atomic decomposition, and a sparsity budget, ATOMO gives a random unbiased sparsification of the atoms minimizing variance. We show that recent methods such as QSGD and TernGrad are special cases of ATOMO and that sparsifiying the singular value decomposition of neural networks gradients, rather than their coordinates, can lead to significantly faster distributed training.

Low Rank Alternating Direction Method of Multipliers Reconstruction for MR Fingerprinting

Abstract: The proposed reconstruction framework addresses the reconstruction accuracy, noise propagation and computation time for magnetic resonance fingerprinting. Based on a singular value decomposition of the signal evolution, magnetic resonance fingerprinting is formulated as a low rank (LR) inverse problem in which one image is reconstructed for each singular value under consideration. This LR approximation of the signal evolution reduces the computational burden by reducing the number of Fourier transformations. Also, the LR approximation improves the conditioning of the problem, which is further improved by extending the LR inverse problem to an augmented Lagrangian that is solved by the alternating direction method of multipliers. The root mean square error and the noise propagation are analyzed in simulations. For verification, in vivo examples are provided. The proposed LR alternating direction method of multipliers approach shows a reduced root mean square error compared to the original fingerprinting reconstruction, to a LR approximation alone and to an alternating direction method of multipliers approach without a LR approximation. Incorporating sensitivity encoding allows for further artifact reduction. The proposed reconstruction provides robust convergence, reduced computational burden and improved image quality compared to other magnetic resonance fingerprinting reconstruction approaches evaluated in this study. Magn Reson Med 79:83-96, 2018. © 2017 International Society for Magnetic Resonance in Medicine.

Improved Robust Tensor Principal Component Analysis via Low-Rank Core Matrix

Abstract: Robust principal component analysis (RPCA) has been widely used for many data analysis problems in matrix data. Robust tensor principal component analysis (RTPCA) aims to extract the low rank and sparse components of multidimensional data, which is a generation of RPCA. The current RTPCA methods are directly based on tensor singular value decomposition (t-SVD), which is a new tensor decomposition method similar to singular value decomposition (SVD) in matrices. These methods focus on utilizing different sparse constraints for real applications and make less analysis for tensor nuclear norm (TNN) defined in t-SVD. However, we find low-rank structure still exists in the core tensor and existing methods can not fully extract the low-rank structure of tensor data. To further exploit the low-rank structures in multiway data, we extract low-rank component for the core matrix whose entries are from the diagonal elements of the core tensor. Based on this idea, we have defined a new TNN that extends TNN with core matrix and propose a creative algorithm to deal with RTPCA problems. The results of numerical experiments show that the proposed method outperforms state-of-the-art methods in terms of both accuracy and computational complexity.

Spatial-temporal traffic speed patterns discovery and incomplete data recovery via SVD-combined tensor decomposition

Abstract: Missing data is an inevitable and ubiquitous problem in data-driven intelligent transportation systems. While there are several studies on the missing traffic data recovery in the last decade, it is still an open issue of making full use of spatial-temporal traffic patterns to improve recovery performance. In this paper, due to the multi-dimensional nature of traffic speed data, we treat missing data recovery as the problem of tensor completion, a three-procedure framework based on Tucker decomposition is proposed to accomplish the recovery task by discovering spatial-temporal patterns and underlying structure from incomplete data. Specifically, in the missing data initialization, intrinsic multi-mode biases based traffic pattern is extracted to perform a robust recovery. Thereby, the truncated singular value decomposition (SVD) is introduced to capture main latent features along each dimension. Finally, applying these latent features, the missing data is eventually estimated by the SVD-combined tensor decomposition (STD). Empirically, relying on the large-scale traffic speed data collected from 214 road segments within two months at 10-min interval, our experiment covers two missing scenarios – element-like random missing and fiber-like random missing. The impacts of different initialization strategies for tensor decomposition are evaluated. From numerical analysis, a sensitivity-driven rank selection can not only choose an appropriate core tensor size but also determine how much features we actually need. By comparison with two baseline tensor decomposition models, our method is shown to successfully recover missing data with the highest accuracy as the missing rate ranges from 20% to 80% under two missing scenarios. Moreover, the results have also indicated that an optimal initialization for tensor decomposition could suggest a better performance.

Patch-Based Image Inpainting via Two-Stage Low Rank Approximation

Abstract: To recover the corrupted pixels, traditional inpainting methods based on low-rank priors generally need to solve a convex optimization problem by an iterative singular value shrinkage algorithm. In this paper, we propose a simple method for image inpainting using low rank approximation, which avoids the time-consuming iterative shrinkage. Specifically, if similar patches of a corrupted image are identified and reshaped as vectors, then a patch matrix can be constructed by collecting these similar patch-vectors. Due to its columns being highly linearly correlated, this patch matrix is low-rank. Instead of using an iterative singular value shrinkage scheme, the proposed method utilizes low rank approximation with truncated singular values to derive a closed-form estimate for each patch matrix. Depending upon an observation that there exists a distinct gap in the singular spectrum of patch matrix, the rank of each patch matrix is empirically determined by a heuristic procedure. Inspired by the inpainting algorithms with component decomposition, a two-stage low rank approximation (TSLRA) scheme is designed to recover image structures and refine texture details of corrupted images. Experimental results on various inpainting tasks demonstrate that the proposed method is comparable and even superior to some state-of-the-art inpainting algorithms.

Quantum singular-value decomposition of nonsparse low-rank matrices

Abstract: We present a method to exponentiate nonsparse indefinite low-rank matrices on a quantum computer. Given access to the elements of the matrix, our method allows one to determine the singular values and their associated singular vectors in time exponentially faster in the dimension of the matrix than known classical algorithms. The method extends to non-Hermitian and nonsquare matrices via matrix embedding. Moreover, our method preserves the phase relations between the singular spaces allowing for efficient algorithms that require operating on the entire singular-value decomposition of a matrix. As an example of such an algorithm, we discuss the Procrustes problem of finding a closest isometry to a given matrix.

Self-supervised Knowledge Distillation Using Singular Value Decomposition

Abstract: To solve deep neural network (DNN)’s huge training dataset and its high computation issue, so-called teacher-student (T-S) DNN which transfers the knowledge of T-DNN to S-DNN has been proposed. However, the existing T-S-DNN has limited range of use, and the knowledge of T-DNN is insufficiently transferred to S-DNN. To improve the quality of the transferred knowledge from T-DNN, we propose a new knowledge distillation using singular value decomposition (SVD). In addition, we define a knowledge transfer as a self-supervised task and suggest a way to continuously receive information from T-DNN. Simulation results show that a S-DNN with a computational cost of 1/5 of the T-DNN can be up to 1.1% better than the T-DNN in terms of classification accuracy. Also assuming the same computational cost, our S-DNN outperforms the S-DNN driven by the state-of-the-art distillation with a performance advantage of 1.79%. code is available on https://github.com/sseung0703/SSKD_SVD.

Essential Tensor Learning for Multi-view Spectral Clustering.

Abstract: Multi-view clustering attracts much attention recently, which aims to take advantage of multi-view information to improve the performance of clustering. However, most recent work mainly focus on self-representation based subspace clustering, which is of high computation complexity. In this paper, we focus on the Markov chain based spectral clustering method and propose a novel essential tensor learning method to explore the high order correlations for multi-view representation. We first construct a tensor based on multi-view transition probability matrices of the Markov chain. By incorporating the idea from robust principle component analysis, tensor singular value decomposition (t-SVD) based tensor nuclear norm is imposed to preserve the low-rank property of the essential tensor, which can well capture the principle information from multiple views. We also employ the tensor rotation operator for this task to better investigate the relationship among views as well as reduce the computation complexity. The proposed method can be efficiently optimized by the alternating direction method of multipliers~(ADMM). Extensive experiments on six real world datasets corresponding to five different applications show that our method achieves superior performance over other state-of-the-art methods.

Grassmann Pooling as Compact Homogeneous Bilinear Pooling for Fine-Grained Visual Classification

Abstract: Designing discriminative and invariant features is the key to visual recognition. Recently, the bilinear pooled feature matrix of Convolutional Neural Network (CNN) has shown to achieve state-of-the-art performance on a range of fine-grained visual recognition tasks. The bilinear feature matrix collects second-order statistics and is closely related to the covariance matrix descriptor. However, the bilinear feature could suffer from the visual burstiness phenomenon similar to other visual representations such as VLAD and Fisher Vector. The reason is that the bilinear feature matrix is sensitive to the magnitudes and correlations of local CNN feature elements which can be measured by its singular values. On the other hand, the singular vectors are more invariant and reasonable to be adopted as the feature representation. Motivated by this point, we advocate an alternative pooling method which transforms the CNN feature matrix to an orthonormal matrix consists of its principal singular vectors. Geometrically, such orthonormal matrix lies on the Grassmann manifold, a Riemannian manifold whose points represent subspaces of the Euclidean space. Similarity measurement of images reduces to comparing the principal angles between these “homogeneous” subspaces and thus is independent of the magnitudes and correlations of local CNN activations. In particular, we demonstrate that the projection distance on the Grassmann manifold deduces a bilinear feature mapping without explicitly computing the bilinear feature matrix, which enables a very compact feature and classifier representation. Experimental results show that our method achieves an excellent balance of model complexity and accuracy on a variety of fine-grained image classification datasets.

A novel hybrid of DCT and SVD in DWT domain for robust and invisible blind image watermarking with optimal embedding strength

Abstract: To optimize the tradeoff between imperceptibility and robustness properties, this paper proposes a robust and invisible blind image watermarking scheme based on a new combination of discrete cosine transform (DCT) and singular value decomposition (SVD) in discrete wavelet transform (DWT) domain using least-square curve fitting and logistic chaotic map. Firstly cover image is decomposed into four subbands using DWT and the low frequency subband LL is partitioned into non-overlapping blocks. Then DCT is applied to each block and several particular middle frequency DCT coefficients are extracted to form a modulation matrix, which is used to embed watermark signal by modifying its largest singular values in SVD domain. Optimal embedding strength for a specific cover image is obtained from an estimation based on least-square curve fitting and provides a good compromise between transparency and robustness of watermarking scheme. The security of the watermarking scheme is ensured by logistic chaotic map. Experimental results demonstrate the better effectiveness of the proposed watermarking scheme in the perceptual quality and the ability of resisting to conventional signal processing and geometric attacks, in comparison with the related existing methods.

Planetary Gears Feature Extraction and Fault Diagnosis Method Based on VMD and CNN

Abstract: Given local weak feature information, a novel feature extraction and fault diagnosis method for planetary gears based on variational mode decomposition (VMD), singular value decomposition (SVD), and convolutional neural network (CNN) is proposed. VMD was used to decompose the original vibration signal to mode components. The mode matrix was partitioned into a number of submatrices and local feature information contained in each submatrix was extracted as a singular value vector using SVD. The singular value vector matrix corresponding to the current fault state was constructed according to the location of each submatrix. Finally, by training a CNN using singular value vector matrices as inputs, planetary gear fault state identification and classification was achieved. The experimental results confirm that the proposed method can successfully extract local weak feature information and accurately identify different faults. The singular value vector matrices of different fault states have a distinct difference in element size and waveform. The VMD-based partition extraction method is better than ensemble empirical mode decomposition (EEMD), resulting in a higher CNN total recognition rate of 100% with fewer training times (14 times). Further analysis demonstrated that the method can also be applied to the degradation recognition of planetary gears. Thus, the proposed method is an effective feature extraction and fault diagnosis technique for planetary gears.

The Singular Value Decomposition: Anatomy of Optimizing an Algorithm for Extreme Scale

Abstract: The computation of the singular value decomposition, or SVD, has a long history with many improvements over the years, both in its implementations and algorithmically. Here, we survey the evolution...

numpywren: serverless linear algebra

Abstract: Linear algebra operations are widely used in scientific computing and machine learning applications. However, it is challenging for scientists and data analysts to run linear algebra at scales beyond a single machine. Traditional approaches either require access to supercomputing clusters, or impose configuration and cluster management challenges. In this paper we show how the disaggregation of storage and compute resources in so-called "serverless" environments, combined with compute-intensive workload characteristics, can be exploited to achieve elastic scalability and ease of management. We present numpywren, a system for linear algebra built on a serverless architecture. We also introduce LAmbdaPACK, a domain-specific language designed to implement highly parallel linear algebra algorithms in a serverless setting. We show that, for certain linear algebra algorithms such as matrix multiply, singular value decomposition, and Cholesky decomposition, numpywren's performance (completion time) is within 33% of ScaLAPACK, and its compute efficiency (total CPU-hours) is up to 240% better due to elasticity, while providing an easier to use interface and better fault tolerance. At the same time, we show that the inability of serverless runtimes to exploit locality across the cores in a machine fundamentally limits their network efficiency, which limits performance on other algorithms such as QR factorization. This highlights how cloud providers could better support these types of computations through small changes in their infrastructure.

Self-supervised Knowledge Distillation Using Singular Value Decomposition

Abstract: To solve deep neural network (DNN)'s huge training dataset and its high computation issue, so-called teacher-student (T-S) DNN which transfers the knowledge of T-DNN to S-DNN has been proposed. However, the existing T-S-DNN has limited range of use, and the knowledge of T-DNN is insufficiently transferred to S-DNN. To improve the quality of the transferred knowledge from T-DNN, we propose a new knowledge distillation using singular value decomposition (SVD). In addition, we define a knowledge transfer as a self-supervised task and suggest a way to continuously receive information from T-DNN. Simulation results show that a S-DNN with a computational cost of 1/5 of the T-DNN can be up to 1.1\% better than the T-DNN in terms of classification accuracy. Also assuming the same computational cost, our S-DNN outperforms the S-DNN driven by the state-of-the-art distillation with a performance advantage of 1.79\%. code is available on this https URL\_SVD.

A secure blind discrete wavelet transform based watermarking scheme using two-level singular value decomposition

Abstract: In this paper, a secure blind image watermarking scheme is proposed to improve efficiency of hybrid Discrete Wavelet Transform (DWT) and Singular Value Decomposition (SVD) based schemes. The main contribution of this research is using two-level singular value decomposition in hybrid DWT + SVD watermarking schemes results to high improvement in both imperceptibility and robustness. Also, security is ensured in this scheme by proposing a two-level authentication system for watermark extraction in which not only the false positive problem is eliminated, when the attacks are very severe, the false negative effect is also detected and prevented. In the proposed scheme, the cover image is initially transformed to DWT and the high frequency sub bands are selected. Subsequently, High (HH) sub band is divided into 8 × 8 non overlapping blocks and the SVD transform is applied to each of them. In SVD transform, the highest energy compaction is in the first row and the first column of the matrix of singular values. These singular values for each block will be collected in a separate matrix and then the second SVD transform will be exerted on the produced matrix. The singular values of the watermark image after exerting one level DWT, is embedded into the singular values of this matrix. Experimental result shows when the scheme is subjected to ten most common types of signal processing and geometric attacks, using the second level of SVD makes it highly robust and imperceptible while it is independent from type of image. So, the proposed scheme is tested and certified for both medical and normal images. This scheme is compared to a range of hybrid schemes using DWT and one level SVD, and experimental results show high improvements in imperceptibility with the average of 46.6950 db and robustness more than 99% and proven efficiency in security.

Heteroskedastic PCA: Algorithm, Optimality, and Applications

Abstract: A general framework for principal component analysis (PCA) in the presence of heteroskedastic noise is introduced. We propose an algorithm called HeteroPCA, which involves iteratively imputing the diagonal entries of the sample covariance matrix to remove estimation bias due to heteroskedasticity. This procedure is computationally efficient and provably optimal under the generalized spiked covariance model. A key technical step is a deterministic robust perturbation analysis on singular subspaces, which can be of independent interest. The effectiveness of the proposed algorithm is demonstrated in a suite of problems in high-dimensional statistics, including singular value decomposition (SVD) under heteroskedastic noise, Poisson PCA, and SVD for heteroskedastic and incomplete data.

Fast Randomized Singular Value Thresholding for Low-Rank Optimization

Abstract: Rank minimization can be converted into tractable surrogate problems, such as Nuclear Norm Minimization (NNM) and Weighted NNM (WNNM). The problems related to NNM, or WNNM, can be solved iteratively by applying a closed-form proximal operator, called Singular Value Thresholding (SVT), or Weighted SVT, but they suffer from high computational cost of Singular Value Decomposition (SVD) at each iteration. We propose a fast and accurate approximation method for SVT, that we call fast randomized SVT (FRSVT), with which we avoid direct computation of SVD. The key idea is to extract an approximate basis for the range of the matrix from its compressed matrix. Given the basis, we compute partial singular values of the original matrix from the small factored matrix. In addition, by developping a range propagation method, our method further speeds up the extraction of approximate basis at each iteration. Our theoretical analysis shows the relationship between the approximation bound of SVD and its effect to NNM via SVT. Along with the analysis, our empirical results quantitatively and qualitatively show that our approximation rarely harms the convergence of the host algorithms. We assess the efficiency and accuracy of the proposed method on various computer vision problems, e.g., subspace clustering, weather artifact removal, and simultaneous multi-image alignment and rectification.

A randomized tensor singular value decomposition based on the t-product

Abstract: The tensor Singular Value Decomposition (t-SVD) for third order tensors that was proposed by Kilmer and Martin~\cite{2011kilmer} has been applied successfully in many fields, such as computed tomography, facial recognition, and video completion. In this paper, we propose a method that extends a well-known randomized matrix method to the t-SVD. This method can produce a factorization with similar properties to the t-SVD, but is more computationally efficient on very large datasets. We present details of the algorithm, theoretical results, and provide numerical results that show the promise of our approach for compressing and analyzing datasets. We also present an improved analysis of the randomized subspace iteration for matrices, which may be of independent interest to the scientific community.