This survey provides an overview of higher-order tensor decompositions, their applications, and available software. A tensor is a multidimensional or $N$-way array. Decompositions of higher-order tensors (i.e., $N$-way arrays with $N \geq 3$) have applications in psycho-metrics, chemometrics, signal processing, numerical linear algebra, computer vision, numerical analysis, data mining, neuroscience, graph analysis, and elsewhere. Two particular tensor decompositions can be considered to be higher-order extensions of the matrix singular value decomposition: CANDECOMP/PARAFAC (CP) decomposes a tensor as a sum of rank-one tensors, and the Tucker decomposition is a higher-order form of principal component analysis. There are many other tensor decompositions, including INDSCAL, PARAFAC2, CANDELINC, DEDICOM, and PARATUCK2 as well as nonnegative variants of all of the above. The N-way Toolbox, Tensor Toolbox, and Multilinear Engine are examples of software packages for working with tensors.

/pdf/tensor-decompositions-and-applications-46vl2ih5gn.pdf

Tensor Decompositions and Applications

The Theory of Matrices. By F R. Gantmacher. Two volumes, pp. 374 and 276. 1959. (Translated from the Russian by K. A. Hirsch; Chelsea Publishing Company, New York)

Tensors or multiway arrays are functions of three or more indices $(i,j,k,\ldots)$ —similar to matrices (two-way arrays), which are functions of two indices $(r,c)$ for (row, column). Tensors have a rich history, stretching over almost a century, and touching upon numerous disciplines; but they have only recently become ubiquitous in signal and data analytics at the confluence of signal processing, statistics, data mining, and machine learning. This overview article aims to provide a good starting point for researchers and practitioners interested in learning about and working with tensors. As such, it focuses on fundamentals and motivation (using various application examples), aiming to strike an appropriate balance of breadth and depth that will enable someone having taken first graduate courses in matrix algebra and probability to get started doing research and/or developing tensor algorithms and software. Some background in applied optimization is useful but not strictly required. The material covered includes tensor rank and rank decomposition; basic tensor factorization models and their relationships and properties (including fairly good coverage of identifiability); broad coverage of algorithms ranging from alternating optimization to stochastic gradient; statistical performance analysis; and applications ranging from source separation to collaborative filtering, mixture and topic modeling, classification, and multilinear subspace learning.

Tensor Decomposition for Signal Processing and Machine Learning

The widespread use of multisensor technology and the emergence of big data sets have highlighted the limitations of standard flat-view matrix models and the necessity to move toward more versatile data analysis tools. We show that higher-order tensors (i.e., multiway arrays) enable such a fundamental paradigm shift toward models that are essentially polynomial, the uniqueness of which, unlike the matrix methods, is guaranteed under very mild and natural conditions. Benefiting from the power of multilinear algebra as their mathematical backbone, data analysis techniques using tensor decompositions are shown to have great flexibility in the choice of constraints which match data properties and extract more general latent components in the data than matrix-based methods.

/pdf/tensor-decompositions-for-signal-processing-applications-2cvd1sgtij.pdf

Tensor Decompositions for Signal Processing Applications: From two-way to multiway component analysis

We prove that multilinear (tensor) analogues of many efficiently computable problems in numerical linear algebra are NP-hard. Our list includes: determining the feasibility of a system of bilinear equations, deciding whether a 3-tensor possesses a given eigenvalue, singular value, or spectral norm; approximating an eigenvalue, eigenvector, singular vector, or the spectral norm; and determining the rank or best rank-1 approximation of a 3-tensor. Furthermore, we show that restricting these problems to symmetric tensors does not alleviate their NP-hardness. We also explain how deciding nonnegative definiteness of a symmetric 4-tensor is NP-hard and how computing the combinatorial hyperdeterminant is NP-, #P-, and VNP-hard.

/pdf/most-tensor-problems-are-np-hard-cqgnykhb6t.pdf

Most Tensor Problems Are NP-Hard

On the Complexity of the Permanent in Various Computational Models

We make an in-depth study of the known border rank (i.e., approximate) algorithms for the matrix multiplication tensor encoding the multiplication of an n × 2 matrix by a 2 × 2 matrix.

On the geometry of border rank algorithms for n x 2 by 2 x 2 matrix multiplication

We introduce vector bundle techniques for finding equations of secant varieties. A test is established that determines when a secant variety is an irreducible component of the zero set of the equations found. We also prove an induction theorem for varieties that are not weakly defective, that allows one to conclude that the zero set of the equations found for s_{r-1}(X) have s_{r-1}(X) as an irreducible component when s_r(X) is an irreducible component of the equations found for it. The techniques are illustrated with examples of homogeneous varieties. We give an algorithm to decompose a general ternary quintic as the sum of seven fifth powers.

Equations for secant varieties via vector bundles

We answer a question, posed implicitly by Coppersmith-Winogrand and Buergisser et. al. and explicitly by Blaeser, showing the border rank of the Kronecker square of the little Coppersmith-Winograd tensor is the square of the border rank of the tensor for all q>2, a negative result for complexity theory. We further show that when q>4, the analogous result holds for the Kronecker cube. In the positive direction, we enlarge the list of explicit tensors potentially useful for the laser method. We observe that a well-known tensor, the 3x3 determinant polynomial regarded as a tensor, could potentially be used in the laser method to prove the exponent of matrix multiplication is two. Because of this, we prove new upper bounds on its Waring rank and rank (both 18), border rank and Waring border rank (both 17), which, in addition to being promising for the laser method, are of interest in their own right. We discuss "skew" cousins of the little Coppersmith-Winograd tensor and indicate whey they may be useful for the laser method. We establish general results regarding border ranks of Kronecker powers of tensors, and make a detailed study of Kronecker squares of tensors of dimensions (3,3,3). In particular we show numerically that for generic tensors in this space, the rank and border rank are strictly sub-multiplicative.

Kronecker Powers of Tensors and Strassen's Laser Method.

The minimal free resolution of the Jacobian ideals of the determinant polynomial were computed by Lascoux, and it is an active area of research to understand the Jacobian ideals of the permanent. As a step in this direction we compute several new cases and completely determine the linear strands of the minimal free resolutions of the ideals generated by sub-permanents. Our motivation is an exploration of the utility and limits of the method of shifted partial derivatives introduced by Kayal and Gupta-Kamath-Kayal-Saptharishi. The method of shifted partial derivatives amounts to computing Hilbert functions of Jacobian ideals, and the Hilbert functions are in turn the Euler characteristics of the minimal free resolutions of the Jacobian ideals. We compute several such Hilbert functions relevant for complexity theory. We show that the method of shifted partial derivatives alone cannot prove the size m padded permanent cannotbe realized inside the orbit closure of the size n determinant when m< 1.5n^2.

Joseph M. Landsberg

Papers

On the Complexity of the Permanent in Various Computational Models

On the geometry of border rank algorithms for n x 2 by 2 x 2 matrix multiplication

Equations for secant varieties via vector bundles

Kronecker Powers of Tensors and Strassen's Laser Method.

On minimal free resolutions and the method of shifted partial derivatives in complexity theory.