This survey provides an overview of higher-order tensor decompositions, their applications, and available software. A tensor is a multidimensional or $N$-way array. Decompositions of higher-order tensors (i.e., $N$-way arrays with $N \geq 3$) have applications in psycho-metrics, chemometrics, signal processing, numerical linear algebra, computer vision, numerical analysis, data mining, neuroscience, graph analysis, and elsewhere. Two particular tensor decompositions can be considered to be higher-order extensions of the matrix singular value decomposition: CANDECOMP/PARAFAC (CP) decomposes a tensor as a sum of rank-one tensors, and the Tucker decomposition is a higher-order form of principal component analysis. There are many other tensor decompositions, including INDSCAL, PARAFAC2, CANDELINC, DEDICOM, and PARATUCK2 as well as nonnegative variants of all of the above. The N-way Toolbox, Tensor Toolbox, and Multilinear Engine are examples of software packages for working with tensors.

/pdf/tensor-decompositions-and-applications-46vl2ih5gn.pdf

Tensor Decompositions and Applications

The Theory of Matrices. By F R. Gantmacher. Two volumes, pp. 374 and 276. 1959. (Translated from the Russian by K. A. Hirsch; Chelsea Publishing Company, New York)

Tensors or multiway arrays are functions of three or more indices $(i,j,k,\ldots)$ —similar to matrices (two-way arrays), which are functions of two indices $(r,c)$ for (row, column). Tensors have a rich history, stretching over almost a century, and touching upon numerous disciplines; but they have only recently become ubiquitous in signal and data analytics at the confluence of signal processing, statistics, data mining, and machine learning. This overview article aims to provide a good starting point for researchers and practitioners interested in learning about and working with tensors. As such, it focuses on fundamentals and motivation (using various application examples), aiming to strike an appropriate balance of breadth and depth that will enable someone having taken first graduate courses in matrix algebra and probability to get started doing research and/or developing tensor algorithms and software. Some background in applied optimization is useful but not strictly required. The material covered includes tensor rank and rank decomposition; basic tensor factorization models and their relationships and properties (including fairly good coverage of identifiability); broad coverage of algorithms ranging from alternating optimization to stochastic gradient; statistical performance analysis; and applications ranging from source separation to collaborative filtering, mixture and topic modeling, classification, and multilinear subspace learning.

Tensor Decomposition for Signal Processing and Machine Learning

The widespread use of multisensor technology and the emergence of big data sets have highlighted the limitations of standard flat-view matrix models and the necessity to move toward more versatile data analysis tools. We show that higher-order tensors (i.e., multiway arrays) enable such a fundamental paradigm shift toward models that are essentially polynomial, the uniqueness of which, unlike the matrix methods, is guaranteed under very mild and natural conditions. Benefiting from the power of multilinear algebra as their mathematical backbone, data analysis techniques using tensor decompositions are shown to have great flexibility in the choice of constraints which match data properties and extract more general latent components in the data than matrix-based methods.

/pdf/tensor-decompositions-for-signal-processing-applications-2cvd1sgtij.pdf

Tensor Decompositions for Signal Processing Applications: From two-way to multiway component analysis

We prove that multilinear (tensor) analogues of many efficiently computable problems in numerical linear algebra are NP-hard. Our list includes: determining the feasibility of a system of bilinear equations, deciding whether a 3-tensor possesses a given eigenvalue, singular value, or spectral norm; approximating an eigenvalue, eigenvector, singular vector, or the spectral norm; and determining the rank or best rank-1 approximation of a 3-tensor. Furthermore, we show that restricting these problems to symmetric tensors does not alleviate their NP-hardness. We also explain how deciding nonnegative definiteness of a symmetric 4-tensor is NP-hard and how computing the combinatorial hyperdeterminant is NP-, #P-, and VNP-hard.

/pdf/most-tensor-problems-are-np-hard-cqgnykhb6t.pdf

Most Tensor Problems Are NP-Hard

We study a deep/multi-layer structured matrix factorization problem It approximates a given matrix by the product of K matrices (called factors) Each factor is obtained by applying a fixed linear operator to a short vector of parameters (thus the name “structured”) We call the model deep or multilayer because the number of factors is not limited In the practical situations we have in mind, we typically have K = 10 or 20 We provide necessary and sufficient conditions for the identifiability of the factors (up to a scale rearrangement) We also provide a sufficient condition that guarantees that the recovery of the factors is stable A practical example where the deep structured factorization is a convolutional tree is provided in an accompanying paper

/pdf/on-the-identifiability-and-stable-recovery-of-deep-multi-3idmz81dz5.pdf

On the identifiability and stable recovery of deep/multi-layer structured matrix factorization

We study the projective geometry of homogeneous varieties $X= G/P\subset P(V)$, where $G$ is a complex simple Lie group, $P$ is a maximal parabolic subgroup and $V$ is the minimal $G$-module associated to $P$. Our study began with the observation that Freudenthal's magic chart could be derived from Zak's theorem on Severi varieties and standard geometric constructions. Our attempt to understand this observation led us to discover further connections between projective geometry and representation theory. 
Among other things, we calculate the variety of tangent directions to lines on $X$ through a point and determine unirulings of $X$. We show this variety is a Hermitian symmetric space if and only if $P$ does not correspond to a short root. We describe the spaces corresponding to the exceptional short roots and their unirulings using the octonions. Further calculations, in the case $X$ is a Hermitian symmetric space, give rise to a strict prolongation property and the appearance of secant varieties at the infinitesimal level. Our work complements and advances that of Freudenthal and Tits, who studied homogeneous varieties in an abstract/axiomatic setting.

On the projective geometry of homogeneous spaces

The method of shifted partial derivatives was used to prove a super-polynomial lower bound on the size of depth four circuits needed to compute the permanent. We show that this method alone cannot prove that the padded permanent $\ell^{n-m} perm_m$ cannot be realized inside the $GL_{n^2}$-orbit closure of the determinant $ det_n$ when $n>2m^2+2m$. Our proof relies on several simple degenerations of the determinant polynomial, Macaulay's theorem that gives a lower bound on the growth of an ideal, and a lower bound estimate from Gupta et. al. regarding the shifted partial derivatives of the determinant.

The method of shifted partial derivatives cannot separate the permanent from the determinant

The geometry of rank decompositions of matrix multiplication II: 3 × 3 matrices

We establish basic techniques for studying the ideals of secant varieties of Segre varieties. We solve a conjecture of Garcia, Stillman and Sturmfels on the generators of the ideal of the first secant variety in the case of three factors and solve the conjecture set-theoretically for an arbitrary number of factors. We determine the low degree components of the ideals of secant varieties of small dimension in a few cases.

Joseph M. Landsberg

Papers

On the identifiability and stable recovery of deep/multi-layer structured matrix factorization

On the projective geometry of homogeneous spaces

The method of shifted partial derivatives cannot separate the permanent from the determinant

The geometry of rank decompositions of matrix multiplication II: 3 × 3 matrices

On the ideals of secant varieties of Segre varieties