This survey provides an overview of higher-order tensor decompositions, their applications, and available software. A tensor is a multidimensional or $N$-way array. Decompositions of higher-order tensors (i.e., $N$-way arrays with $N \geq 3$) have applications in psycho-metrics, chemometrics, signal processing, numerical linear algebra, computer vision, numerical analysis, data mining, neuroscience, graph analysis, and elsewhere. Two particular tensor decompositions can be considered to be higher-order extensions of the matrix singular value decomposition: CANDECOMP/PARAFAC (CP) decomposes a tensor as a sum of rank-one tensors, and the Tucker decomposition is a higher-order form of principal component analysis. There are many other tensor decompositions, including INDSCAL, PARAFAC2, CANDELINC, DEDICOM, and PARATUCK2 as well as nonnegative variants of all of the above. The N-way Toolbox, Tensor Toolbox, and Multilinear Engine are examples of software packages for working with tensors.

/pdf/tensor-decompositions-and-applications-46vl2ih5gn.pdf

Tensor Decompositions and Applications

The Theory of Matrices. By F R. Gantmacher. Two volumes, pp. 374 and 276. 1959. (Translated from the Russian by K. A. Hirsch; Chelsea Publishing Company, New York)

Tensors or multiway arrays are functions of three or more indices $(i,j,k,\ldots)$ —similar to matrices (two-way arrays), which are functions of two indices $(r,c)$ for (row, column). Tensors have a rich history, stretching over almost a century, and touching upon numerous disciplines; but they have only recently become ubiquitous in signal and data analytics at the confluence of signal processing, statistics, data mining, and machine learning. This overview article aims to provide a good starting point for researchers and practitioners interested in learning about and working with tensors. As such, it focuses on fundamentals and motivation (using various application examples), aiming to strike an appropriate balance of breadth and depth that will enable someone having taken first graduate courses in matrix algebra and probability to get started doing research and/or developing tensor algorithms and software. Some background in applied optimization is useful but not strictly required. The material covered includes tensor rank and rank decomposition; basic tensor factorization models and their relationships and properties (including fairly good coverage of identifiability); broad coverage of algorithms ranging from alternating optimization to stochastic gradient; statistical performance analysis; and applications ranging from source separation to collaborative filtering, mixture and topic modeling, classification, and multilinear subspace learning.

Tensor Decomposition for Signal Processing and Machine Learning

The widespread use of multisensor technology and the emergence of big data sets have highlighted the limitations of standard flat-view matrix models and the necessity to move toward more versatile data analysis tools. We show that higher-order tensors (i.e., multiway arrays) enable such a fundamental paradigm shift toward models that are essentially polynomial, the uniqueness of which, unlike the matrix methods, is guaranteed under very mild and natural conditions. Benefiting from the power of multilinear algebra as their mathematical backbone, data analysis techniques using tensor decompositions are shown to have great flexibility in the choice of constraints which match data properties and extract more general latent components in the data than matrix-based methods.

/pdf/tensor-decompositions-for-signal-processing-applications-2cvd1sgtij.pdf

Tensor Decompositions for Signal Processing Applications: From two-way to multiway component analysis

We prove that multilinear (tensor) analogues of many efficiently computable problems in numerical linear algebra are NP-hard. Our list includes: determining the feasibility of a system of bilinear equations, deciding whether a 3-tensor possesses a given eigenvalue, singular value, or spectral norm; approximating an eigenvalue, eigenvector, singular vector, or the spectral norm; and determining the rank or best rank-1 approximation of a 3-tensor. Furthermore, we show that restricting these problems to symmetric tensors does not alleviate their NP-hardness. We also explain how deciding nonnegative definiteness of a symmetric 4-tensor is NP-hard and how computing the combinatorial hyperdeterminant is NP-, #P-, and VNP-hard.

/pdf/most-tensor-problems-are-np-hard-cqgnykhb6t.pdf

Most Tensor Problems Are NP-Hard

It has been known for some time that algebraic geometry and representation theory are useful for proving lower bounds on the complexity of the matrix multiplication operator. In this talk I will explain how geometry can be used to prove both upper and lower bounds. After a discussion of the problem and its history, I will present very recent work with M. Michalek on border rank algorithms. Faculty Host: Eric Allender

/pdf/geometry-and-the-complexity-of-matrix-multiplication-2fvrdkie9q.pdf

Geometry and the complexity of matrix multiplication

Two central problems in computer science are P vs NP and the complexity of matrix multiplication. The first is also a leading candidate for the greatest unsolved problem in mathematics. The second is of enormous practical and theoretical importance. Algebraic geometry and representation theory provide fertile ground for advancing work on these problems and others in complexity. This introduction to algebraic complexity theory for graduate students and researchers in computer science and mathematics features concrete examples that demonstrate the application of geometric techniques to real world problems. Written by a noted expert in the field, it offers numerous open questions to motivate future research. Complexity theory has rejuvenated classical geometric questions and brought different areas of mathematics together in new ways. This book will show the beautiful, interesting, and important questions that have arisen as a result.

Geometry and Complexity Theory

We construct the complex simple Lie algebras using elementary algebraic geometry. We use our construction to obtain a new proof of the classification of complex simple Lie algebras that does not appeal to the classification of root systems.

Construction and classification of complex simple Lie algebras via projective geometry

Moving frames and exterior differential systems

We address special cases of a question of Eisenbud on the ideals of secant varieties of Veronese re-embeddings of arbitrary varieties. Eisenbud's question generalizes a conjecture of Eisenbud, Koh and Stillman (EKS) for curves. We prove that set-theoretic equations of small secant varieties to a high degree Veronese re-embedding of a smooth variety are determined by equations of the ambient Veronese variety and linear equations. However this is false for singular varieties, and we give explicit counter-examples to the EKS conjecture for singular curves. The techniques we use also allow us to prove a gap and uniqueness theorem for symmetric tensor rank. We put Eisenbud's question in a more general context about the behaviour of border rank under specialisation to a linear subspace, and provide an overview of conjectures coming from signal processing and complexity theory in this context.

Joseph M. Landsberg

Papers

Geometry and the complexity of matrix multiplication

Geometry and Complexity Theory

Construction and classification of complex simple Lie algebras via projective geometry

Moving frames and exterior differential systems

Determinantal equations for secant varieties and the Eisenbud-Koh-Stillman conjecture