This survey provides an overview of higher-order tensor decompositions, their applications, and available software. A tensor is a multidimensional or $N$-way array. Decompositions of higher-order tensors (i.e., $N$-way arrays with $N \geq 3$) have applications in psycho-metrics, chemometrics, signal processing, numerical linear algebra, computer vision, numerical analysis, data mining, neuroscience, graph analysis, and elsewhere. Two particular tensor decompositions can be considered to be higher-order extensions of the matrix singular value decomposition: CANDECOMP/PARAFAC (CP) decomposes a tensor as a sum of rank-one tensors, and the Tucker decomposition is a higher-order form of principal component analysis. There are many other tensor decompositions, including INDSCAL, PARAFAC2, CANDELINC, DEDICOM, and PARATUCK2 as well as nonnegative variants of all of the above. The N-way Toolbox, Tensor Toolbox, and Multilinear Engine are examples of software packages for working with tensors.

/pdf/tensor-decompositions-and-applications-46vl2ih5gn.pdf

Tensor Decompositions and Applications

We develop an automated approach for designing matrix multiplication algorithms based on constructions similar to the Coppersmith-Winograd construction. Using this approach we obtain a new improved bound on the matrix multiplication exponent ω

/pdf/multiplying-matrices-faster-than-coppersmith-winograd-1xvjqg5bxt.pdf

Multiplying matrices faster than coppersmith-winograd

There has been continued interest in seeking a theorem describing optimal low-rank approximations to tensors of order 3 or higher that parallels the Eckart-Young theorem for matrices. In this paper, we argue that the naive approach to this problem is doomed to failure because, unlike matrices, tensors of order 3 or higher can fail to have best rank-$r$ approximations. The phenomenon is much more widespread than one might suspect: examples of this failure can be constructed over a wide range of dimensions, orders, and ranks, regardless of the choice of norm (or even Bregman divergence). Moreover, we show that in many instances these counterexamples have positive volume: they cannot be regarded as isolated phenomena.  In one extreme case, we exhibit a tensor space in which no rank-3 tensor has an optimal rank-2 approximation. The notable exceptions to this misbehavior are rank-1 tensors and order-2 tensors (i.e., matrices). In a more positive spirit, we propose a natural way of overcoming the ill-posedness of the low-rank approximation problem, by using weak solutions when true solutions do not exist. For this to work, it is necessary to characterize the set of weak solutions, and we do this  in the case of rank 2, order 3 (in arbitrary dimensions). In our work we emphasize the importance of closely studying concrete low-dimensional examples as a first step toward more general results. To this end, we present a detailed analysis of equivalence classes of $2 \times 2 \times 2$ tensors, and we develop methods for extending results upward to higher orders and dimensions. Finally, we link our work to existing studies of tensors from an algebraic geometric point of view. The rank of a tensor can in theory be given a semialgebraic description; in other words, it can be determined by a system of polynomial inequalities. We study some of these polynomials in cases of interest to us; in particular, we make extensive use of the hyperdeterminant $\Delta$ on $\mathbb{R}^{2\times 2 \times 2}$.

Tensor Rank and the Ill-Posedness of the Best Low-Rank Approximation Problem

A class of interconnection networks based on some existing permutation networks is described with applications to processor to memory communication in multiprocessing systems. These networks, termed delta networks, allow a direct link between any processor to any memory module. The delta networks and full crossbars are analyzed with respect to their effective bandwidth and cost. The analysis shows that delta networks have a far better performance per cost than crossbars in large multiprocessing systems.

Performance of Processor-Memory Interconnections for Multiprocessors

There has been continued interest in seeking a theorem describing optimal low-rank approximations to tensors of order 3 or higher, that parallels the Eckart-Young theorem for matrices. In this paper, we argue that the naive approach to this problem is doomed to failure because, unlike matrices, tensors of order 3 or higher can fail to have best rank-r approximations. The phenomenon is much more widespread than one might suspect: examples of this failure can be constructed over a wide range of dimensions, orders and ranks, regardless of the choice of norm (or even Bregman divergence). Moreover, we show that in many instances these counterexamples have positive volume: they cannot be regarded as isolated phenomena. In one extreme case, we exhibit a tensor space in which no rank-3 tensor has an optimal rank-2 approximation. The notable exceptions to this misbehavior are rank-1 tensors and order-2 tensors. In a more positive spirit, we propose a natural way of overcoming the ill-posedness of the low-rank approximation problem, by using weak solutions when true solutions do not exist. In our work we emphasize the importance of closely studying concrete low-dimensional examples as a first step towards more general results. To this end, we present a detailed analysis of equivalence classes of 2-by-2-by-2 tensors, and we develop methods for extending results upwards to higher orders and dimensions. Finally, we link our work to existing studies of tensors from an algebraic geometric point of view. The rank of a tensor can in theory be given a semialgebraic description; i.e., can be determined by a system of polynomial inequalities. In particular we make extensive use of the 2-by-2-by-2 hyperdeterminant.

Tensor rank and the ill-posedness of the best low-rank approximation problem

O(n 2.7799 ) Complexity for n*n Approximate Matrix Multiplication.

Spectral and computational properties of band symmetric toeplitz matrices

Metodi Numerici per l'Algebra Lineare.

Let $\mathcal{B}_{n,h,k} $ be the class of $n \times n$ band Toeplitz matrices $A = (a_{i,j} )$ such that $a_{i,j} = 0$ if $i - j \geqq k$ or $j - i \geqq h$, $k \leqq h$ and $\mathcal{S}_{n,k} $ the subclass of $\mathcal{B}_{n,k,k} $ made up by symmetric matrices. We show that ${\operatorname{rk}}_{\bf F} (\mathcal{B}_{n,h,k} ) = n + h - 1$ if the field ${\bf F}$ contains a primitive $(n - 1 + h)$th root of unity, ${\operatorname{brk}}_{\bf F} (\mathcal{B}_{n,h,k} ) = n + k - 1,{\operatorname{rk}}_{\bf R} (\mathcal{S}_{n,k} ) = {\operatorname{brk}}_{\bf R} (\mathcal{S}_{n,k} ) = n + 2\lfloor {{(k - 1)} / 2} \rfloor $ if ${\bf R}$ is the real field and $k \leqq {n / 2}$.

Tensor rank and border rank of band Toeplitz matrices

We exhibit fast computational methods for the evaluation of the determinant and the characteristic polynomial of a (2k+1)-diagonal Toeplitz matrix with elements in the complex field, either for sequential or for parallel computations. A fast algorithm, to achieve one step of Newton's method, is shown to be suitable to compute the eigenvalues of such a matrix. Bounds to the eigenvalues and necessary and sufficient conditions for positive definiteness, which are easy to check, are given either for matrices with scalar elements or for matrices with blocks. In the case in which the blocks are themselves band Toeplitz matrices such conditions assume a very simple form.

Milvio Capovani

Papers

O(n 2.7799 ) Complexity for n*n Approximate Matrix Multiplication.

Spectral and computational properties of band symmetric toeplitz matrices

Metodi Numerici per l'Algebra Lineare.

Tensor rank and border rank of band Toeplitz matrices

Fast parallel and sequential computations and spectral properties concerning band Toeplitz matrices