Noname manuscript No.
(will be inserted by the editor)
Tucker factorization with missing data with
application to low-n-rank tensor completion
Marko Filipovi´c · Ante Juki´c
Received: date / Accepted: date
Abstract The problem of tens or completi on arises often in signal p rocessing and
machine learning. It consists of recovering a tenso r from a subset of its entries.
The usual structural assumption on a tensor t h at makes the problem well posed
is that the tensor has low rank in every mode. Several tensor completion methods
based on minimization of nuclear norm, which is the closest convex approximation
of rank, have been proposed recently, with applications mostly in image i n paint-
ing problems. It is often stated in these papers that method s based on Tucker
factorization perform poorly when the true ranks are unknown. In this paper,
we propose a simple algorithm for Tucker factoriz ation of a tensor with missing
data and its application to low-n-ra nk tensor completion . The algorithm is simi lar
to previously proposed metho d for PARAFAC decomposition with mis s ing data.
We demonstrate in several numerical experiments that the pr oposed alg orithm
performs well even when the ranks are significantly overesti mated. Approximate
reconstruction can be obtained when the ranks are underestimated. The algorithm
outperforms nuclear norm minimization methods when the frac tion of known ele-
ments of a t ens or is low.
Keywords Tucker factorization · Tensor co mpletion · Low-n-rank tensor ·
Missing data
Mathematics Subject Classification (2000) MSC 68U99
M. Filipovi´c
Rudjer Boˇskovi´c Institute
Bijeniˇcka 54, 10000 Zagreb, Croatia
Tel.: +385-1-4571241
Fax: +385-1-4680104
E-mail: filipov@irb.hr
A. Juki´c
Department of Medical Physics and Acoustics
University of Oldenburg
26111 Oldenburg, Germany
E-mail: ante.jukic@uni-oldenburg.de
2 Marko Filipovi´c, Ante Juki´c
1 Introduction
Low-rank matrix completion problem was studied extensively in recent years
(Recht et al 2010; Candes and Recht 2009). It arises natur ally in many practi-
cal problems when one would like to recover a matrix from a subset of its entries.
On th e other hand, in ma ny applications one is dealing with multi-way data,
which are n aturally represented by tens ors. Tensor s are multi-dimensional arrays,
i.e. higher-order generalizations of matrices. Multi-way data anal ysis was ori gi-
nally developed in t h e fields of psychometrics and chemometrics, but nowadays
it also has applications in signal processing, machine learning and da ta analysis.
Here, we are interested in the problem of recovering a partially observed tensor, or
tensor completion problem. Examples of applicati ons where the problem arises in-
clude image occlusio n /inpainting problems, social network data analysis , network
traffic data ana lysis, bibliometric data a nalysis, spectroscopy, multidimensional
NMR (Nuclear Magnetic Resonance) data analysis, EEG (electroencephalogram)
data analysis and many others. For a more detailed description of appl ications,
interested reader is referred t o (Acar et al 2011) and references therein.
In the matrix case, it is often realistic to assum e that the matrix that we want
to reconstruct from a subset of its entries has a low rank. This assumption en-
ables matrix completion from only a small number of its entries. However, the
rank function is discrete and nonconvex, which makes its optimization hard in
practice. Therefo re, nuclear norm has been used in many papers as its approxi-
mation. Nuclear norm is defined as the sum of singular values of a matrix, and
it is the tightest convex lower bound of th e rank of a matrix on the set of ma-
trices {Y : kY k
2
≤ 1} (here, k · k
2
denotes usual matrix 2-norm). When the rank
is replaced by nuclear norm, the resulting problem of nuclear norm minimization
is convex, and, as shown in (Candes and R echt 2009), if the matrix rank is low
enough, the solution of the o riginal (rank minimizati on) problem can be f ound
by minimizing the nuclear norm. In several recent papers on tensor completion,
the definitio n o f nuclear norm was extended to tensors. There, it was stated that
methods based on Tucker factorization perform poorly when the true ranks of
the tensor are unknown. In this paper, we propose a method for Tucker factor-
izati on wit h missing data, with application in tensor completion. We demonstrate
in several numerical experiments that t h e method performs well even when th e
true ranks are significantly overestimated. Namely, it can estimate the exact ranks
from the data. Also, it outperform s nuclear norm minimization methods when the
fracti on of known elements of a tenso r is low.
The rest of the paper is or ganized as follows. In Subsection 1.1, we review
basics of tensor notation and t erminol ogy. Problem setting and previou s work are
described in Subsection 1.3. We describe our approach in Section 2. In Subsections
2.1 and 2.2, details related to optimization method and implementation of the
algo rithm are described. Several numerical experiments are presented in Section
3. The emphasis is on s ynthetic experiments, which are used to demonstrate the
efficiency of th e proposed method on exactly low-rank problems. However, we also
perform some ex periments o n realistic data. Conclusions are presented in Section
4.
Low-n-rank tensor completion 3
1.1 Tensor notation and terminology
We denote scalars by regular lowercase o r uppercase, vector s by bold lowercase,
matrices by bo ld uppercase, and tensors by bold Euler script letters. For more de-
tail s on tensor notation and terminol ogy, the reader is also referred to
(Kolda and Bader 2009).
The order of a tensor is the number of its dimens ions (also called ways or
modes). We denote the vector space of tensors of order N and size I
1
× · · · × I
N
by R
I
1
×···×I
N
. Elements o f tensor X of order N are denoted by x
i
1
...i
N
.
A fiber of a tensor is defined as a vector obtained by fixing all indices but
one. Fibers a re generaliz ations of m atrix columns and rows. Mode-n fibers are
obtained by fixing all indices but n-th. Mode-n matricization (unfolding) of tensor
X, denoted as X
(n)
, is obta ined by arranging all mode-n fibers as columns of a
matrix. Precise order in which fibers are stacked as columns is not important as
long as it is consistent. Folding is the inverse operation of ma tricization/ un folding.
Mode-n prod uc t of tensor X and matrix A is denoted by X ×
n
A. It is defined
as
Y = X ×
n
A ⇐⇒ Y
(n)
= AX
(n)
.
Mode-n product is commutative (when applied in distinct modes), i.e.
(X ×
n
A) ×
m
B = (X ×
m
B) ×
n
A
for m 6= n. Repeated mode-n produc t can be expressed as
(X ×
n
A) ×
n
B = X ×
n
(BA) .
There are several definitions of tensor rank. In this pap er, we are interested in
n-rank. For N-way tensor X, n-rank is defined as the rank of X
(n)
. If we denote
r
n
= rank X
(n)
, for n = 1, . . . , N, we say that X is a rank-(r
1
, . . . , r
N
) tensor. In
the experi mental section (Section 3) in this paper we denote an estimation of the
n-rank of given tensor X by ˆr
n
.
For completeness, we al s o st ate the usual definition of the rank of a tensor.
We say tha t an N -way tensor X ∈ R
I
1
×···×I
N
is rank-1 if it can be written as the
outer product of N vectors, i.e.
X = a
(1)
◦ · · · ◦ a
(N)
, (1)
where ◦ denotes the vector outer product. Elementwise, (1) is written as x
i
1
...i
N
=
a
(1)
i
1
· · · a
(N)
i
N
, for all 1 ≤ i
n
≤ I
N
. Tensor rank of X is d efin ed as minimal number
of rank-1 tensors that generate X in the sum. As opposed to n-rank of a tensor,
tensor rank is hard to compute (H˚astad 1990).
The Hadamard prod uc t of tensors is the componentwise product, i .e. for N -way
tensors X, Y, it is defined as (X ∗ Y)
i
1
...ı
N
= x
i
1
...i
N
y
i
1
...i
N
.
The Frobenius norm of tensor X of size I
1
× · · · × I
N
is denoted by kXk
F
and
defined as
kXk
F
=
I
1
X
i
1
=1
· · ·
I
N
X
i
N
=1
x
2
i
1
...i
N
!
1
2
. (2)
4 Marko Filipovi´c, Ante Juki´c
1.2 Tensor factorizations/decompositions
Two of the most often used tensor factorizations/decom positions a re PARAFAC
(parallel factors) decomposition a n d Tucker factorization. PARAFAC decomposi-
tion is also called canonical decomposition (CANDECOMP) or
CANDECOMP/PARA FAC (CP) decomposition. For a given tensor X ∈ R
I
1
×···×I
N
,
it is defined as a decomposition of X as a linear combination of minimal number
of rank-1 tensors
X =
R
X
r=1
λ
r
a
(1)
r
◦ · · · ◦ a
(N)
r
. (3)
For more details regarding the PARAFAC deco mposition, the reader is referred to
(Kolda and Bader 2009), since here we are interested in Tucker factorization.
Tucker factoriza tion (also ca lled N -mode PCA or higher-order SVD) of a tensor
X can be written as
X = G ×
1
A
1
×
2
· · · ×
N
A
N
, (4)
where G ∈ R
J
1
×···×J
N
is the core tensor with J
i
≤ I
i
, for i = 1, . . . , N , an d A
i
,
i = 1, . . . , N a re, usually orthogonal, factor matrices. Factor matrices A
i
are of
size I
i
× r
i
, for i = 1, . . . , N , if X is rank-(r
1
, . . . , r
N
). A tensor that has low ran k
in every mode can be represented with its Tucker fact orization with small core
tensor (whose dimensi ons correspond to r anks in correspo nd ing modes). Mode-n
matricization X
(n)
of X in (4) can be written as
X
(n)
= A
(n)
G
(n)
A
(N)
⊗ · · · ⊗ A
(n+1)
⊗ A
(n−1)
⊗ · · · ⊗ A
(1)
T
, (5)
where G
(n)
denotes mode-n matricization of G, ⊗ denotes Kronecker product of
matrices, and M
T
denotes the transpose of matrix M. If the factor matrices A
(i)
are constrained to be orthogonal, then they can be interpreted as the principa l
components in correspondin g modes, whi le the elem ents of the core tensor G show
the level of interaction between different modes. In general, Tucker factorization is
not unique. However, in practica l applications some constraints are often imposed
on the core and the factors to obtain a meaningful fact orization, for example
orthogonality, non-negativity or sparsity. For m ore details, t h e reader i s referred
to (Tucker 1966; Kolda and Bader 2 009; De La thauwer et al 2000).
1.3 Problem definition and previous work
Let us denote by T ∈ R
I
1
×···×I
N
a tensor that is low-rank in every mode (low-
n-rank tensor ), and by T
Ω
the projection of T onto indexes of observed entries.
Here, Ω is a subset of {1, . . . , I
1
} × {1, . . . , I
2
} × · · · × {1, . . . , I
N
}, consisting of
positions of obs erved tensor entries. The problem of low-n-rank tensor completion
was formulated in (Gandy et al 2011) as
min
X∈R
I
1
×···×I
N
N
X
n=1
rank
X
(n)
subject to X
Ω
= T
Ω
. (6)
Some other functi on o f n-ranks of a tensor can also be considered here, for exam-
ple any linear combination of n-ranks. Nuclear norm mini mization approaches to
Low-n-rank tensor completion 5
tensor completion, described in the following, are based on this type of problem
formulation. Namely, the idea is to replace rank
X
(n)
with nuclear norm of X
(n)
.
This leads to the problem formulation
min
X∈R
I
1
×···×I
N
N
X
n=1
X
(n)
∗
subject to X
Ω
= T
Ω
.
Here, f or given matrix X ∈ R
d
1
×d
2
, kXk
∗
=
P
min(d
1
, d
2
)
i=1
σ
i
(X), where σ
i
(X)
denote singular values of X, denotes nuclear norm (o r trace norm) of X. Corre-
sp onding unconstrained formulation is
min
X∈R
I
1
×···×I
N
N
X
n=1
X
(n)
∗
+
λ
2
kX
Ω
− T
Ω
k
2
2
, (7)
where λ > 0 is a regularization parameter. This problem formulation was used in
(Gandy et al 2011). In (Tomioka et al. (2011)
1
), similar formulation
min
X∈R
I
1
×···×I
N
N
X
n=1
γ
n
X
(n)
∗
+
1
2λ
kX
Ω
− T
Ω
k
2
2
, (8)
where λ > 0 and γ
n
≥ 0 are parameters, was used. In (Liu et al 2013), formulation
min
X∈R
I
1
×···×I
N
N
X
n=1
α
n
X
(n)
∗
+
γ
n
2
X
(n)
− T
(n)
2
F
subject to X
Ω
= T
Ω
,
(9)
where again α
n
≥ 0 and γ
n
> 0 are para meters, was used. In above equations (7),
(8) and (9), we have used the notation from corresponding pap ers. Therefore, note
that λ in (7) and (8), i.e. γ
n
in (8) and (9), have different interpretations.
The first paper tha t proposed an extension of low-rank matrix completion
concept to tensors seems to be (Liu et al 2009). There, the authors introduced
an extension of nuclear norm to tensors. They focused on n-rank, and defined
the nuclear norm of tensor X as the average of nuclear norms of its unfoldings.
In subsequent paper (Liu et al 2013), they defined the nuclear norm of a tensor
more generally, as a convex combination of nuclear norms of its unfoldings. Similar
approaches were used in (Gandy et al 2011) and (Tomioka et al. (2011)
1
).
In (Liu et al 20 13), three algorithms were proposed. Simple low rank tensor
complet ion (SiLRTC) is a block coordinate descent method that is guaranteed to
find the optimal solution since the objective is convex. To improve its convergence
sp eed, the authors in (Liu et al 2013) proposed another algorithm: fast low rank
tensor completion (FaLRTC). FaLRTC uses a smoothing scheme to convert the
original nonsmooth pro blem into a sm ooth one. Then, acceleration scheme is used
to improve the convergence speed of the algorithm. Finally, the authors also pro-
posed the highly accurat e low rank tensor completion (HaLRTC), which applies
the alternating direction method of multipliers (ADMM) algorithm to the low
rank tensor completion p roblems. It was shown to be slower than FaLRTC, but
can achieve higher accuracy. Similar algorithm was derived in (Gandy et al 2011).
1
R. Tomioka, K. Hayashi and H. Kashima: Estimation of low-rank tensors via convex
optimization. Technical report, http://arxiv.org/abs/1010.0789.