This survey provides an overview of higher-order tensor decompositions, their applications, and available software. A tensor is a multidimensional or $N$-way array. Decompositions of higher-order tensors (i.e., $N$-way arrays with $N \geq 3$) have applications in psycho-metrics, chemometrics, signal processing, numerical linear algebra, computer vision, numerical analysis, data mining, neuroscience, graph analysis, and elsewhere. Two particular tensor decompositions can be considered to be higher-order extensions of the matrix singular value decomposition: CANDECOMP/PARAFAC (CP) decomposes a tensor as a sum of rank-one tensors, and the Tucker decomposition is a higher-order form of principal component analysis. There are many other tensor decompositions, including INDSCAL, PARAFAC2, CANDELINC, DEDICOM, and PARATUCK2 as well as nonnegative variants of all of the above. The N-way Toolbox, Tensor Toolbox, and Multilinear Engine are examples of software packages for working with tensors.

/pdf/tensor-decompositions-and-applications-46vl2ih5gn.pdf

Tensor Decompositions and Applications

/pdf/convex-analysisnoer-san-nojin-zhan-nituite-5dl2rbmoyr.pdf

Convex Analysisの二,三の進展について

This monograph is about a class of optimization algorithms called proximal algorithms. Much like Newton's method is a standard tool for solving unconstrained smooth optimization problems of modest size, proximal algorithms can be viewed as an analogous tool for nonsmooth, constrained, large-scale, or distributed versions of these problems. They are very generally applicable, but are especially well-suited to problems of substantial recent interest involving large or high-dimensional datasets. Proximal methods sit at a higher level of abstraction than classical algorithms like Newton's method: the base operation is evaluating the proximal operator of a function, which itself involves solving a small convex optimization problem. These subproblems, which generalize the problem of projecting a point onto a convex set, often admit closed-form solutions or can be solved very quickly with standard or simple specialized methods. Here, we discuss the many different interpretations of proximal operators and algorithms, describe their connections to many other topics in optimization and applied mathematics, survey some popular algorithms, and provide a large number of examples of proximal operators that commonly arise in practice.

Proximal Algorithms

https://wiki.biac.duke.edu/lib/exe/fetch.php?media=biac:jones_etal_2012_the_do_s_and_don_ts_of_diffusion_mri.pdf

White matter integrity, fiber count, and other fallacies: The do's and don'ts of diffusion MRI

Iterative shrinkage/thresholding (1ST) algorithms have been recently proposed to handle a class of convex unconstrained optimization problems arising in image restoration and other linear inverse problems. This class of problems results from combining a linear observation model with a nonquadratic regularizer (e.g., total variation or wavelet-based regularization). It happens that the convergence rate of these 1ST algorithms depends heavily on the linear observation operator, becoming very slow when this operator is ill-conditioned or ill-posed. In this paper, we introduce two-step 1ST (TwIST) algorithms, exhibiting much faster convergence rate than 1ST for ill-conditioned problems. For a vast class of nonquadratic convex regularizers (lscrP norms, some Besov norms, and total variation), we show that TwIST converges to a minimizer of the objective function, for a given range of values of its parameters. For noninvertible observation operators, we introduce a monotonic version of TwIST (MTwIST); although the convergence proof does not apply to this scenario, we give experimental evidence that MTwIST exhibits similar speed gains over IST. The effectiveness of the new methods are experimentally confirmed on problems of image deconvolution and of restoration with missing samples.

/pdf/a-new-twist-two-step-iterative-shrinkage-thresholding-2y9ecuairu.pdf

A New TwIST: Two-Step Iterative Shrinkage/Thresholding Algorithms for Image Restoration

Newton's method for solving a nonlinear equation of several variables is extended to a nonsmooth case by using the generalized Jacobian instead of the derivative. This extension includes the B-derivative version of Newton's method as a special case. Convergence theorems are proved under the condition of semismoothness. It is shown that the gradient function of the augmented Lagrangian forC2-nonlinear programming is semismooth. Thus, the extended Newton's method can be used in the augmented Lagrangian method for solving nonlinear programs.

A nonsmooth version of Newton's method

https://www.polyu.edu.hk/ama/staff/new/qilq/qi-eigen-1.pdf

Eigenvalues of a real supersymmetric tensor

This paper presents convergence analysis of some algorithms for solving systems of nonlinear equations defined by locally Lipschitzian functions. For the directional derivative-based and the generalized Jacobian-based Newton methods, both the iterates and the corresponding function values are locally, superlinearly convergent. Globally, a limiting point of the iterate sequence generated by the damped, directional derivative-based Newton method is a zero of the system if and only if the iterate sequence converges to this point and the stepsize eventually becomes one, provided that the system is strongly BD-regular and semismooth at this point. In this case, the convergence is superlinear. A general attraction theorem is presented, which can be applied to two algorithms proposed by Han, Pang and Rangaraj. A hybrid method, which is both globally convergent in the sense of finding a stationary point of the norm function of the system and locally quadratically convergent, is also presented.

Convergence Analysis of Some Algorithms for Solving Nonsmooth Equations

In this paper we take a new look at smoothing Newton methods for solving the nonlinear complementarity problem (NCP) and the box constrained variational inequalities (BVI). Instead of using an infinite sequence of smoothing approximation functions, we use a single smoothing approximation function and Robinson’s normal equation to reformulate NCP and BVI as an equivalent nonsmooth equation H(u,x)=0, where H:ℜ
 2n
 →ℜ
 2n
 , u∈ℜ
 n
 is a parameter variable and x∈ℜ
 n
 is the original variable. The central idea of our smoothing Newton methods is that we construct a sequence {z

 k
 =(u

 k
 
,x

 k
 )} such that the mapping H(·) is continuously differentiable at each z

 k
 and may be non-differentiable at the limiting point of {z

 k
 }. We prove that three most often used Gabriel-More smoothing functions can generate strongly semismooth functions, which play a fundamental role in establishing superlinear and quadratic convergence of our new smoothing Newton methods. We do not require any function value of F or its derivative value outside the feasible region while at each step we only solve a linear system of equations and if we choose a certain smoothing function only a reduced form needs to be solved. Preliminary numerical results show that the proposed methods for particularly chosen smoothing functions are very promising.

Liqun Qi

Papers

A nonsmooth version of Newton's method

Eigenvalues of a real supersymmetric tensor

Convergence Analysis of Some Algorithms for Solving Nonsmooth Equations

A new look at smoothing Newton methods for nonlinear complementarity problems and box constrained variational inequalities

Tensor Analysis: Spectral Theory and Special Tensors