Open AccessPosted Content
Symmetry, Saddle Points, and Global Geometry of Nonconvex Matrix Factorization
Reads0
Chats0
TLDR
A general theory for studying the geometry of nonconvex objective functions with underlying symmetric structures is proposed and the locations of stationary points and the null space of the associated Hessian matrices are characterized via the lens of invariant groups.Abstract:
We propose a general theory for studying the geometry of nonconvex objective functions with underlying symmetric structures. In specific, we characterize the locations of stationary points and the null space of the associated Hessian matrices via the lens of invariant groups. As a major motivating example, we apply the proposed general theory to characterize the global geometry of the low-rank matrix factorization problem. In particular, we illustrate how the rotational symmetry group gives rise to infinitely many non-isolated strict saddle points and equivalent global minima of the objective function. By explicitly identifying all stationary points, we divide the entire parameter space into three regions: ($\cR_1$) the region containing the neighborhoods of all strict saddle points, where the objective has negative curvatures; ($\cR_2$) the region containing neighborhoods of all global minima, where the objective enjoys strong convexity along certain directions; and ($\cR_3$) the complement of the above regions, where the gradient has sufficiently large magnitudes. We further extend our result to the matrix sensing problem. This allows us to establish strong global convergence guarantees for popular iterative algorithms with arbitrary initial solutions.read more
Citations
More filters
Journal ArticleDOI
Nonconvex Optimization Meets Low-Rank Matrix Factorization: An Overview
Yuejie Chi,Yue Lu,Yuxin Chen +2 more
TL;DR: This tutorial-style overview highlights the important role of statistical models in enabling efficient nonconvex optimization with performance guarantees and reviews two contrasting approaches: two-stage algorithms, which consist of a tailored initialization step followed by successive refinement; and global landscape analysis and initialization-free algorithms.
Journal ArticleDOI
Implicit Regularization in Nonconvex Statistical Estimation: Gradient Descent Converges Linearly for Phase Retrieval, Matrix Completion, and Blind Deconvolution
TL;DR: In this paper, the authors show that gradient descent can achieve near-optimal statistical and computational guarantees without explicit regularization for phase retrieval, low-rank matrix completion, and blind deconvolution.
Journal ArticleDOI
Harnessing Structures in Big Data via Guaranteed Low-Rank Matrix Estimation: Recent Theory and Fast Algorithms via Convex and Nonconvex Optimization
Yudong Chen,Yuejie Chi +1 more
TL;DR: A unified overview of recent advances in low-rank matrix estimation from incomplete measurements is provided, with attention paid to rigorous characterization of the performance of these algorithms and to problems where the lowrank matrix has additional structural properties that require new algorithmic designs and theoretical analysis.
Journal ArticleDOI
Global optimality in low-rank matrix optimization
TL;DR: In this paper, the authors consider the minimization of a general objective function over a set of rectangular matrices that have rank at most r. Despite the resulting nonconvexity, recent studies in matrix completion and sensing have shown that the factored problem has no spurious local minima and obeys the strict saddle property.
Proceedings Article
Deep Hyperspherical Learning
TL;DR: SphereNet as mentioned in this paper adopts SphereConv as its basic convolution operator and is supervised by generalized angular softmax loss, which is a natural loss formulation under Sphere-Conv.
References
More filters
Journal ArticleDOI
A Singular Value Thresholding Algorithm for Matrix Completion
TL;DR: This paper develops a simple first-order and easy-to-implement algorithm that is extremely efficient at addressing problems in which the optimal solution has low rank, and develops a framework in which one can understand these algorithms in terms of well-known Lagrange multiplier algorithms.
Journal ArticleDOI
Exact Matrix Completion via Convex Optimization
TL;DR: It is proved that one can perfectly recover most low-rank matrices from what appears to be an incomplete set of entries, and that objects other than signals and images can be perfectly reconstructed from very limited information.
Journal ArticleDOI
Guaranteed Minimum-Rank Solutions of Linear Matrix Equations via Nuclear Norm Minimization
TL;DR: It is shown that if a certain restricted isometry property holds for the linear transformation defining the constraints, the minimum-rank solution can be recovered by solving a convex optimization problem, namely, the minimization of the nuclear norm over the given affine space.
Foundations of the PARAFAC procedure: Models and conditions for an "explanatory" multi-model factor analysis
TL;DR: It is shown that an extension of Cattell's principle of rotation to Proportional Profiles (PP) offers a basis for determining explanatory factors for three-way or higher order multi-mode data.
Journal Article
Guaranteed Minimum-Rank Solutions of Linear Matrix Equations via Nuclear Norm Minimization
TL;DR: In this paper, it was shown that if a certain restricted isometry property holds for the linear transformation defining the constraints, the minimum-rank solution can be recovered by solving a convex optimization problem, namely, the minimization of the nuclear norm over the given affine space.