When optimizing on a Riemannian manifold, it is important to use an efficient retraction, which maps a point on a tangent space to a point on the manifold. In this paper, we prove a map based on the QR factorization to be a retraction on the generalized Stiefel manifold. In addition, we propose an efficient implementation of the retraction based on the Cholesky QR factorization. Numerical experiments show that the proposed retraction is more efficient than the existing one based on the polar factorization.

Cholesky QR-based retraction on the generalized Stiefel manifold

The phenotypes of complex biological systems are fundamentally driven by various multi-scale mechanisms. Multi-modal data, such as single cell multi-omics data, enables a deeper understanding of underlying complex mechanisms across scales for phenotypes. We developed an interpretable regularized learning model, deepManReg, to predict phenotypes from multi-modal data. First, deepManReg employs deep neural networks to learn cross-modal manifolds and then to align multi-modal features onto a common latent space. Second, deepManReg uses cross-modal manifolds as a feature graph to regularize the classifiers for improving phenotype predictions and also for prioritizing the multi-modal features and cross-modal interactions for the phenotypes. We applied deepManReg to (1) an image dataset of handwritten digits with multi-features and (2) single cell multi-modal data (Patch-seq data) including transcriptomics and electrophysiology for neuronal cells in the mouse brain. We show that deepManReg improved phenotype prediction in both datasets, and also prioritized genes and electrophysiological features for the phenotypes of neuronal cells. 

A deep manifold-regularized learning model for improving phenotype prediction from multi-modal data

We design, analyse and test a class of incomplete orthogonal factorization preconditioners constructed from Givens rotations, incorporating some dropping strategies and updating tricks, for the solution of large sparse systems of linear equations. Comprehensive accounts about how the preconditioners are coded, what storage is required and how the computation is executed for a given accuracy are presented. A number of numerical experiments show that these preconditioners are competitive with standard incomplete triangular factorization preconditioners when they are applied to accelerate Krylov subspace iteration methods such as GMRES and BiCGSTAB.

Numerical study on incomplete orthogonal factorization preconditioners

Variational Bayes (VB) has become a widely-used tool for Bayesian inference in statistics and machine learning. Nonetheless, the development of the existing VB algorithms is so far generally restricted to the case where the variational parameter space is Euclidean, which hinders the potential broad application of VB methods. This paper extends the scope of VB to the case where the variational parameter space is a Riemannian manifold. We develop an efficient manifold-based VB algorithm that exploits both the geometric structure of the constraint parameter space and the information geometry of the manifold of VB approximating probability distributions. Our algorithm is provably convergent and achieves a convergence rate of order $\mathcal O(1/\sqrt{T})$ and $\mathcal O(1/T^{2-2\epsilon})$ for a non-convex evidence lower bound function and a strongly retraction-convex evidence lower bound function, respectively. We develop in particular two manifold VB algorithms, Manifold Gaussian VB and Manifold Neural Net VB, and demonstrate through numerical experiments that the proposed algorithms are stable, less sensitive to initialization and compares favourably to existing VB methods.

Variational Bayes on Manifolds

A variant of IDRstab with reliable update strategies for solving sparse linear systems

Newton’s method for unconstrained optimization problems on the Euclidean space can be generalized to that on Riemannian manifolds. The truncated singular value problem is one particular problem defined on the product of two Stiefel manifolds, and an algorithm of the Riemannian Newton’s method for this problem has been designed. However, this algorithm is not easy to implement in its original form because the Newton equation is expressed by a system of matrix equations which is difficult to solve directly. In the present paper, we propose an effective implementation of the Newton algorithm. A matrix-free Krylov subspace method is used to solve a symmetric linear system into which the Newton equation is rewritten. The presented approach can be used on other problems as well. Numerical experiments demonstrate that the proposed method is effective for the above optimization problem.

A matrix-free implementation of Riemannian Newton’s method on the Stiefel manifold

Short-recurrence Krylov subspace methods, such as conjugate gradient squared-type methods, often exhibit large oscillations in the residual norms, leading to a large residual gap and a loss of attainable accuracy for the approximate solutions. Residual smoothing is useful for obtaining smooth convergence for the residual norms, but it has been shown that this does not improve the maximum attainable accuracy in most cases. In the present study, we reformulate the smoothing scheme from a novel perspective. The smoothed sequences do not usually affect the primary sequences in conventional smoothing schemes. In contrast, we design a variant of residual smoothing in which the primary and smoothed sequences influence each other. This approach enables us to avoid the propagation of large rounding errors, and results in a smaller residual gap, and thus a higher attainable accuracy. We present a rounding error analysis and numerical experiments to demonstrate the effectiveness of our proposed smoothing scheme.

Kensuke Aihara

Papers

Cholesky QR-based retraction on the generalized Stiefel manifold

A matrix-free implementation of Riemannian Newton’s method on the Stiefel manifold

A variant of IDRstab with reliable update strategies for solving sparse linear systems

Variants of residual smoothing with a small residual gap

An alternative implementation of the IDRstab method saving vector updates