We present a new numerical method for computing the GSVD [36, 27] of two matrices A and B. This method is a variation on Paige''s method [30]. It differs from previous algorithms in guaranteeing both backward stability and con- vergence. There are two innovations. The first is a new pre- processing step which reduces A and B to upper triangular forms satisfying certain rank conditions. The second is a new 2 by 2 triangular GSVD algorithm, which constitutes the inner loop of Paige''s method. We present proofs of stability and convergence of our method, and demonstrate examples on which all previous algorithms fail.

Computing the generalized singular value decomposition

Canonical correlation analysis is a family of multivariate statistical methods for the analysis of paired sets of variables. Since its proposition, canonical correlation analysis has, for instance, been extended to extract relations between two sets of variables when the sample size is insufficient in relation to the data dimensionality, when the relations have been considered to be non-linear, and when the dimensionality is too large for human interpretation. This tutorial explains the theory of canonical correlation analysis, including its regularised, kernel, and sparse variants. Additionally, the deep and Bayesian CCA extensions are briefly reviewed. Together with the numerical examples, this overview provides a coherent compendium on the applicability of the variants of canonical correlation analysis. By bringing together techniques for solving the optimisation problems, evaluating the statistical significance and generalisability of the canonical correlation model, and interpreting the relations, we hope that this article can serve as a hands-on tool for applying canonical correlation methods in data analysis.

A Tutorial on Canonical Correlation Methods

The Total Least Squares (TLS) method has been devised as a more global fitting technique than the ordinary least squares technique for solving overdetermined sets of linear equations AX ≈ B when errors occur in all data. If the errors on the measurements A and B are uncorrelated with zero mean and equal variance, TLS is able to compute a strongly consistent estimate of the true solution of the corresponding unperturbed set A 0 X = B 0. In this paper the TLS computations are generalized in order to maintain consistency of the solution in the following cases: first of all, some columns of A may be error-free and secondly, the errors on the remaining data may be correlated and not equally sized. Hereto, a numerically reliable Generalized TLS algorithm GTLS, based on the Generalized Singular Value Decomposition (GSVD), is developed. Additionally, the equivalence between the GTLS solution and alternative expressions of consistent estimators, described in literature, is proven. These relations allow to deduce the main statistical properties of the GTLS solution.

The restricted total least squares problem: formulation, algorithm, and properties

Self-Supervised Learning (SSL) surmises that inputs and pairwise positive relationships are enough to learn meaningful representations. Although SSL has recently reached a milestone: outperforming supervised methods in many modalities\dots the theoretical foundations are limited, method-specific, and fail to provide principled design guidelines to practitioners. In this paper, we propose a unifying framework under the helm of spectral manifold learning to address those limitations. Through the course of this study, we will rigorously demonstrate that VICReg, SimCLR, BarlowTwins et al. correspond to eponymous spectral methods such as Laplacian Eigenmaps, Multidimensional Scaling et al. This unification will then allow us to obtain (i) the closed-form optimal representation for each method, (ii) the closed-form optimal network parameters in the linear regime for each method, (iii) the impact of the pairwise relations used during training on each of those quantities and on downstream task performances, and most importantly, (iv) the first theoretical bridge between contrastive and non-contrastive methods towards global and local spectral embedding methods respectively, hinting at the benefits and limitations of each. For example, (i) if the pairwise relation is aligned with the downstream task, any SSL method can be employed successfully and will recover the supervised method, but in the low data regime, VICReg's invariance hyper-parameter should be high; (ii) if the pairwise relation is misaligned with the downstream task, VICReg with small invariance hyper-parameter should be preferred over SimCLR or BarlowTwins.

Contrastive and Non-Contrastive Self-Supervised Learning Recover Global and Local Spectral Embedding Methods

A nonsingular n*n matrix A is given with its short displacement generator. It has small displacement rank bounded by a fixed constant. The class of such matrices generalizes Toeplitz matrices. A good initial approximation to a short displacement generator for A/sup -1/ is readily available. Ways to refine this approximation and numerically compute a displacement generator of A/sup -1/ and the solution vector x=A/sup -1/b to a linear system Ax=b by using O(log/sup 2/n) parallel arithmetic steps and n processors are presented. These results are extended to some other important classes of dense structure matrices. >

Concurrent iterative algorithm for Toeplitz-like linear systems

In this paper we consider canonical correlations and a generalization of the singular value decomposition (SVD) that involves three matrices. We show how the two matrix problems are related and how they can be used in important applications such as weighted least squares and optimal prediction. We present two new computational procedures for the problems based on implicit SVD methods for triple matrixproducts. Our algorithms are well suited for parallel implementation.

Canonical correlations and generalized SVD: Applications and new algorithms

In this paper we consider canonical correlations and a generalization of the singular value decomposition (SVD) that involves three matrices. We show how the two matrix problems are related and how they can be used in important applications such as weighted least squares and optimal prediction. We present two new computational procedures for the problems based on implicit SVD methods for triple matrix products. Our algorithms are well suited for parallel implementation.

Canonical Correlations And Generalized SVD: Applications And New Algorithms

Four matrix operations: matrix multiplication, the QR decomposition, the singular value decomposition, and the generalized singular value decomposition, form the basic tools for modern signal processing. This paper discusses their implementation on the 65,536-processor Connection Machine, and presents results showing that for n X n matrices, where n <256, an almost linear time performance is obtained. Our other major result is a novel method for computing the generalized singular value decomposition.

Almost Linear Time Matrix Operations On The Connection Machine

In this paper we consider the computation of generalized canonical correlations (gcc), a useful tool for linear prediction. The problem is reduced to a singular value decomposition of a product of three rank deficient matrices, for which a new algorithm is presented. It is then shown how the gcc may be computed directly from the data matrices.

L. Magnus Ewerbring

Papers

Canonical correlations and generalized SVD: Applications and new algorithms

Canonical Correlations And Generalized SVD: Applications And New Algorithms

Almost Linear Time Matrix Operations On The Connection Machine

Computing Generalized Canonical Correlations