Giancarlo Kerg

TL;DR: In this paper, the authors propose a connectivity structure based on the Schur decomposition, which allows to parametrize matrices with unit-norm eigenspectra without orthogonality constraints on eigenbases.

TL;DR: In this paper, a stochastic algorithm called H-detach was proposed to prevent the vanishing gradient problem in LSTM by suppressing the gradient components through the linear path (cell state) in the computational graph, which can prevent LSTMs from capturing long-term dependencies.

TL;DR: This work proposes a novel connectivity structure based on the Schur decomposition and a splitting of theSchur form into normal and non-normal parts that retains the stability advantages and training speed of orthogonal RNNs while enhancing expressivity, especially on tasks that require computations over ongoing input sequences.

TL;DR: A simple stochastic algorithm is introduced that prevents gradients flowing through this path from getting suppressed, thus allowing the LSTM to capture such dependencies better and show significant improvements over vanilla L STM gradient based training in terms of convergence speed, robustness to seed and learning rate, and generalization.

TL;DR: This paper showed that the early value of the trace of the Fisher Information Matrix (FIM) correlates strongly with the final generalization and showed that in the absence of implicit or explicit regularization, the trace can increase to a large value early in training, to which they refer as catastrophic Fisher explosion.