Multilayer neural networks trained with the back-propagation algorithm constitute the best example of a successful gradient based learning technique. Given an appropriate network architecture, gradient-based learning algorithms can be used to synthesize a complex decision surface that can classify high-dimensional patterns, such as handwritten characters, with minimal preprocessing. This paper reviews various methods applied to handwritten character recognition and compares them on a standard handwritten digit recognition task. Convolutional neural networks, which are specifically designed to deal with the variability of 2D shapes, are shown to outperform all other techniques. Real-life document recognition systems are composed of multiple modules including field extraction, segmentation recognition, and language modeling. A new learning paradigm, called graph transformer networks (GTN), allows such multimodule systems to be trained globally using gradient-based methods so as to minimize an overall performance measure. Two systems for online handwriting recognition are described. Experiments demonstrate the advantage of global training, and the flexibility of graph transformer networks. A graph transformer network for reading a bank cheque is also described. It uses convolutional neural network character recognizers combined with global training techniques to provide record accuracy on business and personal cheques. It is deployed commercially and reads several million cheques per day.

Gradient-based learning applied to document recognition

Setting of the learning problem consistency of learning processes bounds on the rate of convergence of learning processes controlling the generalization ability of learning processes constructing learning algorithms what is important in learning theory?.

The Nature of Statistical Learning Theory

There is, I think, something ethereal about i —the square root of minus one. I remember first hearing about it at school. It seemed an odd beast at that time—an intruder hovering on the edge of reality.

Usually familiarity dulls this sense of the bizarre, but in the case of i it was the reverse: over the years the sense of its surreal nature intensified. It seemed that it was impossible to write mathematics that described the real world in …

I and i

抗原变异可使得多种致病微生物易于逃避宿主免疫应答。表达在感染红细胞表面的恶性疟原虫红细胞表面蛋白1（PfPMP1）与感染红细胞、内皮细胞、树突状细胞以及胎盘的单个或多个受体作用，在黏附及免疫逃避中起关键的作用。每个单倍体基因组var基因家族编码约60种成员，通过启动转录不同的var基因变异体为抗原变异提供了分子基础。

疟原虫var基因转换速率变化导致抗原变异[英]／Paul H, Robert P, Christodoulou Z, et al//Proc Natl Acad Sci U S A

https://sccn.ucsd.edu/eeglab/download/eeglab_jnm03.pdf

EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics including independent component analysis.

A manifold of probability distributions naturally possesses a dual pair of affine connections together with the Riemannian metric induced by Fisher information. Such a dual geometrical structure plays a fundamental role in various fields of information sciences such as statistical inference, systems theory, information theory and also stochastic theory of neural networks. This is also a key concept connecting statistical inference and statistical physics. Information geometry is introduced in an intuitive manner and it is applied to elucidate the family of probability distributions realized by stochastic neural networks called the higher-order Boltzmann machines.

Information Geometry and Manifolds of Neural Networks

In a manner similar to the molecular chaos that underlies the stable thermodynamics of gases, a neuronal system may exhibit microscopic instability in individual neuronal dynamics while a macroscopic order of the entire population possibly remains stable. In this study, we analyze the microscopic stability of a network of neurons whose macroscopic activity obeys stable dynamics, expressing either monostable, bistable, or periodic state. We reveal that the network exhibits a variety of dynamical states for microscopic instability residing in a given stable macroscopic dynamics. The presence of a variety of dynamical states in such a simple random network implies more abundant microscopic fluctuations in real neural networks which consist of more complex and hierarchically structured interactions.

Microscopic instability in recurrent neural networks

Blind Decorrelation and SOS for Robust Blind Identification

In a multimedia information providing system, responding to a preview request from a terminal equipment, a server device generates a chaotic random number sequence from an initial value allocated to the terminal equipment, mixes this sequence into a set of multimedia information as a noise signal at a first mixing ratio, and then transmits the result as a first set of multimedia information for distribution. Responding to a quality improvement request, the server device generates a chaotic number sequence from the initial value allocated to the terminal equipment, mixes this sequence into a set of multimedia information as a noise signal at a second mixing ratio, and then transmits the result as a second set of multimedia information for distribution. The terminal equipment performs component analysis on the first and second sets of multimedia information, removing the noise and restoring a high-quality version.

Multimedia information providing system, server device, terminal equipment, multimedia information providing method, and computer-readable medium

The Fisher information matrix (FIM) plays an essential role in statistics and machine learning as a Riemannian metric tensor or a component of the Hessian matrix of loss functions. Focusing on the FIM and its variants in deep neural networks (DNNs), we reveal their characteristic scale dependence on the network width, depth, and sample size when the network has random weights and is sufficiently wide. This study covers two widely used FIMs for regression with linear output and for classification with softmax output. Both FIMs asymptotically show pathological eigenvalue spectra in the sense that a small number of eigenvalues become large outliers depending on the width or sample size, while the others are much smaller. It implies that the local shape of the parameter space or loss landscape is very sharp in a few specific directions while almost flat in the other directions. In particular, the softmax output disperses the outliers and makes a tail of the eigenvalue density spread from the bulk. We also show that pathological spectra appear in other variants of FIMs: one is the neural tangent kernel; another is a metric for the input signal and feature space that arises from feedforward signal propagation. Thus, we provide a unified perspective on the FIM and its variants that will lead to more quantitative understanding of learning in large-scale DNNs.

Shun-ichi Amari

Papers

Information Geometry and Manifolds of Neural Networks

Microscopic instability in recurrent neural networks

Blind Decorrelation and SOS for Robust Blind Identification

Multimedia information providing system, server device, terminal equipment, multimedia information providing method, and computer-readable medium

Pathological Spectra of the Fisher Information Metric and Its Variants in Deep Neural Networks.