Probability Distributions.- Linear Models for Regression.- Linear Models for Classification.- Neural Networks.- Kernel Methods.- Sparse Kernel Machines.- Graphical Models.- Mixture Models and EM.- Approximate Inference.- Sampling Methods.- Continuous Latent Variables.- Sequential Data.- Combining Models.

Pattern Recognition and Machine Learning

Suggested by the structure of the visual nervous system, a new algorithm is proposed for pattern recognition. This algorithm can be realized with a multilayered network consisting of neuron-like cells. The network, “neocognitron”, is self-organized by unsupervised learning, and acquires the ability to recognize stimulus patterns according to the differences in their shapes: Any patterns which we human beings judge to be alike are also judged to be of the same category by the neocognitron. The neocognitron recognizes stimulus patterns correctly without being affected by shifts in position or even by considerable distortions in shape of the stimulus patterns.

Neocognitron--A New Algorithm for Pattern Recognition Tolerant of Deformations and Shifts in Position

https://agupubs.onlinelibrary.wiley.com/doi/epdf/10.1029/2018WR022643

A Transdisciplinary Review of Deep Learning Research and Its Relevance for Water Resources Scientists

The Self-Organizing Map (SOM) algorithm has attracted a great deal of interest among researches and practitioners in a wide variety of fields. The SOM has been analyzed extensively, a number of variants have been developed and, perhaps most notably, it has been applied extensively within fields ranging from engineering sciences to medicine, biology, and economics. We have collected a comprehensive list of 5384 scientific papers that use the algorithms, have benefited from them, or contain analyses of them. The list is intended to serve as a source for literature surveys. The present addendum contains 2092 new articles, mainly from the years 1998-2002. We have provided a keyword index to help finding articles of interest, and additionally a modern automatically constructed variant of a thematic index: a WEBSOM interface to the whole article collection of years 1981-2000. The SOM of SOMs is available at http://websom.hut.fi/websom/somref/search.cgi for browsing and searching the collection.

Bibliography of Self-Organizing Map SOM) Papers: 1998-2001 Addendum

Independent component analysis is a probabilistic method for learning a linear transform of a random vector. The goal is to find components that are maximally independent and non-Gaussian (non-normal). Its fundamental difference to classical multi-variate statistical methods is in the assumption of non-Gaussianity, which enables the identification of original, underlying components, in contrast to classical methods. The basic theory of independent component analysis was mainly developed in the 1990s and summarized, for example, in our monograph in 2001. Here, we provide an overview of some recent developments in the theory since the year 2000. The main topics are: analysis of causal relations, testing independent components, analysing multiple datasets (three-way data), modelling dependencies between the components and improved methods for estimating the basic model.

https://royalsocietypublishing.org/doi/pdf/10.1098/rsta.2011.0534

Independent component analysis: recent advances.

Deep Learning is attracting much attention in object recognition and speech processing. A benefit of using the deep learning is that it provides automatic pre-training. Several proposed methods that include auto-encoder are being successfully used in various applications. Moreover, deep learning uses a multilayer network that consists of many layers, a huge number of units, and huge amount of data. Thus, executing deep learning requires heavy computation, so deep learning is usually utilized with parallel computation with many cores or many machines. Deep learning employs the gradient algorithm, however this traps the learning into the saddle point or local minima. To avoid this difficulty, a rectified linear unit (ReLU) is proposed to speed up the learning convergence. However, the reasons the convergence is speeded up are not well understood. In this paper, we analyze the ReLU by a using simpler network called the soft-committee machine and clarify the reason for the speedup. We also train the network in an on-line manner. The soft-committee machine provides a good test bed to analyze deep learning. The results provide some reasons for the speedup of the convergence of the deep learning.

Analysis of function of rectified linear unit used in deep learning

Deep learning is the state-of-the-art in fields such as visual object recognition and speech recognition. This learning uses a large number of layers, huge number of units, and connections. Therefore, overfitting is a serious problem. To avoid this problem, dropout learning is proposed. Dropout learning neglects some inputs and hidden units in the learning process with a probability, p, and then, the neglected inputs and hidden units are combined with the learned network to express the final output. We find that the process of combining the neglected hidden units with the learned network can be regarded as ensemble learning, so we analyze dropout learning from this point of view.

Analysis of Dropout Learning Regarded as Ensemble Learning

We investigate the accuracy of statistical-mechanical approximations for the estimation of hyperparameters from observable data in probabilistic image processing, which is based on Bayesian statistics and maximum likelihood estimation. Hyperparameters in statistical science correspond to interactions or external fields in the statistical-mechanics context. In this paper, hyperparameters in the probabilistic model are determined so as to maximize a marginal likelihood. A practical algorithm is described for grey-level image restoration based on a Gaussian graphical model and the Bethe approximation. The algorithm corresponds to loopy belief propagation in artificial intelligence. We examine the accuracy of hyperparameter estimation when we use the Bethe approximation. It is well known that a practical algorithm for probabilistic image processing can be prescribed analytically when a Gaussian graphical model is adopted as a prior probabilistic model in Bayes' formula. We are therefore able to compare, in a numerical study, results obtained through mean-field-type approximations with those based on exact calculation.

Accuracy of the Bethe approximation for hyperparameter estimation in probabilistic image processing

Analysis of dropout learning regarded as ensemble learning

Using a large-scale real-world database-the ETL-1 database of the Electrotechnical Laboratory in Japan-we show that a neocognitron trained by unsupervised learning with a winner-take-all process can recognize handwritten digits with a recognition rate higher than 97%. We use the technique of dual thresholds for feature-extracting S-cells, and higher threshold values are used in the learning than in the recognition phase. We discuss how the threshold values affect the recognition rate. The learning method for the cells of the highest stage of the network has been modified from the conventional one, in order to reconcile the unsupervised learning procedure with the use of information about the category names of the training patterns.

http://ieeexplore.ieee.org/iel3/4457/12637/00581680.pdf

Hayaru Shouno

Papers

Analysis of function of rectified linear unit used in deep learning

Analysis of Dropout Learning Regarded as Ensemble Learning

Accuracy of the Bethe approximation for hyperparameter estimation in probabilistic image processing

Analysis of dropout learning regarded as ensemble learning

Training neocognitron to recognize handwritten digits in the real world