From the Publisher:
This is the first comprehensive treatment of feed-forward neural networks from the perspective of statistical pattern recognition. After introducing the basic concepts, the book examines techniques for modelling probability density functions and the properties and merits of the multi-layer perceptron and radial basis function network models. Also covered are various forms of error functions, principal algorithms for error function minimalization, learning and generalization in neural networks, and Bayesian techniques and their applications. Designed as a text, with over 100 exercises, this fully up-to-date work will benefit anyone involved in the fields of neural computation and pattern recognition.

/pdf/neural-networks-for-pattern-recognition-34puigiau1.pdf

Neural networks for pattern recognition

Probability Distributions.- Linear Models for Regression.- Linear Models for Classification.- Neural Networks.- Kernel Methods.- Sparse Kernel Machines.- Graphical Models.- Mixture Models and EM.- Approximate Inference.- Sampling Methods.- Continuous Latent Variables.- Sequential Data.- Combining Models.

Pattern Recognition and Machine Learning

It is clear that the learning speed of feedforward neural networks is in general far slower than required and it has been a major bottleneck in their applications for past decades. Two key reasons behind may be: 1) the slow gradient-based learning algorithms are extensively used to train neural networks, and 2) all the parameters of the networks are tuned iteratively by using such learning algorithms. Unlike these traditional implementations, this paper proposes a new learning algorithm called extreme learning machine (ELM) for single-hidden layer feedforward neural networks (SLFNs) which randomly chooses the input weights and analytically determines the output weights of SLFNs. In theory, this algorithm tends to provide the best generalization performance at extremely fast learning speed. The experimental results based on real-world benchmarking function approximation and classification problems including large complex applications show that the new algorithm can produce best generalization performance in some cases and can learn much faster than traditional popular learning algorithms for feedforward neural networks.

Extreme learning machine: a new learning scheme of feedforward neural networks

According to conventional neural network theories, single-hidden-layer feedforward networks (SLFNs) with additive or radial basis function (RBF) hidden nodes are universal approximators when all the parameters of the networks are allowed adjustable. However, as observed in most neural network implementations, tuning all the parameters of the networks may cause learning complicated and inefficient, and it may be difficult to train networks with nondifferential activation functions such as threshold networks. Unlike conventional neural network theories, this paper proves in an incremental constructive method that in order to let SLFNs work as universal approximators, one may simply randomly choose hidden nodes and then only need to adjust the output weights linking the hidden layer and the output layer. In such SLFNs implementations, the activation functions for additive nodes can be any bounded nonconstant piecewise continuous functions g:R→R and the activation functions for RBF nodes can be any integrable piecewise continuous functions g:R→R and ∫Rg(x)dx≠0. The proposed incremental method is efficient not only for SFLNs with continuous (including nondifferentiable) activation functions but also for SLFNs with piecewise continuous (such as threshold) activation functions. Compared to other popular methods such a new network is fully automatic and users need not intervene the learning process by manually tuning control parameters.

Universal approximation using incremental constructive feedforward networks with random hidden nodes

An artificial neural network (ANN) is a flexible mathematical structure which is capable of identifying complex nonlinear relationships between input and output data sets. ANN models have been found useful and efficient, particularly in problems for which the characteristics of the processes are difficult to describe using physical equations. This study presents a new procedure (entitled linear least squares simplex, or LLSSIM) for identifying the structure and parameters of three-layer feed forward ANN models and demonstrates the potential of such models for simulating the nonlinear hydrologic behavior of watersheds. The nonlinear ANN model approach is shown to provide a better representation of the rainfall-runoff relationship of the medium-size Leaf River basin near Collins, Mississippi, than the linear ARMAX (autoregressive moving average with exogenous inputs) time series approach or the conceptual SAC-SMA (Sacramento soil moisture accounting) model. Because the ANN approach presented here does not provide models that have physically realistic components and parameters, it is by no means a substitute for conceptual watershed modeling. However, the ANN approach does provide a viable and effective alternative to the ARMAX time series approach for developing input-output simulation and forecasting models in situations that do not require modeling of the internal structure of the watershed.

Artificial Neural Network Modeling of the Rainfall‐Runoff Process

Representation of functions by superpositions of a step or sigmoid function and their applications to neural network theory

Approximation of functions on a compact set by finite sums of a sigmoid function without scaling

Original Contribution: Approximation of continuous functions on Rd by linear combinations of shifted rotations of a sigmoid function with and without scaling

With the aim of treating nonlinear systems with inputs being discrete and outputs being generalized functions, generalized Poisson functional are defined and analysed, where the
 -transforms and the renormalizational play essential roles. For Poisson functionals, the differential operators with respect to a Poisson white noise
 
$$\dot P$$

 
(t), their adjoint operators and the multiplication operators by
 
$$\dot P$$

 
(t) are defined. Since these operators involve the time parameter explicitly, they can be used to obtain information concerning the Poisson functional at each point in time. As an example, a new method for measuring the Wiener kernels of such functionals is outlined.

Generalized Poisson Functionals

Recently one of the authors has introduced the concept of generalized Poisson functionals and discussed the differentiation, renormalization, stochastic integrals etc ([8], [9]), analogously to the works of T Hida ([3], [4], [5]) Here we introduce a transformation for Poisson fnnctionals with the idea as in the case of Gaussian white noise (cf [10], [11], [12], [13]) Then we can discuss the differentiation, renormalization, multiple Wiener integrals etc in a way completely parallel with the Gaussian case The only one exceptional point, which is most significant, is that the multiplications are described by for the Gaussian case, for the Poisson case, as will be stated in Section 5 Conversely, those formulae characterize the types of white noises

/pdf/calculus-on-gaussian-and-poisson-white-noises-zs1unfn4d6.pdf

Yoshifusa Ito

Papers

Representation of functions by superpositions of a step or sigmoid function and their applications to neural network theory

Approximation of functions on a compact set by finite sums of a sigmoid function without scaling

Original Contribution: Approximation of continuous functions on Rd by linear combinations of shifted rotations of a sigmoid function with and without scaling

Generalized Poisson Functionals

Calculus on Gaussian and Poisson white noises