From the Publisher:
This is the first comprehensive treatment of feed-forward neural networks from the perspective of statistical pattern recognition. After introducing the basic concepts, the book examines techniques for modelling probability density functions and the properties and merits of the multi-layer perceptron and radial basis function network models. Also covered are various forms of error functions, principal algorithms for error function minimalization, learning and generalization in neural networks, and Bayesian techniques and their applications. Designed as a text, with over 100 exercises, this fully up-to-date work will benefit anyone involved in the fields of neural computation and pattern recognition.

/pdf/neural-networks-for-pattern-recognition-34puigiau1.pdf

Neural networks for pattern recognition

Forecasting with artificial neural networks: the state of the art

Preface * Introduction * The Bayes Error * Inequalities and alternatedistance measures * Linear discrimination * Nearest neighbor rules *Consistency * Slow rates of convergence Error estimation * The regularhistogram rule * Kernel rules Consistency of the k-nearest neighborrule * Vapnik-Chervonenkis theory * Combinatorial aspects of Vapnik-Chervonenkis theory * Lower bounds for empirical classifier selection* The maximum likelihood principle * Parametric classification *Generalized linear discrimination * Complexity regularization *Condensed and edited nearest neighbor rules * Tree classifiers * Data-dependent partitioning * Splitting the data * The resubstitutionestimate * Deleted estimates of the error probability * Automatickernel rules * Automatic nearest neighbor rules * Hypercubes anddiscrete spaces * Epsilon entropy and totally bounded sets * Uniformlaws of large numbers * Neural networks * Other error estimates *Feature extraction * Appendix * Notation * References * Index

/pdf/a-probabilistic-theory-of-pattern-recognition-1p19dg485x.pdf

A Probabilistic Theory of Pattern Recognition

According to conventional neural network theories, single-hidden-layer feedforward networks (SLFNs) with additive or radial basis function (RBF) hidden nodes are universal approximators when all the parameters of the networks are allowed adjustable. However, as observed in most neural network implementations, tuning all the parameters of the networks may cause learning complicated and inefficient, and it may be difficult to train networks with nondifferential activation functions such as threshold networks. Unlike conventional neural network theories, this paper proves in an incremental constructive method that in order to let SLFNs work as universal approximators, one may simply randomly choose hidden nodes and then only need to adjust the output weights linking the hidden layer and the output layer. In such SLFNs implementations, the activation functions for additive nodes can be any bounded nonconstant piecewise continuous functions g:R→R and the activation functions for RBF nodes can be any integrable piecewise continuous functions g:R→R and ∫Rg(x)dx≠0. The proposed incremental method is efficient not only for SFLNs with continuous (including nondifferentiable) activation functions but also for SLFNs with piecewise continuous (such as threshold) activation functions. Compared to other popular methods such a new network is fully automatic and users need not intervene the learning process by manually tuning control parameters.

Universal approximation using incremental constructive feedforward networks with random hidden nodes

/pdf/neural-networks-and-deep-learning-3ccs3ain8p.pdf

Neural Networks and Deep Learning

There have been several recent studies concerning feedforward networks and the problem of approximating arbitrary functionals of a finite number of real variables. Some of these studies deal with cases in which the hidden-layer nonlinearity is not a sigmoid. This was motivated by successful applications of feedforward networks with nonsigmoidal hidden-layer units. This paper reports on a related study of radial-basis-function (RBF) networks, and it is proved that RBF networks having one hidden layer are capable of universal approximation. Here the emphasis is on the case of typical RBF networks, and the results show that a certain class of RBF networks with the same smoothing factor in each kernel node is broad enough for universal approximation.

Universal approximation using radial-basis-function networks

This paper concerns conditions for the approximation of functions in certain general spaces using radial-basis-function networks. It has been shown in recent papers that certain classes of radial-basis-function networks are broad enough for universal approximation. In this paper these results are considerably extended and sharpened.

Approximation and radial-basis-function networks

One of the main results is a proposition to the effect that under some typically mild conditions finite sums of the form

$$\sum\limits_\ell {K_\ell \sigma } \left[ {\sum\limits_m {\eta _{\ell m} Q_m (\cdot) + \rho _\ell } } \right]$$

are dense in an important sense in the set of shift-invariant approximately-finite-memory mapsG(·) that take a certain type of subsetU ofR intoR, whereR is the set of real-valued functions defined onR n orZ n . Here theQ m (·) are linear, σ is any element of a certain set of nonlinear maps fromR toR, and the κl, ρl, and ηlm are real constants. Approximate representations comprising only affine elements and lattice nonlinearities are also presented.

Structure theorems for nonlinear systems

We give results concerning the problem of approximating the input-output maps of nonlinear discrete-time approximately finite-memory systems. Here the focus is on the linear dynamical parts of the approximating structures, and we give examples showing that these linear parts can be derived from asingle prespecified function that meets certain conditions. This is done in the context of an approximation theorem in which attention is focused on what we call "basic sets." We also consider the related but very different problem of approximating nonlinear functionals using lattice operations or the usual linear ring operations. For this problem we give criteria, not just sufficient conditions, for approximation on compact subsets of reflexive Banach spaces (any Hilbert space is a reflexive Banach space).

Network approximation of input-output maps and functionals

We give an expression for the most general input-output map associated with the members of a certain important large family of multidimensional linear shift-invariant systems with bounded Lebesgue-measurable inputs. The expression given is an iterated function-space limit of a convolution. We also give a necessary and sufficient condition under which the limit can be written as a convolution with an integrable impulse-response function. A key role is played by a certain family of weighting operators. It is observed that for the large family of inputs and maps addressed, the Dirac impulse-response concept is in fact not the key concept concerning the representation of H, and that instead the inputoutput properties of H are determined in general by a certain type of family of responses. Some related material concerning engineering education is also given.

Irwin W. Sandberg

Papers

Universal approximation using radial-basis-function networks

Approximation and radial-basis-function networks

Structure theorems for nonlinear systems

Network approximation of input-output maps and functionals

Bounded Inputs and the Representation of Linear System Maps