From the Publisher:
This is the first comprehensive treatment of feed-forward neural networks from the perspective of statistical pattern recognition. After introducing the basic concepts, the book examines techniques for modelling probability density functions and the properties and merits of the multi-layer perceptron and radial basis function network models. Also covered are various forms of error functions, principal algorithms for error function minimalization, learning and generalization in neural networks, and Bayesian techniques and their applications. Designed as a text, with over 100 exercises, this fully up-to-date work will benefit anyone involved in the fields of neural computation and pattern recognition.

/pdf/neural-networks-for-pattern-recognition-34puigiau1.pdf

Neural networks for pattern recognition

The time-frequency and time-scale communities have recently developed a large number of overcomplete waveform dictionaries --- stationary wavelets, wavelet packets, cosine packets, chirplets, and warplets, to name a few. Decomposition into overcomplete systems is not unique, and several methods for decomposition have been proposed, including the method of frames (MOF), Matching pursuit (MP), and, for special dictionaries, the best orthogonal basis (BOB).
Basis Pursuit (BP) is a principle for decomposing a signal into an "optimal" superposition of dictionary elements, where optimal means having the smallest l1 norm of coefficients among all such decompositions. We give examples exhibiting several advantages over MOF, MP, and BOB, including better sparsity and superresolution. BP has interesting relations to ideas in areas as diverse as ill-posed problems, in abstract harmonic analysis, total variation denoising, and multiscale edge denoising.
BP in highly overcomplete dictionaries leads to large-scale optimization problems. With signals of length 8192 and a wavelet packet dictionary, one gets an equivalent linear program of size 8192 by 212,992. Such problems can be attacked successfully only because of recent advances in linear programming by interior-point methods. We obtain reasonable success with a primal-dual logarithmic barrier method and conjugate-gradient solver.

Atomic Decomposition by Basis Pursuit

Data Mining Practical Machine Learning Tools and Techniques

In recent years there has been a growing interest in the study of sparse representation of signals. Using an overcomplete dictionary that contains prototype signal-atoms, signals are described by sparse linear combinations of these atoms. Applications that use sparse representation are many and include compression, regularization in inverse problems, feature extraction, and more. Recent activity in this field has concentrated mainly on the study of pursuit algorithms that decompose signals with respect to a given dictionary. Designing dictionaries to better fit the above model can be done by either selecting one from a prespecified set of linear transforms or adapting the dictionary to a set of training signals. Both of these techniques have been considered, but this topic is largely still open. In this paper we propose a novel algorithm for adapting dictionaries in order to achieve sparse signal representations. Given a set of training signals, we seek the dictionary that leads to the best representation for each member in this set, under strict sparsity constraints. We present a new method-the K-SVD algorithm-generalizing the K-means clustering process. K-SVD is an iterative method that alternates between sparse coding of the examples based on the current dictionary and a process of updating the dictionary atoms to better fit the data. The update of the dictionary columns is combined with an update of the sparse representations, thereby accelerating convergence. The K-SVD algorithm is flexible and can work with any pursuit method (e.g., basis pursuit, FOCUSS, or matching pursuit). We analyze this algorithm and demonstrate its results both on synthetic tests and in applications on real image data

https://users.ece.utexas.edu/~sanghavi/courses/papers/KSVD.pdf

$rm K$ -SVD: An Algorithm for Designing Overcomplete Dictionaries for Sparse Representation

The time-frequency and time-scale communities have recently developed a large number of overcomplete waveform dictionaries---stationary wavelets, wavelet packets, cosine packets, chirplets, and warplets, to name a few. Decomposition into overcomplete systems is not unique, and several methods for decomposition have been proposed, including the method of frames (MOF), matching pursuit (MP), and, for special dictionaries, the best orthogonal basis (BOB).
Basis pursuit (BP) is a principle for decomposing a signal into an "optimal"' superposition of dictionary elements, where optimal means having the smallest l1 norm of coefficients among all such decompositions. We give examples exhibiting several advantages over MOF, MP, and BOB, including better sparsity and superresolution. BP has interesting relations to ideas in areas as diverse as ill-posed problems, abstract harmonic analysis, total variation denoising, and multiscale edge denoising.
BP in highly overcomplete dictionaries leads to large-scale optimization problems. With signals of length 8192 and a wavelet packet dictionary, one gets an equivalent linear program of size 8192 by 212,992. Such problems can be attacked successfully only because of recent advances in linear and quadratic programming by interior-point methods. We obtain reasonable success with a primal-dual logarithmic barrier method and conjugate-gradient solver.

/pdf/atomic-decomposition-by-basis-pursuit-2z0e8wwyak.pdf

Identification algorithms based on the well-known linear least squares methods of gaussian elimination, Cholesky decomposition, classical Gram-Schmidt, modified Gram-Schmidt, Householder transformation, Givens method, and singular value decomposition are reviewed. The classical Gram-Schmidt, modified Gram-Schmidt, and Householder transformation algorithms are then extended to combine structure determination, or which terms to include in the model, and parameter estimation in a very simple and efficient manner for a class of multivariate discrete-time non-linear stochastic systems which are linear in the parameters.

/pdf/orthogonal-least-squares-methods-and-their-application-to-21gya76qb2.pdf

Orthogonal least squares methods and their application to non-linear system identification

Recursive input-output models for non-linear multivariate discrete-time systems are derived, and sufficient conditions for their existence are defined. The paper is divided into two parts. The first part introduces and defines concepts such as Nerode realization, multistructural forms and results from differential geometry which are then used to derive a recursive input-output model for multivariable deterministic non-linear systems. The second part introduces several examples, compares the derived model with other representations and extends the results to create prediction error or innovation input-output models for non-linear stochastic systems. These latter models are the generalization of the multivariable ARM AX models for linear systems and are referred to as NARMAX or Non-linear AutoRegressive Moving Average models with exogenous inputs.

/pdf/input-output-parametric-models-for-non-linear-systems-part-44hf73mgve.pdf

Input-output parametric models for non-linear systems Part II: stochastic non-linear systems

Multi-layered neural networks offer an exciting alternative for modelling complex non-linear systems. This paper investigates the identification of discrete-time nonlinear systems using neural networks with a single hidden layer. New parameter estimation algorithms are derived for the neural network model based on a prediction error formulation and the application to both simulated and real data is included to demonstrate the effectiveness of the neural network approach.

/pdf/non-linear-system-identification-using-neural-networks-4t6814yxut.pdf

Non-linear system identification using neural networks

Input-output representations of non-linear discrete-time systems are discussed. It is shown that the NARMAX (Non-linear AutoRegressive Moving Average with eXogenous inputs) model is a general and natural representation of non-linear systems and contains, as special cases, several existing non-linear models. The problem of approximating non-linear input-output systems is also addressed and several properties of non-linear models are highlighted using simple examples.

Representations of non-linear systems: the NARMAX model

The NARMAX (nonlinear autoregressive moving average with exogenous inputs) model The orthogonal least squares algorithm that allows models to be built term by term where the error reduction ratio reveals the percentage contribution of each model term Statistical and qualitative model validation methods that can be applied to any model class Generalised frequency response functions which provide significant insight into nonlinear behaviours A completely new class of filters that can move, split, spread, and focus energy The response spectrum map and the study of sub harmonic and severely nonlinear systems Algorithms that can track rapid time variation in both linear and nonlinear systems The important class of spatio–temporal systems that evolve over both space and time Many case study examples from modelling space weather, through identification of a model of the visual processing system of fruit flies, to tracking causality in EGG data are all included to demonstrate how easily the methods can be applied in practice and to show the insight that the algorithms reveal even for complex systems

Stephen A. Billings

Papers

Orthogonal least squares methods and their application to non-linear system identification

Input-output parametric models for non-linear systems Part II: stochastic non-linear systems

Non-linear system identification using neural networks

Representations of non-linear systems: the NARMAX model

Nonlinear System Identification: NARMAX Methods in the Time, Frequency, and Spatio-Temporal Domains