Supervised learning plays a key role in the operation of many biological and artificial neural networks. Analysis of the computations underlying supervised learning is facilitated by the relatively simple and uniform architecture of the cerebellum, a brain area that supports numerous motor, sensory, and cognitive functions. We highlight recent discoveries indicating that the cerebellum implements supervised learning using the following organizational principles: ( a) extensive preprocessing of input representations (i.e., feature engineering), ( b) massively recurrent circuit architecture, ( c) linear input-output computations, ( d) sophisticated instructive signals that can be regulated and are predictive, ( e) adaptive mechanisms of plasticity with multiple timescales, and ( f) task-specific hardware specializations. The principles emerging from studies of the cerebellum have striking parallels with those in other brain areas and in artificial neural networks, as well as some notable differences, which can inform future research on supervised learning and inspire next-generation machine-based algorithms.

/pdf/computational-principles-of-supervised-learning-in-the-3bcaecyyz1.pdf

Computational Principles of Supervised Learning in the Cerebellum.

Normalized least mean squares algorithms for FIR adaptive filtering with or without the reuse of past information are known to converge often faster than the conventional least mean squares (LMS) algorithm. This correspondence analyzes an LMS-like algorithm: the binormalized data-reusing least mean squares (BNDR-LMS) algorithm. This algorithm, which corresponds to the affine projection algorithm for the case of two projections, compares favorably with other normalized LMS-like algorithms when the input signal is correlated. Convergence analyses in the mean and in the mean-squared are presented, and a closed-form formula for the mean squared error is provided for white input signals as well as its extension to the case of a colored input signal. A simple model for the input-signal vector that imparts simplicity and tractability to the analysis of second-order statistics is fully described. The methodology is readily applicable to other adaptation algorithms of difficult analysis. Simulation results validate the analysis and ensuing assumptions.

Convergence analysis of the binormalized data-reusing LMS algorithm

An exhaustive review on the use of structured stochastic search approaches towards system identification and digital filter design is presented in this paper. In particular, the paper focuses on the identification of various systems using infinite impulse response adaptive filters and Hammerstein models as well as on the estimation of chaotic systems. In addition to presenting a comprehensive review on the various swarm and evolutionary computing schemes employed for system identification as well as digital filter design, the paper is also envisioned to act as a quick reference for a few popular evolutionary computing algorithms.

Swarm and evolutionary computing algorithms for system identification and filter design: A comprehensive review

Most adaptive filtering algorithms couple performance with complexity. Over the last 15 years, a class of algorithms, termed "affine projection" algorithms, have given system designers the capability to tradeoff performance with complexity. By changing parameters and the size/scale of data used to update the coefficients of an adaptive filter but without fundamentally changing the algorithm structure, a system designer can radically change the performance of the adaptive algorithm. This paper discusses low-complexity data reusing algorithms that are closely related to affine projection algorithms. This paper presents various low-complexity and highly flexible schemes for improving convergence rates of adaptive algorithms that utilize data reusing strategies. All of these schemes are unified by a row projection framework in existence for more than 65 years. This framework leads to the classification of all data reusing and affine projection methods for adaptive filtering into two categories: the Kaczmarz and Cimmino methods. Simulation and convergence analysis results are presented for these methods under a number of conditions. They are compared in terms of convergence rate performance and computational complexity.

Low-complexity data reusing methods in adaptive filtering

Harmony search algorithm for infinite impulse response system identification

Preface. 1: Introduction and Background. 1.1. Common Adaptive Concepts from Different Disciplines. 1.2. Generic Applications of Adaptive Methods. 1.3. Performance Measures in Adaptive Systems. 1.4. The Minimum Mean Squared Error Solution. 1.5. Adaptive Algorithms for FIR Systems. 1.6. Adaptive Algorithms for IIR Systems. 1.7. New Horizons in Adaptive Signal Processing. 1.8. Notation and Conventions. 2: Advanced Algorithms for 1-D Adaptive Filtering. 2.2. Data- Reusing LMS Algorithms. 2.3. Orthogonalization by PR Modulation. 2.3. The Gauss-Newton Adaptive Filtering algorithm. 2.4. Block Adaptive IIR Filters Using the PCG Method. 3: Structures and Algorithms for Two-Dimensional Adaptive Signal Processing. 3.1. Applications of Two-Dimensional Adaptive Filtering. 3.2. Two- Dimensional FIR Adaptive Filtering. 3.3. Two-Dimensional IIR Adaptive Filters. 3.4. Two-Dimensional IIR Adaptive Filtering Experiments. 3.5. Uniqueness Characteristics of the 2-D IIR MSE Minimization. 4: Adaptive Fault Tolerance. 4.1. Application of AFT to FIR Adaptive Filters. 4.2. Adaptive Filter Structures. 4.3. A Simple Fault Tolerant FIR Adaptive Filter. 4.4. The Transform Domain FTAF. 4.5. The DFT-Based TDFTAF with the Conjugate Gradient. 4.6. Robust and Practical TDFTAFs. 4.7. Full Fault Tolerance Transforms. 4.8. Discussion. 5: Adaptive Polynomial Filters. 5.1. The Volterra Series. 5.2. Gradient Based Adaptive Volterra Filters. 5.3. RLSSecond-Order Volterra Adaptive Filter. 5.4. LS Lattice Second-Order Volterra Adaptive Filter. 5.5. QR-Based LS Lattice Second Order Volterra Filter. 5.6. The Adaptive Volterra Filter for Gaussian Signals. 5.7. Other Polynomial-Based Nonlinear Adaptive Filters. 5.8. Discussion. Appendix. Subject Index.

Advanced Concepts in Adaptive Signal Processing

Two-dimensional adaptive digital filters with reduced computational complexity

Linear filtering techniques are known to be effective in a variety of applications in signal processing, telecommunications, and control. However, in some applications where linear filters do not give satisfactory performance, nonlinear filtering techniques are able to provide better performance. For example, in image processing linear filtering methods cannot effectively remove impulsive noise, and they tend to blur edges. On the contrary nonlinear filters, for example the median and order statistic filters, are more effective in removing impulsive noise. In telecommunications it has been shown that the error probability of transmitting data through a telephone channel at rates higher than 4800 bits/s is caused almost entirely by nonlinear distortion [5.1]. Recently there has been a tremendous demand for sending data at much higher rates (64 kbits/s, 128 kbits/s) through subscriber lines [5.2]. In these cases, even a slight nonlinearity may cause problems for a conventional linear echo canceler. In data communication over a fading channel it has been shown that nonlinear equalization must be used to compensate for severe distortion introduced by the transmission channel [5.3].

Adaptive Polynomial Filters

Adaptive Fault Tolerance (APT) is a novel fault tolerance approach applicable to adaptive systems which makes use of the inherent adaptive process as an automatic fault tolerance mechanism [1.14]. Adaptive fault tolerance is motivated as follows. Under normal operating conditions adaptive systems, such as adaptive echo cancelers, adaptive equalizers, and adaptive controllers, adjust their own system parameters to reduce a specified error criterion. Hardware failures in such systems would presumably hamper their ability to minimize the error criterion to the greatest possible extent. However, they will continue to adapt their system parameters to reduce this error to the greatest possible extent despite the occurrence of hardware failures. Since this is the case, it may be possible to design a system that makes use of the ongoing adaptive process to automatically compensate for certain types of hardware failures.

Adaptive Fault Tolerance

A new adaptive 2-D digital filter is presented in which the filter is deterniined by a 1-D FIR prototype, and hence the computational complexity of the coefficient up-date algorithm has an order of complexity O(M), rather than O(M/sup 2/). This new adaptive 2-D filter is particularly attractive for applications in real-time image processing for robotics and computer vision where constrained adaptive filter may be quite useful, and where reduced computational complexity is essential for real-time operation.

Jeffrey C. Strait

Papers

Advanced Concepts in Adaptive Signal Processing

Two-dimensional adaptive digital filters with reduced computational complexity

Adaptive Polynomial Filters

Adaptive Fault Tolerance

Convergence Properties Of A Constrained 2-D Adaptive Digital Filter