Showing papers in "IEEE Transactions on Neural Networks in 1996"

Multilayer neural-net robot controller with guaranteed tracking performance

Abstract: A multilayer neural-net (NN) controller for a general serial-link rigid robot arm is developed. The structure of the NN controller is derived using a filtered error/passivity approach. No off-line learning phase is needed for the proposed NN controller and the weights are easily initialized. The nonlinear nature of the NN, plus NN functional reconstruction inaccuracies and robot disturbances, mean that the standard delta rule using backpropagation tuning does not suffice for closed-loop dynamic control. Novel online weight tuning algorithms, including correction terms to the delta rule plus an added robust signal, guarantee bounded tracking errors as well as bounded NN weights. Specific bounds are determined, and the tracking error bound can be made arbitrarily small by increasing a certain feedback gain. The correction terms involve a second-order forward-propagated wave in the backpropagation network. New NN properties including the notions of a passive NN, a dissipative NN, and a robust NN are introduced.

Learning long-term dependencies in NARX recurrent neural networks

Abstract: It has previously been shown that gradient-descent learning algorithms for recurrent neural networks can perform poorly on tasks that involve long-term dependencies, i.e. those problems for which the desired output depends on inputs presented at times far in the past. We show that the long-term dependencies problem is lessened for a class of architectures called nonlinear autoregressive models with exogenous (NARX) recurrent neural networks, which have powerful representational capabilities. We have previously reported that gradient descent learning can be more effective in NARX networks than in recurrent neural network architectures that have "hidden states" on problems including grammatical inference and nonlinear system identification. Typically, the network converges much faster and generalizes better than other networks. The results in this paper are consistent with this phenomenon. We present some experimental results which show that NARX networks can often retain information for two to three times as long as conventional recurrent neural networks. We show that although NARX networks do not circumvent the problem of long-term dependencies, they can greatly improve performance on long-term dependency problems. We also describe in detail some of the assumptions regarding what it means to latch information robustly and suggest possible ways to loosen these assumptions.

Control of nonlinear dynamical systems using neural networks. II. Observability, identification, and control

Abstract: For pt. I see ibid., vol. 4 (1993). This paper considers the problems of regulation and tracking of a dynamical system when the state variables of the dynamical system are not accessible. The existence of the nonlinear maps describing the identifier and controller are first established and the implications for neural network realizations are described. Simulation results are included to complement the theoretical discussions.

Complex-valued multistate neural associative memory

Abstract: A model of a multivalued associative memory is presented. This memory has the form of a fully connected attractor neural network composed of multistate complex-valued neurons. Such a network is able to perform the task of storing and recalling gray-scale images. It is also shown that the complex-valued fully connected neural network may be considered as a generalization of a Hopfield network containing real-valued neurons. A computational energy function is introduced and evaluated in order to prove network stability for asynchronous dynamics. Storage capacity as related to the number of accessible neuron states is also estimated.

Input-output HMMs for sequence processing

Abstract: We consider problems of sequence processing and propose a solution based on a discrete-state model in order to represent past context. We introduce a recurrent connectionist architecture having a modular structure that associates a subnetwork to each state. The model has a statistical interpretation we call input-output hidden Markov model (IOHMM). It can be trained by the estimation-maximization (EM) or generalized EM (GEM) algorithms, considering state trajectories as missing data, which decouples temporal credit assignment and actual parameter estimation. The model presents similarities to hidden Markov models (HMMs), but allows us to map input sequences to output sequences, using the same processing style as recurrent neural networks. IOHMMs are trained using a more discriminant learning paradigm than HMMs, while potentially taking advantage of the EM algorithm. We demonstrate that IOHMMs are well suited for solving grammatical inference problems on a benchmark problem. Experimental results are presented for the seven Tomita grammars, showing that these adaptive models can attain excellent generalization.

Human expression recognition from motion using a radial basis function network architecture

Abstract: In this paper a radial basis function network architecture is developed that learns the correlation of facial feature motion patterns and human expressions. We describe a hierarchical approach which at the highest level identifies expressions, at the mid level determines motion of facial features, and at the low level recovers motion directions. Individual expression networks were trained to recognize the "smile" and "surprise" expressions. Each expression network was trained by viewing a set of sequences of one expression for many subjects. The trained neural network was then tested for retention, extrapolation, and rejection ability. Success rates were 88% for retention, 88% for extrapolation, and 83% for rejection.

Cooperative-competitive genetic evolution of radial basis function centers and widths for time series prediction

Abstract: In a radial basis function (RBF) network, the RBF centers and widths can be evolved by a cooperative-competitive genetic algorithm The set of genetic strings in one generation of the algorithm represents one REP network, not a population of competing networks This leads to moderate computation times for the algorithm as a whole Selection operates on individual RBFs rather than on whole networks Selection therefore requires a genetic fitness function that promotes competition among RBFs which are doing nearly the same job while at the same time promoting cooperation among RBFs which cover different parts of the domain of the function to be approximated Niche creation resulting from a fitness function of the form |w/sub i/|/sup /spl beta///E(|w/sub i'/|/sup /spl beta//), 1

A new neural network for solving linear and quadratic programming problems

Abstract: A new neural network for solving linear and quadratic programming problems is presented and is shown to be globally convergent. The new neural network improves existing neural networks for solving these problems: it avoids the parameter turning problem, it is capable of achieving the exact solutions, and it uses only simple hardware in which no analog multipliers for variables are required. Furthermore, the network solves both the primal problems and their dual problems simultaneously.

Confidence interval prediction for neural network models

Abstract: To derive an estimate of a neural network's accuracy as an empirical modeling tool, a method to quantify the confidence intervals of a neural network model of a physical system is desired. In general, a model of a physical system has error associated with its predictions due to the dependence of the physical system's output on uncontrollable or unobservable quantities. A confidence interval can be computed for a neural network model with the assumption of normally distributed error for the neural network. The proposed method accounts for the accuracy of the data with which the neural network model is trained.

Dynamic structure neural networks for stable adaptive control of nonlinear systems

Abstract: An adaptive control technique, using dynamic structure Gaussian radial basis function neural networks, that grow in time according to the location of the system's state in space is presented for the affine class of nonlinear systems having unknown or partially known dynamics. The method results in a network that is "economic" in terms of network size, for cases where the state spans only a small subset of state space, by utilizing less basis functions than would have been the case if basis functions were centered on discrete locations covering the whole, relevant region of state space. Additionally, the system is augmented with sliding control so as to ensure global stability if and when the state moves outside the region of state space spanned by the basis functions, and to ensure robustness to disturbances that arise due to the network inherent approximation errors and to the fact that for limiting the network size, a minimal number of basis functions are actually being used. Adaptation laws and sliding control gains that ensure system stability in a Lyapunov sense are presented, together with techniques for determining which basis functions are to form part of the network structure. The effectiveness of the method is demonstrated by experiment simulations.

Multilayer discrete-time neural-net controller with guaranteed performance

Abstract: A family of novel multilayer discrete-time neural-net (NN) controllers is presented for the control of a class of multi-input multi-output (MIMO) dynamical systems. The neural net controller includes modified delta rule weight tuning and exhibits a learning while-functioning-features. The structure of the NN controller is derived using a filtered error/passivity approach. Linearity in the parameters is not required and certainty equivalence is not used. This overcomes several limitations of standard adaptive control. The notion of persistency of excitation (PE) for multilayer NN is defined and explored. New online improved tuning algorithms for discrete-time systems are derived, which are similar to /spl sigma/ or /spl epsiv/-modification for the case of continuous-time systems, that include a modification to the learning rate parameter plus a correction term. These algorithms guarantee tracking as well as bounded NN weights in nonideal situations so that PE is not needed. An extension of these novel weight tuning updates to NN with an arbitrary number of hidden layers is discussed. The notions of discrete-time passive NN, dissipative NN, and robust NN are introduced. The NN makes the closed-loop system passive.

A review of Bayesian neural networks with an application to near infrared spectroscopy

Abstract: MacKay's (1992) Bayesian framework for backpropagation is a practical and powerful means to improve the generalization ability of neural networks. It is based on a Gaussian approximation to the posterior weight distribution. The framework is extended, reviewed, and demonstrated in a pedagogical way. The notation is simplified using the ordinary weight decay parameter, and a detailed and explicit procedure for adjusting several weight decay parameters is given. Bayesian backprop is applied in the prediction of fat content in minced meat from near infrared spectra. It outperforms "early stopping" as well as quadratic regression. The evidence of a committee of differently trained networks is computed, and the corresponding improved generalization is verified. The error bars on the predictions of the fat content are computed. There are three contributors: The random noise, the uncertainty in the weights, and the deviation among the committee members. The Bayesian framework is compared to Moody's GPE (1992). Finally, MacKay and Neal's automatic relevance determination, in which the weight decay parameters depend on the input number, is applied to the data with improved results.

Comparison of adaptive methods for function estimation from samples

Abstract: The problem of estimating an unknown function from a finite number of noisy data points has fundamental importance for many applications. This problem has been studied in statistics, applied mathematics, engineering, artificial intelligence, and, more recently, in the fields of artificial neural networks, fuzzy systems, and genetic optimization. In spite of many papers describing individual methods, very little is known about the comparative predictive (generalization) performance of various methods. We discuss subjective and objective factors contributing to the difficult problem of meaningful comparisons. We also describe a pragmatic framework for comparisons between various methods, and present a detailed comparison study comprising several thousand individual experiments. Our approach to comparisons is biased toward general (nonexpert) users. Our study uses six representative methods described using a common taxonomy. Comparisons performed on artificial data sets provide some insights on applicability of various methods. No single method proved to be the best, since a method's performance depends significantly on the type of the target function, and on the properties of training data.

An optimal tracking neuro-controller for nonlinear dynamic systems

Abstract: Multilayer neural networks are used to design an optimal tracking neuro-controller (OTNC) for discrete-time nonlinear dynamic systems with quadratic cost function. The OTNC is made of two controllers: feedforward neuro-controller (FFNC) and feedback neuro-controller (FBNC). The FFNC controls the steady-state output of the plant, while the FBNC controls the transient-state output of the plant. The FFNC is designed using a novel inverse mapping concept by using a neuro-identifier. A generalized backpropagation-through-time (GBTT) algorithm is developed to minimize the general quadratic cost function for the FBNC training. The proposed methodology is useful as an off-line control method where the plant is first identified and then a controller is designed for it. A case study for a typical plant with nonlinear dynamics shows good performance of the proposed OTNC.

Hopfield neural network implementation of the optimal CDMA multiuser detector

Abstract: We investigate the application of Hopfield neural networks (HNN's) to the problem of multiuser detection in spread spectrum/CDMA (code division multiple access) communication systems. It is shown that the NP-complete problem of minimizing the objective function of the optimal multiuser detector (OMD) can be translated into minimizing an HNN "energy" function, thus allowing to take advantage of the ability of HNN's to perform very fast gradient descent algorithms in analog hardware and produce in real-time suboptimal solutions to hard combinatorial optimization problems. The performance of the proposed HNN receiver is evaluated via computer simulations and compared to that of other suboptimal schemes as well as to that of the OMD for both the synchronous and the asynchronous CDMA transmission cases. It is shown that the HNN detector exhibits a number of attractive properties and that it provides a powerful generalization of a well-known and extensively studied suboptimal scheme, namely the multistage detector.

Worst-case quadratic loss bounds for prediction using linear functions and gradient descent

Abstract: Studies the performance of gradient descent (GD) when applied to the problem of online linear prediction in arbitrary inner product spaces. We prove worst-case bounds on the sum of the squared prediction errors under various assumptions concerning the amount of a priori information about the sequence to predict. The algorithms we use are variants and extensions of online GD. Whereas our algorithms always predict using linear functions as hypotheses, none of our results requires the data to be linearly related. In fact, the bounds proved on the total prediction loss are typically expressed as a function of the total loss of the best fixed linear predictor with bounded norm. All the upper bounds are tight to within constants. Matching lower bounds are provided in some cases. Finally, we apply our results to the problem of online prediction for classes of smooth functions.

Extending the functional equivalence of radial basis function networks and fuzzy inference systems

Abstract: We establish the functional equivalence of a generalized class of Gaussian radial basis function (RBFs) networks and the full Takagi-Sugeno model (1983) of fuzzy inference. This generalizes an existing result which applies to the standard Gaussian RBF network and a restricted form of the Takagi-Sugeno fuzzy system. The more general framework allows the removal of some of the restrictive conditions of the previous result.

Stability analysis of dynamical neural networks

Abstract: In this paper, we use the matrix measure technique to study the stability of dynamical neural networks. Testable conditions for global exponential stability of nonlinear dynamical systems and dynamical neural networks are given. It shows how a few well-known results can be unified and generalized in a straightforward way. Local exponential stability of a class of dynamical neural networks is also studied; we point out that the local exponential stability of any equilibrium point of dynamical neural networks is equivalent to the stability of the linearized system around that equilibrium point. From this, some well-known and new sufficient conditions for local exponential stability of neural networks are obtained.

A rapid supervised learning neural network for function interpolation and approximation

Abstract: This paper presents a neural-network architecture and an instant learning algorithm that rapidly decides the weights of the designed single-hidden layer neural network. For an n-dimensional N-pattern training set, with a constant bias, a maximum of N-r-1 hidden nodes is required to learn the mapping within a given precision (where r is the rank, usually the dimension, of the input patterns). For off-line training, the proposed network and algorithm is able to achieve "one-shot" training as opposed to most iterative training algorithms in the literature. An online training algorithm is also presented. Similar to most of the backpropagation type of learning algorithms, the given algorithm also interpolates the training data. To eliminate outlier data which may appear in some erroneous training data, a robust weighted least squares method is proposed. The robust weighted least squares learning algorithm can eliminate outlier samples and the algorithm approximates the training data rather than interpolates them. The advantage of the designed network architecture is also mathematically proved. Several experiments show very promising results.

Gradient radial basis function networks for nonlinear and nonstationary time series prediction

Abstract: We present a method of modifying the structure of radial basis function (RBF) network to work with nonstationary series that exhibit homogeneous nonstationary behavior. In the original RBF network, the hidden node's function is to sense the trajectory of the time series and to respond when there is a strong correlation between the input pattern and the hidden node's center. This type of response, however, is highly sensitive to changes in the level and trend of the time series. To counter these effects, the hidden node's function is modified to one which detects and reacts to the gradient of the series. We call this new network the gradient RBF (GRBF) model. Single and multistep predictive performance for the Mackey-Glass chaotic time series were evaluated using the classical RBF and GRBF models. The simulation results for the series without and with a tine-varying mean confirm the superior performance of the GRBF predictor over the RBF predictor.

Median radial basis function neural network

Abstract: Radial basis functions (RBFs) consist of a two-layer neural network, where each hidden unit implements a kernel function. Each kernel is associated with an activation region from the input space and its output is fed to an output unit. In order to find the parameters of a neural network which embeds this structure we take into consideration two different statistical approaches. The first approach uses classical estimation in the learning stage and it is based on the learning vector quantization algorithm and its second-order statistics extension. After the presentation of this approach, we introduce the median radial basis function (MRBF) algorithm based on robust estimation of the hidden unit parameters. The proposed algorithm employs the marginal median for kernel location estimation and the median of the absolute deviations for the scale parameter estimation. A histogram-based fast implementation is provided for the MRBF algorithm. The theoretical performance of the two training algorithms is comparatively evaluated when estimating the network weights. The network is applied in pattern classification problems and in optical flow segmentation.

Robust error measure for supervised neural network learning with outliers

Abstract: Most supervised neural networks (NNs) are trained by minimizing the mean squared error (MSE) of the training set. In the presence of outliers, the resulting NN model can differ significantly from the underlying system that generates the data. Two different approaches are used to study the mechanism by which outliers affect the resulting models: influence function and maximum likelihood. The mean log squared error (MLSE) is proposed as the error criteria that can be easily adapted by most supervised learning algorithms. Simulation results indicate that the proposed method is robust against outliers.

An analog VLSI recurrent neural network learning a continuous-time trajectory

Abstract: Real-time algorithms for gradient descent supervised learning in recurrent dynamical neural networks fail to support scalable VLSI implementation, due to their complexity which grows sharply with the network dimension. We present an alternative implementation in analog VLSI, which employs a stochastic perturbation algorithm to observe the gradient of the error index directly on the network in random directions of the parameter space, thereby avoiding the tedious task of deriving the gradient from an explicit model of the network dynamics. The network contains six fully recurrent neurons with continuous-time dynamics, providing 42 free parameters which comprise connection strengths and thresholds. The chip implementing the network includes local provisions supporting both the learning and storage of the parameters, integrated in a scalable architecture which can be readily expanded for applications of learning recurrent dynamical networks requiring larger dimensionality. We describe and characterize the functional elements comprising the implemented recurrent network and integrated learning system, and include experimental results obtained from training the network to represent a quadrature-phase oscillator.

An analysis of noise in recurrent neural networks: convergence and generalization

Abstract: Concerns the effect of noise on the performance of feedforward neural nets. We introduce and analyze various methods of injecting synaptic noise into dynamically driven recurrent nets during training. Theoretical results show that applying a controlled amount of noise during training may improve convergence and generalization performance. We analyze the effects of various noise parameters and predict that best overall performance can be achieved by injecting additive noise at each time step. Noise contributes a second-order gradient term to the error function which can be viewed as an anticipatory agent to aid convergence. This term appears to find promising regions of weight space in the beginning stages of training when the training error is large and should improve convergence on error surfaces with local minima. The first-order term is a regularization term that can improve generalization. Specifically, it can encourage internal representations where the state nodes operate in the saturated regions of the sigmoid discriminant function. While this effect can improve performance on automata inference problems with binary inputs and target outputs, it is unclear what effect it will have on other types of problems. To substantiate these predictions, we present simulations on learning the dual parity grammar from temporal strings for all noise models, and present simulations on learning a randomly generated six-state grammar using the predicted best noise model.

A new neural network for solving linear programming problems and its application

Abstract: Presents a new neural network which improves existing neural networks for solving general linear programming problems. The network, without setting parameter, uses only simple hardware in which no analog multipliers are required, and is proved to be completely stable to the exact solutions. Moreover, using this network the author can solve linear programming problems and its dual simultaneously, and cope with problems with nonunique solutions whose set is allowed to be unbounded.

A neural network-based model for paper currency recognition and verification

Abstract: This paper describes the neural-based recognition and verification techniques used in a banknote machine, recently implemented for accepting paper currency of different countries. The perception mechanism is based on low-cost optoelectronic devices which produce a signal associated with the light refracted by the banknotes. The classification and verification steps are carried out by a society of multilayer perceptrons whose operation is properly scheduled by an external controlling algorithm, which guarantees real-time implementation on a standard microcontroller-based platform. The verification relies mainly on the property of autoassociators to generate closed separation surfaces in the pattern space. The experimental results are very interesting, particularly when considering that the recognition and verification steps are based on low-cost sensors.

A note on stability of analog neural networks with time delays

Abstract: This note presents a generalized sufficient condition which guarantees stability of analog neural networks with time delays. The condition is derived using a Lyapunov functional and the stability criterion is stated as: the equilibrium of analog neural networks with delays is globally asymptotically stable if the product of the norm of connection matrix and the maximum neuronal gain is less than one.

An approach to stability criteria of neural-network control systems

Abstract: This paper discusses stability of neural network (NN)-based control systems using Lyapunov approach. First, it is pointed out that the dynamics of NN systems can be represented by a class of nonlinear systems treated as linear differential inclusions (LDI). Next, stability conditions for the class of nonlinear systems are derived and applied to the stability analysis of single NN systems and feedback NN control systems. Furthermore, a method of parameter region (PR) representation, which graphically shows the location of parameters of nonlinear systems, is proposed by introducing new concepts of vertex point and minimum representation. From these concepts, an important theorem, which is useful for effectively finding a Lyapunov function, is derived. Stability criteria of single NN systems are illustrated in terms of PR representation. Finally, stability of feedback NN control systems, which consist of a plant represented by an NN and an NN controller, is analyzed.

On the efficiency of the orthogonal least squares training method for radial basis function networks

Abstract: The efficiency of the orthogonal least squares (OLS) method for training approximation networks is examined using the criterion of energy compaction. We show that the selection of basis vectors produced by the procedure is not the most compact when the approximation is performed using a nonorthogonal basis. Hence, the algorithm does not produce the smallest possible networks for a given approximation error. Specific examples are given using the Gaussian radial basis functions type of approximation networks.

Synchronization and desynchronization in a network of locally coupled Wilson-Cowan oscillators

Abstract: A network of Wilson-Cowan (WC) oscillators is constructed, and its emergent properties of synchronization and desynchronization are investigated by both computer simulation and formal analysis. The network is a 2D matrix, where each oscillator is coupled only to its neighbors. We show analytically that a chain of locally coupled oscillators (the piecewise linear approximation to the WC oscillator) synchronizes, and we present a technique to rapidly entrain finite numbers of oscillators. The coupling strengths change on a fast time scale based on a Hebbian rule. A global separator is introduced which receives input from and sends feedback to each oscillator in the matrix. The global separator is used to desynchronize different oscillator groups. Unlike many other models, the properties of this network emerge from local connections that preserve spatial relationships among components and are critical for encoding Gestalt principles of feature grouping. The ability to synchronize and desynchronize oscillator groups within this network offers a promising approach for pattern segmentation and figure/ground segregation based on oscillatory correlation.