Showing papers on "Recurrent neural network published in 1992"

Neural networks and the bias/variance dilemma

Abstract: Feedforward neural networks trained by error backpropagation are examples of nonparametric regression estimators. We present a tutorial on nonparametric inference and its relation to neural networks, and we use the statistical viewpoint to highlight strengths and weaknesses of neural models. We illustrate the main points with some recognition experiments involving artificial data as well as handwritten numerals. In way of conclusion, we suggest that current-generation feedforward neural networks are largely inadequate for difficult problems in machine perception and machine learning, regardless of parallel-versus-serial hardware or other implementation issues. Furthermore, we suggest that the fundamental challenges in neural modeling are about representation rather than learning per se. This last point is supported by additional experiments with handwritten numerals.

Theory of the backpropagation neural network

Abstract: Publisher Summary This chapter presents a survey of the elementary theory of the basic backpropagation neural network architecture, covering the areas of architectural design, performance measurement, function approximation capability, and learning. The survey includes a formulation of the backpropagation neural network architecture to make it a valid neural network and a proof that the backpropagation mean squared error function exists and is differentiable. Also included in the survey is a theorem showing that any L2 function can be implemented to any desired degree of accuracy with a three-layer backpropagation neural network. An appendix presents a speculative neurophysiological model illustrating the way in which the backpropagation neural network architecture might plausibly be implemented in the mammalian brain for corticocortical learning between nearby regions of cerebral cortex. One of the crucial decisions in the design of the backpropagation architecture is the selection of a sigmoidal activation function.

Functional-link net computing: theory, system architecture, and functionalities

Abstract: A system architecture and a network computational approach compatible with the goal of devising a general-purpose artificial neural network computer are described. The functionalities of supervised learning and optimization are illustrated, and cluster analysis and associative recall are briefly mentioned. >

Neural Networks for Nonlinear Dynamic System Modelling and Identification

Abstract: Many real-world systems exhibit complex nonlinear characteristics and cannot be treated satisfactorily using linear systems theory. A neural network which has the ability to learn sophisticated nonlinear relationships provides an ideal means of modelling complicated nonlinear systems. This paper addresses the issues related to the identification of nonlinear discrete-time dynamic systems using neural networks. Three network architectures, namely the multi-layer perceptron, the radial basis function network and the functional-link network, are presented and several learning or identification algorithms are derived. Advantages and disadvantages of these structures are discussed and illustrated using simulated and real data. Particular attention is given to the connections between existing techniques for nonlinear systems identification and some aspects of neural network methodology, and this demonstrates that certain techniques employed in the neural network context have long been developed by the control e...

Evolving dynamical neural networks for adaptive behavior

Abstract: We would like the behavior of the artificial agents that we construct to be as well-adapted to their environments as natural animals are to theirs. Unfortunately, designing controllers with these properties is a very difficult task. In this article, we demonstrate that continuous-time recurrent neural networks are a viable mechanism for adaptive agent control and that the genetic algorithm can be used to evolve effective neural controllers. A significant advantage of this approach is that one need specify only a measure of an agent's overall performance rather than the precise motor output trajectories by which it is achieved. By manipulating the performance evaluation, one can place selective pressure on the development of controllers with desired properties. Several novel controllers have been evolved, including a chemotaxis controller that switches between different strategies depending on environmental conditions, and a locomotion controller that takes advantage of sensory feedback if available but th...

Feedforward neural networks with random weights

Abstract: In the field of neural network research a number of experiments described seem to be in contradiction with the classical pattern recognition or statistical estimation theory. The authors attempt to give some experimental understanding why this could be possible by showing that a large fraction of the parameters (the weights of neural networks) are of less importance and do not need to be measured with high accuracy. The remaining part is capable to implement the desired classifier and because this is only a small fraction of the total number of weights, the reported experiments seem to be more realistic from a classical point of view. >

Learning and extracting finite state automata with second-order recurrent neural networks

Abstract: We show that a recurrent, second-order neural network using a real-time, forward training algorithm readily learns to infer small regular grammars from positive and negative string training samples. We present simulations that show the effect of initial conditions, training set size and order, and neural network architecture. All simulations were performed with random initial weight strengths and usually converge after approximately a hundred epochs of training. We discuss a quantization algorithm for dynamically extracting finite state automata during and after training. For a well-trained neural net, the extracted automata constitute an equivalence class of state machines that are reducible to the minimal machine of the inferred grammar. We then show through simulations that many of the neural net state machines are dynamically stable, that is, they correctly classify many long unseen strings. In addition, some of these extracted automata actually outperform the trained neural network for classification...

Learning with the Recurrent Random Neural Network

Abstract: The capacity to learn from examples is one of the most desirable features of neural network models. We present a learning algorithm for the recurrent random network model (Gelenbe 1989, 1990) using gradient descent of a quadratic error function. The analytical properties of the model lead to a "backpropagation" type algorithm that requires the solution of a system of n linear and n nonlinear equations each time the n-neuron network "learns" a new input-output pair.

Long-term predictions of chemical processes using recurrent neural networks: a parallel training approach

Abstract: Multilayer feedforward networks have been used successfully for nonlinear system identification by using them as discrete-time dynamic models. In the past, feedforward networks have been adapted as one-step-ahead predictors; however, in model predictive control the model has to be iterated to predict many time steps ahead into the future. Therefore, the feedforward network is chained to itself to go as far as needed in the future, and this chaining may result in large errors. As an alternative to using the one-step-ahead approach, a feedforward network is chained to itself during the training. This training procedure is referred to as a parallel identification method since the network is in parallel with the system to be identified. A feedforward network used in the parallel approach results in an external recurrent network. The learning algorithm for external recurrent networks is derived using ordered derivatives. The network is used to identify the dynamic behavior of a biological wastewater treatment plant and a catalytic reformer in a petroleum refinery

Bifurcations in the learning of recurrent neural networks

Abstract: Gradient descent algorithms in recurrent neural networks can have problems when the network dynamics experience bifurcations in the course of learning. The possible hazards caused by the bifurcations of the network dynamics and the learning equations are investigated. The roles of teacher forcing, preprogramming of network structures, and the approximate learning algorithms are discussed. >

Modelling, Identification and Stable Adaptive Control of Continuous-Time Nonlinear Dynamical Systems Using Neural Networks

Abstract: Several empirical studies have demonstrated the feasibility of employing neural networks as models of nonlinear dynamical systems. This paper develops the appropriate mathematical tools for synthesizing and analyzing stable neural network based identification and control schemes. Feedforward network architectures are combined with dynamical elements, in the form of stable filters, to construct a general recurrent network configuration which is shown to be capable of approximating a large class of dynamical systems. Adaptive identification and control schemes, based on neural network models, are developed using the Lyapunov synthesis approach with the projection modification method. These schemes are shown to guarantee stability of the overall system, even in the presence of modelling errors. A crucial characteristic of the methods and formulations developed in this paper is the generality of the results which allows their application to various neural network models as well as other approximators.

Sensitivity analysis for feedforward artificial neural networks with differentiable activation functions

Abstract: A method for computing the network output sensitivities with respect to variations in the inputs for multilayer feedforward artificial neural networks with differentiable activation functions is presented. It is applied to obtain expressions for the first- and second-order sensitivities. An example is introduced along with a discussion to illustrate how the sensitivities are calculated and to show how they compare to the actual derivatives of the function being modeled by the neural network. >

A neural network controller for a temperature control system

Abstract: A backpropagation neural network is trained to learn the inverse dynamics model of a temperature control system and then configured as a direct controller to the process. The ability of the neural network to learn the inverse model of the process plant is based on input vectors with no a priori knowledge regarding dynamics. Based on these characteristics, the neural network is compared to a conventional proportional-plus-integral (PI) controller. Experimental results show that the neural network controller performs very well and offers worthwhile advantages. >

Improved nuclear reactor temperature control using diagonal recurrent neural networks

Abstract: A novel approach to wide-range optimal reactor temperature control using diagonal recurrent neural networks (DRNNs) with an adaptive learning rate scheme is presented. The drawback of the usual feedforward neural network (FNN) is that it is a static mapping and requires a large number of neurons and takes a long training time. The usual fixed learning rate based on an empirical trial and error scheme is slow and does not guarantee convergence. The DRNN is for dynamic mapping and requires much fewer neurons and weights, and thus converges faster than FNN. A dynamic backpropagation algorithm coupled with an adaptive learning rate guarantees even faster convergence. The DRNN controller described includes both a neurocontroller and a neuroidentifier. A reference model which incorporates an optimal control law with improved reactor temperature response is used for training of the neurocontroller and neuroidentifier. Rapid convergence of this DRNN-based control system is demonstrated when used to improve reactor temperature performance. >

Neural Networks: Current Applications

Abstract: Neural network basics using adaptive networks for resource allocation in changing environments medical risk assessment for insurance underwriting modelling chemical process systems via neural computation the application of neural networks to robotics neural networks in vision image labelling with a neural network object recognition with optimum neural networks handwritten digit recognition with a backpropagation network higher order networks for invariant pattern recognition the bionic retina and beyond.

Neural networks: theory and applications

Abstract: Weightless neural tools - toward cognitive macrostructures, L. Aleksander an estimation theoretic basis for the design of sorting and classification network, R.W. Brockett a self organizing ARTMAP neural architecture for supervized learning and pattern recognition, G.A. Carpenter et al hybrid neural network architectures - equilibrium systems that pay attention, L.N. Cooper neural networks for internal representation of movements in primates and robots, R. Eckmiller et al recognition and segmentation of characters in handwriting with selective attention, K. Fukushima et al adaptive acquisition of language, A.L. Gorin et al what connectionist models learn - learning and representation in connectionist networks, S.J. Hanson and D.J. Burr early vision, focal attention and neural nets, B. Julesz toward hierarchical matched filtering, R. Hecht-Nielsen some variations on training of recurrent networks, G.M. Kuhn and N.P. Herzberg generalized perception networks with nonlinear discriminant functions, S.Y. Kung et al neural tree networks, A. Sankar and R. Mammone capabilities and training of feedforward nets, E.D. Sontag a fast learning algorithm for multilayer neural network based on projection methods, S.J. Yeh and H. Stark.

An application of a recurrent network to an intrusion detection system

Abstract: The authors present an application of recurrent neural networks for intrusion detection. A partially recurrent network has been chosen for this particular application. The neural network acts as a data filter that highlights anomalous or suspicious data according to previously learned patterns. It has proven adaptive, because the same results for several users have been obtained with varying activities. The network cosine function was tested, and a hetero-associative version of the network was used to analyze the flipflop problem. >

Structure-unknown non-linear dynamic systems: identification through neural networks

Abstract: Explores the potential of using parallel distributed processing (neural network) approaches to identify the internal forces of structure-unknown non-linear dynamic systems typically encountered in the field of applied mechanics. The relevant characteristics of neural networks, such as the processing elements, network topology, and learning algorithms, are discussed in the context of system identification. The analogy of the neural network procedure to a qualitatively similar non-parametric identification approach, which was previously developed by the authors for handling arbitrary non-linear systems, is discussed. The utility of the neural network approach is demonstrated by application to several illustrative problems.

Modelling Hebbian cell assemblies comprised of cortical neurons

Abstract: A highly simplified network model of cortical associative memory, based on Hebb's theory of cell assemblies, has been developed and simulated. The network comprises realistically modelled pyramidal...

Electronic realisation of recurrent neural network for solving simultaneous linear equations

Abstract: An electronic neural network for solving simultaneous linear equations is presented. The proposed electronic neural network is able to generate real-time solutions to large-scale problems. The operating characteristics of an opamp based neural network is demonstrated via an illustrative example.

Off-line signature verification using directional PDF and neural networks

Abstract: The first stage of a complete automatic handwritten signature verification system (AHSVS) is described in this paper Since only random forgeries are taken into account in this first stage of decision, the directional probability density function (PDF) which is related to the overall shape of the handwritten signature has been taken into account as feature vector Experimental results show that using both directional PDFs and the completely connected feedforward neural network classifier are valuable to build the first stage of a complete AHSVS >

Training Second-Order Recurrent Neural Networks using Hints

Abstract: We investigate a method for inserting rules into discrete-time second-order recurrent neural networks which are trained to recognize regular languages. The rules defining regular languages can be expressed in the form of transitions in the corresponding deterministic finite-state automaton. Inserting these rules as hints into networks with second-order connections is straightforward. Our simulation results show that even weak hints seem to improve the convergence time by an order of magnitude.

Machine fault classification: a neural network approach

Abstract: This paper presents a neural network approach for machine fault diagnosis. Specifically, two tasks are explained and discussed: (1) a neural network-based machine fault diagnosis model is developed using the back propagation (BP) learning paradigm; (2) network training efficiency is studied by varying the learning rate and learning momentum of the activation function. The results are presented and discussed.

Proposal of chaotic steepest descent method for neural networks and analysis of their dynamics

Abstract: Most of the neural network models based on the Lyapunov stability contain various problems such as the trap of the local minimum, and the limit of their dynamic performances has been pointed out. This paper attempts to provide a new dynamic performance to such a neural network model by introducing chaos dynamics. The features of the chaotic dynamic model proposed here is that the dynamical equation describing the trajectory on the energy curve has a periodically varying nonlinear resistance in the dissipation term. By iterating the stable and unstable phases, the chaotic transitions of the state can be realized. The proposed dynamic model is applied to the error backpropagation learning and the memory recall in the Hopfield-type network, and the chaotic minimum transitions in the dynamic process are verified, it is verified further that the intermittent chaos generated by an appropriate parameter manipulation can introduce useful dynamic behaviors into the network, e.g., the flexible learning and the memory recall with a structure containing both stability and plasticity.

Universality in neural networks: the importance of the `mean firing rate'

Abstract: We present a general analysis of highly connected recurrent neural networks which are able to learn and retrieve a finite number of static patterns. The arguments are based on spike trains and their interval distribution and require no specific model of a neuron. In particular, they apply to formal two-state neurons as well as to more refined models like the integrate-and-fire neuron or the Hodgkin-Huxley equations. We show that the mean firing rate defined as the inverse of the mean interval length is the only relevant parameter (apart from the synaptic weights) that determines the existence of retrieval solutions with a large overlap with one of the learnt patterns. The statistics of the spiking noise (Gaussian, Poisson or other) and hence the shape of the interval distribution does not matter. Thus our unifying approach explains why, and when, all the different associative networks which treat static patterns yield basically the same results, i.e., belong to the same universality class.