Showing papers on "Artificial neural network published in 2000"

Statistical pattern recognition: a review

Abstract: The primary goal of pattern recognition is supervised or unsupervised classification. Among the various frameworks in which pattern recognition has been traditionally formulated, the statistical approach has been most intensively studied and used in practice. More recently, neural network techniques and methods imported from statistical learning theory have been receiving increasing attention. The design of a recognition system requires careful attention to the following issues: definition of pattern classes, sensing environment, pattern representation, feature extraction and selection, cluster analysis, classifier design and learning, selection of training and test samples, and performance evaluation. In spite of almost 50 years of research and development in this field, the general problem of recognizing complex patterns with arbitrary orientation, location, and scale remains unsolved. New and emerging applications, such as data mining, web searching, retrieval of multimedia data, face recognition, and cursive handwriting recognition, require robust and efficient pattern recognition techniques. The objective of this review paper is to summarize and compare some of the well-known methods used in various stages of a pattern recognition system and identify research topics and applications which are at the forefront of this exciting and challenging field.

Statistical Pattern Recognition

Abstract: The primary goal of pattern recognition is supervised or unsupervised classification. Among the various frameworks in which pattern recognition has been traditionally formulated, the statistical ap...

Neural networks for the prediction and forecasting of water resources variables: a review of modelling issues and applications

Abstract: Artificial Neural Networks (ANNs) are being used increasingly to predict and forecast water resources variables. In this paper, the steps that should be followed in the development of such models are outlined. These include the choice of performance criteria, the division and pre-processing of the available data, the determination of appropriate model inputs and network architecture, optimisation of the connection weights (training) and model validation. The options available to modellers at each of these steps are discussed and the issues that should be considered are highlighted. A review of 43 papers dealing with the use of neural network models for the prediction and forecasting of water resources variables is undertaken in terms of the modelling process adopted. In all but two of the papers reviewed, feedforward networks are used. The vast majority of these networks are trained using the backpropagation algorithm. Issues in relation to the optimal division of the available data, data pre-processing and the choice of appropriate model inputs are seldom considered. In addition, the process of choosing appropriate stopping criteria and optimising network geometry and internal network parameters is generally described poorly or carried out inadequately. All of the above factors can result in non-optimal model performance and an inability to draw meaningful comparisons between different models. Future research efforts should be directed towards the development of guidelines which assist with the development of ANN models and the choice of when ANNs should be used in preference to alternative approaches, the assessment of methods for extracting the knowledge that is contained in the connection weights of trained ANNs and the incorporation of uncertainty into ANN models.

Neural networks for classification: a survey

Abstract: Classification is one of the most active research and application areas of neural networks. The literature is vast and growing. This paper summarizes some of the most important developments in neural network classification research. Specifically, the issues of posterior probability estimation, the link between neural and conventional classifiers, learning and generalization tradeoff in classification, the feature variable selection, as well as the effect of misclassification costs are examined. Our purpose is to provide a synthesis of the published research in this area and stimulate further research interests and efforts in the identified topics.

Nonlinear System Identification: From Classical Approaches to Neural Networks and Fuzzy Models

Abstract: 1 Introduction- I Optimization Techniques- 2 Introduction to Optimization- 3 Linear Optimization- 4 Nonlinear Local Optimization- 5 Nonlinear Global Optimization- 6 Unsupervised Learning Techniques- 7 Model Complexity Optimization- II Static Models- 9 Introduction to Static Models- 10 Linear, Polynomial, and Look-Up Table Models- 11 Neural Networks- 12 Fuzzy and Neuro-Fuzzy Models- 13 Local Linear Neuro-Fuzzy Models: Fundamentals- 14 Local Linear Neuro-Fuzzy Models: Advanced Aspects- III Dynamic Models- 16 Linear Dynamic System Identification- 17 Nonlinear Dynamic System Identification- 18 Classical Polynomial Approaches- 19 Dynamic Neural and Fuzzy Models- 20 Dynamic Local Linear Neuro-Fuzzy Models- 21 Neural Networks with Internal Dynamics- IV Applications- 22 Applications of Static Models- 23 Applications of Dynamic Models- 24 Applications of Advanced Methods- A Vectors and Matrices- A1 Vector and Matrix Derivatives- A2 Gradient, Hessian, and Jacobian- B Statistics- B1 Deterministic and Random Variables- B2 Probability Density Function (pdf)- B3 Stochastic Processes and Ergodicity- B4 Expectation- B5 Variance- B6 Correlation and Covariance- B7 Properties of Estimators- References

Online Ensemble Learning

Abstract: Ensemble learning methods train combinations of base models, which may be decision trees, neural networks, or others traditionally used in supervised learning. Ensemble methods have gained popularity because many researchers have demonstrated their superior prediction performance relative to single models on a variety of problems especially when the correlations of the errors made by the base models are low (e.g., (Freund & Schapire 1996; Tumer & Oza 1999)). However, these learning methods have largely operated in batch mode—that is, they repeatedly process the entire set of training examples as a whole. These methods typically require at least one pass through the data for each base model in the ensemble. We would instead prefer to learn the entire ensemble in an online fashion, i.e., using only one pass through the entire dataset. This would make ensemble methods practical when data is being generated continuously so that storing data for batch learning is impractical, or in data mining tasks where the datasets are large enough that multiple passes would require a prohibitively large training time. We have so far developed online versions of the popular bagging (Breiman 1994) and boosting (Freund & Schapire 1996) algorithms. We have shown empirically that both online algorithms converge to the same prediction performance as the batch versions and proved this convergence for online bagging (Oza 2000). However, significant empirical and theoretical work remains to be done. There are several traditional ensemble learning issues that remain in our online ensemble learning framework such as the number and types of base models to use, the combining method to use, and how to maintain diversity among the base models. When learning large datasets, we may hope to avoid using all of the training examples and/or input features. We have developed input decimation (Tumer & Oza 1999), a technique that uses different subsets of the input features in different base models. We have shown that this method performs better than combinations of base models that use all the input features because of two characteristics of our base models: they overfit less by using only a small number of highly-relevant input features, and they have lower correlations in their errors because they use different input feature subsets. However, our method of selecting input features

Neural Networks for Modelling and Control of Dynamic Systems

Abstract: 1. Introduction.- 1.1 Background.- 1.1.1 Inferring Models and Controllers from Data.- 1.1.2 Why Use Neural Networks?.- 1.2 Introduction to Multilayer Perceptron Networks.- 1.2.1 The Neuron.- 1.2.2 The Multilayer Perceptron.- 1.2.3 Choice of Neural Network Architecture.- 1.2.4 Models of Dynamic Systems.- 1.2.5 Recurrent Networks.- 1.2.6 Other Neural Network Architectures.- 2. System Identification with Neural Networks.- 2.1 Introduction to System Identification.- 2.1.1 The Procedure.- 2.2 Model Structure Selection.- 2.2.1 Some Linear Model Structures.- 2.2.2 Nonlinear Model Structures Based on Neural Networks.- 2.2.3 A Few Remarks on Stability.- 2.2.4 Terminology.- 2.2.5 Selecting the Lag Space.- 2.2.6 Section Summary.- 2.3 Experiment.- 2.3.1 When is a Linear Model Insufficient?.- 2.3.2 Issues in Experiment Design.- 2.3.3 Preparing the Data for Modelling.- 2.3.4 Section Summary.- 2.4 Determination of the Weights.- 2.4.1 The Prediction Error Method.- 2.4.2 Regularization and the Concept of Generalization.- 2.4.3 Remarks on Implementation.- 2.4.4 Section Summary.- 2.5 Validation.- 2.5.1 Looking for Correlations.- 2.5.2 Estimation of the Average Generalization Error.- 2.5.3 Visualization of the Predictions.- 2.5.4 Section Summary.- 2.6 Going Backwards in the Procedure.- 2.6.1 Training the Network Again.- 2.6.2 Finding the Optimal Network Architecture.- 2.6.3 Redoing the Experiment.- 2.6.4 Section Summary.- 2.7 Recapitulation of System Identification.- 3. Control with Neural Networks.- 3.1 Introduction to Neural-Network-based Control.- 3.1.1 The Benchmark System.- 3.2 Direct Inverse Control.- 3.2.1 General Training.- 3.2.2 Direct Inverse Control of the Benchmark System.- 3.2.3 Specialized Training.- 3.2.4 Specialized Training and Direct Inverse Control of the Benchmark System.- 3.2.5 Section Summary.- 3.3 Internal Model Control (IMC).- 3.3.1 Internal Model Control with Neural Networks.- 3.3.2 Section Summary.- 3.4 Feedback Linearization.- 3.4.1 The Basic Principle of Feedback Linearization.- 3.4.2 Feedback Linearization Using Neural Network Models..- 3.4.3 Feedback Linearization of the Benchmark System.- 3.4.4 Section Summary.- 3.5 Feedforward Control.- 3.5.1 Feedforward for Optimizing an Existing Control System.- 3.5.2 Feedforward Control of the Benchmark System.- 3.5.3 Section Summary.- 3.6 Optimal Control.- 3.6.1 Training of an Optimal Controller.- 3.6.2 Optimal Control of the Benchmark System.- 3.6.3 Section Summary.- 3.7 Controllers Based on Instantaneous Linearization.- 3.7.1 Instantaneous Linearization.- 3.7.2 Applying Instantaneous Linearization to Control.- 3.7.3 Approximate Pole Placement Design.- 3.7.4 Pole Placement Control of the Benchmark System.- 3.7.5 Approximate Minimum Variance Design.- 3.7.6 Section Summary.- 3.8 Predictive Control.- 3.8.1 Nonlinear Predictive Control (NPC).- 3.8.2 NPC Applied to the Benchmark System.- 3.8.3 Approximate Predictive Control (APC).- 3.8.4 APC applied to the Benchmark System.- 3.8.5 Extensions to the Predictive Controller.- 3.8.6 Section Summary.- 3.9 Recapitulation of Control Design Methods.- 4. Case Studies.- 4.1 The Sunspot Benchmark.- 4.1.1 Modelling with a Fully Connected Network.- 4.1.2 Pruning of the Network Architecture.- 4.1.3 Section Summary.- 4.2 Modelling of a Hydraulic Actuator.- 4.2.1 Estimation of a Linear Model.- 4.2.2 Neural Network Modelling of the Actuator.- 4.2.3 Section Summary.- 4.3 Pneumatic Servomechanism.- 4.3.1 Identification of the Pneumatic Servomechanism.- 4.3.2 Nonlinear Predictive Control of the Servo.- 4.3.3 Approximate Predictive Control of the Servo.- 4.3.4 Section Summary.- 4.4 Control of Water Level in a Conic Tank.- 4.4.1 Linear Analysis and Control.- 4.4.2 Direct Inverse Control of the Water Level.- 4.4.3 Section Summary.- References.

Neural and adaptive systems : fundamentals through simulations

Abstract: Data Fitting with Linear Models Pattern Recognition Multilayer Perceptrons Designing and Training MLPs Function Approximation with MLPs, Radial Basis Functions, and Support Vector Machines Hebbian Learning and Principal Component Analysis Competitive and Kohonen Networks Principles of Digital Signal Processing Adaptive Filters Temporal Processing with Neural Networks Training and Using Recurrent Networks Appendices Glossary Index

Computational Explorations in Cognitive Neuroscience: Understanding the Mind by Simulating the Brain

Abstract: From the Publisher: The goal of computational cognitive neuroscience is to understand how the brain embodies the mind by using biologically based computational models comprising networks of neuronlike units. This text, based on a course taught by Randall O'Reilly and Yuko Munakata over the past several years, provides an in-depth introduction to the main ideas in the field. The neural units in the simulations use equations based directly on the ion channels that govern the behavior of real neurons, and the neural networks incorporate anatomical and physiological properties of the neocortex. Thus the text provides the student with knowledge of the basic biology of the brain as well as the computational skills needed to simulate large-scale cognitive phenomena. The text consists of two parts. The first part covers basic neural computation mechanisms: individual neurons, neural networks, and learning mechanisms. The second part covers large-scale brain area organization and cognitive phenomena: perception and attention, memory, language, and higher-level cognition. The second part is relatively self-contained and can be used separately for mechanistically oriented cognitive neuroscience courses. Integrated throughout the text are more than forty different simulation models, many of them full-scale research-grade models, with friendly interfaces and accompanying exercises. The simulation software (PDP++, available for all major platforms) and simulations can be downloaded free of charge from the Web. Exercise solutions are available, and the text includes full information on the software.

Tandem connectionist feature extraction for conventional HMM systems

Abstract: Hidden Markov model speech recognition systems typically use Gaussian mixture models to estimate the distributions of decorrelated acoustic feature vectors that correspond to individual subword units. By contrast, hybrid connectionist-HMM systems use discriminatively-trained neural networks to estimate the probability distribution among subword units given the acoustic observations. In this work we show a large improvement in word recognition performance by combining neural-net discriminative feature processing with Gaussian-mixture distribution modeling. By training the network to generate the subword probability posteriors, then using transformations of these estimates as the base features for a conventionally-trained Gaussian-mixture based system, we achieve relative error rate reductions of 35% or more on the multicondition Aurora noisy continuous digits task.

Neural Networks for RF and Microwave Design

Abstract: Introduction and Overview. Modeling and Optimization for Design. Neural Network Structures. Training of Neural Networks. Models for RF/Microwave Components. Modeling of Interconnects. Active Device Modeling. Circuit Analysis and Optimization. Knowledge-Based ANN Models. Concluding Remarks and Emerging Trends. Appendix: ANN Modeling Software Available.

Genetic algorithms approach to feature discretization in artificial neural networks for the prediction of stock price index

Abstract: This paper proposes genetic algorithms (GAs) approach to feature discretization and the determination of connection weights for artificial neural networks (ANNs) to predict the stock price index. Previous research proposed many hybrid models of ANN and GA for the method of training the network, feature subset selection, and topology optimization. In most of these studies, however, GA is only used to improve the learning algorithm itself. In this study, GA is employed not only to improve the learning algorithm, but also to reduce the complexity in feature space. GA optimizes simultaneously the connection weights between layers and the thresholds for feature discretization. The genetically evolved weights mitigate the well-known limitations of the gradient descent algorithm. In addition, globally searched feature discretization reduces the dimensionality of the feature space and eliminates irrelevant factors. Experimental results show that GA approach to the feature discretization model outperforms the other two conventional models.

Principles of Neurocomputing for Science and Engineering

Abstract: From the Publisher: This exciting new text covers artificial neural networks,but more specifically,neurocomputing. Neurocomputing is concerned with processing information,which involves a learning process within an artificial neural network architecture. This neural architecture responds to inputs according to a defined learning rule and then the trained network can be used to perform certain tasks depending on the application. Neurocomputing can play an important role in solving certain problems such as pattern recognition,optimization,event classification,control and identification of nonlinear systems,and statistical analysis. "Principles of Neurocomputing for Science and Engineering," unlike other neural networks texts,is written specifically for scientists and engineers who want to apply neural networks to solve complex problems. For each neurocomputing concept,a solid mathematical foundation is presented along with illustrative examples to accompany that particular architecture and associated training algorithm. The book is primarily intended for graduate-level neural networks courses,but in some instances may be used at the undergraduate level. The book includes many detailed examples and an extensive set of end-of-chapter problems.

Identification and control of dynamic systems using recurrent fuzzy neural networks

Abstract: Proposes a recurrent fuzzy neural network (RFNN) structure for identifying and controlling nonlinear dynamic systems. The RFNN is inherently a recurrent multilayered connectionist network for realizing fuzzy inference using dynamic fuzzy rules. Temporal relations are embedded in the network by adding feedback connections in the second layer of the fuzzy neural network (FNN). The RFNN expands the basic ability of the FNN to cope with temporal problems. In addition, results for the FNN-fuzzy inference engine, universal approximation, and convergence analysis are extended to the RFNN. For the control problem, we present the direct and indirect adaptive control approaches using the RFNN. Based on the Lyapunov stability approach, rigorous proofs are presented to guarantee the convergence of the RFNN by choosing appropriate learning rates. Finally, the RFNN is applied in several simulations (time series prediction, identification, and control of nonlinear systems). The results confirm the effectiveness of the RFNN.

Clustering ECG complexes using Hermite functions and self-organizing maps

Abstract: An integrated method for clustering of QRS complexes is presented which includes basis function representation and self-organizing neural networks (NN's). Each QRS complex is decomposed into Hermite basis functions and the resulting coefficients and width parameter are used to represent the complex. By means of this representation, unsupervised self-organizing NNs are employed to cluster the data into 25 groups. Using the MIT-BIH arrhythmia database, the resulting clusters are found to exhibit a very low degree of misclassification (1.5%). The integrated method outperforms, on the MIT-BIH database, both a published supervised learning method as well as a conventional template cross-correlation clustering method.

Recent developments of induction motor drives fault diagnosis using AI techniques

Abstract: This paper presents a review of the developments in the field of diagnosis of electrical machines and drives based on artificial intelligence (AI). It covers the application of expert systems, artificial neural networks (ANNs), and fuzzy logic systems that can be integrated into each other and also with more traditional techniques. The application of genetic algorithms is considered as well. In general, a diagnostic procedure starts from a fault tree developed on the basis of the physical behavior of the electrical system under consideration. In this phase, the knowledge of well-tested models able to simulate the electrical machine in different fault conditions is fundamental to obtain the patterns characterizing the faults. The fault tree navigation performed by an expert system inference engine leads to the choice of suitable diagnostic indexes, referred to a particular fault, and relevant to build an input data set for specific AI (NNs, fuzzy logic, or neuro-fuzzy) systems. The discussed methodologies, that play a general role in the diagnostic field, are applied to an induction machine, utilizing as input signals the instantaneous voltages and currents. In addition, the supply converter is also considered to incorporate in the diagnostic procedure the most typical failures of power electronic components. A brief description of the various AI techniques is also given; this highlights the advantages and the limitations of using AI techniques. Some applications examples are also discussed and areas for future research are also indicated.

Evolutionary ensembles with negative correlation learning

Abstract: Based on negative correlation learning and evolutionary learning, this paper presents evolutionary ensembles with negative correlation learning (EENCL) to address the issues of automatic determination of the number of individual neural networks (NNs) in an ensemble and the exploitation of the interaction between individual NN design and combination. The idea of EENCL is to encourage different individual NNs in the ensemble to learn different parts or aspects of the training data so that the ensemble can learn better the entire training data. The cooperation and specialization among different individual NNs are considered during the individual NN design. This provides an opportunity for different NNs to interact with each other and to specialize. Experiments on two real-world problems demonstrate that EENCL can produce NN ensembles with good generalization ability.

Dynamic fuzzy neural networks-a novel approach to function approximation

Abstract: In this paper, an architecture of dynamic fuzzy neural networks (D-FNN) implementing Takagi-Sugeno-Kang (TSK) fuzzy systems based on extended radial basis function (RBF) neural networks is proposed. A novel learning algorithm based on D-FNN is also presented. The salient characteristics of the algorithm are: 1) hierarchical on-line self-organizing learning is used; 2) neurons can be recruited or deleted dynamically according to their significance to the system's performance; and 3) fast learning speed can be achieved. Simulation studies and comprehensive comparisons with some other learning algorithms demonstrate that a more compact structure with higher performance can be achieved by the proposed approach.

Robust backstepping control of nonlinear systems using neural networks

Abstract: A controller is proposed for the robust backstepping control of a class of general nonlinear systems using neural networks (NNs). A tuning scheme is proposed which can guarantee the boundedness of tracking error and weight updates. Compared with adaptive backstepping control schemes, we do not require the unknown parameters to be linear parametrizable. No regression matrices are needed, so no preliminary dynamical analysis is needed. One salient feature of our NN approach is that there is no need for the off-line learning phase. Three nonlinear systems, including a one-link robot, an induction motor, and a rigid-link flexible-joint robot, were used to demonstrate the effectiveness of the proposed scheme.

A neural network classifier based on Dempster-Shafer theory

Abstract: A new adaptive pattern classifier based on the Dempster-Shafer theory of evidence is presented This method uses reference patterns as items of evidence regarding the class membership of each input pattern under consideration This evidence is represented by basic belief assignments (BBA) and pooled using the Dempster's rule of combination This procedure can be implemented in a multilayer neural network with specific architecture consisting of one input layer, two hidden layers and one output layer The weight vector, the receptive field and the class membership of each prototype are determined by minimizing the mean squared differences between the classifier outputs and target values After training, the classifier computes for each input vector a BBA that provides a description of the uncertainty pertaining to the class of the current pattern, given the available evidence This information may be used to implement various decision rules allowing for ambiguous pattern rejection and novelty detection The outputs of several classifiers may also be combined in a sensor fusion context, yielding decision procedures which are very robust to sensor failures or changes in the system environment Experiments with simulated and real data demonstrate the excellent performance of this classification scheme as compared to existing statistical and neural network techniques

Improving the Rprop Learning Algorithm

Abstract: The Rprop algorithm proposed by Riedmiller and Braun is one of the best performing first-order learning methods for neural networks. We introduce modifications of the algorithm that improve its learning speed. The resulting speedup is experimentally shown for a set of neural network learning tasks as well as for artificial error surfaces.

On the Computational Power of Winner-Take-All

Abstract: This article initiates a rigorous theoretical analysis of the computational power of circuits that employ modules for computing winner-take-all. Computational models that involve competitive stages have so far been neglected in computational complexity theory, although they are widely used in computational brain models, artificial neural networks, and analog VLSI. Our theoretical analysis shows that winner-take-all is a surprisingly powerful computational module in comparison with threshold gates (also referred to as McCulloch-Pitts neurons) and sigmoidal gates. We prove an optimal quadratic lower bound for computing winner-take-all in any feedforward circuit consisting of threshold gates. In addition we show that arbitrary continuous functions can be approximated by circuits employing a single soft winner-take-all gate as their only nonlinear operation. Our theoretical analysis also provides answers to two basic questions raised by neurophysiologists in view of the well-known asymmetry between excitatory and inhibitory connections in cortical circuits: how much computational power of neural networks is lost if only positive weights are employed in weighted sums and how much adaptive capability is lost if only the positive weights are subject to plasticity.

Handbook of Neural Network Signal Processing

Abstract: From the Publisher: The use of neural networks is permeating every area of signal processing. They can provide powerful means for solving many problems, especially in nonlinear, real-time, adaptive, and blind signal processing. The Handbook of Neural Network Signal Processing brings together applications that were previously scattered among various publications to provide an up-to-date, detailed treatment of the subject from an engineering point of view.The authors cover basic principles, modeling, algorithms, architectures, implementation procedures, and well-designed simulation examples of audio, video, speech, communication, geophysical, sonar, radar, medical, and many other signals. The subject of neural networks and their application to signal processing is constantly improving. You need a handy reference that will inform you of current applications in this new area. The Handbook of Neural Network Signal Processing provides this much needed service for all engineers and scientists in the field.

Pattern recognition via synchronization in phase-locked loop neural networks

Abstract: We propose a novel architecture of an oscillatory neural network that consists of phase-locked loop (PLL) circuits. It stores and retrieves complex oscillatory patterns as synchronized states with appropriate phase relations between neurons.

Stereo- and neural network-based pedestrian detection

Abstract: Pedestrian detection is essential to avoid dangerous traffic situations. We present a fast and robust algorithm for detecting pedestrians in a cluttered scene from a pair of moving cameras. This is achieved through stereo-based segmentation and neural network-based recognition. The algorithm includes three steps. First, we segment the image into sub-image object candidates using disparities discontinuity. Second, we merge and split the sub-image object candidates into sub-images that satisfy pedestrian size and shape constraints. Third, we use intensity gradients of the candidate sub-images as input to a trained neural network for pedestrian recognition. The experiments on a large number of urban street scenes demonstrate that the proposed algorithm: (1) can detect pedestrians in various poses, shapes, sizes, clothing, and occlusion status; (2) runs in real-time; and (3) is robust to illumination and background changes.