Showing papers on "Deep learning published in 2004"

Cellular Neural Networks: Theory And Applications

Abstract: CONTENTS: Preface Introduction to Cellular Neural Networks Theory of Cellular Neural Networks: Mathematical Point of View Stability Analysis of Bidirectional Associative Memory CNNs with time delays On the Dynamics of Some Classes of Cellular Neural Networks Spatio-Temporal Phenomena in Two-dimensional Cellular Nonlinear Networks Travelling Waves in FitzHugh-Nagumo Cellular Neural Network Model CNN Applications in Modeling and Solving Non-Electrical Problems CNN for Obstacle Detection in Stereo Vision Imagery Object Tracking and Exact Colour Reproduction for Medical Imaging Criteria for Trained Neural Networks with Appliance in Passive Radiolocation Index.

Bayesian Methods for Neural Networks and Related Models

Abstract: Models such as feed-forward neural networks and certain other structures investigated in the computer science literature are not amenable to closed-form Bayesian analysis. The paper reviews the various approaches taken to overcome this difficulty, involving the use of Gaussian approximations, Markov chain Monte Carlo simulation routines and a class of non-Gaussian but “deterministic” approximations called variational approximations.

A one-layer recurrent neural network for support vector machine learning

Abstract: This paper presents a one-layer recurrent neural network for support vector machine (SVM) learning in pattern classification and regression. The SVM learning problem is first converted into an equivalent formulation, and then a one-layer recurrent neural network for SVM learning is proposed. The proposed neural network is guaranteed to obtain the optimal solution of support vector classification and regression. Compared with the existing two-layer neural network for the SVM classification, the proposed neural network has a low complexity for implementation. Moreover, the proposed neural network can converge exponentially to the optimal solution of SVM learning. The rate of the exponential convergence can be made arbitrarily high by simply turning up a scaling parameter. Simulation examples based on benchmark problems are discussed to show the good performance of the proposed neural network for SVM learning.

Nonlinear system modelling via optimal design of neural trees.

Abstract: This paper introduces a flexible neural tree model. The model is computed as a flexible multi-layer feed-forward neural network. A hybrid learning/evolutionary approach to automatically optimize the neural tree model is also proposed. The approach includes a modified probabilistic incremental program evolution algorithm (MPIPE) to evolve and determine a optimal structure of the neural tree and a parameter learning algorithm to optimize the free parameters embedded in the neural tree. The performance and effectiveness of the proposed method are evaluated using function approximation, time series prediction and system identification problems and compared with the related methods.

Recurrent neural network with backpropagation through time for speech recognition

Abstract: Speech recognition and understanding have been studied for many years. The neural network is well-known as a technique that is able to classify nonlinear problems. Much research has been done in applying neural networks to solving the problem of recognizing speech such as Arabic. Arabic offers a number of challenges to speech recognition. We propose a fully-connected hidden layer between the input and state nodes and the output. We also investigate and show that this hidden layer makes the learning of complex classification tasks more efficient. We also investigate the difference between LPCC (linear predictive cepstrum coefficients) and MFCC (Mel-frequency cepstral coefficients) in the feature extraction process. The aim of the study was to observe the differences in the 29 letters of the Arabic alphabet from "alif" to "ya". The purpose of this research is to upgrade the knowledge and understanding of Arabic alphabet or words using a fully-connected recurrent neural network (FCRNN) and backpropagation through time (BPTT) learning algorithm. Six speakers (a mixture of male and female) in a quiet environment are used in training.

Rule extraction: using neural networks or for neural networks?

Abstract: In the research of rule extraction from neural networks, fidelity describes how well the rules mimic the behavior of a neural network while accuracy describes how well the rules can be generalized. This paper identifies the fidelity-accuracy dilemma. It argues to distinguish rule extraction using neural networks and rule extraction for neural networks according to their different goals, where fidelity and accuracy should be excluded from the rule quality evaluation framework, respectively.

A neural net based approach to Test Oracle

Abstract: In this paper an attempt has been made to explore the possibility of the usage of artificial neural networks as Test Oracle. The triangle classification problem has been used as a case study. Results for the usage of unsupervised artificial networks indicate that they are not suitable for this purpose. The Feed-forward back propagation neural networks are demonstrated to be suitable.

Simultaneous optimization of neural network function and architecture algorithm

Abstract: A major limitation to current artificial neural network (NN) research is the inability to adequately identify unnecessary weights in the solution. If a method were found that would allow unnecessary weights to be identified, decision-makers would gain crucial information about the problem at hand as well as benefit by having a network that was more effective and efficient. The Neural Network Simultaneous Optimization Algorithm (NNSOA) is proposed for supervised training in multilayer feedforward neural networks. We demonstrate with Monte Carlo studies that the NNSOA can be used to obtain both a global solution and simultaneously identify a parsimonious network structure.

Neural networks for protein classification.

Abstract: This paper describes a biomolecular classification methodology based on multilayer perceptron neural networks. The system developed is used to classify enzymes found in the Protein Data Bank. The primary goal of classification, here, is to infer the function of an (unknown) enzyme by analysing its structural similarity to a given family of enzymes. A new codification scheme was devised to convert the primary structure of enzymes into a real-valued vector. The system was tested with a different number of neural networks, training set sizes and training epochs. For all experiments, the proposed system achieved a higher accuracy rate when compared with profile hidden Markov models. Results demonstrated the robustness of this approach and the possibility of implementing fast and efficient biomolecular classification using neural networks.

MBNR: Case-Based Reasoning with Local Feature Weighting by Neural Network

Abstract: Our aim is to build an integrated learning framework of neural network and case-based reasoning. The main idea is that feature weights for case-based reasoning can be evaluated by neural networks. In this paper, we propose MBNR (Memory-Based Neural Reasoning), case-based reasoning with local feature weighting by neural network. In our method, the neural network guides the case-based reasoning by providing case-specific weights to the learning process. We developed a learning algorithm to train the neural network to learn the case-specific local weighting patterns for case-based reasoning. We showed the performance of our learning system using four datasets.

Method of setting optimum-partitioned classified neural network and method and apparatus for automatic labeling using optimum-partitioned classified neural network

Abstract: A method of automatic labeling using an optimum-partitioned classified neural network includes searching for neural networks having minimum errors with respect to a number of L phoneme combinations from a number of K neural network combinations generated at an initial stage or updated, updating weights during learning of the K neural networks by K phoneme combination groups searched with the same neural networks, and composing an optimum-partitioned classified neural network combination using the K neural networks of which a total error sum has converged; and tuning a phoneme boundary of a first label file by using the phoneme combination group classification result and the optimum-partitioned classified neural network combination, and generating a final label file reflecting the tuning result.

Neural methods for non-standard data

Abstract: Standard pattern recognition provides effective and noise-tolerant tools for machine learning tasks; however, most approaches only deal with real vectors of a finite and fixed dimensionality In this tutorial paper, we give an overview about extensions of pattern recognition towards non-standard data which are not contained in a finite dimensional space, such as strings, sequences, trees, graphs, or functions Two major directions can be distinguished in the neural networks literature: models can be based on a similarity measure adapted to non-standard data, including kernel methods for structures as a very prominent approach, but also alternative metric based algorithms and functional networks; alternatively, non-standard data can be processed recursively within supervised and unsupervised recurrent and recursive networks and fully recurrent systems

High-order neural network structure selection for function approximation applications using genetic algorithms

Abstract: Neural network literature for function approximation is by now sufficiently rich. In its complete form, the problem entails both parametric (i.e., weights determination) and structural learning (i.e., structure selection). The majority of works deal with parametric uncertainty assuming knowledge of the appropriate neural structure. In this paper we present an algorithmic approach to determine the structure of high order neural networks (HONNs), to solve function approximation problems. The method is based on a genetic algorithm (GA) and is equipped with a stable update law to guarantee parametric learning. Simulation results on an illustrative example highlight the performance and give some insight of the proposed approach.

Evolving Neural Network Ensembles by Minimization of Mutual Information

Abstract: Learning and evolution are two fundamental forms of adaptation. There has been a great interest in combining learning and evolution with neural networks in recent years. This paper presents a hybrid learning system for learning and designing of neural network ensembles based on negative correlation learning and evolutionary learning. The idea of the hybrid learning system is to regard the population of neural networks as an ensemble, and the evolutionary process as the design of neural network ensembles. Two fitness sharing techniques have been used in the evolutionary process. One is based on the covering set. The other is to use the concept of mutual information. The effectiveness of such hybrid learning approach was tested on two real-world problems.

Adaptive control using combined online and background learning neural network

Abstract: A new adaptive neural network (NN) control concept is proposed with proof of stability properties. The NN learns the plant dynamics with online training, and then combines this with background learning from previously recorded data, which can be advantageous to the NN adaptation convergence characteristics. The network adaptation characteristics of the new combined online and background learning adaptive NN is demonstrated through simulations.

Neural Information Processing: Research and Development

Abstract: 1: Architectures.- Scale Independence in the Visual System.- Dynamic Neuronal Information Processing of Vowel Sounds in Auditory Cortex.- Convolutional Spiking Neural Network for Robust Object Detection with Population Code using Structured Pulse Packets.- Networks Constructed of Neuroid Elements Capable of Temporal Summation of Signals.- Predictive Synchrony Organized by Spike-Based Hebbian Learning with Time-Representing Synfire Activities.- Improving Chow-Liu Tree Performance by Mining Association Rules.- A Reconstructed Missing Data-Finite Impulse Response Selective Ensemble (RMD-FSE) Network.- Higher Order Multidirectional Associative Memory with Decreasing Energy Function.- Fast Indexing of Codebook Vectors Using Dynamic Binary Search Trees with Fat Decision Hyperplanes.- 2: Learning Algorithms.- On Some External Characteristics of Brain-like Learning and Some Logical Flaws of Connectionism.- Superlinear Learning Algorithm Design.- Extension of Binary Neural Networks for Multi-class Output and Finite Automata.- A Memory-Based Reinforcement Learning Algorithm to Prevent Unlearning in Neural Networks.- Structural Optimization of Neural Networks by Genetic Algorithm with Degeneration (GAd).- Adaptive Training for Combining Classifier Ensembles.- Combination Strategies for Finding Optimal Neural Network Architecture and Weights.- 3: Applications.- Biologically Inspired Recognition System for Car Detection from Real-Time Video Streams.- Financial Time Series Prediction Using Non-Fixed and Asymmetrical Margin Setting with Momentum in Support Vector Regression.- A Method for Applying Neural Networks to Control of Nonlinear Systesm.- Robot Manipulator Control via Recurrent Neural Networks.- Gesture Recognition Based on SOM Using Multiple Sensors.- Enhanced Phrase-Based Document Clustering Using Self-Organizing Map (SOM) Architectures.- Discovering Gene Regulatory Networks from Gene Expression Data with the Use of Evolving Connectionist Systems.- Experimental Analysis of Knowledge Based Multiagent Credit Assignment.- Implementation of Visual Tracking System Using Artificial Retina Chip and Shape Memory Alloy Actuator.

Resilient back propagation learning algorithm for recurrent fuzzy neural networks

Abstract: An efficient training method for recurrent fuzzy neural networks is proposed. The method modifies the RPROP algorithm, originally developed for static neural networks, in order to be applied to dynamic systems. A comparative analysis with the standard back propagation through time is given, indicating the effectiveness of the proposed algorithm.

Optimizing the Performance of Probabilistic Neural Networks in a BioinformaticsTa sk

Abstract: A self adaptive probabilistic neural network model is proposed The model incorporates the Particle Swarm Optimization algorithm to optimize the spread parameter of the probabilistic neural network, enhancing thus its perfor- mance The proposed approach is tested on two data sets from the eld of bioinformatics, with promising results The performance of the proposed model is compared to probabilistic neural networks, as well as to four different feedfor- ward neural networks Different sampling techniques are used, and statistical tests are performed to justify the statistical signicance of the results

3D object reconstruction and representation using neural networks

Abstract: 3D object reconstruction is frequent used in various fields such as product design, engineering, medical and artistic applications. Numerous reconstruction techniques and software were introduced and developed. However, the purpose of this paper is to fully integrate an adaptive artificial neural network (ANN) based method in reconstructing and representing 3D objects. This study explores the ability of neural networks in learning through experience when reconstructing an object by estimating it's z-coordinate. Neural networks' capability in representing most classes of 3D objects used in computer graphics is also proven. Simple affined transformation is applied on different objects using this approach and compared with the real objects. The results show that neural network is a promising approach for reconstruction and representation of 3D objects.

Modeling syllable duration in Indian languages using neural networks

Abstract: We propose a neural network model for predicting the syllable duration in Indian languages. A four layer feedforward neural network trained with a backpropagation algorithm is used for modeling the syllable duration. Analysis is performed on broadcast news data in Hindi, Telugu and Tamil in order to predict the duration of syllables in these languages using a neural network model. The input to the neural network consists of a set of phonological, positional and contextual features extracted from the text. About 88% of the syllable durations are predicted within 25% of the actual duration. The relative importance of the positional and contextual features are examined separately.

A principled approach for building and evaluating neural network classification models

Abstract: In this paper, we propose a principled approach to building and evaluating neural network classification models for decision support system (DSS) implementations. First, the usefulness of neural networks for use with e-commerce data and for Bayesian classification is discussed. Next, the theory concerning model accuracy and generalization is presented. Then, the principled approach, which is developed with consideration of these issues, is described. Through an illustrative problem, it is seen that when problem complexity is considered, the classification performance of the neural networks can be much better than what is observed. Furthermore, it is seen that model order selection processes based upon a single dataset can lead to an incorrect conclusion concerning the best model, which impacts model error and utility.

Binary factorization in hopfield-like neural networks: single-step approximation and computer simulations

Abstract: The unsupervised learning of feature extraction in high-dimesional pat- terns is a central problem for the neural network approach. Feature extraction is a procedure which maps original patterns into the feature (or factor) space of reduced dimension. In this paper we demonstrate that Hebbian learning in Hopfield-like neural network is a natural procedure for unsupervised learning of feature extrac- tion. Due to this learning, factors become the attractors of network dynamics, hence they can be revealed by the random search. The neurodynamics is anal- ysed by Single-Step approximation which is known (8) to be rather accurate for sparsely encoded Hopfield-network. Thus, the analysis is restricted by the case of sparsely encoded factors. The accuracy of Single-Step approximation is confirmed by computer simulations.

Fast neural networks for sub-matrix (object/face) detection

Abstract: In recent years, fast neural networks for sub-matrix (object/face) detection have been introduced based on cross correlation in frequency domain between the input image and the weights of neural networks. In H. M. El-Bakry (2003), it has been proved that for those fast neural networks, either the weights of neural networks or the input image must be symmetric. In case of converting the input image into a symmetric one, those fast neural networks become slower than conventional neural networks. In this paper, a new form of symmetry for he input image to fast the operation of neural nets is presented. Simulation results using Matlab confirm the theoretical computations.

An Evolutionary Race: A Comparison of Genetic Algorithms and Particle Swarm Optimization for Training Neural Networks.

Abstract: This paper compares the performance of genetic algorithms (GA) and particle swarm optimization (PSO) when used to train artificial neural networks. The networks are used to control virtual racecars, with the aim of successfully navigating around a track in the shortest possible period of time. Each car is mounted with multiple straight-line distance sensors, which provide the input to the networks. The cars act as autonomous agents for the duration of the training run: they record the distance traveled and rely on this for fitness evaluations. Both evolutionary algorithms are well suited to this unsupervised learning task, and the networks learn to successfully navigate the course in a minimal number of generations. The paper shows that PSO is superior for this application: it trains networks faster and more accurately than GAs do, once properly optimized.

Dynamically weighted ensemble neural networks for regression problems

Abstract: Combining the outputs of several neural networks into an aggregate output often gives improved accuracy over any individual output. The set of networks is known as an ensemble. This work presents an ensemble method for regression that has advantages over simple weighted or weighted average combining techniques. Generally, the output of an ensemble is a weighted sum whose weights are fixed. Our ensemble is weighted dynamically, the weights dynamically determined from the predicting accuracies of the trained networks with training dataset. The more accurate a network seems to be of its prediction, the higher the weight. This is implemented by generalized regression neural network. Empirical results show that this method improved on predicting accuracy.