Showing papers on "Backpropagation published in 2011"

Extensions of recurrent neural network language model

Abstract: We present several modifications of the original recurrent neural network language model (RNN LM).While this model has been shown to significantly outperform many competitive language modeling techniques in terms of accuracy, the remaining problem is the computational complexity. In this work, we show approaches that lead to more than 15 times speedup for both training and testing phases. Next, we show importance of using a backpropagation through time algorithm. An empirical comparison with feedforward networks is also provided. In the end, we discuss possibilities how to reduce the amount of parameters in the model. The resulting RNN model can thus be smaller, faster both during training and testing, and more accurate than the basic one.

Stochastic gradient descent on Riemannian manifolds

Abstract: Stochastic gradient descent is a simple approach to find the local minima of a cost function whose evaluations are corrupted by noise. In this paper, we develop a procedure extending stochastic gradient descent algorithms to the case where the function is defined on a Riemannian manifold. We prove that, as in the Euclidian case, the gradient descent algorithm converges to a critical point of the cost function. The algorithm has numerous potential applications, and is illustrated here by four examples. In particular a novel gossip algorithm on the set of covariance matrices is derived and tested numerically.

A hybrid method for MRI brain image classification

Abstract: Automated and accurate classification of MR brain images is of importance for the analysis and interpretation of these images and many methods have been proposed. In this paper, we present a neural network (NN) based method to classify a given MR brain image as normal or abnormal. This method first employs wavelet transform to extract features from images, and then applies the technique of principle component analysis (PCA) to reduce the dimensions of features. The reduced features are sent to a back propagation (BP) NN, with which scaled conjugate gradient (SCG) is adopted to find the optimal weights of the NN. We applied this method on 66 images (18 normal, 48 abnormal). The classification accuracies on both training and test images are 100%, and the computation time per image is only 0.0451s.

On the training of recurrent neural networks

Abstract: This paper proposes a new approach which combines unsupervised and supervised learning for training recurrent neural networks (RNNs). In this approach, the weights between input and hidden layers were determined according to an unsupervised procedure relying on the Kohonen algorithm and the weights between hidden and output layers were updated according to a supervised procedure based on dynamic gradient descent method. The simulation results show that the proposed method performs well in comparison with the back propagation through time algorithm.

A differential evolution based neural network approach to nonlinear system identification

Abstract: This paper addresses the effectiveness of soft computing approaches such as evolutionary computation (EC) and neural network (NN) to system identification of nonlinear systems. In this work, two evolutionary computing approaches namely differential evolution (DE) and opposition based differential evolution (ODE) combined with Levenberg Marquardt algorithm have been considered for training the feed-forward neural network applied for nonlinear system identification. Results obtained envisage that the proposed combined opposition based differential evolution neural network (ODE-NN) approach to identification of nonlinear system exhibits better model identification accuracy compared to differential evolution neural network (DE-NN) approach. The above method is finally tested on a one degree of freedom (1DOF) highly nonlinear twin rotor multi-input-multi-output system (TRMS) to verify the identification performance.

Multilayer perceptrons and radial basis function neural network methods for the solution of differential equations: A survey

Abstract: Since neural networks have universal approximation capabilities, therefore it is possible to postulate them as solutions for given differential equations that define unsupervised errors. In this paper, we present a wide survey and classification of different Multilayer Perceptron (MLP) and Radial Basis Function (RBF) neural network techniques, which are used for solving differential equations of various kinds. Our main purpose is to provide a synthesis of the published research works in this area and stimulate further research interest and effort in the identified topics. Here, we describe the crux of various research articles published by numerous researchers, mostly within the last 10 years to get a better knowledge about the present scenario.

Deep belief nets for natural language call-routing

Abstract: This paper considers application of Deep Belief Nets (DBNs) to natural language call routing. DBNs have been successfully applied to a number of tasks, including image, audio and speech classification, thanks to the recent discovery of an efficient learning technique. DBNs learn a multi-layer generative model from unlabeled data and the features discovered by this model are then used to initialize a feed-forward neural network which is fine-tuned with backpropagation. We compare a DBN-initialized neural network to three widely used text classification algorithms; Support Vector machines (SVM), Boosting and Maximum Entropy (MaxEnt). The DBN-based model gives a call-routing classification accuracy that is equal to the best of the other models even though it currently uses an impoverished representation of the input.

Electromygraphy (EMG) signal based hand gesture recognition using artificial neural network (ANN)

Abstract: Electromyography (EMG) signal is a measure of muscles' electrical activity and usually represented as a function of time, defined in terms of amplitude, frequency and phase. This biosignal can be employed in various applications including diagnoses of neuromuscular diseases, controlling assistive devices like prosthetic/orthotic devices, controlling machines, robots, computer etc. EMG signal based reliable and efficient hand gesture identification can help to develop good human computer interface which in turn will increase the quality of life of the disabled or aged people. The purpose of this paper is to describe the process of detecting different predefined hand gestures (left, right, up and down) using artificial neural network (ANN). ANNs are particularly useful for complex pattern recognition and classification tasks. The capability of learning from examples, the ability to reproduce arbitrary non-linear functions of input, and the highly parallel and regular structure of ANNs make them especially suitable for pattern recognition tasks. The EMG pattern signatures are extracted from the signals for each movement and then ANN utilized to classify the EMG signals based on features. A back-propagation (BP) network with Levenberg-Marquardt training algorithm has been used for the detection of gesture. The conventional and most effective time and time-frequency based features (namely MAV, RMS, VAR, SD, ZC, SSC and WL) have been chosen to train the neural network.

A New Elman Neural Network-Based Control Algorithm for Adjustable-Pitch Variable-Speed Wind-Energy Conversion Systems

Abstract: This paper presents an improved Elman neural network (IENN)-based algorithm for optimal wind-energy control with maximum power point tracking. An online training IENN controller using back-propagation (BP) learning algorithm with modified particle swarm optimization (MPSO) is designed to allow the pitch adjustment for power regulation. The node connecting weights of the IENN are trained online by BP methodology. MPSO is adopted to adjust the learning rates in the BP process to improve the learning capability. Performance of the proposed ENN with MPSO is verified by many experimental results.

Development and Implementation of Parameterized FPGA-Based General Purpose Neural Networks for Online Applications

Abstract: This paper presents the development and implementation of a generalized backpropagation multilayer perceptron (MLP) architecture described in VLSI hardware description language (VHDL). The development of hardware platforms has been complicated by the high hardware cost and quantity of the arithmetic operations required in online artificial neural networks (ANNs), i.e., general purpose ANNs with learning capability. Besides, there remains a dearth of hardware platforms for design space exploration, fast prototyping, and testing of these networks. Our general purpose architecture seeks to fill that gap and at the same time serve as a tool to gain a better understanding of issues unique to ANNs implemented in hardware, particularly using field programmable gate array (FPGA). The challenge is thus to find an architecture that minimizes hardware costs, while maximizing performance, accuracy, and parameterization. This work describes a platform that offers a high degree of parameterization, while maintaining generalized network design with performance comparable to other hardware-based MLP implementations. Application of the hardware implementation of ANN with backpropagation learning algorithm for a realistic application is also presented.

Comparing performances of backpropagation and genetic algorithms in the data classification

Abstract: Artificial neural networks (ANN) have a wide ranging usage area in the data classification problems. Backpropagation algorithm is classical technique used in the training of the artificial neural networks. Since this algorithm has many disadvantages, the training of the neural networks has been implemented with the binary and real-coded genetic algorithms. These algorithms can be used for the solutions of the classification problems. The real-coded genetic algorithm has been compared with other training methods in the few works. It is known that the comparison of the approaches is as important as proposing a new classification approach. For this reason, in this study, a large-scale comparison of performances of the neural network training methods is examined on the data classification datasets. The experimental comparison contains different real classification data taken from the literature and a simulation study. A comparative analysis on the real data sets and simulation data shows that the real-coded genetic algorithm may offer efficient alternative to traditional training methods for the classification problem.

A New Formulation for Feedforward Neural Networks

Abstract: Feedforward neural network is one of the most commonly used function approximation techniques and has been applied to a wide variety of problems arising from various disciplines. However, neural networks are black-box models having multiple challenges/difficulties associated with training and generalization. This paper initially looks into the internal behavior of neural networks and develops a detailed interpretation of the neural network functional geometry. Based on this geometrical interpretation, a new set of variables describing neural networks is proposed as a more effective and geometrically interpretable alternative to the traditional set of network weights and biases. Then, this paper develops a new formulation for neural networks with respect to the newly defined variables; this reformulated neural network (ReNN) is equivalent to the common feedforward neural network but has a less complex error response surface. To demonstrate the learning ability of ReNN, in this paper, two training methods involving a derivative-based (a variation of backpropagation) and a derivative-free optimization algorithms are employed. Moreover, a new measure of regularization on the basis of the developed geometrical interpretation is proposed to evaluate and improve the generalization ability of neural networks. The value of the proposed geometrical interpretation, the ReNN approach, and the new regularization measure are demonstrated across multiple test problems. Results show that ReNN can be trained more effectively and efficiently compared to the common neural networks and the proposed regularization measure is an effective indicator of how a network would perform in terms of generalization.

A Survey on Artificial Intelligence Approaches for Medical Image Classification

Abstract: In this paper, a survey has been made on the applications of intelligent computing techniques for diagnostic sciences in biomedical image classification. Several state-of-the-art Artificial Intelligence (AI) techniques for automation of biomedical image classification are investigated. This study gathers representative works that exhibit how AI is applied to the solution of very different problems related to different diagnostic science analysis. It also detects the methods of artificial intelligence that are used frequently together to solve the special problems of medicine. SVM neural network is used in almost all imaging modalities of medical image classification. Similarly fuzzy C means and improvements to it are important tool in segmentation of brain images. Various diagnostic studies like mammogram analysis, MRI brain analysis, bone and retinal analysis etc., using neural network approach result in use of back propagation network, probabilistic neural network, and extreme learning machine recurrently. Hybrid approach of GA and PSO are also commonly used for feature extraction and feature selection.

Evolving neural network using real coded genetic algorithm for permeability estimation of the reservoir

Abstract: In this work we investigate how artificial neural network (ANN) evolution with genetic algorithm (GA) improves the reliability and predictability of artificial neural network This strategy is applied to predict permeability of Mansuri Bangestan reservoir located in Ahwaz, Iran utilizing available geophysical well log data Our methodology utilizes a hybrid genetic algorithm-neural network strategy (GA-ANN) The proposed algorithm combines the local searching ability of the gradient-based back-propagation (BP) strategy with the global searching ability of genetic algorithms Genetic algorithms are used to decide the initial weights of the gradient decent methods so that all the initial weights can be searched intelligently The genetic operators and parameters are carefully designed and set avoiding premature convergence and permutation problems For an evaluation purpose, the performance and generalization capabilities of GA-ANN are compared with those of models developed with the common technique of BP The results demonstrate that carefully designed genetic algorithm-based neural network outperforms the gradient descent-based neural network

The use of neural network and discrete Fourier transform for real-time evaluation of friction stir welding

Abstract: This paper introduces a novel real-time approach to detecting wormhole defects in friction stir welding in a nondestructive manner. The approach is to evaluate feedback forces provided by the welding process using the discrete Fourier transform and a multilayer neural network. It is asserted here that the oscillations of the feedback forces are related to the dynamics of the plasticized material flow, so that the frequency spectra of the feedback forces can be used for detecting wormhole defects. A one-hidden-layer neural network trained with the backpropagation algorithm is used for classifying the frequency patterns of the feedback forces. The neural network is trained and optimized with a data set of forge-load control welds, and the generality is tested with novel data set of position control welds. Overall, about 95% classification accuracy is achieved with no bad welds classified as good. Accordingly, the present paper demonstrates an approach for providing important feedback information about weld quality in real-time to a control system for friction stir welding.

Implementation efficient nonlinear equalizer based on correlated digital backpropagation

Abstract: One stage per span is commonly considered as the lower boundary for implementation complexity of nonlinear equalizer based on digital backpropagation. The proposed method overcomes this boundary and improves the efficiency by about four times.

Decision Support System for Congenital Heart Disease Diagnosis based on Signs and Symptoms using Neural Networks

Abstract: Congenital Heart Disease is one of the major causes of deaths in children. However, a proper diagnosis at an early stage can result in significant life saving. Unfortunately, all the physicians are not equally skilled, which can cause for time delay, inaccuracy of the diagnosis. A system for automated medical diagnosis would enhance the accuracy of the diagnosis and reduce the cost effects. In the present paper, a Decision Support System has been proposed for diagnosis of Congenital Heart Disease. The proposed system is designed and developed by using MATLAB’s GUI feature with the implementation of Backpropagation Neural Network. The Backpropagation Neural Network used in this study is a multi layered Feed Forward Neural Network, which is trained by a supervised Delta Learning Rule. The dataset used in this study are the signs, symptoms and the results of physical evaluation of a patient. The proposed system achieved an accuracy of 90%.

Neural-Network Security-Boundary Constrained Optimal Power Flow

Abstract: This paper proposes a new approach to model stability and security constraints in optimal power flow (OPF) problems based on a neural network (NN) representation of the system security boundary (SB). The novelty of this proposal is that a closed form, differentiable function derived from the system's SB is used to represent security constraints in an OPF model. The procedure involves two main steps: First, an NN representation of the SB is obtained based on back-propagation neural network (BPNN) training. Second, a differentiable mapping function extracted from the BPNN is used to directly incorporate this function as a constraint in the OPF model. This approach ensures that the operating points resulting from the OPF solution process are within a feasible and secure region, whose limits are better represented using the proposed technique compared to typical security-constrained OPF models. The effectiveness and feasibility of the proposed approach is demonstrated through the implementation, as well as testing and comparison using the IEEE two-area and 118-bus benchmark systems, of an optimal dispatch technique that guarantees system security in the context of competitive electricity markets.

Feed-forward neural networks training: a comparison between genetic algorithm and back-propagation learning algorithm

Abstract: This study discusses the advantages and characteristics of the genetic algo- rithm and back-propagation neural network to train a feed-forward neural network to cope with weighting adjustment problems. We compare the performances of a back-propagation neural network and genetic algorithm in the training outcomes of three examples by re- ferring to the measurement indicators and experiment data. The results show that the back-propagation neural network is superior to the genetic algorithm. Also, the back- propagation neural network has faster training speed than the genetic algorithm. How- ever, the back-propagation neural network has the shortcoming of overtraining, while the genetic algorithm does not. The experiment result proves that the back-propagation neu- ral network yields better outcomes than the genetic algorithm.

Uniformly Stable Backpropagation Algorithm to Train a Feedforward Neural Network

Abstract: Neural networks (NNs) have numerous applications to online processes, but the problem of stability is rarely discussed. This is an extremely important issue because, if the stability of a solution is not guaranteed, the equipment that is being used can be damaged, which can also cause serious accidents. It is true that in some research papers this problem has been considered, but this concerns continuous-time NN only. At the same time, there are many systems that are better described in the discrete time domain such as population of animals, the annual expenses in an industry, the interest earned by a bank, or the prediction of the distribution of loads stored every hour in a warehouse. Therefore, it is of paramount importance to consider the stability of the discrete-time NN. This paper makes several important contributions. 1) A theorem is stated and proven which guarantees uniform stability of a general discrete-time system. 2) It is proven that the backpropagation (BP) algorithm with a new time-varying rate is uniformly stable for online identification and the identification error converges to a small zone bounded by the uncertainty. 3) It is proven that the weights' error is bounded by the initial weights' error, i.e., overfitting is eliminated in the proposed algorithm. 4) The BP algorithm is applied to predict the distribution of loads that a transelevator receives from a trailer and places in the deposits in a warehouse every hour, so that the deposits in the warehouse are reserved in advance using the prediction results. 5) The BP algorithm is compared with the recursive least square (RLS) algorithm and with the Takagi-Sugeno type fuzzy inference system in the problem of predicting the distribution of loads in a warehouse, giving that the first and the second are stable and the third is unstable. 6) The BP algorithm is compared with the RLS algorithm and with the Kalman filter algorithm in a synthetic example.

Price Prediction of Share Market using Artificial Neural Network (ANN)

Abstract: Share Market is an untidy place for predicting since there are no significant rules to estimate or predict the price of share in the share market. Many methods like technical analysis, fundamental analysis, time series analysis and statistical analysis etc are all used to attempt to predict the price in the share market but none of these methods are proved as a consistently acceptable prediction tool. Artificial Neural Network (ANN), a field of Artificial Intelligence (AI), is a popular way to identify unknown and hidden patterns in data which is suitable for share market prediction. For predicting of share price using ANN, there are two modules, one is training session and other is predicting price based on previously trained data. We used Backpropagation algorithm for training session and Multilayer Feedforward network as a network model for predicting price. In this paper, we introduce a method which can predict share market price using Backpropagation algorithm and Multilayer Feedforward network. General Terms Artificial Neural Network, Machine Learning, Back propagation Algorithms, Share Market.