Showing papers on "Activation function published in 2006"

Universal approximation using incremental constructive feedforward networks with random hidden nodes

Abstract: According to conventional neural network theories, single-hidden-layer feedforward networks (SLFNs) with additive or radial basis function (RBF) hidden nodes are universal approximators when all the parameters of the networks are allowed adjustable. However, as observed in most neural network implementations, tuning all the parameters of the networks may cause learning complicated and inefficient, and it may be difficult to train networks with nondifferential activation functions such as threshold networks. Unlike conventional neural network theories, this paper proves in an incremental constructive method that in order to let SLFNs work as universal approximators, one may simply randomly choose hidden nodes and then only need to adjust the output weights linking the hidden layer and the output layer. In such SLFNs implementations, the activation functions for additive nodes can be any bounded nonconstant piecewise continuous functions g:R→R and the activation functions for RBF nodes can be any integrable piecewise continuous functions g:R→R and ∫Rg(x)dx≠0. The proposed incremental method is efficient not only for SFLNs with continuous (including nondifferentiable) activation functions but also for SLFNs with piecewise continuous (such as threshold) activation functions. Compared to other popular methods such a new network is fully automatic and users need not intervene the learning process by manually tuning control parameters.

A Fast and Accurate Online Sequential Learning Algorithm for Feedforward Networks

Abstract: In this paper, we develop an online sequential learning algorithm for single hidden layer feedforward networks (SLFNs) with additive or radial basis function (RBF) hidden nodes in a unified framework. The algorithm is referred to as online sequential extreme learning machine (OS-ELM) and can learn data one-by-one or chunk-by-chunk (a block of data) with fixed or varying chunk size. The activation functions for additive nodes in OS-ELM can be any bounded nonconstant piecewise continuous functions and the activation functions for RBF nodes can be any integrable piecewise continuous functions. In OS-ELM, the parameters of hidden nodes (the input weights and biases of additive nodes or the centers and impact factors of RBF nodes) are randomly selected and the output weights are analytically determined based on the sequentially arriving data. The algorithm uses the ideas of ELM of Huang developed for batch learning which has been shown to be extremely fast with generalization performance better than other batch training methods. Apart from selecting the number of hidden nodes, no other control parameters have to be manually chosen. Detailed performance comparison of OS-ELM is done with other popular sequential learning algorithms on benchmark problems drawn from the regression, classification and time series prediction areas. The results show that the OS-ELM is faster than the other sequential algorithms and produces better generalization performance

Can threshold networks be trained directly

Abstract: Neural networks with threshold activation functions are highly desirable because of the ease of hardware implementation. However, the popular gradient-based learning algorithms cannot be directly used to train these networks as the threshold functions are nondifferentiable. Methods available in the literature mainly focus on approximating the threshold activation functions by using sigmoid functions. In this paper, we show theoretically that the recently developed extreme learning machine (ELM) algorithm can be used to train the neural networks with threshold functions directly instead of approximating them with sigmoid functions. Experimental results based on real-world benchmark regression problems demonstrate that the generalization performance obtained by ELM is better than other algorithms used in threshold networks. Also, the ELM method does not need control variables (manually tuned parameters) and is much faster.

Multilayer Feedforward Neural Network Based on Multi-valued Neurons (MLMVN) and a Backpropagation Learning Algorithm

Abstract: A multilayer neural network based on multi-valued neurons (MLMVN) is considered in the paper. A multi-valued neuron (MVN) is based on the principles of multiple-valued threshold logic over the field of the complex numbers. The most important properties of MVN are: the complex-valued weights, inputs and output coded by the kth roots of unity and the activation function, which maps the complex plane into the unit circle. MVN learning is reduced to the movement along the unit circle, it is based on a simple linear error correction rule and it does not require a derivative. It is shown that using a traditional architecture of multilayer feedforward neural network (MLF) and the high functionality of the MVN, it is possible to obtain a new powerful neural network. Its training does not require a derivative of the activation function and its functionality is higher than the functionality of MLF containing the same number of layers and neurons. These advantages of MLMVN are confirmed by testing using parity n, two spirals and "sonar" benchmarks and the Mackey---Glass time series prediction.

Improved conditions for global exponential stability of recurrent neural networks with time-varying delays

Abstract: This paper presents new theoretical results on global exponential stability of recurrent neural networks with bounded activation functions and time-varying delays. The stability conditions depend on external inputs, connection weights, and time delays of recurrent neural networks. Using these results, the global exponential stability of recurrent neural networks can be derived, and the estimated location of the equilibrium point can be obtained. As typical representatives, the Hopfield neural network (HNN) and the cellular neural network (CNN) are examined in detail

Application of a radial basis function neural network for diagnosis of diabetes mellitus

Abstract: In this article an attempt is made to study the appl i- cability of a general purpose, supervised feed forward neural network with one hidden layer, namely. Radial Basis Function (RBF) neural network. It uses rela- tively smaller number of locally tuned units and is adaptive in nature. RBFs are suitable for pattern re cog- nition and classification. Performance of the RBF neural network was also compared with the most commonly used multilayer perceptron network model and the classical logistic regression. Diabetes database was used for empirical comparisons and the results show that RBF network performs better than other models. 1 . In MLP, the weighted sum of the inputs and bias term are passed to activation level through a transfer function to produce the output, and the units are arranged in a layered feed-forward topology called Feed Forward Neural Network (FFNN). The schematic representation of FFNN with n inputs, m hidden units and one output unit along with the bias term of the input unit and hidden unit is given in Figure 1. An artificial neural ne twork (ANN) has three layers: input layer, hidden layer and output layer. The hidden layer vastly increases the lear ning power of the MLP. The transfer or activation function of the ne twork modifies the input to give a desired output. The transfer function is chosen such that the algorithm requires a r e- sponse function with a continuous, single-valued with first derivative existence. Choice of the number of the hi d- den layers, hidden nodes and type of activ ation function plays an important role in model constructions 2-4 . Radial basis function (RBF) neural network is based on supervised learning. RBF networks were independently proposed by many researchers 5-9 and are a popular alter- native to the MLP. RBF networks are also good at modelling nonlinear data and can be trained in one stage rather than using an iterative process as in MLP and also learn the given application quickly. They are useful in sol ving problems where the input data are corrupted with add itive noise. The transformation functions used are based on a Gau ssian distribution. If the error of the network is minimized appropr i- ately, it will produce outputs that sum to unity, which will represent a probability for the outputs. The objective of this article is to study the applic ability of RBF to diabetes data and compare the results with MLP and logistic r e- gression.

Dynamical Behaviors of Delayed Neural Network Systems with Discontinuous Activation Functions

Abstract: In this letter, without assuming the boundedness of the activation functions, we discuss the dynamics of a class of delayed neural networks with discontinuous activation functions. A relaxed set of sufficient conditions is derived, guaranteeing the existence, uniqueness, and global stability of the equilibrium point. Convergence behaviors for both state and output are discussed. The constraints imposed on the feedback matrix are independent of the delay parameter and can be validated by the linear matrix inequality technique. We also prove that the solution of delayed neural networks with discontinuous activation functions can be regarded as a limit of the solutions of delayed neural networks with high-slope continuous activation functions.

A novel radial basis function neural network for discriminant analysis

Abstract: A novel radial basis function neural network for discriminant analysis is presented in this paper. In contrast to many other researches, this work focuses on the exploitation of the weight structure of radial basis function neural networks using the Bayesian method. It is expected that the performance of a radial basis function neural network with a well-explored weight structure can be improved. As the weight structure of a radial basis function neural network is commonly unknown, the Bayesian method is, therefore, used in this paper to study this a priori structure. Two weight structures are investigated in this study, i.e., a single-Gaussian structure and a two-Gaussian structure. An expectation-maximization learning algorithm is used to estimate the weights. The simulation results showed that the proposed radial basis function neural network with a weight structure of two Gaussians outperformed the other algorithms

Stability analysis of Cohen-Grossberg neural networks

Abstract: Without assuming boundedness and differentiability of the activation functions and any symmetry of interconnections, we employ Lyapunov functions to establish some sufficient conditions ensuring existence, uniqueness, global asymptotic stability, and even global exponential stability of equilibria for the Cohen-Grossberg neural networks with and without delays. Our results are not only presented in terms of system parameters and can be easily verified and also less restrictive than previously known criteria and can be applied to neural networks, including Hopfield neural networks, bidirectional association memory neural networks, and cellular neural networks.

The application of a radial basis function neural network for predicting the surface roughness in a turning process

Abstract: This study considers the performance of a radial basis function neural network for predicting the surface roughness in a turning process. A simple algorithm is proposed for finding the upper and lower estimates of the surface roughness. A code is developed that automatically fits the best network architecture for a given training and testing dataset. The validation of the methodology is carried out for dry and wet turning of mild steel using HSS and carbide tools, and is compared to the performance of the studied network with the reported performance of a multi-layer perception neural network. It is observed that the performance of the radial basis function network is slightly inferior compared to multi-layer perceptron neural network. However, the training procedure is simpler and requires less computational time.

An efficient hardware architecture for a neural network activation function generator

Abstract: This paper proposes an efficient hardware architecture for a function generator suitable for an artificial neural network (ANN). A spline-based approximation function is designed that provides a good trade-off between accuracy and silicon area, whilst also being inherently scalable and adaptable for numerous activation functions. This has been achieved by using a minimax polynomial and through optimal placement of the approximating polynomials based on the results of a genetic algorithm. The approximation error of the proposed method compares favourably to all related research in this field. Efficient hardware multiplication circuitry is used in the implementation, which reduces the area overhead and increases the throughput.

Training radial basis function networks with differential evolution

Abstract: In this paper, Differential Evolution (DE) algorithm, a new promising evolutionary algorithm, is proposed to train Radial Basis Function (RBF) network related to automatic configuration of network architecture. Classification tasks on data sets: Iris, Wine, New-thyroid and Glass are conducted to measure the performance of neural networks. Compared with a standard RBF training algorithm in Matlab neural network toolbox, DE achieves more rational architecture for RBF networks. The resulting networks hence obtain strong generalization abilities.

An Experiment on the Design of Radial Basis Function Neural Networks for Software Cost Estimation

Abstract: This paper is concerned with the use of Radial Basis Function (RBF) neural networks for software cost estimation The study is devoted to the design of these networks, especially their middle layer composed of receptive fields, using two clustering techniques the C-means and the APC-III algorithms A comparison between an RBFN using C-means and an RBFN using APC-III, in terms of estimates accuracy, is hence presented This study is based on the COCOMO' 81 dataset

New Criteria of Global Exponential Stability for a Class of Generalized Neural Networks with Time-Varying Delays

Abstract: In this paper, we essentially drop the requirement of Lipschitz condition on the activation functions. By employing Lyapunov functional and several new inequalities, some new criteria concerning global exponential stability for a class of generalized neural networks with time-varying delays are obtained, which only depend on physical parameters of neural networks. Since these new criteria do not require the activation functions to be differentiable, bounded or monotone nondecreasing and the connection weight matrices to be symmetric, they are mild and more general than previously known criteria.

A quantitative comparison of different MLP activation functions in classification

Abstract: Multilayer perceptrons (MLP) has been proven to be very successful in many applications including classification. The activation function is the source of the MLP power. Careful selection of the activation function has a huge impact on the network performance. This paper gives a quantitative comparison of the four most commonly used activation functions, including the Gaussian RBF network, over ten real different datasets. Results show that the sigmoid activation function substantially outperforms the other activation functions. Also, using only the needed number of hidden units in the MLP, we improved its conversion time to be competitive with the RBF networks most of the time.

GA-based Neural Fuzzy Control of Flexible-link Manipulators

Abstract: The limitations of conventional model-based control mechanisms for flexible manipulator systems have stimulated the development of intelligent control mechanisms incorporating fuzzy logic and neural networks. Problems have been encountered in applying the traditional PD-, PI-, and PID-type fuzzy controllers to flexible-link manipulators. A PD-PI-type fuzzy controller has been developed where the membership functions are adjusted by tuning the scaling factors using a neural network. Such a network needs a sufficient number of neurons in the hidden layer to approximate the nonlinearity of the system. A simple realisable network is desirable and hence a single neuron network with a nonlinear activation function is used. It has been demonstrated that the sigmoidal function and its shape can represent the nonlinearity of the system. A genetic algorithm is used to learn the weights, biases and shape of the sigmoidal function of the neural network.

Global Robust Exponential Stability of Interval General BAM Neural Network with Delays

Abstract: In this paper, a class of interval general bidirectional associative memory (BAM) neural networks with delays are introduced and studied, which include many well-known neural networks as special cases. By using fixed point technic, we prove an existence and uniqueness of the equilibrium point for the interval general BAM neural networks with delays. By using a proper Lyapunov functions, we get a sufficient condition to ensure the global robust exponential stability for the interval general BAM neural networks with delays, and we just require that activation function is globally Lipschitz continuous, which is less conservative and less restrictive than the monotonic assumption in previous results. In the last section, we also give an example to demonstrate the validity of our stability result for interval neural networks with delays.

Single-Step Prediction of Chaotic Time Series Using Wavelet-Networks

Abstract: This paper presents a wavelet neural-network for chaotic time series prediction. Wavelet-networks are inspired by both the feed-forward neural network and the theory underlying wavelet decompositions. Wavelet-networks are a class of neural network that take advantage of good localization properties of multiresolution analysis and combine them with the approximation abilities of neural networks. This kind of networks uses wavelets as activation functions in the hidden layer and a type of backpropagation algorithm is used for its learning. Comparisons are made between a wavelet-network and the typical feedforward network trained with the back-propagation algorithm. The results reported in this paper show that wavelet-networks have better prediction properties than its similar back-propagation networks

Real-time video based finger spelling recognition system using low computational complexity Artificial Neural Networks

Abstract: The automatic sign language translation still is the most complex and challenging task for video recognition and processing This work presents the Brazilian Sign Language Automatic Translation project and specifically focuses on low complexity Artificial Neural Networks dedicated to real-time video processing A new approach for reducing the computational complexity of the activation function of the Multi-Layer Perceptron is proposed in this work, allowing complex processing of video signals be done in real-time The low complexity neural networks are used in two stages of the system In the color detection and hand posture classification blocks The obtained results indicate an increase of the frame rate from 86 fps to 281 fps using a personal microcomputer with a USB webcam, without reduction of the correct recognition rate

Wavelets Based Neural Network for Function Approximation

Abstract: In this paper, a new type of WNN is proposed to enhance the function approximation capability. In the proposed WNN, the nonlinear activation function is a linear combination of wavelets, that can be updated during the networks training process. As a result the approximate error is significantly decreased. The BP algorithm and the QR decomposition based training method for the proposed WNN is derived. The obtained results indicate that this new type of WNN exhibits excellent learning ability compared to the conventional ones.

RBF ANN nonlinear prediction model based adaptive PID control of switched reluctance motor drive

Abstract: The inherent nonlinear of switched reluctance motor (SRM) makes it hard to get a good performance by using the conventional PID controller to the speed control of SRM. This paper develops a radial basis function (RBF) artificial neural network (ANN) nonlinear prediction model based adaptive PID controller for SRM. ANN, under certain condition, can approximate any nonlinear function with arbitrary precision. It also has a strong ability of adaptive, selflearning and self-organization. So, combining it with the conventional PID controller, a neural network based adaptive PID controller can be developed. Appling it to the speed control of SRM, a good control performance can be gotten. At the same time, the nonlinear mapping property and high parallel operation ability of ANN make it suitable to be applied to establish nonlinear prediction model performing parameter prediction. In this paper, two ANN - NNC and NNI are employed. The former is a back propagation (BP) ANN with sigmoid activation function. The later is an ANN using RBF as activation function. The former is used to adaptively adjust the parameters of the PID controller on line. The later is used to establish nonlinear prediction model performing parameter prediction. Compared with BP ANN with sigmoid activation function, the RBF ANN has a more fast convergence speed and can avoid getting stuck in a local optimum. Through parameter prediction, response speed of the system can be improved. To increase the convergence speed of ANN, an adaptive learning algorithm is adopted in this paper that is to adjust the learning rate according to the error. This can increase the convergence speed of ANN and make the system response quick. The experimental results demonstrate that a high control performance is achieved. The system responds quickly with little overshoot. The steady state error is zero. The system shows robust performance to the load torque disturbance.