Recently, a novel learning algorithm for single-hidden-layer feedforward neural networks (SLFNs) named extreme learning machine (ELM) was proposed by Huang et al. The essence of ELM is that the learning parameters of hidden nodes, including input weights and biases, are randomly assigned and need not be tuned while the output weights can be analytically determined by the simple generalized inverse operation. The only parameter needed to be defined is the number of hidden nodes. Compared with other traditional learning algorithms for SLFNs, ELM provides extremely faster learning speed, better generalization performance and with least human intervention. This paper firstly introduces a brief review of ELM, describing the principle and algorithm of ELM. Then, we put emphasis on the improved methods or the typical variants of ELM, especially on incremental ELM, pruning ELM, error-minimized ELM, two-stage ELM, online sequential ELM, evolutionary ELM, voting-based ELM, ordinal ELM, fully complex ELM, and symmetric ELM. Next, the paper summarized the applications of ELM on classification, regression, function approximation, pattern recognition, forecasting and diagnosis, and so on. In the last, the paper discussed several open issues of ELM, which may be worthy of exploring in the future.

Extreme learning machine and its applications

An extreme learning machine (ELM) is a feedforward neural network (FNN) like learning system whose connections with output neurons are adjustable, while the connections with and within hidden neurons are randomly fixed. Numerous applications have demonstrated the feasibility and high efficiency of ELM-like systems. It has, however, been open if this is true for any general applications. In this two-part paper, we conduct a comprehensive feasibility analysis of ELM. In Part I, we provide an answer to the question by theoretically justifying the following: 1) for some suitable activation functions, such as polynomials, Nadaraya-Watson and sigmoid functions, the ELM-like systems can attain the theoretical generalization bound of the FNNs with all connections adjusted, i.e., they do not degrade the generalization capability of the FNNs even when the connections with and within hidden neurons are randomly fixed; 2) the number of hidden neurons needed for an ELM-like system to achieve the theoretical bound can be estimated; and 3) whenever the activation function is taken as polynomial, the deduced hidden layer output matrix is of full column-rank, therefore the generalized inverse technique can be efficiently applied to yield the solution of an ELM-like system, and, furthermore, for the nonpolynomial case, the Tikhonov regularization can be applied to guarantee the weak regularity while not sacrificing the generalization capability. In Part II, however, we reveal a different aspect of the feasibility of ELM: there also exists some activation functions, which makes the corresponding ELM degrade the generalization capability. The obtained results underlie the feasibility and efficiency of ELM-like systems, and yield various generalizations and improvements of the systems as well.

Is Extreme Learning Machine Feasible? A Theoretical Assessment (Part I)

This paper deals with the recognition of handwritten Malayalam character using wavelet energy feature (WEF) and extreme learning machine (ELM). The wavelet energy (WE) is a new and robust parameter, and is derived using wavelet transform. It can reduce the influences of different types of noise at different levels. WEF can reflect the WE distribution of characters in several directions at different scales. To a non oscillating pattern, the amplitudes of wavelet coefficients increase when the scale of wavelet decomposition increase. WE of different decomposition levels have different powers to discriminate the character images. These features constitute patterns of handwritten characters for classification. The traditional learning algorithms of the different classifiers are far slower than required. So we have used an extremely fast leaning algorithm called ELM for single hidden layer feed forward networks (SLFN), which randomly chooses the input weights and analytically determines the output weights of SLFN. This algorithm learns much faster than traditional popular learning algorithms for feed forward neural networks. This feature vector, classifier combination gave good recognition accuracy at level 6 of the wavelet decomposition.

Handwritten character recognition using wavelet energy and extreme learning machine

Conventional extreme learning machines (ELMs) solve a Moore–Penrose generalized inverse of hidden layer activated matrix and analytically determine the output weights to achieve generalized performance, by assuming the same loss from different types of misclassification. The assumption may not hold in cost-sensitive recognition tasks, such as face recognition-based access control system, where misclassifying a stranger as a family member may result in more serious disaster than misclassifying a family member as a stranger. Though recent cost-sensitive learning can reduce the total loss with a given cost matrix that quantifies how severe one type of mistake against another, in many realistic cases, the cost matrix is unknown to users. Motivated by these concerns, this paper proposes an evolutionary cost-sensitive ELM, with the following merits: 1) to the best of our knowledge, it is the first proposal of ELM in evolutionary cost-sensitive classification scenario; 2) it well addresses the open issue of how to define the cost matrix in cost-sensitive learning tasks; and 3) an evolutionary backtracking search algorithm is induced for adaptive cost matrix optimization. Experiments in a variety of cost-sensitive tasks well demonstrate the effectiveness of the proposed approaches, with about 5%–10% improvements.

Evolutionary Cost-Sensitive Extreme Learning Machine

Boosting weighted ELM for imbalanced learning

The machine recognition of handwriting has been studied several decades, and still is an active research area due to the presence of many challenging issues. Several measures have been taken to improve the recognition accuracy and decrease computational burden and time involved in the overall recognition process. This work is carried out in this context, and tried to achieve the above goals. A discriminative feature set is an important component in any pattern recognition system. We have used division point features generated by recursive subdivision of chamaracter images. These features are extracted from a size normalized binary image. In many of the applications data arrive chunk by chunk or one by one, and we need a classifier which handles the data sequentially. In this paper, a fast and accurate online sequential algorithm known as OS-ELM is used for single hidden layer feed forward neural networks (SLFN) with both additive and radial basis function (RBF) hidden nodes in a unified way. We have also considered the fact that the complexity of the system increases as the number of hidden layers increases. An SLFN trained with OS-ELM is used for the classification of 44 class handwritten Malayalam characters. The only control parameter required for this system is the size of the hidden layer. After several practices we have achieved a recognition accuracy of 96.83% with 1200 neurons in the hidden layer.

Online sequential extreme learning machine based handwritten character recognition

This paper deals with the recognition of handwritten Malayalam characters. Most of the pattern recognition systems will go through the steps like preprocessing, feature extraction and classification. Here, we have presented edge detection in the preprocessing stage. It is important that the edge detector should give the character edges without fragmentation and displacement of edge pixels. Canny edge detector is used to produce thinned edges of the character. But for high resolution images, it gives spurious branches corresponding to noise and texture of the image. So a technique is applied prior to this process to focus the edge regions using nonlinear an isotropic diffusion via partial differential equations (PDE). Even though this method helped to get accurate edge points, the final result shows some broken edges. These broken parts are then linked using ant colony optimization (ACO) method. This image is further partitioned into different zones for the purpose of feature extraction. Multi layer perceptron (MLP) used these features and classified the characters with a recognition accuracy of 95.16%.

Pre and Post Processing Approaches in Edge Detection for Character Recognition

This paper compares the recognition accuracy of handwritten numerals achieved using statistical and structural features. Both features are trained and tested using neural network. In order to get good features, digit images are undergone various preprocessing activities. The operations such as noise removal, thresholding, linking broken digit, rotation, pruning and cropping are done before feature extraction. The recognition rate obtained using statistical and structural features are 93.3% and 95.7% respectively.

Comparison of Statistical and Structural Features for Handwritten Numeral Recognition

Generally, gradient based learning algorithms have showed reasonable performance in the training of multi layer feed forward neural networks; but they are still relatively slow in learning. In this context, a hybrid learning algorithm is used by combining differential evolution (DE) algorithm and Moore-Penrose (MP) generalized inverse to classify handwritten Malayalam characters. DE is used to select the input weights and biases in a single layer feed forward neural network (SLFN), and the output weights are analytically determined with MP inverse. A new set of features known as division point distance from centroid (DPDC) is used to generate patterns for the classifier. The system could provide overall recognition accuracy 83.98% by spending only 184 seconds for training.

Binu P. Chacko

Papers

Handwritten character recognition using wavelet energy and extreme learning machine

Online sequential extreme learning machine based handwritten character recognition

Pre and Post Processing Approaches in Edge Detection for Character Recognition

Comparison of Statistical and Structural Features for Handwritten Numeral Recognition

A Hybrid Learning Algorithm for Handwriting Recognition