Computational intelligence techniques have been used in wide applications. Out of numerous computational intelligence techniques, neural networks and support vector machines (SVMs) have been playing the dominant roles. However, it is known that both neural networks and SVMs face some challenging issues such as: (1) slow learning speed, (2) trivial human intervene, and/or (3) poor computational scalability. Extreme learning machine (ELM) as emergent technology which overcomes some challenges faced by other techniques has recently attracted the attention from more and more researchers. ELM works for generalized single-hidden layer feedforward networks (SLFNs). The essence of ELM is that the hidden layer of SLFNs need not be tuned. Compared with those traditional computational intelligence techniques, ELM provides better generalization performance at a much faster learning speed and with least human intervene. This paper gives a survey on ELM and its variants, especially on (1) batch learning mode of ELM, (2) fully complex ELM, (3) online sequential ELM, (4) incremental ELM, and (5) ensemble of ELM.

Extreme learning machines: a survey

Letters: Convex incremental extreme learning machine

Enhanced random search based incremental extreme learning machine

One of the open problems in neural network research is how to automatically determine network architectures for given applications. In this brief, we propose a simple and efficient approach to automatically determine the number of hidden nodes in generalized single-hidden-layer feedforward networks (SLFNs) which need not be neural alike. This approach referred to as error minimized extreme learning machine (EM-ELM) can add random hidden nodes to SLFNs one by one or group by group (with varying group size). During the growth of the networks, the output weights are updated incrementally. The convergence of this approach is proved in this brief as well. Simulation results demonstrate and verify that our new approach is much faster than other sequential/incremental/growing algorithms with good generalization performance.

Error Minimized Extreme Learning Machine With Growth of Hidden Nodes and Incremental Learning

The emergent machine learning technique—extreme learning machines (ELMs)—has become a hot area of research over the past years, which is attributed to the growing research activities and significant contributions made by numerous researchers around the world. Recently, it has come to our attention that a number of misplaced notions and misunderstandings are being dissipated on the relationships between ELM and some earlier works. This paper wishes to clarify that (1) ELM theories manage to address the open problem which has puzzled the neural networks, machine learning and neuroscience communities for 60 years: whether hidden nodes/neurons need to be tuned in learning, and proved that in contrast to the common knowledge and conventional neural network learning tenets, hidden nodes/neurons do not need to be iteratively tuned in wide types of neural networks and learning models (Fourier series, biological learning, etc.). Unlike ELM theories, none of those earlier works provides theoretical foundations on feedforward neural networks with random hidden nodes; (2) ELM is proposed for both generalized single-hidden-layer feedforward network and multi-hidden-layer feedforward networks (including biological neural networks); (3) homogeneous architecture-based ELM is proposed for feature learning, clustering, regression and (binary/multi-class) classification. (4) Compared to ELM, SVM and LS-SVM tend to provide suboptimal solutions, and SVM and LS-SVM do not consider feature representations in hidden layers of multi-hidden-layer feedforward networks either.

What are Extreme Learning Machines? Filling the Gap Between Frank Rosenblatt’s Dream and John von Neumann’s Puzzle

We investigate the approximation ability of a multilayer perceptron (MLP) network when it is extended to the complex domain. The main challenge for processing complex data with neural networks has been the lack of bounded and analytic complex nonlinear activation functions in the complex domain, as stated by Liouville's theorem. To avoid the conflict between the boundedness and the analyticity of a nonlinear complex function in the complex domain, a number of ad hoc MLPs that include using two real-valued MLPs, one processing the real part and the other processing the imaginary part, have been traditionally employed. However, since nonanalytic functions do not meet the Cauchy-Riemann conditions, they render themselves into degenerative backpropagation algorithms that compromise the efficiency of nonlinear approximation and learning in the complex vector field. A number of elementary transcendental functions (ETFs) derivable from the entire exponential function ez that are analytic are defined as fully complex activation functions and are shown to provide a parsimonious structure for processing data in the complex domain and address most of the shortcomings of the traditional approach. The introduction of ETFs, however, raises a new question in the approximation capability of this fully complex MLP. In this letter, three proofs of the approximation capability of the fully complex MLP are provided based on the characteristics of singularity among ETFs. First, the fully complex MLPs with continuous ETFs over a compact set in the complex vector field are shown to be the universal approximator of any continuous complex mappings. The complex universal approximation theorem extends to bounded measurable ETFs possessing a removable singularity. Finally, it is shown that the output of complex MLPs using ETFs with isolated and essential singularities uniformly converges to any nonlinear mapping in the deleted annulus of singularity nearest to the origin.

Approximation by fully complex multilayer perceptrons

Designing a neural network (NN) to process complex-valued signals is a challenging task since a complex nonlinear activation function (AF) cannot be both analytic and bounded everywhere in the complex plane {\bb C}. To avoid this difficulty, ‘splitting’, i.e., using a pair of real sigmoidal functions for the real and imaginary components has been the traditional approach. However, this ‘ad hoc’ compromise to avoid the unbounded nature of nonlinear complex functions results in a nowhere analytic AF that performs the error back-propagation (BP) using the split derivatives of the real and imaginary components instead of relying on well-defined fully complex derivatives. In this paper, a fully complex multi-layer perceptron (MLP) structure that yields a simplified complex-valued back-propagation (BP) algorithm is presented. The simplified BP verifies that the fully complex BP weight update formula is the complex conjugate form of real BP formula and the split complex BP is a special case of the fully complex BP. This generalization is possible by employing elementary transcendental functions (ETFs) that are almost everywhere (a.e.) bounded and analytic in {\bb C}. The properties of fully complex MLP are investigated and the advantage of ETFs over split complex AF is shown in numerical examples where nonlinear magnitude and phase distortions of non-constant modulus modulated signals are successfully restored.

Fully Complex Multi-Layer Perceptron Network for Nonlinear Signal Processing

A number of complex nonlinear functions are proposed for the independent component analysis (ICA) of complex-valued data. We discuss the properties of these nonlinearities and show their efficiency in generating the higher order statistics needed for ICA.

Independent component analysis by complex nonlinearities

Designing a neural network (NN) for processing complex signals is a challenging task due to the lack of bounded and differentiable nonlinear activation functions in the entire complex domain C. To avoid this difficulty, 'splitting', i.e., using uncoupled real sigmoidal functions for the real and imaginary components has been the traditional approach, and a number of fully complex activation functions introduced can only correct for magnitude distortion but can not handle phase distortion. We have previously introduced a fully complex NN that uses a hyperbolic tangent function defined in the entire complex domain and showed that for most practical signal processing problems, it is sufficient to have an activation function that is bounded and differentiable almost everywhere in the complex domain. In this paper, the fully complex NN design is extended to employ other complex activation functions of the hyperbolic, circular, and their inverse function family. They are shown to successfully restore the nonlinear amplitude and phase distortions of non-constant modulus modulated signals.

Complex backpropagation neural network using elementary transcendental activation functions

One of the challenges in designing a neural network to process complex-valued signals is finding a suitable nonlinear complex activation function. The main reason for this difficulty is the conflict between the boundedness and the differentiability of complex functions in the entire complex plane, stated by Louiville's theorem. To avoid this difficulty, splitting, i.e., using two separate real nonlinear activation functions for the real and imaginary signal components has been the traditional approach. We introduce a feedforward neural network (FNN) architecture employing hyperbolic tangent tanh(z) function defined in the entire complex domain, and compare its performance with the FNN that uses a split complex structure. Since tanh(z) is analytic and bounded almost everywhere in the complex plane, when trained by backpropagation, it can easily outperform the non-analytic split complex activation function in convergence speed and achievable minimum squared error when the domain is bounded around the unit circle. We demonstrate this property by an equalization example, equalization of multi-phase shift keying (MPSK) signals corrupted by a multipath channel. The properties of tanh(z) and future directions to combat nonlinear distortions in complex transmission schemes are discussed.

Taehwan Kim

Papers

Approximation by fully complex multilayer perceptrons

Fully Complex Multi-Layer Perceptron Network for Nonlinear Signal Processing

Independent component analysis by complex nonlinearities

Complex backpropagation neural network using elementary transcendental activation functions

Fully complex backpropagation for constant envelope signal processing