Showing papers in "IEEE Transactions on Neural Networks in 2011"
TL;DR: This work proposes a novel dimensionality reduction framework for reducing the distance between domains in a latent space for domain adaptation and proposes both unsupervised and semisupervised feature extraction approaches, which can dramatically reduce thedistance between domain distributions by projecting data onto the learned transfer components.
Abstract: Domain adaptation allows knowledge from a source domain to be transferred to a different but related target domain. Intuitively, discovering a good feature representation across domains is crucial. In this paper, we first propose to find such a representation through a new learning method, transfer component analysis (TCA), for domain adaptation. TCA tries to learn some transfer components across domains in a reproducing kernel Hilbert space using maximum mean miscrepancy. In the subspace spanned by these transfer components, data properties are preserved and data distributions in different domains are close to each other. As a result, with the new representations in this subspace, we can apply standard machine learning methods to train classifiers or regression models in the source domain for use in the target domain. Furthermore, in order to uncover the knowledge hidden in the relations between the data labels from the source and target domains, we extend TCA in a semisupervised learning setting, which encodes label information into transfer components learning. We call this extension semisupervised TCA. The main contribution of our work is that we propose a novel dimensionality reduction framework for reducing the distance between domains in a latent space for domain adaptation. We propose both unsupervised and semisupervised feature extraction approaches, which can dramatically reduce the distance between domain distributions by projecting data onto the learned transfer components. Finally, our approach can handle large datasets and naturally lead to out-of-sample generalization. The effectiveness and efficiency of our approach are verified by experiments on five toy datasets and two real-world applications: cross-domain indoor WiFi localization and cross-domain text classification.
3,195 citations
TL;DR: An ensemble of classifiers-based approach for incremental learning of concept drift, characterized by nonstationary environments (NSEs), where the underlying data distributions change over time, which indicates that Learn++.NSE can track the changing environments very closely, regardless of the type of concept Drift.
Abstract: We introduce an ensemble of classifiers-based approach for incremental learning of concept drift, characterized by nonstationary environments (NSEs), where the underlying data distributions change over time. The proposed algorithm, named Learn++.NSE, learns from consecutive batches of data without making any assumptions on the nature or rate of drift; it can learn from such environments that experience constant or variable rate of drift, addition or deletion of concept classes, as well as cyclical drift. The algorithm learns incrementally, as other members of the Learn++ family of algorithms, that is, without requiring access to previously seen data. Learn++.NSE trains one new classifier for each batch of data it receives, and combines these classifiers using a dynamically weighted majority voting. The novelty of the approach is in determining the voting weights, based on each classifier's time-adjusted accuracy on current and past environments. This approach allows the algorithm to recognize, and act accordingly, to the changes in underlying data distributions, as well as to a possible reoccurrence of an earlier distribution. We evaluate the algorithm on several synthetic datasets designed to simulate a variety of nonstationary environments, as well as a real-world weather prediction dataset. Comparisons with several other approaches are also included. Results indicate that Learn++.NSE can track the changing environments very closely, regardless of the type of concept drift. To allow future use, comparison and benchmarking by interested researchers, we also release our data used in this paper.
782 citations
TL;DR: It is shown that a simple deterministically constructed cycle reservoir is comparable to the standard echo state network methodology and the (short-term) of linear cyclic reservoirs can be made arbitrarily close to the proved optimal value.
Abstract: Reservoir computing (RC) refers to a new class of state-space models with a fixed state transition structure (the reservoir) and an adaptable readout form the state space. The reservoir is supposed to be sufficiently complex so as to capture a large number of features of the input stream that can be exploited by the reservoir-to-output readout mapping. The field of RC has been growing rapidly with many successful applications. However, RC has been criticized for not being principled enough. Reservoir construction is largely driven by a series of randomized model-building stages, with both researchers and practitioners having to rely on a series of trials and errors. To initialize a systematic study of the field, we concentrate on one of the most popular classes of RC methods, namely echo state network, and ask: What is the minimal complexity of reservoir construction for obtaining competitive models and what is the memory capacity (MC) of such simplified reservoirs? On a number of widely used time series benchmarks of different origin and characteristics, as well as by conducting a theoretical analysis we show that a simple deterministically constructed cycle reservoir is comparable to the standard echo state network methodology. The (short-term) of linear cyclic reservoirs can be made arbitrarily close to the proved optimal value.
578 citations
TL;DR: A new, fast, yet reliable method for the construction of PIs for NN predictions, and the quantitative comparison with three traditional techniques for prediction interval construction reveals that the LUBE method is simpler, faster, and more reliable.
Abstract: Prediction intervals (PIs) have been proposed in the literature to provide more information by quantifying the level of uncertainty associated to the point forecasts. Traditional methods for construction of neural network (NN) based PIs suffer from restrictive assumptions about data distribution and massive computational loads. In this paper, we propose a new, fast, yet reliable method for the construction of PIs for NN predictions. The proposed lower upper bound estimation (LUBE) method constructs an NN with two outputs for estimating the prediction interval bounds. NN training is achieved through the minimization of a proposed PI-based objective function, which covers both interval width and coverage probability. The method does not require any information about the upper and lower bounds of PIs for training the NN. The simulated annealing method is applied for minimization of the cost function and adjustment of NN parameters. The demonstrated results for 10 benchmark regression case studies clearly show the LUBE method to be capable of generating high-quality PIs in a short time. Also, the quantitative comparison with three traditional techniques for prediction interval construction reveals that the LUBE method is simpler, faster, and more reliable.
533 citations
TL;DR: A novel data-driven robust approximate optimal tracking control scheme is proposed for unknown general nonlinear systems by using the adaptive dynamic programming (ADP) method and a robustifying term is developed to compensate for the NN approximation errors introduced by implementing the ADP method.
Abstract: In this paper, a novel data-driven robust approximate optimal tracking control scheme is proposed for unknown general nonlinear systems by using the adaptive dynamic programming (ADP) method. In the design of the controller, only available input-output data is required instead of known system dynamics. A data-driven model is established by a recurrent neural network (NN) to reconstruct the unknown system dynamics using available input-output data. By adding a novel adjustable term related to the modeling error, the resultant modeling error is first guaranteed to converge to zero. Then, based on the obtained data-driven model, the ADP method is utilized to design the approximate optimal tracking controller, which consists of the steady-state controller and the optimal feedback controller. Further, a robustifying term is developed to compensate for the NN approximation errors introduced by implementing the ADP method. Based on Lyapunov approach, stability analysis of the closed-loop system is performed to show that the proposed controller guarantees the system state asymptotically tracking the desired trajectory. Additionally, the obtained control input is proven to be close to the optimal control input within a small bound. Finally, two numerical examples are used to demonstrate the effectiveness of the proposed control scheme.
530 citations
TL;DR: The quality of PIs produced by the combiners is dramatically better than the quality ofPIs obtained from each individual method and a new method for generating combined PIs using the traditional PIs is proposed.
Abstract: This paper evaluates the four leading techniques proposed in the literature for construction of prediction intervals (PIs) for neural network point forecasts. The delta, Bayesian, bootstrap, and mean-variance estimation (MVE) methods are reviewed and their performance for generating high-quality PIs is compared. PI-based measures are proposed and applied for the objective and quantitative assessment of each method's performance. A selection of 12 synthetic and real-world case studies is used to examine each method's performance for PI construction. The comparison is performed on the basis of the quality of generated PIs, the repeatability of the results, the computational requirements and the PIs variability with regard to the data uncertainty. The obtained results in this paper indicate that: 1) the delta and Bayesian methods are the best in terms of quality and repeatability, and 2) the MVE and bootstrap methods are the best in terms of low computational load and the width variability of PIs. This paper also introduces the concept of combinations of PIs, and proposes a new method for generating combined PIs using the traditional PIs. Genetic algorithm is applied for adjusting the combiner parameters through minimization of a PI-based cost function subject to two sets of restrictions. It is shown that the quality of PIs produced by the combiners is dramatically better than the quality of PIs obtained from each individual method.
481 citations
TL;DR: A data-driven model-free adaptive control approach based on a new dynamic linearization technique (DLT) with a novel concept called pseudo-partial derivative for a class of general multiple-input and multiple-output nonlinear discrete-time systems.
Abstract: In this paper, a data-driven model-free adaptive control (MFAC) approach is proposed based on a new dynamic linearization technique (DLT) with a novel concept called pseudo-partial derivative for a class of general multiple-input and multiple-output nonlinear discrete-time systems. The DLT includes compact form dynamic linearization, partial form dynamic linearization, and full form dynamic linearization. The main feature of the approach is that the controller design depends only on the measured input/output data of the controlled plant. Analysis and extensive simulations have shown that MFAC guarantees the bounded-input bounded-output stability and the tracking error convergence.
477 citations
TL;DR: An improved version of the TBSVM is proposed, named twin bounded support vector machines (TBSVM), based on TWSVM, that the structural risk minimization principle is implemented by introducing the regularization term.
Abstract: For classification problems, the generalized eigenvalue proximal support vector machine (GEPSVM) and twin support vector machine (TWSVM) are regarded as milestones in the development of the powerful SVMs, as they use the nonparallel hyperplane classifiers. In this brief, we propose an improved version, named twin bounded support vector machines (TBSVM), based on TWSVM. The significant advantage of our TBSVM over TWSVM is that the structural risk minimization principle is implemented by introducing the regularization term. This embodies the marrow of statistical learning theory, so this modification can improve the performance of classification. In addition, the successive overrelaxation technique is used to solve the optimization problems to speed up the training procedure. Experimental results show the effectiveness of our method in both computation time and classification accuracy, and therefore confirm the above conclusion further.
476 citations
TL;DR: This brief investigates globally exponential synchronization for linearly coupled neural networks (NNs) with time-varying delay and impulsive disturbances with results that extend and improve upon earlier work.
Abstract: This brief investigates globally exponential synchronization for linearly coupled neural networks (NNs) with time-varying delay and impulsive disturbances. Since the impulsive effects discussed in this brief are regarded as disturbances, the impulses should not happen too frequently. The concept of average impulsive interval is used to formalize this phenomenon. By referring to an impulsive delay differential inequality, we investigate the globally exponential synchronization of linearly coupled NNs with impulsive disturbances. The derived sufficient condition is closely related with the time delay, impulse strengths, average impulsive interval, and coupling structure of the systems. The obtained criterion is given in terms of an algebraic inequality which is easy to be verified, and hence our result is valid for large-scale systems. The results extend and improve upon earlier work. As a numerical example, a small-world network composing of impulsive coupled chaotic delayed NN nodes is given to illustrate our theoretical result.
358 citations
TL;DR: In this paper, passivity analysis is conducted for discrete-time stochastic neural networks with both Markovian jumping parameters and mixed time delays by introducing a Lyapunov functional that accounts for the Mixed time delays.
Abstract: In this paper, passivity analysis is conducted for discrete-time stochastic neural networks with both Markovian jumping parameters and mixed time delays. The mixed time delays consist of both discrete and distributed delays. The Markov chain in the underlying neural networks is finite piecewise homogeneous. By introducing a Lyapunov functional that accounts for the mixed time delays, a delay-dependent passivity condition is derived in terms of the linear matrix inequality approach. The case of Markov chain with partially unknown transition probabilities is also considered. All the results presented depend upon not only discrete delay but also distributed delay. A numerical example is included to demonstrate the effectiveness of the proposed methods.
355 citations
TL;DR: By using Lyapunov analysis, it is proven that all the signals in the closed-loop system is the semi-globally uniformly ultimately bounded and the output errors converge to a compact set.
Abstract: This brief studies an adaptive neural output feedback tracking control of uncertain nonlinear multi-input-multi-output (MIMO) systems in the discrete-time form. The considered MIMO systems are composed of n subsystems with the couplings of inputs and states among subsystems. In order to solve the noncausal problem and decouple the couplings, it needs to transform the systems into a predictor form. The higher order neural networks are utilized to approximate the desired controllers. By using Lyapunov analysis, it is proven that all the signals in the closed-loop system is the semi-globally uniformly ultimately bounded and the output errors converge to a compact set. In contrast to the existing results, the advantage of the scheme is that the number of the adjustable parameters is highly reduced. The effectiveness of the scheme is verified by a simulation example.
TL;DR: This paper proposes the spectral embedded clustering (SEC) framework, in which a linearity regularization is explicitly added into the objective function of SC methods, and presents a new Laplacian matrix constructed from a local regression of each pattern to capture both local and global discriminative information for clustering.
Abstract: Spectral clustering (SC) methods have been successfully applied to many real-world applications. The success of these SC methods is largely based on the manifold assumption, namely, that two nearby data points in the high-density region of a low-dimensional data manifold have the same cluster label. However, such an assumption might not always hold on high-dimensional data. When the data do not exhibit a clear low-dimensional manifold structure (e.g., high-dimensional and sparse data), the clustering performance of SC will be degraded and become even worse than K -means clustering. In this paper, motivated by the observation that the true cluster assignment matrix for high-dimensional data can be always embedded in a linear space spanned by the data, we propose the spectral embedded clustering (SEC) framework, in which a linearity regularization is explicitly added into the objective function of SC methods. More importantly, the proposed SEC framework can naturally deal with out-of-sample data. We also present a new Laplacian matrix constructed from a local regression of each pattern and incorporate it into our SEC framework to capture both local and global discriminative information for clustering. Comprehensive experiments on eight real-world high-dimensional datasets demonstrate the effectiveness and advantages of our SEC framework over existing SC methods and K-means-based clustering methods. Our SEC framework significantly outperforms SC using the Nystrom algorithm on unseen data.
TL;DR: A no-reference image quality assessment (QA) algorithm that deploys a general regression neural network (GRNN) trained on and successfully assesses image quality, relative to human subjectivity, across a range of distortion types.
Abstract: We develop a no-reference image quality assessment (QA) algorithm that deploys a general regression neural network (GRNN). The new algorithm is trained on and successfully assesses image quality, relative to human subjectivity, across a range of distortion types. The features deployed for QA include the mean value of phase congruency image, the entropy of phase congruency image, the entropy of the distorted image, and the gradient of the distorted image. Image quality estimation is accomplished by approximating the functional relationship between these features and subjective mean opinion scores using a GRNN. Our experimental results show that the new method accords closely with human subjective judgment.
TL;DR: By applying the adaptive approach on the diagonal submatrices of the asymmetric coupling matrix, the corresponding cluster synchronization result is obtained by adding some simple intermittent pinning controls.
Abstract: In this paper, we investigate the cluster synchronization problem for linearly coupled networks, which can be recurrently connected neural networks, cellular neural networks, Hodgkin-Huxley models, Lorenz chaotic oscillators, etc., by adding some simple intermittent pinning controls. We assume the nodes in the network to be identical and the coupling matrix to be asymmetric. Some sufficient conditions to guarantee global cluster synchronization are presented. Furthermore, a centralized adaptive intermittent control is introduced and theoretical analysis is provided. Then, by applying the adaptive approach on the diagonal submatrices of the asymmetric coupling matrix, we also get the corresponding cluster synchronization result. Finally, numerical simulations are given to verify the theoretical results.
TL;DR: It is proved that the two proposed control approaches can guarantee that all the signals of the closed-loop system are semi-globally uniformly ultimately bounded, and the observer errors and the tracking errors converge to a small neighborhood of the origin.
Abstract: In this paper, two adaptive neural network (NN) decentralized output feedback control approaches are proposed for a class of uncertain nonlinear large-scale systems with immeasurable states and unknown time delays. Using NNs to approximate the unknown nonlinear functions, an NN state observer is designed to estimate the immeasurable states. By combining the adaptive backstepping technique with decentralized control design principle, an adaptive NN decentralized output feedback control approach is developed. In order to overcome the problem of “explosion of complexity” inherent in the proposed control approach, the dynamic surface control (DSC) technique is introduced into the first adaptive NN decentralized control scheme, and a simplified adaptive NN decentralized output feedback DSC approach is developed. It is proved that the two proposed control approaches can guarantee that all the signals of the closed-loop system are semi-globally uniformly ultimately bounded, and the observer errors and the tracking errors converge to a small neighborhood of the origin. Simulation results are provided to show the effectiveness of the proposed approaches.
TL;DR: This paper studies the finite-horizon optimal control problem for discrete-time nonlinear systems using the adaptive dynamic programming (ADP) approach and uses an iterative ADP algorithm to obtain the optimal control law.
Abstract: In this paper, we study the finite-horizon optimal control problem for discrete-time nonlinear systems using the adaptive dynamic programming (ADP) approach. The idea is to use an iterative ADP algorithm to obtain the optimal control law which makes the performance index function close to the greatest lower bound of all performance indices within an -error bound. The optimal number of control steps can also be obtained by the proposed ADP algorithms. A convergence analysis of the proposed ADP algorithms in terms of performance index function and control policy is made. In order to facilitate the implementation of the iterative ADP algorithms, neural networks are used for approximating the performance index function, computing the optimal control policy, and modeling the nonlinear system. Finally, two simulation examples are employed to illustrate the applicability of the proposed method.
TL;DR: In this paper, new synchronization and state estimation problems are considered for an array of coupled discrete time-varying stochastic complex networks over a finite horizon with a novel concept of bounded H∞ synchronization.
Abstract: In this paper, new synchronization and state estimation problems are considered for an array of coupled discrete time-varying stochastic complex networks over a finite horizon. A novel concept of bounded H∞ synchronization is proposed to handle the time-varying nature of the complex networks. Such a concept captures the transient behavior of the time-varying complex network over a finite horizon, where the degree of bounded synchronization is quantified in terms of the H∞-norm. A general sector-like nonlinear function is employed to describe the nonlinearities existing in the network. By utilizing a timevarying real-valued function and the Kronecker product, criteria are established that ensure the bounded H∞ synchronization in terms of a set of recursive linear matrix inequalities (RLMIs), where the RLMIs can be computed recursively by employing available MATLAB toolboxes. The bounded H∞ state estimation problem is then studied for the same complex network, where the purpose is to design a state estimator to estimate the network states through available output measurements such that, over a finite horizon, the dynamics of the estimation error is guaranteed to be bounded with a given disturbance attenuation level. Again, an RLMI approach is developed for the state estimation problem. Finally, two simulation examples are exploited to show the effectiveness of the results derived in this paper.
TL;DR: It is theoretically proven that some novel delay-independent and delay-dependent stability criteria are more effective than some existing ones either for SNNs or for LFNNs, which is confirmed by some numerical examples.
Abstract: This paper is concerned with global asymptotic stability for a class of generalized neural networks (NNs) with interval time-varying delays, which include two classes of fundamental NNs, i.e., static neural networks (SNNs) and local field neural networks (LFNNs), as their special cases. Some novel delay-independent and delay-dependent stability criteria are derived. These stability criteria are applicable not only to SNNs but also to LFNNs. It is theoretically proven that these stability criteria are more effective than some existing ones either for SNNs or for LFNNs, which is confirmed by some numerical examples.
TL;DR: An adaptive output feedback controller is constructively developed for uncertain nonlinear single-input-single-output systems with partial unmeasured states by employing radial basis function neural networks and incorporating the ROO into a new backstepping design.
Abstract: An adaptive output feedback control is studied for uncertain nonlinear single-input-single-output systems with partial unmeasured states. In the scheme, a reduced-order observer (ROO) is designed to estimate those unmeasured states. By employing radial basis function neural networks and incorporating the ROO into a new backstepping design, an adaptive output feedback controller is constructively developed. A prominent advantage is its ability to balance the control action between the state feedback and the output feedback. In addition, the scheme can be still implemented when all the states are not available. The stability of the closed-loop system is guaranteed in the sense that all the signals are semiglobal uniformly ultimately bounded and the system output tracks the reference signal to a bounded compact set. A simulation example is given to validate the effectiveness of the proposed scheme.
TL;DR: By introducing an integral partitioning technique, two new forms of Lyapunov-Krasovskii functionals are constructed, and improved distributed delay-dependent stability conditions are established in terms of linear matrix inequalities.
Abstract: In this brief, the problems of delay-dependent stability analysis and strict (Q,S,ℜ)-α-dissipativity analysis are investigated for cellular neural networks (CNNs) with distributed delay. First, by introducing an integral partitioning technique, two new forms of Lyapunov-Krasovskii functionals are constructed, and improved distributed delay-dependent stability conditions are established in terms of linear matrix inequalities. Based on this criterion, a new sufficient delay and α-dependent condition is given to guarantee that the CNNs with distributed delay are strictly (Q,S,ℜ)-α-dissipative. The results developed in this brief can tolerate larger allowable delay than existing ones in the literature, which is demonstrated by several examples.
TL;DR: It is proved that the equilibrium point set of the proposed neural network is equivalent to the optimal solution of the original optimization problem by using the Lagrangian saddle-point theorem.
Abstract: A recurrent neural network is proposed for solving the non-smooth convex optimization problem with the convex inequality and linear equality constraints. Since the objective function and inequality constraints may not be smooth, the Clarke's generalized gradients of the objective function and inequality constraints are employed to describe the dynamics of the proposed neural network. It is proved that the equilibrium point set of the proposed neural network is equivalent to the optimal solution of the original optimization problem by using the Lagrangian saddle-point theorem. Under weak conditions, the proposed neural network is proved to be stable, and the state of the neural network is convergent to one of its equilibrium points. Compared with the existing neural network models for non-smooth optimization problems, the proposed neural network can deal with a larger class of constraints and is not based on the penalty method. Finally, the proposed neural network is used to solve the identification problem of genetic regulatory networks, which can be transformed into a non-smooth convex optimization problem. The simulation results show the satisfactory identification accuracy, which demonstrates the effectiveness and efficiency of the proposed approach.
TL;DR: This paper uses an isolated word recognition task with babble noise to evaluate the performance of the photonic reservoirs with respect to traditional software reservoir implementations, and shows that the use of coherent light in a well-tuned reservoir architecture offers significant performance benefits.
Abstract: Reservoir computing (RC), a computational paradigm inspired on neural systems, has become increasingly popular in recent years for solving a variety of complex recognition and classification problems. Thus far, most implementations have been software-based, limiting their speed and power efficiency. Integrated photonics offers the potential for a fast, power efficient and massively parallel hardware implementation. We have previously proposed a network of coupled semiconductor optical amplifiers as an interesting test case for such a hardware implementation. In this paper, we investigate the important design parameters and the consequences of process variations through simulations. We use an isolated word recognition task with babble noise to evaluate the performance of the photonic reservoirs with respect to traditional software reservoir implementations, which are based on leaky hyperbolic tangent functions. Our results show that the use of coherent light in a well-tuned reservoir architecture offers significant performance benefits. The most important design parameters are the delay and the phase shift in the system's physical connections. With optimized values for these parameters, coherent semiconductor optical amplifier (SOA) reservoirs can achieve better results than traditional simulated reservoirs. We also show that process variations hardly degrade the performance, but amplifier noise can be detrimental. This effect must therefore be taken into account when designing SOA-based RC implementations.
TL;DR: A novel framework using a Bayesian approach for content-based phishing web page detection is presented, which takes into account textual and visual contents to measure the similarity between the protected web page and suspicious web pages.
Abstract: A novel framework using a Bayesian approach for content-based phishing web page detection is presented. Our model takes into account textual and visual contents to measure the similarity between the protected web page and suspicious web pages. A text classifier, an image classifier, and an algorithm fusing the results from classifiers are introduced. An outstanding feature of this paper is the exploration of a Bayesian model to estimate the matching threshold. This is required in the classifier for determining the class of the web page and identifying whether the web page is phishing or not. In the text classifier, the naive Bayes rule is used to calculate the probability that a web page is phishing. In the image classifier, the earth mover's distance is employed to measure the visual similarity, and our Bayesian model is designed to determine the threshold. In the data fusion algorithm, the Bayes theory is used to synthesize the classification results from textual and visual content. The effectiveness of our proposed approach was examined in a large-scale dataset collected from real phishing cases. Experimental results demonstrated that the text classifier and the image classifier we designed deliver promising results, the fusion algorithm outperforms either of the individual classifiers, and our model can be adapted to different phishing cases.
TL;DR: An improved LDA framework is proposed, the local LDA (LLDA), which can perform well without needing to satisfy the above two assumptions, and can effectively capture the local structure of samples.
Abstract: The linear discriminant analysis (LDA) is a very popular linear feature extraction approach. The algorithms of LDA usually perform well under the following two assumptions. The first assumption is that the global data structure is consistent with the local data structure. The second assumption is that the input data classes are Gaussian distributions. However, in real-world applications, these assumptions are not always satisfied. In this paper, we propose an improved LDA framework, the local LDA (LLDA), which can perform well without needing to satisfy the above two assumptions. Our LLDA framework can effectively capture the local structure of samples. According to different types of local data structure, our LLDA framework incorporates several different forms of linear feature extraction approaches, such as the classical LDA and principal component analysis. The proposed framework includes two LLDA algorithms: a vector-based LLDA algorithm and a matrix-based LLDA (MLLDA) algorithm. MLLDA is directly applicable to image recognition, such as face recognition. Our algorithms need to train only a small portion of the whole training set before testing a sample. They are suitable for learning large-scale databases especially when the input data dimensions are very high and can achieve high classification accuracy. Extensive experiments show that the proposed algorithms can obtain good classification results.
TL;DR: This paper focuses on symmetric NMF (SNMF), which is a special case of NMF decomposition, and proposes another two fast parallel methods: α-SNMF and β -SNMF algorithms, which are applied to probabilistic clustering.
Abstract: Nonnegative matrix factorization (NMF) is an unsupervised learning method useful in various applications including image processing and semantic analysis of documents. This paper focuses on symmetric NMF (SNMF), which is a special case of NMF decomposition. Three parallel multiplicative update algorithms using level 3 basic linear algebra subprograms directly are developed for this problem. First, by minimizing the Euclidean distance, a multiplicative update algorithm is proposed, and its convergence under mild conditions is proved. Based on it, we further propose another two fast parallel methods: α-SNMF and β -SNMF algorithms. All of them are easy to implement. These algorithms are applied to probabilistic clustering. We demonstrate their effectiveness for facial image clustering, document categorization, and pattern clustering in gene expression.
TL;DR: A fast gradient descent (FGD) is proposed to overcome the problem of slow multiplicative update rule in NPAF and uses the Newton method to search the optimal step size, and thus converges faster than MUR.
Abstract: In this paper, we present a non-negative patch alignment framework (NPAF) to unify popular non-negative matrix factorization (NMF) related dimension reduction algorithms. It offers a new viewpoint to better understand the common property of different NMF algorithms. Although multiplicative update rule (MUR) can solve NPAF and is easy to implement, it converges slowly. Thus, we propose a fast gradient descent (FGD) to overcome the aforementioned problem. FGD uses the Newton method to search the optimal step size, and thus converges faster than MUR. Experiments on synthetic and real-world datasets confirm the efficiency of FGD compared with MUR for optimizing NPAF. Based on NPAF, we develop non-negative discriminative locality alignment (NDLA). Experiments on face image and handwritten datasets suggest the effectiveness of NDLA in classification tasks and its robustness to image occlusions, compared with representative NMF-related dimension reduction algorithms.
TL;DR: This paper proposes a general adaptive incremental learning framework named ADAIN that is capable of learning from continuous raw data, accumulating experience over time, and using such knowledge to improve future learning and prediction performance.
Abstract: Recent years have witnessed an incredibly increasing interest in the topic of incremental learning Unlike conventional machine learning situations, data flow targeted by incremental learning becomes available continuously over time Accordingly, it is desirable to be able to abandon the traditional assumption of the availability of representative training data during the training period to develop decision boundaries Under scenarios of continuous data flow, the challenge is how to transform the vast amount of stream raw data into information and knowledge representation, and accumulate experience over time to support future decision-making process In this paper, we propose a general adaptive incremental learning framework named ADAIN that is capable of learning from continuous raw data, accumulating experience over time, and using such knowledge to improve future learning and prediction performance Detailed system level architecture and design strategies are presented in this paper Simulation results over several real-world data sets are used to validate the effectiveness of this method
TL;DR: It is shown that the quaternion-valued exponential function is locally analytic, and, since local analyticity extends to polynomials, products, and ratios, it is showed that a class of transcendental nonlinear functions can serve as activation functions in nonlinear and neural adaptive models.
Abstract: A class of nonlinear quaternion-valued adaptive filtering algorithms is proposed based on locally analytic nonlinear activation functions. To circumvent the stringent standard analyticity conditions which are prohibitive to the development of nonlinear adaptive quaternion-valued estimation models, we use the fact that stochastic gradient learning algorithms require only local analyticity at the operating point in the estimation space. It is shown that the quaternion-valued exponential function is locally analytic, and, since local analyticity extends to polynomials, products, and ratios, we show that a class of transcendental nonlinear functions can serve as activation functions in nonlinear and neural adaptive models. This provides a unifying framework for the derivation of gradient-based learning algorithms in the quaternion domain, and the derived algorithms are shown to have the same generic form as their real- and complex-valued counterparts. To make such models second-order optimal for the generality of quaternion signals (both circular and noncircular), we use recent developments in augmented quaternion statistics to introduce widely linear versions of the proposed nonlinear adaptive quaternion valued filters. This allows full exploitation of second-order information in the data, contained both in the covariance and pseudocovariances to cater rigorously for second-order noncircularity (improperness), and the corresponding power mismatch in the signal components. Simulations over a range of circular and noncircular synthetic processes and a real world 3-D noncircular wind signal support the approach.
TL;DR: A novel heuristic dynamic programming (HDP) iteration algorithm is proposed to solve the optimal tracking control problem for a class of nonlinear discrete-time systems with time delays.
Abstract: In this paper, a novel heuristic dynamic programming (HDP) iteration algorithm is proposed to solve the optimal tracking control problem for a class of nonlinear discrete-time systems with time delays. The novel algorithm contains state updating, control policy iteration, and performance index iteration. To get the optimal states, the states are also updated. Furthermore, the “backward iteration” is applied to state updating. Two neural networks are used to approximate the performance index function and compute the optimal control policy for facilitating the implementation of HDP iteration algorithm. At last, we present two examples to demonstrate the effectiveness of the proposed HDP iteration algorithm.
TL;DR: The experimental segmentation results of the gray natural images from the Berkeley Segmentation Dataset, rather than synthetic images, prove the validity and efficiency of the proposed automatic parameter setting method of SPCNN.
Abstract: An automatic parameter setting method of a simplified pulse coupled neural network (SPCNN) is proposed here. Our method successfully determines all the adjustable parameters in SPCNN and does not need any training and trials as required by previous methods. In order to achieve this goal, we try to derive the general formulae of dynamic threshold and internal activity of the SPCNN according to the dynamic properties of neurons, and then deduce the sub-intensity range expression of each segment based on the general formulae. Besides, we extract information from an input image, such as the standard deviation and the optimal histogram threshold of the image, and attempt to build a direct relation between the dynamic properties of neurons and the static properties of each input image. Finally, the experimental segmentation results of the gray natural images from the Berkeley Segmentation Dataset, rather than synthetic images, prove the validity and efficiency of our proposed automatic parameter setting method of SPCNN.