Showing papers in "IEEE Transactions on Neural Networks in 2017"
TL;DR: This paper presents the first large-scale analysis of eight LSTM variants on three representative tasks: speech recognition, handwriting recognition, and polyphonic music modeling, and observes that the studied hyperparameters are virtually independent and derive guidelines for their efficient adjustment.
Abstract: Several variants of the long short-term memory (LSTM) architecture for recurrent neural networks have been proposed since its inception in 1995. In recent years, these networks have become the state-of-the-art models for a variety of machine learning problems. This has led to a renewed interest in understanding the role and utility of various computational components of typical LSTM variants. In this paper, we present the first large-scale analysis of eight LSTM variants on three representative tasks: speech recognition, handwriting recognition, and polyphonic music modeling. The hyperparameters of all LSTM variants for each task were optimized separately using random search, and their importance was assessed using the powerful functional ANalysis Of VAriance framework. In total, we summarize the results of 5400 experimental runs ( $\approx 15$ years of CPU time), which makes our study the largest of its kind on LSTM networks. Our results show that none of the variants can improve upon the standard LSTM architecture significantly, and demonstrate the forget gate and the output activation function to be its most critical components. We further observe that the studied hyperparameters are virtually independent and derive guidelines for their efficient adjustment.
4,746 citations
TL;DR: In this article, a general methodology based on region perturbation for evaluating ordered collections of pixels such as heatmaps is presented, and the authors compare heatmaps computed by three different methods on the SUN397, ILSVRC2012, and MIT Places data sets.
Abstract: Deep neural networks (DNNs) have demonstrated impressive performance in complex machine learning tasks such as image classification or speech recognition. However, due to their multilayer nonlinear structure, they are not transparent, i.e., it is hard to grasp what makes them arrive at a particular classification or recognition decision, given a new unseen data sample. Recently, several approaches have been proposed enabling one to understand and interpret the reasoning embodied in a DNN for a single test image. These methods quantify the “importance” of individual pixels with respect to the classification decision and allow a visualization in terms of a heatmap in pixel/input space. While the usefulness of heatmaps can be judged subjectively by a human, an objective quality measure is missing. In this paper, we present a general methodology based on region perturbation for evaluating ordered collections of pixels such as heatmaps. We compare heatmaps computed by three different methods on the SUN397, ILSVRC2012, and MIT Places data sets. Our main result is that the recently proposed layer-wise relevance propagation algorithm qualitatively and quantitatively provides a better explanation of what made a DNN arrive at a particular classification decision than the sensitivity-based approach or the deconvolution method. We provide theoretical arguments to explain this result and discuss its practical implications. Finally, we investigate the use of heatmaps for unsupervised assessment of the neural network performance.
866 citations
TL;DR: A multiobjective deep belief networks ensemble (MODBNE) method that employs a multiobjectives evolutionary algorithm integrated with the traditional DBN training technique to evolve multiple DBNs simultaneously subject to accuracy and diversity as two conflicting objectives is proposed.
Abstract: In numerous industrial applications where safety, efficiency, and reliability are among primary concerns, condition-based maintenance (CBM) is often the most effective and reliable maintenance policy. Prognostics, as one of the key enablers of CBM, involves the core task of estimating the remaining useful life (RUL) of the system. Neural networks-based approaches have produced promising results on RUL estimation, although their performances are influenced by handcrafted features and manually specified parameters. In this paper, we propose a multiobjective deep belief networks ensemble (MODBNE) method. MODBNE employs a multiobjective evolutionary algorithm integrated with the traditional DBN training technique to evolve multiple DBNs simultaneously subject to accuracy and diversity as two conflicting objectives. The eventually evolved DBNs are combined to establish an ensemble model used for RUL estimation, where combination weights are optimized via a single-objective differential evolution algorithm using a task-oriented objective function. We evaluate the proposed method on several prognostic benchmarking data sets and also compare it with some existing approaches. Experimental results demonstrate the superiority of our proposed method.
569 citations
TL;DR: This work introduces a recurrent deep neural network for real-time financial signal representation and trading and proposes a task-aware backpropagation through time method to cope with the gradient vanishing issue in deep training.
Abstract: Can we train the computer to beat experienced traders for financial assert trading? In this paper, we try to address this challenge by introducing a recurrent deep neural network (NN) for real-time financial signal representation and trading. Our model is inspired by two biological-related learning concepts of deep learning (DL) and reinforcement learning (RL). In the framework, the DL part automatically senses the dynamic market condition for informative feature learning. Then, the RL module interacts with deep representations and makes trading decisions to accumulate the ultimate rewards in an unknown environment. The learning system is implemented in a complex NN that exhibits both the deep and recurrent structures. Hence, we propose a task-aware backpropagation through time method to cope with the gradient vanishing issue in deep training. The robustness of the neural system is verified on both the stock and the commodity future markets under broad testing conditions.
522 citations
TL;DR: A robust regularization path algorithm is proposed for LaTeX vector classification, based on lower upper decomposition with partial pivoting, that can avoid the exceptions completely, handle the singularities in the key matrix, and fit the entire solution path in a finite number of steps.
Abstract: The $
u $ -support vector classification has the advantage of using a regularization parameter $
u $ to control the number of support vectors and margin errors. Recently, a regularization path algorithm for $
u $ -support vector classification ( $
u $ -SvcPath) suffers exceptions and singularities in some special cases. In this brief, we first present a new equivalent dual formulation for $
u $ -SVC and, then, propose a robust $
u $ -SvcPath, based on lower upper decomposition with partial pivoting. Theoretical analysis and experimental results verify that our proposed robust regularization path algorithm can avoid the exceptions completely, handle the singularities in the key matrix, and fit the entire solution path in a finite number of steps. Experimental results also show that our proposed algorithm fits the entire solution path with fewer steps and less running time than original one does.
451 citations
TL;DR: The aim of this paper is to contribute with a Markov switching estimator design method, which ensures that the resulting error system is extended stochastically dissipative, in the simultaneous presences of packet dropouts and signal quantization stemmed from unreliable communication links.
Abstract: This paper is concerned with the problem of extended dissipativity-based state estimation for discrete-time Markov jump neural networks (NNs), where the variation of the piecewise time-varying transition probabilities of Markov chain is subject to a set of switching signals satisfying an average dwell-time property. The communication links between the NNs and the estimator are assumed to be imperfect, where the phenomena of signal quantization and data packet dropouts occur simultaneously. The aim of this paper is to contribute with a Markov switching estimator design method, which ensures that the resulting error system is extended stochastically dissipative, in the simultaneous presences of packet dropouts and signal quantization stemmed from unreliable communication links. Sufficient conditions for the solvability of such a problem are established. Based on the derived conditions, an explicit expression of the desired Markov switching estimator is presented. Finally, two illustrated examples are given to show the effectiveness of the proposed design method.
434 citations
TL;DR: A multitask deep convolutional network is developed, which simultaneously detects the presence of the target and the geometric attributes of thetarget with respect to the region of interest and a recurrent neuron layer is adopted for structured visual detection.
Abstract: Hierarchical neural networks have been shown to be effective in learning representative image features and recognizing object classes. However, most existing networks combine the low/middle level cues for classification without accounting for any spatial structures. For applications such as understanding a scene, how the visual cues are spatially distributed in an image becomes essential for successful analysis. This paper extends the framework of deep neural networks by accounting for the structural cues in the visual signals. In particular, two kinds of neural networks have been proposed. First, we develop a multitask deep convolutional network, which simultaneously detects the presence of the target and the geometric attributes (location and orientation) of the target with respect to the region of interest. Second, a recurrent neuron layer is adopted for structured visual detection. The recurrent neurons can deal with the spatial distribution of visible cues belonging to an object whose shape or structure is difficult to explicitly define. Both the networks are demonstrated by the practical task of detecting lane boundaries in traffic scenes. The multitask convolutional neural network provides auxiliary geometric information to help the subsequent modeling of the given lane structures. The recurrent neural network automatically detects lane boundaries, including those areas containing no marks, without any explicit prior knowledge or secondary modeling.
385 citations
TL;DR: This paper uses two finite mixture models to capture the structural information of the data from binary classification and proposes a structural MPM, which can be interpreted as a large margin classifier and can be transformed to support vector machine and maxi–min margin machine under certain special conditions.
Abstract: Minimax probability machine (MPM) is an interesting discriminative classifier based on generative prior knowledge. It can directly estimate the probabilistic accuracy bound by minimizing the maximum probability of misclassification. The structural information of data is an effective way to represent prior knowledge, and has been found to be vital for designing classifiers in real-world problems. However, MPM only considers the prior probability distribution of each class with a given mean and covariance matrix, which does not efficiently exploit the structural information of data. In this paper, we use two finite mixture models to capture the structural information of the data from binary classification. For each subdistribution in a finite mixture model, only its mean and covariance matrix are assumed to be known. Based on the finite mixture models, we propose a structural MPM (SMPM). SMPM can be solved effectively by a sequence of the second-order cone programming problems. Moreover, we extend a linear model of SMPM to a nonlinear model by exploiting kernelization techniques. We also show that the SMPM can be interpreted as a large margin classifier and can be transformed to support vector machine and maxi–min margin machine under certain special conditions. Experimental results on both synthetic and real-world data sets demonstrate the effectiveness of SMPM.
337 citations
TL;DR: This paper proposes a new unsupervised spectral feature selection model by embedding a graph regularizer into the framework of joint sparse regression for preserving the local structures of data by proposing a novel joint graph sparse coding (JGSC) model.
Abstract: In this paper, we propose a new unsupervised spectral feature selection model by embedding a graph regularizer into the framework of joint sparse regression for preserving the local structures of data. To do this, we first extract the bases of training data by previous dictionary learning methods and, then, map original data into the basis space to generate their new representations, by proposing a novel joint graph sparse coding (JGSC) model. In JGSC, we first formulate its objective function by simultaneously taking subspace learning and joint sparse regression into account, then, design a new optimization solution to solve the resulting objective function, and further prove the convergence of the proposed solution. Furthermore, we extend JGSC to a robust JGSC (RJGSC) via replacing the least square loss function with a robust loss function, for achieving the same goals and also avoiding the impact of outliers. Finally, experimental results on real data sets showed that both JGSC and RJGSC outperformed the state-of-the-art algorithms in terms of ${k}$ -nearest neighbor classification performance.
321 citations
TL;DR: A two-phase recommendation process is proposed to utilize deep learning to determinate the initialization in MF for trust-aware social recommendations and to differentiate the community effect in user’s trusted friendships.
Abstract: With the emergence of online social networks, the social network-based recommendation approach is popularly used. The major benefit of this approach is the ability of dealing with the problems with cold-start users. In addition to social networks, user trust information also plays an important role to obtain reliable recommendations. Although matrix factorization (MF) becomes dominant in recommender systems, the recommendation largely relies on the initialization of the user and item latent feature vectors. Aiming at addressing these challenges, we develop a novel trust-based approach for recommendation in social networks. In particular, we attempt to leverage deep learning to determinate the initialization in MF for trust-aware social recommendations and to differentiate the community effect in user’s trusted friendships. A two-phase recommendation process is proposed to utilize deep learning in initialization and to synthesize the users’ interests and their trusted friends’ interests together with the impact of community effect for recommendations. We perform extensive experiments on real-world social network data to demonstrate the accuracy and effectiveness of our proposed approach in comparison with other state-of-the-art methods.
261 citations
TL;DR: This paper identifies two limitations of the existing RNN solutions for manipulator control, i.e., position error accumulation and the convex restriction on the projection set, and overcomes them by proposing two modified neural network models.
Abstract: Redundancy resolution is a critical problem in the control of robotic manipulators. Recurrent neural networks (RNNs), as inherently parallel processing models for time-sequence processing, are potentially applicable for the motion control of manipulators. However, the development of neural models for high-accuracy and real-time control is a challenging problem. This paper identifies two limitations of the existing RNN solutions for manipulator control, i.e., position error accumulation and the convex restriction on the projection set, and overcomes them by proposing two modified neural network models. Our method allows nonconvex sets for projection operations, and control error does not accumulate over time in the presence of noise. Unlike most works in which RNNs are used to process time sequences, the proposed approach is model-based and training-free, which makes it possible to achieve fast tracking of reference signals with superior robustness and accuracy. Theoretical analysis reveals the global stability of a system under the control of the proposed neural networks. Simulation results confirm the effectiveness of the proposed control method in both the position regulation and tracking control of redundant PUMA 560 manipulators.
TL;DR: A novel NN adaptive output-feedback FTC approach is developed that can guarantee that all signals in all subsystems are bounded, and the tracking errors for each subsystem converge to a small neighborhood of zero.
Abstract: The problem of active fault-tolerant control (FTC) is investigated for the large-scale nonlinear systems in nonstrict-feedback form. The nonstrict-feedback nonlinear systems considered in this paper consist of unstructured uncertainties, unmeasured states, unknown interconnected terms, and actuator faults (e.g., bias fault and gain fault). A state observer is designed to solve the unmeasurable state problem. Neural networks (NNs) are used to identify the unknown lumped nonlinear functions so that the problems of unstructured uncertainties and unknown interconnected terms can be solved. By combining the adaptive backstepping design principle with the combination Nussbaum gain function property, a novel NN adaptive output-feedback FTC approach is developed. The proposed FTC controller can guarantee that all signals in all subsystems are bounded, and the tracking errors for each subsystem converge to a small neighborhood of zero. Finally, numerical results of practical examples are presented to further demonstrate the effectiveness of the proposed control strategy.
TL;DR: This paper compares the differences and commonalities of these methods based on regression and regularization strategies, but also provides useful guidelines to practitioners working in related fields to guide them how to do feature selection.
Abstract: Feature selection (FS) is an important component of many pattern recognition tasks. In these tasks, one is often confronted with very high-dimensional data. FS algorithms are designed to identify the relevant feature subset from the original features, which can facilitate subsequent analysis, such as clustering and classification. Structured sparsity-inducing feature selection (SSFS) methods have been widely studied in the last few years, and a number of algorithms have been proposed. However, there is no comprehensive study concerning the connections between different SSFS methods, and how they have evolved. In this paper, we attempt to provide a survey on various SSFS methods, including their motivations and mathematical representations. We then explore the relationship among different formulations and propose a taxonomy to elucidate their evolution. We group the existing SSFS methods into two categories, i.e., vector-based feature selection (feature selection based on lasso) and matrix-based feature selection (feature selection based on ${l_{r,p}}$ -norm). Furthermore, FS has been combined with other machine learning algorithms for specific applications, such as multitask learning, multilabel learning, multiview learning, classification, and clustering. This paper not only compares the differences and commonalities of these methods based on regression and regularization strategies, but also provides useful guidelines to practitioners working in related fields to guide them how to do feature selection.
TL;DR: The adaptive supplementary control approach versus the traditional SMC in the cruising flight is verified, and three simulation studies are provided to illustrate the improved performance with the proposed approach.
Abstract: In this paper, we propose a data-driven supplementary control approach with adaptive learning capability for air-breathing hypersonic vehicle tracking control based on action-dependent heuristic dynamic programming (ADHDP). The control action is generated by the combination of sliding mode control (SMC) and the ADHDP controller to track the desired velocity and the desired altitude. In particular, the ADHDP controller observes the differences between the actual velocity/altitude and the desired velocity/altitude, and then provides a supplementary control action accordingly. The ADHDP controller does not rely on the accurate mathematical model function and is data driven. Meanwhile, it is capable to adjust its parameters online over time under various working conditions, which is very suitable for hypersonic vehicle system with parameter uncertainties and disturbances. We verify the adaptive supplementary control approach versus the traditional SMC in the cruising flight, and provide three simulation studies to illustrate the improved performance with the proposed approach.
TL;DR: This paper provides a tractable way to extend existing results on individual manipulator control using recurrent neural networks to the scenario with the coordination of multiple manipulators and proves global stability and solution optimality of the proposed neural networks.
Abstract: This paper considers cooperative kinematic control of multiple manipulators using distributed recurrent neural networks and provides a tractable way to extend existing results on individual manipulator control using recurrent neural networks to the scenario with the coordination of multiple manipulators. The problem is formulated as a constrained game, where energy consumptions for each manipulator, saturations of control input, and the topological constraints imposed by the communication graph are considered. An implicit form of the Nash equilibrium for the game is obtained by converting the problem into its dual space. Then, a distributed dynamic controller based on recurrent neural networks is devised to drive the system toward the desired Nash equilibrium to seek the optimal solution of the cooperative control. Global stability and solution optimality of the proposed neural networks are proved in the theory. Simulations demonstrate the effectiveness of the proposed method.
TL;DR: A new supervised classification algorithm, called neural dynamic classification (NDC), is presented with the goal of finding the optimum number of features required for accurate classification using the patented robust neural dynamic optimization model of Adeli and Park.
Abstract: The keys for the development of an effective classification algorithm are: 1) discovering feature spaces with large margins between clusters and close proximity of the classmates and 2) discovering the smallest number of the features to perform accurate classification. In this paper, a new supervised classification algorithm, called neural dynamic classification (NDC), is presented with the goal of: 1) discovering the most effective feature spaces and 2) finding the optimum number of features required for accurate classification using the patented robust neural dynamic optimization model of Adeli and Park. The new classification algorithm is compared with the probabilistic neural network (PNN), enhanced PNN (EPNN), and support vector machine using two sets of classification problems. The first set consists of five standard benchmark problems. The second set is a large benchmark problem called Mixed National Institute of Standards and Technology database of handwritten digits. In general, NDC yields the most accurate classification results followed by EPNN. A beauty of the new algorithm is the smoothness of convergence curves which is an indication of robustness and good performance of the algorithm. The main aim is to maximize the prediction accuracy.
TL;DR: This paper investigates the problem of exponential passive filtering for a class of stochastic neutral-type neural networks with both semi-Markovian jump parameters and mixed time delays by designing a Luenberger-type observer, and develops a convex optimization algorithm for the filter design.
Abstract: This paper investigates the problem of exponential passive filtering for a class of stochastic neutral-type neural networks with both semi-Markovian jump parameters and mixed time delays. Our aim is to estimate the states by designing a Luenberger-type observer, such that the filter error dynamics are mean-square exponentially stable with an expected decay rate and an attenuation level. Sufficient conditions for the existence of passive filters are obtained, and a convex optimization algorithm for the filter design is given. In addition, a cone complementarity linearization procedure is employed to cast the nonconvex feasibility problem into a sequential minimization problem, which can be readily solved by the existing optimization techniques. Numerical examples are given to demonstrate the effectiveness of the proposed techniques.
TL;DR: The long-short-term memory (LSTM) recurrent neural network is proposed to accomplish fault detection and identification tasks based on the commonly available measurement signals by considering the signals from multiple track circuits in a geographic area.
Abstract: Timely detection and identification of faults in railway track circuits are crucial for the safety and availability of railway networks. In this paper, the use of the long-short-term memory (LSTM) recurrent neural network is proposed to accomplish these tasks based on the commonly available measurement signals. By considering the signals from multiple track circuits in a geographic area, faults are diagnosed from their spatial and temporal dependences. A generative model is used to show that the LSTM network can learn these dependences directly from the data. The network correctly classifies 99.7% of the test input sequences, with no false positive fault detections. In addition, the t-Distributed Stochastic Neighbor Embedding (t-SNE) method is used to examine the resulting network, further showing that it has learned the relevant dependences in the data. Finally, we compare our LSTM network with a convolutional network trained on the same task. From this comparison, we conclude that the LSTM network architecture is better suited for the railway track circuit fault detection and identification tasks than the convolutional network.
TL;DR: This paper addresses the problem of state estimation for a class of discrete-time stochastic complex networks with a constrained and randomly varying coupling and uncertain measurements with the help of a Markov chain and a logarithmic quantizer.
Abstract: This paper addresses the problem of state estimation for a class of discrete-time stochastic complex networks with a constrained and randomly varying coupling and uncertain measurements. The randomly varying coupling is governed by a Markov chain, and the capacity constraint is handled by introducing a logarithmic quantizer. The uncertainty of measurements is modeled by a multiplicative noise. An asynchronous estimator is designed to overcome the difficulty that each node cannot access to the coupling information, and an augmented estimation error system is obtained using the Kronecker product. Sufficient conditions are established, which guarantee that the estimation error system is stochastically stable and achieves the strict ( $Q,S,R$ )- $\gamma $ -dissipativity. Then, the estimator gains are derived using the linear matrix inequality method. Finally, a numerical example is provided to illustrate the effectiveness of the proposed new design techniques.
TL;DR: An event-triggered state estimator is constructed and a sufficient condition is given under which the estimation error dynamics is exponentially ultimately bounded in the mean square, and the characterization of the desired estimator gain is designed in terms of the solution to a certain matrix inequality.
Abstract: In this paper, the event-triggered state estimation problem is investigated for a class of discrete-time multidelayed neural networks with stochastic parameters and incomplete measurements. In order to cater for more realistic transmission process of the neural signals, we make the first attempt to introduce a set of stochastic variables to characterize the random fluctuations of system parameters. In the addressed neural network model, the delays among the interconnections are allowed to be different, which are more general than those in the existing literature. The incomplete information under consideration includes randomly occurring sensor saturations and quantizations. For the purpose of energy saving, an event-triggered state estimator is constructed and a sufficient condition is given under which the estimation error dynamics is exponentially ultimately bounded in the mean square. It is worth noting that the ultimate boundedness of the error dynamics is explicitly estimated. The characterization of the desired estimator gain is designed in terms of the solution to a certain matrix inequality. Finally, a numerical simulation example is presented to illustrate the effectiveness of the proposed event-triggered state estimation scheme.
TL;DR: A novel model, stacked autoencoder Levenberg-Marquardt model, which is a type of deep architecture of neural network approach aiming to improve forecasting accuracy, and an optimized structure of the traffic flow forecasting model with a deep learning approach is presented.
Abstract: Forecasting accuracy is an important issue for successful intelligent traffic management, especially in the domain of traffic efficiency and congestion reduction. The dawning of the big data era brings opportunities to greatly improve prediction accuracy. In this paper, we propose a novel model, stacked autoencoder Levenberg-Marquardt model, which is a type of deep architecture of neural network approach aiming to improve forecasting accuracy. The proposed model is designed using the Taguchi method to develop an optimized structure and to learn traffic flow features through layer-by-layer feature granulation with a greedy layerwise unsupervised learning algorithm. It is applied to real-world data collected from the M6 freeway in the U.K. and is compared with three existing traffic predictors. To the best of our knowledge, this is the first time that an optimized structure of the traffic flow forecasting model with a deep learning approach is presented. The evaluation results demonstrate that the proposed model with an optimized structure has superior performance in traffic flow forecasting.
TL;DR: A novel semisupervised feature selection framework by mining correlations among multiple tasks and apply it to different multimedia applications is proposed, which outperforms the other state-of-the-art feature selection algorithms.
Abstract: In this paper, we propose a novel semisupervised feature selection framework by mining correlations among multiple tasks and apply it to different multimedia applications. Instead of independently computing the importance of features for each task, our algorithm leverages shared knowledge from multiple related tasks, thus improving the performance of feature selection. Note that the proposed algorithm is built upon an assumption that different tasks share some common structures. The proposed algorithm selects features in a batch mode, by which the correlations between various features are taken into consideration. Besides, considering the fact that labeling a large amount of training data in real world is both time-consuming and tedious, we adopt manifold learning, which exploits both labeled and unlabeled training data for a feature space analysis. Since the objective function is nonsmooth and difficult to solve, we propose an iteractive algorithm with fast convergence. Extensive experiments on different applications demonstrate that our algorithm outperforms the other state-of-the-art feature selection algorithms.
TL;DR: Several sufficient conditions are derived to guarantee the exponential synchronization of drive-response memristive neural networks with heterogeneous time-varying delays based on some novel Lyapunov functionals and interval matrix inequalities.
Abstract: This paper considers the global exponential synchronization of drive-response memristive neural networks (MNNs) with heterogeneous time-varying delays. Because the parameters of MNNs are state-dependent, the MNNs may exhibit unexpected parameter mismatch when different initial conditions are chosen. Therefore, traditional robust control scheme cannot guarantee the synchronization of MNNs. Under the framework of Filippov solution, the drive and response MNNs are first transformed into systems with interval parameters. Then suitable controllers are designed to overcome the problem of mismatched parameters and synchronize the coupled MNNs. Based on some novel Lyapunov functionals and interval matrix inequalities, several sufficient conditions are derived to guarantee the exponential synchronization. Moreover, adaptive control is also investigated for the exponential synchronization. Numerical simulations are provided to illustrate the effectiveness of the theoretical analysis.
TL;DR: The purpose of this paper is to derive consensus criteria, such that linear multi-agent systems with sampled-data and packet losses can reach consensus, by means of the Lyapunov function approach and the decomposition method and to reveal the intrinsic effect of communication topologies on consensus performance.
Abstract: In this paper, the consensus problem is studied for a class of multi-agent systems with sampled data and packet losses, where random and deterministic packet losses are considered, respectively. For random packet losses, a Bernoulli-distributed white sequence is used to describe packet dropouts among agents in a stochastic way. For deterministic packet losses, a switched system with stable and unstable subsystems is employed to model packet dropouts in a deterministic way. The purpose of this paper is to derive consensus criteria, such that linear multi-agent systems with sampled-data and packet losses can reach consensus. By means of the Lyapunov function approach and the decomposition method, the design problem of a distributed controller is solved in terms of convex optimization. The interplay among the allowable bound of the sampling interval, the probability of random packet losses, and the rate of deterministic packet losses are explicitly derived to characterize consensus conditions. The obtained criteria are closely related to the maximum eigenvalue of the Laplacian matrix versus the second minimum eigenvalue of the Laplacian matrix, which reveals the intrinsic effect of communication topologies on consensus performance. Finally, simulations are given to show the effectiveness of the proposed results.
TL;DR: This paper investigates the global Mittag-Leffler stabilization for a class of fractional-order memristive neural networks, and two types of control rules are designed for the stabilization of fractionalsized neural networks.
Abstract: According to conventional memristive neural network theories, neurodynamic properties are powerful tools for solving many problems in the areas of brain-like associative learning, dynamic information storage or retrieval, etc. However, as have often been noted in most fractional-order systems, system analysis approaches for integral-order systems could not be directly extended and applied to deal with fractional-order systems, and consequently, it raises difficult issues in analyzing and controlling the fractional-order memristive neural networks. By using the set-valued maps and fractional-order differential inclusions, then aided by a newly proposed fractional derivative inequality, this paper investigates the global Mittag–Leffler stabilization for a class of fractional-order memristive neural networks. Two types of control rules (i.e., state feedback stabilizing control and output feedback stabilizing control) are designed for the stabilization of fractional-order memristive neural networks, while a list of stabilization criteria is established. Finally, two numerical examples are given to show the effectiveness and characteristics of the obtained theoretical results.
TL;DR: This paper is concerned with the state estimation problem for a class of nonlinear dynamical networks with time-varying delays subject to the round-robin protocol, and designs an estimator, such that the estimation error is exponentially ultimately bounded with a certain asymptotic upper bound in mean squaresubject to the process noise and exogenous disturbance.
Abstract: This paper is concerned with the state estimation problem for a class of nonlinear dynamical networks with time-varying delays subject to the round-robin protocol. The communication between the state estimator and the nodes of the dynamical networks is implemented through a shared constrained network, in which only one node is allowed to send data at each time instant. The round-robin protocol is utilized to orchestrate the transmission order of nodes. By using a switch-based approach, the dynamics of the estimation error is modeled by a periodic parameter-switching system with time-varying delays. The purpose of the problem addressed is to design an estimator, such that the estimation error is exponentially ultimately bounded with a certain asymptotic upper bound in mean square subject to the process noise and exogenous disturbance. Furthermore, such a bound is subsequently minimized by the designed estimator parameters. A novel Lyapunov-like functional is employed to deal with the dynamics analysis issue of the estimation error. Sufficient conditions are established to guarantee the ultimate boundedness of the estimation error in mean square by applying the stochastic analysis approach. Then, the desired estimator gains are characterized by solving a convex problem. Finally, a numerical example is given to illustrate the effectiveness of the estimator design scheme.
TL;DR: This paper presents a collective neurodynamic approach with multiple interconnected recurrent neural networks (RNNs) for distributed constrained optimization, capable of solving more general distributed optimization problems.
Abstract: This paper presents a collective neurodynamic approach with multiple interconnected recurrent neural networks (RNNs) for distributed constrained optimization. The objective function of the distributed optimization problems to be solved is a sum of local convex objective functions, which may be nonsmooth. Subject to its local constraints, each local objective function is minimized individually by using an RNN, with consensus among others. In contrast to existing continuous-time distributed optimization methods, the proposed collective neurodynamic approach is capable of solving more general distributed optimization problems. Simulation results on three numerical examples are discussed to substantiate the effectiveness and characteristics of the proposed approach. In addition, an application to the optimal placement problem is delineated to demonstrate the viability of the approach.
TL;DR: This paper presents a predictor-based neural dynamic surface control design method for a class of uncertain nonlinear systems in a strict-feedback form where a predictor is proposed for every subsystem, and the prediction errors are employed to update the neural adaptation laws.
Abstract: This paper presents a predictor-based neural dynamic surface control (PNDSC) design method for a class of uncertain nonlinear systems in a strict-feedback form. In contrast to existing NDSC approaches where the tracking errors are commonly used to update neural network weights, a predictor is proposed for every subsystem, and the prediction errors are employed to update the neural adaptation laws. The proposed scheme enables smooth and fast identification of system dynamics without incurring high-frequency oscillations, which are unavoidable using classical NDSC methods. Furthermore, the result is extended to the PNDSC with observer feedback, and its robustness against measurement noise is analyzed. Numerical and experimental results are given to demonstrate the efficacy of the proposed PNDSC architecture.
TL;DR: A novel discriminative sparse representation method is proposed and its noticeable performance in image classification is demonstrated by the experimental results, and the proposed method outperforms the existing state-of-the-art sparse representation methods.
Abstract: Sparse representation has shown an attractive performance in a number of applications. However, the available sparse representation methods still suffer from some problems, and it is necessary to design more efficient methods. Particularly, to design a computationally inexpensive, easily solvable, and robust sparse representation method is a significant task. In this paper, we explore the issue of designing the simple, robust, and powerfully efficient sparse representation methods for image classification. The contributions of this paper are as follows. First, a novel discriminative sparse representation method is proposed and its noticeable performance in image classification is demonstrated by the experimental results. More importantly, the proposed method outperforms the existing state-of-the-art sparse representation methods. Second, the proposed method is not only very computationally efficient but also has an intuitive and easily understandable idea. It exploits a simple algorithm to obtain a closed-form solution and discriminative representation of the test sample. Third, the feasibility, computational efficiency, and remarkable classification accuracy of the proposed $l_{2}$ regularization-based representation are comprehensively shown by extensive experiments and analysis. The code of the proposed method is available at http://www.yongxu.org/lunwen.html .
TL;DR: An off-policy integral reinforcement learning (IRL) method to solve nonlinear continuous-time (CT) nonzero-sum (NZS) games with unknown system dynamics with asymptotic stability and Nash equilibrium is established.
Abstract: This paper establishes an off-policy integral reinforcement learning (IRL) method to solve nonlinear continuous-time (CT) nonzero-sum (NZS) games with unknown system dynamics. The IRL algorithm is presented to obtain the iterative control and off-policy learning is used to allow the dynamics to be completely unknown. Off-policy IRL is designed to do policy evaluation and policy improvement in the policy iteration algorithm. Critic and action networks are used to obtain the performance index and control for each player. The gradient descent algorithm makes the update of critic and action weights simultaneously. The convergence analysis of the weights is given. The asymptotic stability of the closed-loop system and the existence of Nash equilibrium are proved. The simulation study demonstrates the effectiveness of the developed method for nonlinear CT NZS games with unknown system dynamics.