Adaptation and learning in automatic systems

The year 2015 marks the 30th birthday of DC (Difference of Convex functions) programming and DCA (DC Algorithms) which constitute the backbone of nonconvex programming and global optimization. In this article we offer a short survey on thirty years of developments of these theoretical and algorithmic tools. The survey is comprised of three parts. In the first part we present a brief history of the field, while in the second we summarize the state-of-the-art results and recent advances. We focus on main theoretical results and DCA solvers for important classes of difficult nonconvex optimization problems, and then give an overview of real-world applications whose solution methods are based on DCA. The third part is devoted to new trends and important open issues, as well as suggestions for future developments.

DC programming and DCA: thirty years of developments

This paper provides a review and commentary on the past, present, and future of numerical optimization algorithms in the context of machine learning applications. Through case studies on text classification and the training of deep neural networks, we discuss how optimization problems arise in machine learning and what makes them challenging. A major theme of our study is that large-scale machine learning represents a distinctive setting in which the stochastic gradient (SG) method has traditionally played a central role while conventional gradient-based nonlinear optimization techniques typically falter. Based on this viewpoint, we present a comprehensive theory of a straightforward, yet versatile SG algorithm, discuss its practical behavior, and highlight opportunities for designing algorithms with improved performance. This leads to a discussion about the next generation of optimization methods for large-scale machine learning, including an investigation of two main streams of research on techniques that diminish noise in the stochastic directions and methods that make use of second-order derivative approximations.

Optimization Methods for Large-Scale Machine Learning

Recent years have witnessed an incredibly increasing interest in the topic of incremental learning Unlike conventional machine learning situations, data flow targeted by incremental learning becomes available continuously over time Accordingly, it is desirable to be able to abandon the traditional assumption of the availability of representative training data during the training period to develop decision boundaries Under scenarios of continuous data flow, the challenge is how to transform the vast amount of stream raw data into information and knowledge representation, and accumulate experience over time to support future decision-making process In this paper, we propose a general adaptive incremental learning framework named ADAIN that is capable of learning from continuous raw data, accumulating experience over time, and using such knowledge to improve future learning and prediction performance Detailed system level architecture and design strategies are presented in this paper Simulation results over several real-world data sets are used to validate the effectiveness of this method

/pdf/incremental-learning-from-stream-data-1dzr83glth.pdf

Incremental Learning From Stream Data

Anomaly detection has attracted much attention in recent years since it plays a crucial role in many domains. Various anomaly detection approaches have been proposed, among which one-class support vector machine (OCSVM) is a popular one. In practice, data used for anomaly detection can be distributively collected via wireless sensor networks. Besides, as the data usually arrive at the nodes sequentially, online detection method that can process streaming data is preferred. In this paper, we formulate a distributed online OCSVM for anomaly detection over networks and get a decentralized cost function. To get the decentralized implementation without transmitting the original data, we use a random approximate function to replace the kernel function. Furthermore, to find an appropriate approximate dimension, we add a sparse constraint into the decentralized cost function to get another one. Then we minimize these two cost functions by stochastic gradient descent and derive two distributed algorithms. Some theoretical analysis and experiments are performed to show the effectiveness of the proposed algorithms. Experimental results on both synthetic and real datasets reveal that both of the proposed algorithms achieve low misdetection rates and high true positive rates. Compared with other state-of-the-art anomaly detection methods, the proposed distributed algorithms not only show good anomaly detection performance, but also require relatively short running time and low CPU memory consumption.

Distributed Online One-Class Support Vector Machine for Anomaly Detection Over Networks

We describe and evaluate two algorithms for Neyman-Pearson (NP) classification problem which has been recently shown to be of a particular importance for bipartite ranking problems. NP classification is a nonconvex problem involving a constraint on false negatives rate. We investigated batch algorithm based on DC programming and stochastic gradient method well suited for large-scale datasets. Empirical evidences illustrate the potential of the proposed methods.

/pdf/batch-and-online-learning-algorithms-for-nonconvex-neyman-16424wnkyy.pdf

Aristidis Pappaioannou

Papers

Batch and online learning algorithms for nonconvex neyman-pearson classification