Open AccessProceedings Article
In-Network PCA and Anomaly Detection
Ling Huang,Long Nguyen,Minos Garofalakis,Michael I. Jordan,Anthony D. Joseph,Nina Taft +5 more
- Vol. 19, pp 617-624
TLDR
A PCA-based anomaly detector in which adaptive local data filters send to a coordinator just enough data to enable accurate global detection is developed, based on a stochastic matrix perturbation analysis that characterizes the tradeoff between the accuracy of anomaly detection and the amount of data communicated over the network.Abstract:
We consider the problem of network anomaly detection in large distributed systems. In this setting, Principal Component Analysis (PCA) has been proposed as a method for discovering anomalies by continuously tracking the projection of the data onto a residual subspace. This method was shown to work well empirically in highly aggregated networks, that is, those with a limited number of large nodes and at coarse time scales. This approach, however, has scalability limitations. To overcome these limitations, we develop a PCA-based anomaly detector in which adaptive local data filters send to a coordinator just enough data to enable accurate global detection. Our method is based on a stochastic matrix perturbation analysis that characterizes the tradeoff between the accuracy of anomaly detection and the amount of data communicated over the network.read more
Citations
More filters
Book
Outlier Analysis
TL;DR: Outlier Analysis is a comprehensive exposition, as understood by data mining experts, statisticians and computer scientists, and emphasis was placed on simplifying the content, so that students and practitioners can also benefit.
Proceedings ArticleDOI
Adversarial machine learning
TL;DR: In this article, the authors discuss an emerging field of study: adversarial machine learning (AML), the study of effective machine learning techniques against an adversarial opponent, and give a taxonomy for classifying attacks against online machine learning algorithms.
Journal ArticleDOI
Adversarial Machine Learning
TL;DR: The author briefly introduces the emerging field of adversarial machine learning, in which opponents can cause traditional machine learning algorithms to behave poorly in security applications.
Proceedings ArticleDOI
ANTIDOTE: understanding and defending against poisoning of anomaly detectors
Benjamin I. P. Rubinstein,Blaine Nelson,Ling Huang,Anthony D. Joseph,Shing-hon Lau,Satish Rao,Nina Taft,J. D. Tygar +7 more
TL;DR: This work proposes an antidote based on techniques from robust statistics and presents a new robust PCA-based detector that substantially reduces the effectiveness of poisoning for a variety of scenarios and indeed maintains a significantly better balance between false positives and false negatives than the original method when under attack.
Journal ArticleDOI
A Unifying Review of Deep and Shallow Anomaly Detection
Lukas Ruff,Jacob R. Kauffmann,Robert A. Vandermeulen,Grégoire Montavon,Wojciech Samek,Marius Kloft,Thomas G. Dietterich,Klaus-Robert Müller +7 more
TL;DR: This review aims to identify the common underlying principles and the assumptions that are often made implicitly by various methods in deep learning, and draws connections between classic “shallow” and novel deep approaches and shows how this relation might cross-fertilize or extend both directions.
References
More filters
Book
Matrix perturbation theory
G. W. Stewart,Ji-guang Sun +1 more
TL;DR: In this article, the Perturbation of Eigenvalues and Generalized Eigenvalue Problems are studied. But they focus on linear systems and Least Squares problems and do not consider invariant subspaces.
Proceedings ArticleDOI
Diagnosing network-wide traffic anomalies
TL;DR: A general method based on a separation of the high-dimensional space occupied by a set of network traffic measurements into disjoint subspaces corresponding to normal and anomalous network conditions to diagnose anomalies is proposed.
Journal ArticleDOI
Control Procedures for Residuals Associated With Principal Component Analysis
TL;DR: In this article, the treatment of residuals associated with principal component analysis (PCA) is discussed, i.e., the difference between the original observations and the predictions of them using less than a full set of principal components.
Book ChapterDOI
Matrix Perturbation Theory
V. N. Bogaevski,A. Povzner +1 more
TL;DR: X is the vector space which acts in the n-dimensional (complex) vector space R.1.1 and is related to Varepsilon by the following inequality.
Proceedings ArticleDOI
Structural analysis of network traffic flows
Anukool Lakhina,Konstantina Papagiannaki,Mark Crovella,Christophe Diot,Eric D. Kolaczyk,Nina Taft +5 more
TL;DR: This work presents the first analysis of complete sets of OD flow time-series, taken from two different backbone networks (Abilene and Sprint-Europe) and finds that the set of OD flows has small intrinsic dimension, and shows how to use PCA to systematically decompose the structure ofOD flow timeseries into three main constituents.