Stable non-Gaussian random processes , by G. Samorodnitsky and M. S. Taqqu. Pp. 632. £49.50. 1994. ISBN 0-412-05171-0 (Chapman and Hall).

The Transmission Control Protocol.

Improved complete ensemble EMD: A suitable tool for biomedical signal processing

An empirical comparison of botnet detection methods

Anomaly detection is an important problem that has been well-studied within diverse research areas and application domains. The aim of this survey is two-fold, firstly we present a structured and comprehensive overview of research methods in deep learning-based anomaly detection. Furthermore, we review the adoption of these methods for anomaly across various application domains and assess their effectiveness. We have grouped state-of-the-art research techniques into different categories based on the underlying assumptions and approach adopted. Within each category we outline the basic anomaly detection technique, along with its variants and present key assumptions, to differentiate between normal and anomalous behavior. For each category, we present we also present the advantages and limitations and discuss the computational complexity of the techniques in real application domains. Finally, we outline open issues in research and challenges faced while adopting these techniques.

Deep Learning for Anomaly Detection: A Survey.

Evaluating anomaly detectors is a crucial task in traffic monitoring made particularly difficult due to the lack of ground truth. The goal of the present article is to assist researchers in the evaluation of detectors by providing them with labeled anomaly traffic traces. We aim at automatically finding anomalies in the MAWI archive using a new methodology that combines different and independent detectors. A key challenge is to compare the alarms raised by these detectors, though they operate at different traffic granularities. The main contribution is to propose a reliable graph-based methodology that combines any anomaly detector outputs. We evaluated four unsupervised combination strategies; the best is the one that is based on dimensionality reduction. The synergy between anomaly detectors permits to detect twice as many anomalies as the most accurate detector, and to reject numerous false positive alarms reported by the detectors. Significant anomalous traffic features are extracted from reported alarms, hence the labels assigned to the MAWI archive are concise. The results on the MAWI traffic are publicly available and updated daily. Also, this approach permits to include the results of upcoming anomaly detectors so as to improve over time the quality and variety of labels.

http://conferences.sigcomm.org/co-next/2010/CoNEXT_papers/08-Fontugne.pdf

MAWILab: combining diverse anomaly detectors for automated anomaly labeling and performance benchmarking

In the mid 1990s, it was shown that the statistics of aggregated time series from Internet traffic departed from those of traditional short range-dependent models, and were instead characterized by asymptotic self-similarity. Following this seminal contribution, over the years, many studies have investigated the existence and form of scaling in Internet traffic. This contribution first aims at presenting a methodology, combining multiscale analysis (wavelet and wavelet leaders) and random projections (or sketches), permitting a precise, efficient and robust characterization of scaling, which is capable of seeing through non-stationary anomalies. Second, we apply the methodology to a data set spanning an unusually long period: 14 years, from the MAWI traffic archive, thereby allowing an in-depth longitudinal analysis of the form, nature, and evolutions of scaling in Internet traffic, as well as network mechanisms producing them. We also study a separate three-day long trace to obtain complementary insight into intra-day behavior. We find that a biscaling (two ranges of independent scaling phenomena) regime is systematically observed: long-range dependence over the large scales, and multifractallike scaling over the fine scales. We quantify the actual scaling ranges precisely, verify to high accuracy the expected relationship between the long range dependent parameter and the heavy tail parameter of the flow size distribution, and relate fine scale multifractal scaling to typical IP packet inter-arrival and to round-trip time distributions.

/pdf/scaling-in-internet-traffic-a-14-year-and-3-day-longitudinal-15v5oarx2w.pdf

Scaling in Internet Traffic: A 14 Year and 3 Day Longitudinal Study, With Multiscale Analyses and Random Projections

A typical large building contains thousands of sensors, monitoring the HVAC system, lighting, and other operational sub-systems. With the increased push for operational efficiency, operators are relying more on historical data processing to uncover opportunities for energy-savings. However, they are overwhelmed with the deluge of data and seek more efficient ways to identify potential problems. In this paper, we present a new approach called the Strip, Bind and Search (SBS); a method for uncovering abnormal equipment behavior and in-concert usage patterns. SBS uncovers relationships between devices and constructs a model for their usage pattern relative to other devices. It then flags deviations from the model. We run SBS on a set of building sensor traces; each containing hundred sensors reporting data flows over 18 weeks from two separate buildings with fundamentally different infrastructures. We demonstrate that, in many cases, SBS uncovers misbehavior corresponding to inefficient device usage that leads to energy waste. The average waste uncovered is as high as 2500~kWh per device.

/pdf/strip-bind-and-search-a-method-for-identifying-abnormal-qxwu3log0q.pdf

Strip, bind, and search: a method for identifying abnormal energy consumption in buildings

Anomaly detection is essential for preventing network outages and maintaining the network resources available. However, to cope with the increasing growth of Internet traffic, network anomaly detectors are only exposed to sampled traffic, so harmful traffic may avoid detector examination. In this paper, we investigate the benefits of recent distributed computing approaches for real-time analysis of non-sampled Internet traffic. Focusing on the MapReduce model, our study uncovers a fundamental difficulty in order to detect network traffic anomalies by using Hadoop. Since MapReduce requires the dataset to be divided into small splits and anomaly detectors compute statistics from spatial and temporal traffic structures, special care should be taken when splitting traffic. We propose Hashdoop, a MapReduce framework that splits traffic with a hash function to preserve traffic structures and, hence, profits of distributed computing infrastructures to detect network anomalies. The benefits of Hashdoop are evaluated with two anomaly detectors and fifteen traces of Internet backbone traffic captured between 2001 and 2013. Using a 6-node cluster Hashdoop increased the throughput of the slowest detector with a speed-up of 15; thus, enabling real-time detection for the largest analyzed traces. Hashdoop also improved the overall detectors accuracy as splits emphasized anomalies by reducing the surrounding traffic.

/pdf/hashdoop-a-mapreduce-framework-for-network-anomaly-detection-1kofyx7fx1.pdf

Hashdoop: A MapReduce framework for network anomaly detection

We thus present a new taxonomy of network anomalies with wide coverage of existing work. We also provide a set of signatures that assign taxonomy labels to events. We present a preliminary study applying this taxonomy with six years of real network traffic from the MAWI repository. We classify previously documented anomalous events and draw to main conclusions. First, the taxonomy-based analysis provides new insights regarding events previous classified by heuristic rule labeling. For example, some RST events are now classified as network scan response and the majority of ICMP events are split into network scans and network scan responses. Moreover, some previously unknown events now account for a substantial number of all UDP network scans, network scan responses and port scans. Second, the number of unknown events decreases from 20 to 10% of all events with the proposed taxonomy as compared to the heuristic approach.

http://www.necoma-project.eu/m/filer_public/0d/34/0d34ebf0-0e45-46ea-a40b-852bc6eb758f/johan_taxonomy.pdf

Romain Fontugne

Papers

MAWILab: combining diverse anomaly detectors for automated anomaly labeling and performance benchmarking

Scaling in Internet Traffic: A 14 Year and 3 Day Longitudinal Study, With Multiscale Analyses and Random Projections

Strip, bind, and search: a method for identifying abnormal energy consumption in buildings

Hashdoop: A MapReduce framework for network anomaly detection

A taxonomy of anomalies in backbone network traffic