Anomalies are unusual and significant changes in a network's traffic levels, which can often span multiple links. Diagnosing anomalies is critical for both network operators and end users. It is a difficult problem because one must extract and interpret anomalous patterns from large amounts of high-dimensional, noisy data.In this paper we propose a general method to diagnose anomalies. This method is based on a separation of the high-dimensional space occupied by a set of network traffic measurements into disjoint subspaces corresponding to normal and anomalous network conditions. We show that this separation can be performed effectively by Principal Component Analysis.Using only simple traffic measurements from links, we study volume anomalies and show that the method can: (1) accurately detect when a volume anomaly is occurring; (2) correctly identify the underlying origin-destination (OD) flow which is the source of the anomaly; and (3) accurately estimate the amount of traffic involved in the anomalous OD flow.We evaluate the method's ability to diagnose (i.e., detect, identify, and quantify) both existing and synthetically injected volume anomalies in real traffic from two backbone networks. Our method consistently diagnoses the largest volume anomalies, and does so with a very low false alarm rate.

/pdf/diagnosing-network-wide-traffic-anomalies-2oqvh3kj5a.pdf

Diagnosing network-wide traffic anomalies

One of the fundamental difficulties with trying to characterize the Internet''s behavior lies in its immense diversity. To date, efforts to characterize end-to-end Internet dynamics such as packet delay and loss have studied at most a handful of traffic traces and Internet paths, making it difficult to assess the generality of the findings. Yet solid assessments of Internet path properties are crucial for engineering transport protocols for high performance. In this work, we attempt to tackle this problem using a distributed framework we developed for conducting end-to-end measurements. In the first part, we report on an end-to-end routing study conducted using the framework, in which 37 sites participated, yielding about 40,000 routing measurements along 1,000 Internet paths. We characterize routing pathologies and failures, stability over time, and symmetry. In the second part of the study, we analyze 20,000 TCP bulk transfers measured using the same framework. We discuss: 1) errors introduced by the packet filters used to make the measurements; 2) dealing with unsynchronized and skewed clocks; 3) separating out TCP behavior from network behavior; 4) the surprisingly large variations between different TCP implementations; 5) pathological network behavior, such as out-of-order delivery, replication, and corruption; 6) a robust algorithm for estimating bottleneck bandwidth; and 7) the resulting characterizations of Internet packet loss and delay, including general assessments of queueing time scales and available bandwidth.

/pdf/measurements-and-analysis-of-end-to-end-internet-dynamics-2g9j633ch4.pdf

Measurements and analysis of end-to-end Internet dynamics

A matrix giving the traffic volumes between origin and destination in a network has tremendously potential utility for network capacity planning and management. Unfortunately, traffic matrices are generally unavailable in large operational IP networks. On the other hand, link load measurements are readily available in IP networks. In this paper, we propose a new method for practical and rapid inference of traffic matrices in IP networks from link load measurements, augmented by readily available network and routing configuration information. We apply and validate the method by computing backbone-router to backbone-router traffic matrices on a large operational tier-1 IP network -- a problem an order of magnitude larger than any other comparable method has tackled. The results show that the method is remarkably fast and accurate, delivering the traffic matrix in under five seconds.

/pdf/fast-accurate-computation-of-large-scale-ip-traffic-matrices-2qyjoqrsy6.pdf

Fast accurate computation of large-scale IP traffic matrices from link loads

Very few techniques have been proposed for estimating traffic matrices in the context of Internet traffic. Our work on POP-to-POP traffic matrices (TM) makes two contributions. The primary contribution is the outcome of a detailed comparative evaluation of the three existing techniques. We evaluate these methods with respect to the estimation errors yielded, sensitivity to prior information required and sensitivity to the statistical assumptions they make. We study the impact of characteristics such as path length and the amount of link sharing on the estimation errors. Using actual data from a Tier-1 backbone, we assess the validity of the typical assumptions needed by the TM estimation techniques. The secondary contribution of our work is the proposal of a new direction for TM estimation based on using choice models to model POP fanouts. These models allow us to overcome some of the problems of existing methods because they can incorporate additional data and information about POPs and they enable us to make a fundamentally different kind of modeling assumption. We validate this approach by illustrating that our modeling assumption matches actual Internet data well. Using two initial simple models we provide a proof of concept showing that the incorporation of knowledge of POP features (such as total incoming bytes, number of customers, etc.) can reduce estimation errors. Our proposed approach can be used in conjunction with existing or future methods in that it can be used to generate good priors that serve as inputs to statistical inference techniques.

Traffic matrix estimation: existing techniques and new directions

Network traffic arises from the superposition of Origin-Destination (OD) flows. Hence, a thorough understanding of OD flows is essential for modeling network traffic, and for addressing a wide variety of problems including traffic engineering, traffic matrix estimation, capacity planning, forecasting and anomaly detection. However, to date, OD flows have not been closely studied, and there is very little known about their properties.We present the first analysis of complete sets of OD flow time-series, taken from two different backbone networks (Abilene and Sprint-Europe). Using Principal Component Analysis (PCA), we find that the set of OD flows has small intrinsic dimension. In fact, even in a network with over a hundred OD flows, these flows can be accurately modeled in time using a small number (10 or less) of independent components or dimensions.We also show how to use PCA to systematically decompose the structure of OD flow timeseries into three main constituents: common periodic trends, short-lived bursts, and noise. We provide insight into how the various constitutents contribute to the overall structure of OD flows and explore the extent to which this decomposition varies over time.

/pdf/structural-analysis-of-network-traffic-flows-47iodguryo.pdf

Structural analysis of network traffic flows

The problem of estimating the node-to-node traffic intensity from repeated measurements of traffic on the links of a network is formulated and discussed under Poisson assumptions and two types of traffic-routing regimens: deterministic (a fixed known path between each directed pair of nodes) and Markovian (a random path between each directed pair of nodes, determined according to a known Markov chain fixed for that pair). Maximum likelihood estimation and related approximations are discussed, and computational difficulties are pointed out. A detailed methodology is presented for estimates based on the method of moments. The estimates are derived algorithmically, taking advantage of the fact that the first and second moment equations give rise to a linear inverse problem with positivity restrictions that can be approached by an EM algorithm, resulting in a particularly simple solution to a hard problem. A small simulation study is carried out.

Y. Vardi

Papers

Network Tomography: Estimating Source-Destination Traffic Intensities from Link Data