Today's Internet is a massive, distributed network which continues to explode in size as e-commerce and related activities grow. The heterogeneous and largely unregulated structure of the Internet renders tasks such as dynamic routing, optimized service provision, service-level verification, and detection of anomalous/malicious behavior increasingly challenging tasks. The problem is compounded by the fact that one cannot rely on the cooperation of individual servers and routers to aid in the collection of network traffic measurements vital for these tasks. In many ways, network monitoring and inference problems bear a strong resemblance to other "inverse problems" in which key aspects of a system are not directly observable. Familiar signal processing problems such as tomographic image reconstruction, system identification, and array processing all have interesting interpretations in the networking context. This article introduces the new field of network tomography, a field which we believe will benefit greatly from the wealth of signal processing theory and algorithms.

http://ieeexplore.ieee.org/iel5/79/21538/00998081.pdf

Internet tomography

Today's Internet is a massive, distributed network which contin- ues to explode in size as e-commerce and related activities grow. The hetero- geneous and largely unregulated structure of the Internet renders tasks such as dynamic routing, optimized service provision, service level verification and detection of anomalous/malicious behavior extremely challenging. The problem is compounded by the fact that one cannot rely on the cooperation of individual servers and routers to aid in the collection of network traffic measurements vital for these tasks. In many ways, network monitoring and inference problems bear a strong resemblance to other "inverse problems" in which key aspects of a system are not directly observable. Familiar sig- nal processing or statistical problems such as tomographic image reconstruc- tion and phylogenetic tree identification have interesting connections to those arising in networking. This article introduces network tomography, a new field which we believe will benefit greatly from the wealth of statistical the- ory and algorithms. It focuses especially on recent developments in the field including the application of pseudo-likelihood methods and tree estimation formulations.

/pdf/network-tomography-recent-developments-2sa7wg7tlp.pdf

Network Tomography: Recent Developments

The substantial overhead of performing internal network monitoring motivates techniques for inferring spatially localized information about performance using only end-to-end measurements. In this paper, we present a novel methodology for inferring the queuing delay distributions across internal links in the network based solely on unicast, end-to-end measurements. The major contributions are: 1) we formulate a measurement procedure for estimation and localization of delay distribution based on end-to-end packet pairs; 2) we develop a simple way to compute maximum likelihood estimates (MLEs) using the expectation-maximization (EM) algorithm; 3) we develop a new estimation methodology based on recently proposed nonparametric, wavelet-based density estimation method; and 4) we optimize the computational complexity of the EM algorithm by developing a new fast Fourier transform implementation. Realistic network simulations are carried out using network-level simulator ns-2 to demonstrate the accuracy of the estimation procedure.

/pdf/network-delay-tomography-4yw2u6kc98.pdf

Network delay tomography

Network tomography has been regarded as one of the most promising methodologies for performance evaluation and diagnosis of the massive and decentralized Internet It can be used to infer unobservable network behaviors from directly measurable metrics and it does not require cooperation between the network internal elements and the end users For instance, the Internet users may estimate the link level characteristics such as loss and delay from end-to-end measurements, whereas the network operators can evaluate the Internet path-level traffic intensity based on link-level traffic measurements In this paper, we present a novel estimation approach for the network tomography problem Unlike previous likelihood based methods, our approach is based on characteristic functions, ie the Fourier transform, of the distributions of observed measurements We focus on network delay tomography and develop a Fourier domain inference algorithm based on flexible mixture models of link delays Through extensive model simulation and simulation using real Internet trace, we are able to demonstrate that the new algorithm is computationally more efficient and yields more accurate estimates than previous methods, especially for a network with heterogeneous link delays In addition, we obtain some identifiability results that can be applied to general distribution models for link delays

Network Tomography: Identifiability and Fourier Domain Estimation

The modeling and analysis of computer communications networks give rise to a variety of interesting statistical problems. This paper focuses on network tomography, a term used to characterize two classes of large-scale inverse problems. The first deals with passive tomography where aggregate data are collected at the individual router/node level and the goal is to recover path-level information. The main problem of interest here is the estimation of the origin-destination traffic matrix. The second, referred to as active tomography, deals with reconstructing link-level information from end-to-end path-level measurements obtained by actively probing the network. The primary application in this case is estimation of quality-of-service parameters such as loss rates and delay distributions. The paper provides a review of the statistical issues and developments in network tomography with an emphasis on active tomography. An application to Internet telephony is used to illustrate the results.

Network tomography: A review and recent developments

Providers of high quality-of-service over telecommunication networks require accurate methods for remote measurement of link-level performance. Recent research in network tomography has demonstrated that it is possible to estimate internal link characteristics, e.g., link delays and packet losses, using unicast probing schemes in which probes are exchanged between several pairs of sites in the network. In this paper we present a new method for estimation of internal link delay distributions using the end-to-end packet delay statistics gathered by such unicast probes. Our method is based on a variant of the penalized maximum likelihood expectation-maximization (PML-EM) algorithm applied to an additive finite mixture model for the link delay probability density functions. The mixture model incorporates a combination of discrete and continuous components and we use a minimum message length (MML) penalty for selection of model order. We present results of matlab and ns-2 simulations to illustrate the promise of our network tomography algorithm for light cross-traffic scenarios.

/pdf/unicast-based-inference-of-network-link-delay-distributions-3o3o8bxiyx.pdf

Unicast-Based Inference of Network Link Delay Distributions with Finite Mixture Models (Revision)

Providers of high quality-of-service over telecommunication networks require accurate methods for remote measurement of link-level performance Recent research in network tomography has demonstrated that it is possible to estimate internal link characteristics, eg, link delays and packet losses, using unicast probing schemes in which probes are exchanged between several pairs of sites in the network We present a new method for estimation of internal link delay distributions using the end-to-end packet pair delay statistics gathered by back-to-back packet-pair unicast probes Our method is based on a variant of the penalized maximum likelihood expectation-maximization (PML-EM) algorithm applied to an additive finite mixture model for the link delay probability density functions The mixture model incorporates a combination of discrete and continuous components, and we use a minimum message length (MML) penalty for selection of model order We present results of Matlab and ns-2 simulations to illustrate the promise of our network tomography algorithm for light cross-traffic scenarios

/pdf/unicast-based-inference-of-network-link-delay-distributions-3cbubt1ydq.pdf

Unicast-based inference of network link delay distributions with finite mixture models

Inference of network internal link characteristics has become an increasingly important issue for operating and evaluating large telecommunication networks. Since it is usually impractical to directly monitor each link along a specific path, end-to-end probes are sometimes used to collect link characteristic information at edge nodes of the network. This paper deals with unicast probing methods for estimation of link delay characteristics. Unicast traffic is easy to generate and is supported by almost every network currently in operation. Under the assumptions that link delays are spatially and temporally independent, we propose a bias corrected estimator for the internal link delay cumulant generating function (CGF) based on unicast probe end-to-end delay measurements. Through simulation we show that the proposed estimator attains a level of mean squared error comparable to link delay CGF estimates obtained from directly measured link delay statistics. We can use these CGF estimates to estimate delay mean, variance and level exceedance probabilities for each link.

/pdf/unicast-inference-of-network-link-delay-distributions-from-3ihmlf3zvp.pdf

Unicast inference of network link delay distributions from edge measurements

In this paper, we address the problem of topology discovery in unicast logical tree networks using end-to-end measurements. Without any cooperation from the internal routers, topology estimation can be formulated as hierarchical clustering of the leaf nodes based on pairwise correlations as similarity metrics. Unlike previous work that first assumes the network topology is a binary tree and then tries to generalize to a nonbinary tree, we provide a framework that directly deals with general logical tree topologies. A hierarchical algorithm to estimate the topology is developed in a recursive manner by finding the best partitions of the leaf nodes level by level. Our simulations show that the algorithm is more robust than binary-tree based methods

/pdf/hierarchical-inference-of-unicast-network-topologies-based-58v207w5tb.pdf

Hierarchical Inference of Unicast Network Topologies Based on End-to-End Measurements

As telecommunication networks grow larger and more complex, it is important to monitor internal link characteristics for operation, monitoring, and diagnosis purposes. Since router link monitoring is not practical due to high communication overhead, there has been considerable interest in monitoring from edge (end-to-end) observations. This paper focuses on the estimation of internal link delay distributions from edge measurements. Discrete and continuous delay models are introduced and we propose a new mixed finite mixture model for link delay probability density functions (p.d.f.). When collecting end-to-end unicast packet delays from edge nodes, we are able to estimate internal link delay distributions using the EM algorithm. Simulation results are given to illustrate our method.

/pdf/unicast-based-inference-of-network-link-delay-distributions-3s6yh1sk85.pdf

Meng-Fu Shih

Papers

Unicast-Based Inference of Network Link Delay Distributions with Finite Mixture Models (Revision)

Unicast-based inference of network link delay distributions with finite mixture models

Unicast inference of network link delay distributions from edge measurements

Hierarchical Inference of Unicast Network Topologies Based on End-to-End Measurements

Unicast-based inference of network link delay distributions using mixed finite mixture models