Statistical inverse problems in active network tomography
TL;DR: This paper is concerned with active network tomography where the goal is to recover information about quality-of-service parameters at the link level from aggregate data measured on end-to- end network paths.
Abstract: The analysis of computer and communication networks gives rise to some interesting inverse problems. This paper is concerned with active network tomography where the goal is to recover information about quality-of-service (QoS) parameters at the link level from aggregate data measured on end-to- end network paths. The estimation and monitoring of QoS parameters, such as loss rates and delays, are of considerable interest to network engineers and Internet service providers. The paper provides a review of the inverse problems and recent research on inference for loss rates and delay distributions. Some new results on parametric inference for delay distributions are also developed. In addition, a real application on Internet telephony is discussed.
Citations
More filters
01 May 2007
TL;DR: A unifying theory on the identifiability of the distribution of X is developed and a novel mixture model for link delays is proposed and a fast algorithm for estimation based on the General Method of Moments is developed.
Abstract: Network tomography has been regarded as one of the most promising methodologies for performance evaluation and diagnosis of the massive and decentralized Internet It can be used to infer unobservable network behaviors from directly measurable metrics and it does not require cooperation between the network internal elements and the end users For instance, the Internet users may estimate the link level characteristics such as loss and delay from end-to-end measurements, whereas the network operators can evaluate the Internet path-level traffic intensity based on link-level traffic measurements In this paper, we present a novel estimation approach for the network tomography problem Unlike previous likelihood based methods, our approach is based on characteristic functions, ie the Fourier transform, of the distributions of observed measurements We focus on network delay tomography and develop a Fourier domain inference algorithm based on flexible mixture models of link delays Through extensive model simulation and simulation using real Internet trace, we are able to demonstrate that the new algorithm is computationally more efficient and yields more accurate estimates than previous methods, especially for a network with heterogeneous link delays In addition, we obtain some identifiability results that can be applied to general distribution models for link delays
108 citations
Cites background or methods from "Statistical inverse problems in act..."
...See Lawrence et al. (2006) and Denby et al. (2007) for examples of how unicast and multicast probes can be designed and sent....
[...]
...The main tool we use is characteristic function whose basic properties are reviewed below....
[...]
...Liang and Yu (2003) proposed a pseudo-likelihood method with multicast measurements, and recently Lawrence et al. (2006) proposed local likelihood method with both unicast and multicast measurements, both of which were shown to be fast and quite efficient compared with the MLE....
[...]
TL;DR: This is the first work that derives the necessary and sufficient conditions on the network topology for identifying additive link metrics and develops a polynomial-time algorithm to compute linearly independent cycles and paths.
Abstract: In this paper, we study the problem of identifying constant additive link metrics using linearly independent monitoring cycles and paths. A monitoring cycle starts and ends at the same monitoring station, while a monitoring path starts and ends at distinct monitoring stations. We show that three-edge connectivity is a necessary and sufficient condition to identify link metrics using one monitoring station and employing monitoring cycles. We develop a polynomial-time algorithm to compute the set of linearly independent cycles. For networks that are less than three-edge-connected, we show how the minimum number of monitors required and their placement may be computed. For networks with symmetric directed links, we show the relationship between the number of monitors employed, the number of directed links for which metric is known a priori, and the identifiability for the remaining links. To the best of our knowledge, this is the first work that derives the necessary and sufficient conditions on the network topology for identifying additive link metrics and develops a polynomial-time algorithm to compute linearly independent cycles and paths.
69 citations
TL;DR: Methods for estimating edge-level parameters from end-to-end path-level measurements are discussed, an important engineering problem that raises interesting statistical modeling issues.
Abstract: Assessing and monitoring the performance of computer and communications networks is an important problem for network engineers. A considerable amount of work has been done on tools and techniques for data collection, modeling, and analysis within the network research community. This article presents an overview of the engineering problems and statistical issues, describes recent research developments, and summarizes ongoing work and areas for further research. Although there are many interesting issues related to network analysis, our focus here is on estimating and monitoring network quality-of-service parameters. We discuss methods for estimating edge-level parameters from end-to-end path-level measurements, an important engineering problem that raises interesting statistical modeling issues. Other topics include network monitoring, network visualization, and discovery of network topology. Data from a corporate network are used to illustrate the problems and techniques. As in any overview, the discussio...
41 citations
TL;DR: This paper focuses on network delay tomography and develops a Fourier domain inference algorithm based on flexible mixture models of link delays that is computationally more efficient and yields more accurate estimates than previous methods, especially for a network with heterogeneous link delays.
Abstract: The statistical problem for network tomography is to infer the distribution of X, with mutually independent components, from a measurement model Y = AX, where A is a given binary matrix representing the routing topology of a network under consideration. The challenge is that the dimension of X is much larger than that of Y and thus the problem is often ill-posed. This paper studies some statistical aspects of network tomography. We first develop a unifying theory on the identifiability of the distribution of X. We then focus on an important instance of network tomography-network delay tomography, where the problem is to infer internal link delay distributions using end-to-end delay measurements. We propose a novel mixture model for link delays and develop a fast algorithm for estimation based on the General Method of Moments. Through extensive model simulations and real Internet trace driven simulation, the proposed approach is shown to be favorable to previous methods using simple discretization for inferring link delays in a heterogeneous network.
41 citations
TL;DR: A general approach for estimating the density of the delay in any link of the network, based on continuous-time bivariate Markov chain modeling, which also provides the estimates of the packet routing probability at each node, and the probability of each source-destination path in the network.
Abstract: Estimation of link delay densities in a computer network, from source-destination delay measurements, is of great importance in analyzing and improving the operation of the network. In this paper, we develop a general approach for estimating the density of the delay in any link of the network, based on continuous-time bivariate Markov chain modeling. The proposed approach also provides the estimates of the packet routing probability at each node, and the probability of each source-destination path in the network. In this approach, the states of one process of the bivariate Markov chain are associated with nodes of the network, while the other process serves as an underlying process that affects statistical properties of the node process. The node process is not Markov, and the sojourn time in each of its states is phase-type. Phase-type densities are dense in the set of densities with non-negative support. Hence, they can be used to approximate arbitrarily well any sojourn time distribution. Furthermore, the class of phase-type densities is closed under convolution and mixture operations. We adopt the expectation-maximization (EM) algorithm of Asmussen, Nerman, and Olsson for estimating the parameter of the bivariate Markov chain. We demonstrate the performance of the approach in a numerical study.
17 citations
Cites background from "Statistical inverse problems in act..."
...Some schemes rely on active network tomography in which test probes are transmitted across the network (see, for example, [6], [8], [12], [14], [19]–[21], [25], [29], [32]), and the references therein....
[...]
References
More filters
Book•
01 Aug 1988
TL;DR: This book offers a balanced presentation of the theoretical, practical, and computational aspects of nonlinear regression and provides background material on linear regression, including the geometrical development for linear and nonlinear least squares.
Abstract: Wiley-Interscience Paperback Series The Wiley-Interscience Paperback Series consists of selected books that have been made more accessible to consumers in an effort to increase global appeal and general circulation. With these new unabridged softcover volumes, Wiley hopes to extend the lives of these works by making them available to future generations of statisticians, mathematicians, and scientists. "The authors have put together an extraordinary presentation of concepts and methods concerning the use and analysis of nonlinear regression models ...highly recommend[ed] ...for anyone needing to use and/or understand issues concerning the analysis of nonlinear regression models." -Technometrics "[This book] provides a good balance of relevant theory and application with many examples ...[and it] provides the most balanced approach to theory and application appropriate for a first course in nonlinear regression modeling for graduate statistics students." -Mathematical Reviews "[This book] joins a distinguished list of publications with a reputation for balancing technical rigor with readability, and theory with application. [It] upholds tradition ...[and is] a worthwhile reference for the marketing researcher with a serious interest in linear models. " -Journal of Marketing Research This book offers a balanced presentation of the theoretical, practical, and computational aspects of nonlinear regression and provides background material on linear regression, including the geometrical development for linear and nonlinear least squares. The authors employ real data sets throughout, and their extensive use of geometric constructs and continuing examples makes the progression of ideas appear very natural. The book also includes pseudocode for computing algorithms.
3,202 citations
01 Aug 2000
TL;DR: Self-similar Network Traffic: An Overview (K. Park & W. Willinger).
Abstract: Self-Similar Network Traffic: An Overview (K. Park & W. Willinger). Wavelets for the Analysis, Estimation, and Synthesis of Scaling Data (P. Abry, et al.). Simulations with Heavy-Tailed Workloads (M. Crovella & L. Lipsky). Queueing Behavior Under Fractional Brownian Traffic (I. Norros). Heavy Load Queueing Analysis with LRD On/Off Sources (F. Brichet, et al.). The Single Server Queue: Heavy Tails and Heavy Traffic (O. Boxma & J. Cohen). Fluid Queues, On/Off Processes, and Teletraffic Modeling with Highly Variable and Correlated Inputs (S. Resnick & G. Samorodnitsky). Bounds on the Buffer Occupancy Probability with Self-Similar Input Traffic (N. Likhanov). Buffer Asymptotics for M/G/ Input Processes (A. Makowski & M. Parulekar). Asymptotic Analysis of Queues with Subexponential Arrival Processes (P. Jelenkovi). Traffic and Queueing from an Unbounded Set of Independent Memoryless On/Off Sources (P. Jacquet). Long-Range Dependence and Queueing Effects for VBR Video (D. Heyman & T. Lakshman). Analysis of Transient Loss Performance Impact of Long-Range Dependence in Network Traffic (G.-L. Li & V. Li). The Protocol Stack and Its Modulating Effect on Self-Similar Traffic (K. Park, et al.). Characteristics of TCP Connection Arrivals (A. Feldmann). Engineering for Quality of Service (J. Roberts). Network Design and Control Using On/Off and Multilevel Source Traffic Models with Heavy-Tailed Distributions (N. Duffield & W. Whitt). Congestion Control for Self-Similar Network Traffic (T. Tuan & K. Park). Quality of Service Provisioning for Long-Range-Dependent Real-Time Traffic (A. Adas & A. Mukherjhee). Toward an Improved Understanding of Network Traffic Dynamics (R. Riedi & W. Willinger). Future Directions and Open Problems in Performance Evaluation and Control of Self-Similar Network Traffic (K. Park). Index.
1,329 citations
"Statistical inverse problems in act..." refers background in this paper
...This is somewhat unrealistic in real network situations where traffic tends to be fairly bursty [11], but it provides a simple scenario for our purposes....
[...]
TL;DR: In this article, the problem of estimating the node-to-node traffic intensity from repeated measurements of traffic on the links of a network is formulated and discussed under Poisson assumptions and two types of traffic-routing regimens: deterministic (a fixed known path between each directed pair of nodes) and Markovian (a random path between a pair of vertices, determined according to a known Markov chain fixed for that pair).
Abstract: The problem of estimating the node-to-node traffic intensity from repeated measurements of traffic on the links of a network is formulated and discussed under Poisson assumptions and two types of traffic-routing regimens: deterministic (a fixed known path between each directed pair of nodes) and Markovian (a random path between each directed pair of nodes, determined according to a known Markov chain fixed for that pair). Maximum likelihood estimation and related approximations are discussed, and computational difficulties are pointed out. A detailed methodology is presented for estimates based on the method of moments. The estimates are derived algorithmically, taking advantage of the fact that the first and second moment equations give rise to a linear inverse problem with positivity restrictions that can be approached by an EM algorithm, resulting in a particularly simple solution to a hard problem. A small simulation study is carried out.
801 citations
723 citations
"Statistical inverse problems in act..." refers methods in this paper
...This is a special case of the nonlinear least squares problem and can be solved using iterative methods such as the Gauss-Newton procedure (see [1] for example)....
[...]
TL;DR: This article introduces network tomography, a new field which it is believed will benefit greatly from the wealth of statistical methods and algorithms including the application of pseudo-likelihood methods and tree estimation formulations.
Abstract: Today's Internet is a massive, distributed network which contin- ues to explode in size as e-commerce and related activities grow. The hetero- geneous and largely unregulated structure of the Internet renders tasks such as dynamic routing, optimized service provision, service level verification and detection of anomalous/malicious behavior extremely challenging. The problem is compounded by the fact that one cannot rely on the cooperation of individual servers and routers to aid in the collection of network traffic measurements vital for these tasks. In many ways, network monitoring and inference problems bear a strong resemblance to other "inverse problems" in which key aspects of a system are not directly observable. Familiar sig- nal processing or statistical problems such as tomographic image reconstruc- tion and phylogenetic tree identification have interesting connections to those arising in networking. This article introduces network tomography, a new field which we believe will benefit greatly from the wealth of statistical the- ory and algorithms. It focuses especially on recent developments in the field including the application of pseudo-likelihood methods and tree estimation formulations.
483 citations