10 Lessons from 10 Years of Measuring and Modeling the Internet's Autonomous Systems
TL;DR: By presenting a BGP-focused state-of-the-art treatment of the aspects that are critical for a rigorous study of this inter-domain topology, this paper demystify in this paper many "controversial" observations reported in the existing literature and illustrate the benefits and richness of new scientific approaches to measuring, modeling, and analyzing the inter- domain topology.
Abstract: Formally, the Internet inter-domain routing system is a collection of networks, their policies, peering relationships and organizational affiliations, and the addresses they advertize. It also includes components like Internet exchange points. By its very definition, each and every aspect of this system is impacted by BGP, the de-facto standard inter-domain routing protocol. The element of this inter-domain routing system that has attracted the single-most attention within the research community has been the "inter-domain topology". Unfortunately, almost from the get go, the vast majority of studies of this topology, from definition, to measurement, to modeling and analysis, have ignored the central role of BGP in this problem. The legacy is a set of specious findings, unsubstantiated claims, and ill-conceived ideas about the Internet as a whole. By presenting a BGP-focused state-of-the-art treatment of the aspects that are critical for a rigorous study of this inter-domain topology, we demystify in this paper many "controversial" observations reported in the existing literature. At the same time, we illustrate the benefits and richness of new scientific approaches to measuring, modeling, and analyzing the inter-domain topology that are faithful to the BGP-specific nature of this problem domain.
Citations
More filters
13 Aug 2012
TL;DR: A first-of-its-kind and in-depth analysis of one of the largest IXPs worldwide based on nine months' worth of sFlow records collected at that IXP in 2011 suggests that these large IXPs can be viewed as a microcosm of the Internet ecosystem itself and argues for a re-assessment of the mental picture the community has about this ecosystem.
Abstract: The largest IXPs carry on a daily basis traffic volumes in the petabyte range, similar to what some of the largest global ISPs reportedly handle. This little-known fact is due to a few hundreds of member ASes exchanging traffic with one another over the IXP's infrastructure. This paper reports on a first-of-its-kind and in-depth analysis of one of the largest IXPs worldwide based on nine months' worth of sFlow records collected at that IXP in 2011.A main finding of our study is that the number of actual peering links at this single IXP exceeds the number of total AS links of the peer-peer type in the entire Internet known as of 2010! To explain such a surprisingly rich peering fabric, we examine in detail this IXP's ecosystem and highlight the diversity of networks that are members at this IXP and connect there with other member ASes for reasons that are similarly diverse, but can be partially inferred from their business types and observed traffic patterns. In the process, we investigate this IXP's traffic matrix and illustrate what its temporal and structural properties can tell us about the member ASes that generated the traffic in the first place. While our results suggest that these large IXPs can be viewed as a microcosm of the Internet ecosystem itself, they also argue for a re-assessment of the mental picture that our community has about this ecosystem.
278 citations
23 Oct 2013
TL;DR: A new algorithm to infer business relationships between ASes using BGP paths, relying on three assumptions about the Internet's inter-domain structure, and evaluating three algorithms for inferring each AS's customer cone, defined as the set of ASes an AS can reach using customer links.
Abstract: Business relationships between ASes in the Internet are typically confidential, yet knowledge of them is essential to understand many aspects of Internet structure, performance, dynamics, and evolution. We present a new algorithm to infer these relationships using BGP paths. Unlike previous approaches, our algorithm does not assume the presence (or seek to maximize the number) of valley-free paths, instead relying on three assumptions about the Internet's inter-domain structure: (1) an AS enters into a provider relationship to become globally reachable; and (2) there exists a peering clique of ASes at the top of the hierarchy, and (3) there is no cycle of p2c links. We assemble the largest source of validation data for AS-relationship inferences to date, validating 34.6% of our 126,082 c2p and p2p inferences to be 99.6% and 98.7% accurate, respectively. Using these inferred relationships, we evaluate three algorithms for inferring each AS's customer cone, defined as the set of ASes an AS can reach using customer links. We demonstrate the utility of our algorithms for studying the rise and fall of large transit providers over the last fifteen years, including recent claims about the flattening of the AS-level topology and the decreasing influence of tier-1 ASes on the global Internet.
256 citations
TL;DR: This survey reviews selected approaches, algorithms, and results on shortest-path queries from these fields, with the main focus lying on the tradeoff between the index size and the query time.
Abstract: We consider the point-to-point (approximate) shortest-path query problem, which is the following generalization of the classical single-source (SSSP) and all-pairs shortest-path (APSP) problems: we are first presented with a network (graph). A so-called preprocessing algorithm may compute certain information (a data structure or index) to prepare for the next phase. After this preprocessing step, applications may ask shortest-path or distance queries, which should be answered as fast as possible.Due to its many applications in areas such as transportation, networking, and social science, this problem has been considered by researchers from various communities (sometimes under different names): algorithm engineers construct fast route planning methods; database and information systems researchers investigate materialization tradeoffs, query processing on spatial networks, and reachability queries; and theoretical computer scientists analyze distance oracles and sparse spanners. Related problems are considered for compact routing and distance labeling schemes in networking and distributed computing and for metric embeddings in geometry as well.In this survey, we review selected approaches, algorithms, and results on shortest-path queries from these fields, with the main focus lying on the tradeoff between the index size and the query time. We survey methods for general graphs as well as specialized methods for restricted graph classes, in particular for those classes with arguable practical significance such as planar graphs and complex networks.
249 citations
Cites background from "10 Lessons from 10 Years of Measuri..."
...5Whether or not many of these degree sequences actually obey power laws is a controversial question [Faloutsos et al. 1999; Clauset et al. 2009; Achlioptas et al. 2009; Roughan et al. 2011]....
[...]
...…carefully selecting landmarks [Potamias et al. 2009; Das Sarma et al. 2010; Gubichev et al. 2010; Tretyakov 5Whether or not many of these degree sequences actually obey power laws is a controversial question [Faloutsos et al. 1999; Clauset et al. 2009; Achlioptas et al. 2009; Roughan et al. 2011]....
[...]
TL;DR: This paper examines social network analysis and potential implications for the design of technological networks, and vice versa, and suggests new directions for future research in networking.
Abstract: Social networks overlaid on technological networks account for a significant fraction of Internet use. Through graph theoretic and functionality models, this paper examines social network analysis and potential implications for the design of technological networks, and vice versa. Such interplay between social networks and technological networks suggests new directions for future research in networking.
105 citations
Cites background from "10 Lessons from 10 Years of Measuri..."
...The Internet exchange points lead to many peering links among ASs, shortcuts to enable settlement-free exchange of Internet traffic, that cannot be readily measured using standard network measurement probes [45]....
[...]
05 Nov 2014
TL;DR: BGP, traceroute, and geolocation data are used to extend CAIDA's AS relationship inference algorithm to infer two types of complex relationships: hybrid relationships, where two ASes have different relationships at different interconnection points, and partial transit relationships, which restrict the scope of a customer relationship to the provider's peers and customers.
Abstract: The traditional approach of modeling relationships between ASes abstracts relationship types into three broad categories: transit, peering, and sibling. More complicated configurations exist, and understanding them may advance our knowledge of Internet economics and improve models of routing. We use BGP, traceroute, and geolocation data to extend CAIDA's AS relationship inference algorithm to infer two types of complex relationships: hybrid relationships, where two ASes have different relationships at different interconnection points, and partial transit relationships, which restrict the scope of a customer relationship to the provider's peers and customers. Using this new algorithm, we find 4.5% of the 90,272 provider-customer relationships observed in March 2014 were complex, including 1,071 hybrid relationships and 2,955 partial-transit relationships. Because most peering relationships are invisible, we believe these numbers are lower bounds. We used feedback from operators, and relationships encoded in BGP communities and RPSL, to validate 20% and 6.9% of our partial transit and hybrid inferences, respectively, and found our inferences have 92.9% and 97.0% positive predictive values. Hybrid relationships are not only established betweenlarge transit providers; in 57% of the inferred hybrid transit/peering relationships the customer had a customer cone of fewer than 5 ASes.
100 citations
Cites background from "10 Lessons from 10 Years of Measuri..."
...Using only conventional transit and peering relationships to study the AS topology is problematic [31]....
[...]
...This oversimplification ignores more complex relationships and may introduce artifacts into the study of inter-domain routing [31], such as spurious relationship cycles [11], artificial policy violations [24, 30], and generally inaccurate AS path prediction [9,23]....
[...]
...Despite the many complex relationships we inferred, and positive validation, we are limited by the well-known AS topology incompleteness problem [31], as our algorithm can only infer complex relationships where the peering component is revealed in public BGP data....
[...]
References
More filters
TL;DR: A model based on these two ingredients reproduces the observed stationary scale-free distributions, which indicates that the development of large networks is governed by robust self-organizing phenomena that go beyond the particulars of the individual systems.
Abstract: Systems as diverse as genetic networks or the World Wide Web are best described as networks with complex topology. A common property of many large networks is that the vertex connectivities follow a scale-free power-law distribution. This feature was found to be a consequence of two generic mechanisms: (i) networks expand continuously by the addition of new vertices, and (ii) new vertices attach preferentially to sites that are already well connected. A model based on these two ingredients reproduces the observed stationary scale-free distributions, which indicates that the development of large networks is governed by robust self-organizing phenomena that go beyond the particulars of the individual systems.
33,771 citations
TL;DR: Developments in this field are reviewed, including such concepts as the small-world effect, degree distributions, clustering, network correlations, random graph models, models of network growth and preferential attachment, and dynamical processes taking place on networks.
Abstract: Inspired by empirical studies of networked systems such as the Internet, social networks, and biological networks, researchers have in recent years developed a variety of techniques and models to help us understand or predict the behavior of these systems. Here we review developments in this field, including such concepts as the small-world effect, degree distributions, clustering, network correlations, random graph models, models of network growth and preferential attachment, and dynamical processes taking place on networks.
17,647 citations
TL;DR: It is found that scale-free networks, which include the World-Wide Web, the Internet, social networks and cells, display an unexpected degree of robustness, the ability of their nodes to communicate being unaffected even by unrealistically high failure rates.
Abstract: Many complex systems display a surprising degree of tolerance against errors. For example, relatively simple organisms grow, persist and reproduce despite drastic pharmaceutical or environmental interventions, an error tolerance attributed to the robustness of the underlying metabolic network1. Complex communication networks2 display a surprising degree of robustness: although key components regularly malfunction, local failures rarely lead to the loss of the global information-carrying ability of the network. The stability of these and other complex systems is often attributed to the redundant wiring of the functional web defined by the systems' components. Here we demonstrate that error tolerance is not shared by all redundant systems: it is displayed only by a class of inhomogeneously wired networks, called scale-free networks, which include the World-Wide Web3,4,5, the Internet6, social networks7 and cells8. We find that such networks display an unexpected degree of robustness, the ability of their nodes to communicate being unaffected even by unrealistically high failure rates. However, error tolerance comes at a high price in that these networks are extremely vulnerable to attacks (that is, to the selection and removal of a few nodes that play a vital role in maintaining the network's connectivity). Such error tolerance and attack vulnerability are generic properties of communication networks.
7,697 citations
"10 Lessons from 10 Years of Measuri..." refers background in this paper
...This fact has often been ignored when considering topics such as reliability of the AS graph under link or node failures [48], although it is a necessary ingredient for such studies....
[...]
...Other analyses of the AS-graph have included studies of its reliability [48]....
[...]
30 Aug 1999
TL;DR: These power-laws hold for three snapshots of the Internet, between November 1997 and December 1998, despite a 45% growth of its size during that period, and can be used to generate and select realistic topologies for simulation purposes.
Abstract: Despite the apparent randomness of the Internet, we discover some surprisingly simple power-laws of the Internet topology. These power-laws hold for three snapshots of the Internet, between November 1997 and December 1998, despite a 45% growth of its size during that period. We show that our power-laws fit the real data very well resulting in correlation coefficients of 96% or higher.Our observations provide a novel perspective of the structure of the Internet. The power-laws describe concisely skewed distributions of graph properties such as the node outdegree. In addition, these power-laws can be used to estimate important parameters such as the average neighborhood size, and facilitate the design and the performance analysis of protocols. Furthermore, we can use them to generate and select realistic topologies for simulation purposes.
5,023 citations
"10 Lessons from 10 Years of Measuri..." refers background in this paper
...In fact, starting with [69], theapproximatenature of the inferred inter-domain topology and the limitations of the underlying BGP route monitor data emphasized in [64] have been largely ignored, and the majority of later papers in thi s area typically only cite [69] and no longer [64]....
[...]
TL;DR: Despite the apparent randomness of the Internet, some surprisingly simple power-laws of theInternet topology are discovered, which hold for three snapshots of the internet.
Abstract: Despite the apparent randomness of the Internet, we discover some surprisingly simple power-laws of the Internet topology. These power-laws hold for three snapshots of the Internet, between Novembe...
2,270 citations