In the data stream scenario, input arrives very rapidly and there is limited memory to store the input. Algorithms have to work with one or few passes over the data, space less than linear in the input size or time significantly less than the input size. In the past few years, a new theory has emerged for reasoning about algorithms that work within these constraints on space, time, and number of passes. Some of the methods rely on metric embeddings, pseudo-random computations, sparse approximation theory and communication complexity. The applications for this scenario include IP network traffic analysis, mining text message streams and processing massive data sets in general. Researchers in Theoretical Computer Science, Databases, IP Networking and Computer Systems are working on the data stream challenges. This article is an overview and survey of data stream algorithmics and is an updated version of [1].

/pdf/data-streams-algorithms-and-applications-5fo85u5tcy.pdf

Data streams: algorithms and applications

CQL, a continuous query language, is supported by the STREAM prototype data stream management system (DSMS) at Stanford. CQL is an expressive SQL-based declarative language for registering continuous queries against streams and stored relations. We begin by presenting an abstract semantics that relies only on “black-box” mappings among streams and relations. From these mappings we define a precise and general interpretation for continuous queries. CQL is an instantiation of our abstract semantics using SQL to map from relations to relations, window specifications derived from SQL-99 to map from streams to relations, and three new operators to map from relations to streams. Most of the CQL language is operational in the STREAM system. We present the structure of CQL's query execution plans as well as details of the most important components: operators, interoperator queues, synopses, and sharing of components among multiple operators and queries. Examples throughout the paper are drawn from the Linear Road benchmark recently proposed for DSMSs. We also curate a public repository of data stream applications that includes a wide variety of queries expressed in CQL. The relative ease of capturing these applications in CQL is one indicator that the language contains an appropriate set of constructs for data stream processing.

/pdf/the-cql-continuous-query-language-semantic-foundations-and-i8wz91bcbt.pdf

The CQL continuous query language: semantic foundations and query execution

The CQL Continuous Query Language : Semantic Foundations and Query Execution

We explore the nature of traffic in data centers, designed to support the mining of massive data sets. We instrument the servers to collect socket-level logs, with negligible performance impact. In a 1500 server operational cluster, we thus amass roughly a petabyte of measurements over two months, from which we obtain and report detailed views of traffic and congestion conditions and patterns. We further consider whether traffic matrices in the cluster might be obtained instead via tomographic inference from coarser-grained counter data.

/pdf/the-nature-of-data-center-traffic-measurements-analysis-1db3il1pin.pdf

The nature of data center traffic: measurements & analysis

Traditional databases store sets of relatively static records with no pre-defined notion of time, unless timestamp attributes are explicitly added. While this model adequately represents commercial catalogues or repositories of personal information, many current and emerging applications require support for on-line analysis of rapidly changing data streams. Limitations of traditional DBMSs in supporting streaming applications have been recognized, prompting research to augment existing technologies and build new systems to manage streaming data. The purpose of this paper is to review recent work in data stream management systems, with an emphasis on application requirements, data models, continuous query languages, and query evaluation.

/pdf/issues-in-data-stream-management-4o590m5t36.pdf

Issues in data stream management

We have developed Gigascope, a stream database for network applications including traffic analysis, intrusion detection, router configuration analysis, network research, network monitoring, and performance monitoring and debugging. Gigascope is undergoing installation at many sites within the AT&T network, including at OC48 routers, for detailed monitoring. In this paper we describe our motivation for and constraints in developing Gigascope, the Gigascope architecture and query language, and performance issues. We conclude with a discussion of stream database research problems we have found in our application.

/pdf/gigascope-a-stream-database-for-network-applications-535fwuier2.pdf

Gigascope: a stream database for network applications

The present invention provides an architecture that advantageously leverages multiple content distribution networks to provide enhanced services. In accordance with an embodiment of the present invention, a share of content requests are served by each of a plurality of content distribution networks. The fraction of content requests served by a particular content distribution network can be determined dynamically, depending on the offered load or other traffic characteristics.

Method and apparatus for content distribution network brokering and peering

No single encoding scheme or fault model is optimal for all data. A versatile storage system allows them to be matched to access patterns, reliability requirements, and cost goals on a per-data item basis. Ursa Minor is a cluster-based storage system that allows data-specific selection of, and on-line changes to, encoding schemes and fault models. Thus, different data types can share a scalable storage infrastructure and still enjoy specialized choices, rather than suffering from "one size fits all." Experiments with Ursa Minor show performance benefits of 2-3× when using specialized choices as opposed to a single, more general, configuration. Experiments also show that a single cluster supporting multiple workloads simultaneously is much more efficient when the choices are specialized for each distribution rather than forced to use a "one size fits all" configuration. When using the specialized distributions, aggregate cluster throughput nearly doubled.

/pdf/ursa-minor-versatile-cluster-based-storage-49er76hwv8.pdf

Ursa minor: versatile cluster-based storage

Content Distribution Networks (CDNs) attempt to improve Web performance by delivering Web content to end-users from servers located at the edge of the network. An important factor contributing to the performance improvement is the ability of a CDN to select servers in the proximity of the requesting clients. Most CDNs today use the Domain Name System (DNS) to make such server selection decisions. However, DNS provides only the IP address of the client’s local DNS server to the CDN, rather than the client’s IP address. Therefore, CDNs using DNS-based server selection assume that clients are “close” to their local DNS servers. To quantify the proximity between clients and their local DNS servers, we propose a novel, precise, and efficient technique for finding the associations of client to local DNS servers. We collected more than 4.2 million such unique associations in three months. From this data, we study the impact of proximity on DNS-based server selection using four different proximity metrics. We conclude that DNS is good for very coarse-grained server selection, since 64% of the associations belong to the same Autonomous System. DNS is less useful for finergrained server selection, since only 16% of the client and local DNS associations are in the same network-aware cluster [13] (based on BGP routing information from a wide set of routers). As an application of this methodology, we evaluate DNS-based server selection in three of the largest commercially deployed CDNs to study its accuracy.

A Precise and Efficient Evaluation of the Proximity Between Web Clients and Their Local DNS Servers

The present invention is directed to mechanisms for content-aware redirection and content exchange/content discovery that permit a request for content to be redirected to a particular advantageous server that can serve the content.

Charles D. Cranor

Papers

Gigascope: a stream database for network applications

Method and apparatus for content distribution network brokering and peering

Ursa minor: versatile cluster-based storage

A Precise and Efficient Evaluation of the Proximity Between Web Clients and Their Local DNS Servers

Method for content-aware redirection and content renaming