In this paper, we review the background and state-of-the-art of big data. We first introduce the general background of big data and review related technologies, such as could computing, Internet of Things, data centers, and Hadoop. We then focus on the four phases of the value chain of big data, i.e., data generation, data acquisition, data storage, and data analysis. For each phase, we introduce the general background, discuss the technical challenges, and review the latest advances. We finally examine the several representative applications of big data, including enterprise management, Internet of Things, online social networks, medial applications, collective intelligence, and smart grid. These discussions aim to provide a comprehensive overview and big-picture to readers of this exciting area. This survey is concluded with a discussion of open problems and future directions.

Big Data: A Survey

Recent technological advancements have led to a deluge of data from distinctive domains (e.g., health care and scientific sensors, user-generated data, Internet and financial companies, and supply chain systems) over the past two decades. The term big data was coined to capture the meaning of this emerging trend. In addition to its sheer volume, big data also exhibits other unique characteristics as compared with traditional data. For instance, big data is commonly unstructured and require more real-time analysis. This development calls for new system architectures for data acquisition, transmission, storage, and large-scale data processing mechanisms. In this paper, we present a literature survey and system tutorial for big data analytics platforms, aiming to provide an overall picture for nonexpert readers and instill a do-it-yourself spirit for advanced audiences to customize their own big-data solutions. First, we present the definition of big data and discuss big data challenges. Next, we present a systematic framework to decompose big data systems into four sequential modules, namely data generation, data acquisition, data storage, and data analytics. These four modules form a big data value chain. Following that, we present a detailed survey of numerous approaches and mechanisms from research and industry communities. In addition, we present the prevalent Hadoop framework for addressing big data challenges. Finally, we outline several evaluation benchmarks and potential research directions for big data systems.

Toward Scalable Systems for Big Data Analytics: A Technology Tutorial

Data centers are experiencing an exponential increase in the amount of network traffic that they have to sustain due to cloud computing and several emerging web applications. To face this network load, large data centers are required with thousands of servers interconnected with high bandwidth switches. Current data center networks, based on electronic packet switches, consume excessive power to handle the increased communication bandwidth of emerging applications. Optical interconnects have gained attention recently as a promising solution offering high throughput, low latency and reduced energy consumption compared to current networks based on commodity switches. This paper presents a thorough survey on optical interconnects for next generation data center networks. Furthermore, the paper provides a qualitative categorization and comparison of the proposed schemes based on their main features such as connectivity and scalability. Finally, the paper discusses the cost and the power consumption of these schemes that are of primary importance in the future data center networks.

A Survey on Optical Interconnects for Data Centers

When network is undergoing problems such as congestion, scan attack, DDoS attack, etc, measurements are much more important than usual In this case, traffic characteristics including available bandwidth, packet rate, and flow size distribution vary drastically, significantly degrading the performance of measurements To address this issue, we propose the Elastic sketch It is adaptive to currently traffic characteristics Besides, it is generic to measurement tasks and platforms We implement the Elastic sketch on six platforms: P4, FPGA, GPU, CPU, multi-core CPU, and OVS, to process six typical measurement tasks Experimental results and theoretical analysis show that the Elastic sketch can adapt well to traffic characteristics Compared to the state-of-the-art, the Elastic sketch achieves 446 ∼ 452 times faster speed and 20 ∼ 2737 smaller error rate

Elastic sketch: adaptive and fast network-wide measurements

In this paper, we show that up to hundreds of software load balancer (SLB) servers can be replaced by a single modern switching ASIC, potentially reducing the cost of load balancing by over two orders of magnitude. Today, large data centers typically employ hundreds or thousands of servers to load-balance incoming traffic over application servers. These software load balancers (SLBs) map packets destined to a service (with a virtual IP address, or VIP), to a pool of servers tasked with providing the service (with multiple direct IP addresses, or DIPs). An SLB is stateful, it must always map a connection to the same server, even if the pool of servers changes and/or if the load is spread differently across the pool. This property is called per-connection consistency or PCC. The challenge is that the load balancer must keep track of millions of connections simultaneously.Until recently, it was not possible to implement a load balancer with PCC in a merchant switching ASIC, because high-performance switching ASICs typically can not maintain per-connection states with PCC. Newer switching ASICs provide resources and primitives to enable PCC at a large scale. In this paper, we explore how to use switching ASICs to build much faster load balancers than have been built before. Our system, called SilkRoad, is defined in a 400 line P4 program and when compiled to a state-of-the-art switching ASIC, we show it can load-balance ten million connections simultaneously at line rate.

https://dl.acm.org/doi/pdf/10.1145/3098822.3098824

SilkRoad: Making Stateful Layer-4 Load Balancing Fast and Cheap Using Switching ASICs

In this article we review the growing trend of warehouse-scale mega-datacenter computing, the Internet transformation driven by mega-datacenter applications, and the opportunities and challenges for fiber optic communication technologies to support the growth of mega-datacenter computing in the next three to four years.

Fiber optic communication technologies: What's needed for datacenter network operations

We present the design of Espresso, Google's SDN-based Internet peering edge routing infrastructure. This architecture grew out of a need to exponentially scale the Internet edge cost-effectively and to enable application-aware routing at Internet-peering scale. Espresso utilizes commodity switches and host-based routing/packet processing to implement a novel fine-grained traffic engineering capability. Overall, Espresso provides Google a scalable peering edge that is programmable, reliable, and integrated with global traffic systems. Espresso also greatly accelerated deployment of new networking features at our peering edge. Espresso has been in production for two years and serves over 22% of Google's total traffic to the Internet.

https://dl.acm.org/doi/pdf/10.1145/3098822.3098854

Taking the Edge off with Espresso: Scale, Reliability and Programmability for Global Internet Peering

Maintaining the highest levels of availability for content providers is challenging in the face of scale, network evolution and complexity. Little, however, is known about failures large content providers are susceptible to, and what mechanisms they employ to ensure high availability. From a detailed analysis of over 100 high-impact failure events in a global-scale content provider encompassing several data centers and two WANs, we quantify several dimensions of availability failures. We find that failures are evenly distributed across different network types and planes, but that a large number of failures happen when a management operation is in progress within the network. We discuss some of these failures in detail, and also describe our design principles for high availability motivated by these failures, including using defense in depth, maintaining consistency across planes, failing open on large failures, carefully preventing and avoiding failures, and assessing root cause quickly. Our findings suggest that, as networks become more complicated, failures lurk everywhere, and, counter-intuitively, continuous incremental evolution of the network can, when applied together with our design principles, result in a more robust network.

https://dl.acm.org/doi/pdf/10.1145/2934872.2934891

Evolve or Die: High-Availability Design Principles Drawn from Googles Network Infrastructure

Mega data centers and their interconnection networks have drawn great attention in recent years because of the rapid public adoption of cloud-based services. The unprecedented amount of data that needs to be communicated between data centers imposes new requirements and challenges to inter-data-center optical networks. In this article, we discuss the traffic growth trends and capacity demands of Google?s inter-data-center network, and how they drive the network architectures and technologies to scale capacities and operational ease on existing fiber plants. We extensively review recent research findings and emerging technologies, such as digital coherent detection and the flexgrid dense wavelength-division multiplexed channel plan, and propose practical implementations, such as C+L-band transmission, packet and optical layer integration, and a software-defined networking enabled network architecture for both capacity and operational scaling. In addition, we point out a few critical areas that require more attention and research to improve efficiency and flexibility of an inter-data-center optical network: optical regeneration, data rate mismatch between Ethernet and optical transport, and real-time optical performance monitoring.

/pdf/the-prospect-of-inter-data-center-optical-networks-3f22qxf4o4.pdf

The prospect of inter-data-center optical networks

Described are methods and system for network analysis. A network analyzer for a first network is configured to receive network assessment information from a network metric monitors situated in third-party networks, the network assessment information indicating values for characteristics of one or more network paths from the respective network metric monitor to a node in a second network. The network analyzer aggregates the received network assessment information and identifies, from the aggregated network assessment information, a route from the first network to the node in the second network. The identified route is then selected from among a plurality of potential routes from the first network to the node in the second network and used in setting a routing policy for data flows from the first network through the node in the second network.

Bikash Koley

Papers

Fiber optic communication technologies: What's needed for datacenter network operations

Taking the Edge off with Espresso: Scale, Reliability and Programmability for Global Internet Peering

Evolve or Die: High-Availability Design Principles Drawn from Googles Network Infrastructure

The prospect of inter-data-center optical networks

Systems and methods for inferring network topology and path metrics in wide area networks