https://hal.inria.fr/hal-01653842/document

Distributed data stream processing and edge computing

This thesis has two aims. The first one is to elucidate how Alice’s Adventures in Wonderland (1865) functions as a Bildungsroman, and the other one is to demonstrate how the novel also has a coming of age aspect based on feminism. Whilst Alice matures in the traditional sense, she also in parallel does so as a stronger female fighting for gender rights with signs of feminism. The feminist angle as well as the surreal world of Wonderland makes the novel a not very obvious Bildungsroman in a genre dominated by male protagonists. For Alice to be a young female child who ends up in a fantasy world thus makes her a very fascinating character. The central hypothesis of this thesis is that what Alice is exposed to and reacts to in Wonderland generally reflects the genre of a Bildungsroman and also specifically a feminist Bildungsroman. Theoretical framework is based on the ideas of Franco Moretti, Mikhail Bakhtin, Thomas Jeffers, Carol Lazzaro-Weis, George Eliot and Elizabeth Drew Stoddard, as well as novels by Eliot and Stoddard. This includes dynamic protagonists, unpredictable development, symbols of modernity, the quest for universality, and minor characters who make sure that the protagonist develops, as well as feminist struggle by means of disregarding the „cult of true womanhood‟ in a genre and society dominated by men.

不思議の国のアリス = Alice's adventures in woderland

Data Stream Processing (DSP) applications are widely used to timely extract information from distributed data sources, such as sensing devices, monitoring stations, and social networks. To successfully handle this ever increasing amount of data, recent trends investigate the possibility of exploiting decentralized computational resources (e.g., Fog computing) to define the applications placement. Several placement policies have been proposed in the literature, but they are based on different assumptions and optimization goals and, as such, they are not completely comparable to each other.In this paper we study the placement problem for distributed DSP applications. Our contributions are twofold. We provide a general formulation of the optimal DSP placement (for short, ODP) as an Integer Linear Programming problem which takes explicitly into account the heterogeneity of computing and networking resources and which encompasses - as special cases - the different solutions proposed in the literature. We present an ODP-based scheduler for the Apache Storm DSP framework. This allows us to compare some well-known centralized and decentralized placement solutions. We also extensively analyze the ODP scalability with respect to various parameter settings.

Optimal operator placement for distributed stream processing applications

Under several emerging application scenarios, such as in smart cities, operational monitoring of large infrastructure, wearable assistance, and Internet of Things, continuous data streams must be processed under very short delays. Several solutions, including multiple software engines, have been developed for processing unbounded data streams in a scalable and efficient manner. More recently, architecture has been proposed to use edge computing for data stream processing. This paper surveys state of the art on stream processing engines and mechanisms for exploiting resource elasticity features of cloud computing in stream processing. Resource elasticity allows for an application or service to scale out/in according to fluctuating demands. Although such features have been extensively investigated for enterprise applications, stream processing poses challenges on achieving elastic systems that can make efficient resource management decisions based on current load. Elasticity becomes even more challenging in highly distributed environments comprising edge and cloud computing resources. This work examines some of these challenges and discusses solutions proposed in the literature to address them.

Distributed Data Stream Processing and Edge Computing: A Survey on Resource Elasticity and Future Directions

The integration of network function virtualization (NFV) and software defined networks (SDN) seeks to create a more flexible and dynamic software-based network environment. The line between entities involved in forwarding and those involved in more complex middle box functionality in the network is blurred by the use of high-performance virtualized platforms capable of performing these functions. A key problem is how and where network functions should be placed in the network and how traffic is routed through them. An efficient placement and appropriate routing increases system capacity while also minimizing the delay seen by flows. In this paper, we formulate the problem of network function placement and routing as a mixed integer linear programming problem. This formulation not only determines the placement of services and routing of the flows, but also seeks to minimize the resource utilization. We develop heuristicsto solve the problem incrementally, allowing us to support a large number of flows and to solve the problem for incoming flows without impacting existing flows.

Virtual function placement and traffic steering in flexible and dynamic software defined networks

Storm has emerged as a promising computation platform for stream data processing. In this paper, we first show inefficiencies of the current practice of Storm scheduling and challenges associated with applying traffic-aware online scheduling in Storm via experimental results and analysis. Motivated by our observations, we design and implement a new stream data processing system based on Storm, namely, T-Storm. Compared to Storm, T-Storm has the following desirable features: 1) based on runtime states, it accelerates data processing by leveraging effective traffic-aware scheduling for assigning/re-assigning tasks dynamically, which minimizes inter-node and inter-process traffic while ensuring no worker nodes are overloaded, 2) it enables fine-grained control over worker node consolidation such that T-Storm can achieve better performance with even fewer worker nodes, 3) it allows hot-swapping of scheduling algorithms and adjustment of scheduling parameters on the fly, and 4) it is transparent to Storm users (i.e., Storm applications can be ported to run on T-Storm without any changes). We conducted real experiments in a cluster using well-known data processing applications for performance evaluation. Extensive experimental results show that compared to Storm (with the default scheduler), T-Storm can achieve over 84% and 27% speedup on lightly and heavily loaded topologies respectively (in terms of average processing time) with 30% less number of worker nodes.

T-Storm: Traffic-Aware Online Scheduling in Storm

The Single Instruction Multiple Data (SIMD) architecture of Graphic Processing Units (GPUs) makes them perfect for parallel processing of big data. In this paper, we present the design, implementation and evaluation of G-Storm, a GPU-enabled parallel system based on Storm, which harnesses the massively parallel computing power of GPUs for high-throughput online stream data processing. G-Storm has the following desirable features: 1) G-Storm is designed to be a general data processing platform as Storm, which can handle various applications and data types. 2) G-Storm exposes GPUs to Storm applications while preserving its easy-to-use programming model. 3) G-Storm achieves high-throughput and low-overhead data processing with GPUs. We implemented G-Storm based on Storm 0.9.2 and tested it using two different applications: continuous query and matrix multiplication. Extensive experimental results show that compared to Storm, G-Storm achieves over 7x improvement on throughput for continuous query, while maintaining reasonable average tuple processing time. It also leads to 2.3x throughput improvement for the matrix multiplication application.

G-Storm: GPU-enabled high-throughput online data processing in Storm

The Single Instruction Multiple Data (SIMD) architecture of Graphic Processing Units (GPUs) makes them perfect for parallel processing of big data. In this paper, we present the design, implementation and evaluation of G-Storm , a GPU-enabled parallel system based on Storm, which harnesses the massively parallel computing power of GPUs for high-throughput online stream data processing. G-Storm has the following desirable features: 1) G-Storm is designed to be a general data processing platform as Storm, which can handle various applications and data types. 2) G-Storm exposes GPUs to Storm applications while preserving its easy-to-use programming model. 3) G-Storm achieves high-throughput and low-overhead data processing with GPUs. 4) G-Storm accelerates data processing further by enabling Direct Data Transfer (DDT), between two executors that process data at a common GPU. We implemented G-Storm based on Storm 0.9.2 and tested it using three different applications, including continuous query, matrix multiplication and image resizing. Extensive experimental results show that 1) Compared to Storm, G-Storm achieves over 7× improvement on throughput for continuous query, while maintaining reasonable average tuple processing time. It also leads to 2.3× and 1.3× throughput improvements on the other two applications, respectively. 2) DDT significantly reduces data processing time.

GPU-Accelerated High-Throughput Online Stream Data Processing

This paper presents a novel parallel platform, C-Storm (Copula-based Storm), for the computationally complex problem of fusion of heterogeneous data streams for inference C-Storm is designed by marrying copula-based dependence modeling for highly accurate inference and a highly-regarded parallel computing platform Storm for fast stream data processing C-Storm has the following desirable features: 1) C-Storm offers fast inference responses 2) C-Storm provides high inference accuracies 3) C-Storm is a general-purpose inference platform that can support data fusion applications 4) C-Storm is easy to use and its users do not need to know deep knowledge of Storm or copula theory We implemented C-Storm based on Apache Storm 102 and conducted extensive experiments using a typical data fusion application Experimental results show that C-Storm offers a significant 47x speedup over a commonly used sequential baseline and higher degree of parallelism leads to better performance

A Parallel Platform for Fusion of Heterogeneous Stream Data

In this paper, we present a novel cross-job framework for MapReduce scheduling, which aims to minimize the total processing time of a sequence of related jobs by combining reduce and map phases of two consecutive jobs and streaming data between them. The proposed framework has the following desirable properties: (1) It can accelerate the execution of a sequence of related MapReduce jobs by achieving a good tradeoff between data locality and parallelism. (2) It can support all the existing MapReduce applications with no changes to their source code. (3) It is a general framework, which can work with different scheduling algorithms. We built a new MapReduce runtime system called cross-job Hadoop by integrating the proposed cross-job framework into Hadoop. We conducted extensive experiments to evaluate its performance using PageRank and an Apache Pig application. Our experimental results show that the cross-job Hadoop can significantly reduce both the total processing time of a job sequence and the size of data transferred over the network.

Zhenhua Chen

Papers

T-Storm: Traffic-Aware Online Scheduling in Storm

G-Storm: GPU-enabled high-throughput online data processing in Storm

GPU-Accelerated High-Throughput Online Stream Data Processing

A Parallel Platform for Fusion of Heterogeneous Stream Data

A cross-job framework for MapReduce scheduling