This paper proposes a learned cost estimation model for Distributed Stream Processing Systems (DSPS) with an aim to provide accurate cost predictions of executing queries. A major premise of this work is that the proposed learned model can generalize to the dynamics of streaming workloads out-of-the-box. This means a model once trained can accurately predict performance metrics such as latency and throughput even if the characteristics of the data and workload or the deployment of operators to hardware changes at runtime. That way the model can be used to solve tasks such as optimizing the placement of operators to minimize the end-to-end latency of a streaming query or maximize its throughput even under varying conditions. Our evaluation on a well-known DSPS, Apache Storm, shows that the model can predict accurately for unseen workloads and queries while generalizing across real-world benchmarks.

/pdf/zero-shot-cost-models-for-distributed-stream-processing-4mzjhcju.pdf

Zero-shot cost models for distributed stream processing

Complex event processing (CEP) enables the detection of situations of interest by evaluating queries over event streams. When applied in a networked application, events generated by distributed nodes are sent over the network to evaluate CEP queries. To reduce the transmission of events, push-based communication that sends each event immediately upon generation may be complemented with a pull-based model that buffers events locally until they are requested for query evaluation. Existing approaches that leverage push-pull communication to reduce transmission costs for distributed CEP, however, exploit solely temporal constraints imposed by a query. As such, they are not applicable in scenarios, in which relevant events may occur in each time window defined by a query. In this paper, we propose predicate-based push-pull (PrePP) plans for CEP queries to overcome the above limitations. Our idea is to construct pull requests that enable fine-granular filtering at event sources based on query predicates, thereby reducing event transmission. Since the construction of optimal PrePP plans is NP-hard, we introduce a set of algorithms to speed up the plan construction by up to five orders of magnitude compared to a brute-force approach, while producing near-optimal results. In extensive experiments, we demonstrate that PrePP plans reduce event transmission by up to three orders of magnitude over baseline techniques.

/pdf/predicate-based-push-pull-communication-for-distributed-cep-2vxobt7k.pdf

Predicate-based push-pull communication for distributed CEP

Stream processing systems (SPSs) have been designed to process data streams in real-time, allowing organizations to analyze and act upon data on-the-fly, as it is generated. However, handling sensitive or personal data in these multilayered SPSs that distribute resources across sensor, fog, and cloud layers raises privacy concerns, as the data may be subject to unauthorized access and attacks that can violate user privacy, hence facing regulations such as the GDPR across the SPS layers. To address these issues, different privacy-preserving mechanisms (PPMs) are proposed to protect user privacy in SPSs. Yet, selecting and applying such PPMs in SPSs is challenging, since they must operate in real-time while tolerating little overhead. The multilayered nature of SPSs complicates privacy protection because each layer may confront different privacy threats, which must be addressed by specific PPMs. To overcome these challenges, we present Prinseps, our comprehensive privacy vision for SPSs. Towards this vision, we (1) identify critical privacy threats on different layers of the multilayered SPS, (2) evaluate the effectiveness of existing PPMs in addressing such threats, and (3) integrate privacy considerations into the decision-making processes of SPSs.

No One Size (PPM) Fits All: Towards Privacy in Stream Processing Systems

Complex event processing (CEP) detects situations of interest by evaluating queries over event streams. Once CEP is used in networked applications, the distribution of query evaluation among the event sources enables performance optimization. Instead of collecting all events at one location for query evaluation, sub-queries are placed at network nodes to reduce the data transmission overhead. Yet, existing techniques either place such sub-queries at exactly one node in the network, which neglects the benefits of truly distributed evaluation, or are agnostic to the network structure, which ignores transmission costs due to the absence of direct network links. To overcome the above limitations, we propose INEV graphs for in-network evaluation of CEP queries with rich semantics, including Kleene closure and negation. Our idea is to introduce fine-granular routing of partial results of sub-queries as an additional degree of freedom in query evaluation: We exploit events already disseminated in the network as part of one sub-query, when evaluating another one. We show how to instantiate INEv graphs by splitting a query workload into sub-queries, placing them at network nodes, and forwarding of their results to other nodes. Also, we characterize INEv graphs that guarantee correct and complete query evaluation, and discuss their construction based on a cost model that unifies transmission and processing latency. Our experimental results indicate that INEv graphs can reduce transmission costs for distributed CEP by up to eight orders of magnitude compared to baseline strategies.

/pdf/inev-in-network-evaluation-for-event-stream-processing-3o4xe4kd.pdf

Pascal Weisenberger

Papers

TCEP: Transitions in operator placement to adapt to dynamic network environments