This open source computing framework unifies streaming, batch, and interactive big data workloads to unlock new applications

https://dl.acm.org/doi/pdf/10.1145/2934664

Apache Spark: a unified engine for big data processing

Spark SQL is a new module in Apache Spark that integrates relational processing with Spark's functional programming API. Built on our experience with Shark, Spark SQL lets Spark programmers leverage the benefits of relational processing (e.g. declarative queries and optimized storage), and lets SQL users call complex analytics libraries in Spark (e.g. machine learning). Compared to previous systems, Spark SQL makes two main additions. First, it offers much tighter integration between relational and procedural processing, through a declarative DataFrame API that integrates with procedural Spark code. Second, it includes a highly extensible optimizer, Catalyst, built using features of the Scala programming language, that makes it easy to add composable rules, control code generation, and define extension points. Using Catalyst, we have built a variety of features (e.g. schema inference for JSON, machine learning types, and query federation to external databases) tailored for the complex needs of modern data analysis. We see Spark SQL as an evolution of both SQL-on-Spark and of Spark itself, offering richer APIs and optimizations while keeping the benefits of the Spark programming model.

http://people.csail.mit.edu/matei/papers/2015/sigmod_spark_sql.pdf

Spark SQL: Relational Data Processing in Spark

Modern enterprise applications are currently undergoing a complete paradigm shift away from traditional transactional processing to combined analytical and transactional processing. This challenge of combining two opposing query types in a single database management system results in additional requirements for transaction management as well. In this paper, we discuss our approach to achieve high throughput for transactional query processing while allowing concurrent analytical queries. We present our approach to distributed snapshot isolation and optimized two-phase commit protocols.

/pdf/apache-flink-stream-and-batch-processing-in-a-single-engine-3u2381y8rg.pdf

Apache flink : Stream and batch processing in a single engine

The broad success of Hadoop has led to a fast-evolving and diverse ecosystem of application engines that are building upon the YARN resource management layer. The open-source implementation of MapReduce is being slowly replaced by a collection of engines dedicated to specific verticals. This has led to growing fragmentation and repeated efforts with each new vertical engine re-implementing fundamental features (e.g. fault-tolerance, security, stragglers mitigation, etc.) from scratch. In this paper, we introduce Apache Tez, an open-source framework designed to build data-flow driven processing runtimes. Tez provides a scaffolding and library components that can be used to quickly build scalable and efficient data-flow centric engines. Central to our design is fostering component re-use, without hindering customizability of the performance-critical data plane. This is in fact the key differentiator with respect to the previous generation of systems (e.g. Dryad, MapReduce) and even emerging ones (e.g. Spark), that provided and mandated a fixed data plane implementation. Furthermore, Tez provides native support to build runtime optimizations, such as dynamic partition pruning for Hive. Tez is deployed at Yahoo!, Microsoft Azure, LinkedIn and numerous Hortonworks customer sites, and a growing number of engines are being integrated with it. This confirms our intuition that most of the popular vertical engines can leverage a core set of building blocks. We complement qualitative accounts of real-world adoption with quantitative experimental evidence that Tez-based implementations of Hive, Pig, Spark, and Cascading on YARN outperform their original YARN implementation on popular benchmarks (TPC-DS, TPC-H) and production workloads.

/pdf/apache-tez-a-unifying-framework-for-modeling-and-building-3axdtiegve.pdf

Apache Tez: A Unifying Framework for Modeling and Building Data Processing Applications

A system and computer-implemented is provided for displaying a configurable metric relating to an environment in a graphical display along with a value of the metric calculated over a configurable time period. The metric is used to identify events of interest in the environment based on processing real time machine data from one or more sources. The configurable metric is selected and a corresponding value is calculated based on the events of interest over the configurable time period. The value of the metric may be continuously updated in real time based on receiving additional real-time machine data and displayed in a graphical interface as time progresses. Statistical trends in the value of the metric may also be determined over the configurable time period and displayed in the graphical interface as well as an indication if the value of the metric exceeds a configurable threshold value. Further, a selection of one or more thresholds for the value of the metric may be applied and an indication displayed indicating if the threshold(s) have been exceeded.

Key indicators view

Cloudera Impala is a modern, open-source MPP SQL engine architected from the ground up for the Hadoop data processing environment. Impala provides low latency and high concurrency for BI/analytic read-mostly queries on Hadoop, not delivered by batch frameworks such as Apache Hive. This paper presents Impala from a user’s perspective, gives an overview of its architecture and main components and briefly demonstrates its superior performance compared against other popular SQL-on-Hadoop systems.

/pdf/impala-a-modern-open-source-sql-engine-for-hadoop-20x8qmti8r.pdf

Impala: A Modern, Open-Source SQL Engine for Hadoop.

Systems and methods of collecting and aggregating log data with fault tolerance are disclosed. One embodiment includes, one or more devices that generate log data, the one or more machines each associated with an agent node to collect the log data, wherein, the agent node generates a batch comprising multiple messages from the log data and assigns a tag to the batch. In one embodiment, the agent node further computes a checksum for the batch of multiple messages. The system may further include a collector device, the collector device being associated with a collector tier having a collector node to which the agent sends the log data; wherein, the collector determines the checksum for the batch of multiple messages received from the agent node.

Collecting and aggregating log data with fault tolerance

Methods for configuring a system to collect and aggregate datasets are disclosed. One embodiment includes, identifying a data source in the system from where dataset is to be collected, configuring a machine in the system that generates the dataset to be collected, to send the dataset to the data source, identifying an arrival location where the dataset that is collected is to be aggregated or written, and/or configuring an agent node by specifying a source for the agent node as the data source in the system and specifying a sink for the agent node as the arrival location.

Configuring a system to collect and aggregate datasets

A low latency query engine for Apache Hadoop that provides real-time or near real-time, ad hoc query capability, while completing batch-processing of MapReduce. In one embodiment, the low latency query engine comprises a daemon that is installed on data nodes in a Hadoop cluster for handling query requests and all internal requests related to query execution. In a further embodiment, the low latency query engine comprises a daemon for providing name service and metadata distribution. The low latency query engine receives a query request via client, turns the request into collections of plan fragments and coordinates parallel and optimized execution of the plan fragments on remote daemons to generate results at a much faster speed than existing batch-oriented processing frameworks.

Low latency query engine for apache hadoop

Impala von Cloudera ist ein modernes, massiv paralleles Datenbanksystem, welches von Grund auf fur die Bedurfnisse und Anforderungen einer Big Data Umgebung wie Hadoop entworfen wurde. Das Ziel von Impala ist es, klassische SQL-Abfragen mit geringer Latenz und Laufzeit auszufuhren, so wie man es von typischen BI/DW Losungen gewohnt ist. Gleichzeitig sollen dabei sehr grose Quelldaten in Hadoop gelesen werden, ohne dass ein weiterer Extraktionsprozess in zusatzliche Systemlandschaften notwendig ist. Dieses Kapitel soll einen Uberblick uber Impala aus der Benutzerperspektive geben und detaillierter auf die Hauptkomponenten und deren Entwurfsentscheidungen eingehen. Zusatzlich werden wir einen Geschwindigkeitsvergleich mit anderen bekannten SQL-auf-Hadoop Losungen vorstellen, der den besonderen Ansatz von Impala unterstreicht.

Henry Noel Robinson

Papers

Impala: A Modern, Open-Source SQL Engine for Hadoop.

Collecting and aggregating log data with fault tolerance

Configuring a system to collect and aggregate datasets

Low latency query engine for apache hadoop

Impala: Eine moderne, quellen-offene SQL Engine für Hadoop