Data visualization is by far the most commonly used mechanism to explore and extract insights from datasets, especially by novice data scientists. And yet, current visual analytics tools are rather limited in their ability to operate on collections of visualizations---by composing, filtering, comparing, and sorting them---to find those that depict desired trends or patterns. The process of visual data exploration remains a tedious process of trial-and-error. We propose zenvisage, a visual analytics platform for effortlessly finding desired visual patterns from large datasets. We introduce zenvisage's general purpose visual exploration language, ZQL ("zee-quel") for specifying the desired visual patterns, drawing from use-cases in a variety of domains, including biology, mechanical engineering, climate science, and commerce. We formalize the expressiveness of ZQL via a visual exploration algebra---an algebra on collections of visualizations---and demonstrate that ZQL is as expressive as that algebra. zenvisage exposes an interactive front-end that supports the issuing of ZQL queries, and also supports interactions that are "short-cuts" to certain commonly used ZQL queries. To execute these queries, zenvisage uses a novel ZQL graph-based query optimizer that leverages a suite of optimizations tailored to the goal of processing collections of visualizations in certain pre-defined ways. Lastly, a user survey and study demonstrates that data scientists are able to effectively use zenvisage to eliminate error-prone and tedious exploration and directly identify desired visualizations.

/pdf/effortless-data-exploration-with-zenvisage-an-expressive-and-1k04g95wtv.pdf

Effortless data exploration with zenvisage: an expressive and interactive visual analytics system

Debugging data processing logic in Data-Intensive Scalable Computing (DISC) systems is a difficult and time consuming effort. Today's DISC systems offer very little tooling for debugging programs, and as a result programmers spend countless hours collecting evidence (e.g., from log files) and performing trial and error debugging. To aid this effort, we built Titian, a library that enables data provenance---tracking data through transformations---in Apache Spark. Data scientists using the Titian Spark extension will be able to quickly identify the input data at the root cause of a potential bug or outlier result. Titian is built directly into the Spark platform and offers data provenance support at interactive speeds---orders-of-magnitude faster than alternative solutions---while minimally impacting Spark job performance; observed overheads for capturing data lineage rarely exceed 30% above the baseline job execution time.

/pdf/titian-data-provenance-support-in-spark-4phmttb7zw.pdf

Titian: data provenance support in Spark

Mobile and sensing devices have already become ubiquitous. They have made tracking moving objects an easy task. As a result, mobile applications like Uber and many IoT projects have generated massive amounts of trajectory data that can no longer be processed by a single machine efficiently. Among the typical query operations over trajectories, similarity search is a common yet expensive operator in querying trajectory data. It is useful for applications in different domains such as traffic and transportation optimizations, weather forecast and modeling, and sports analytics. It is also a fundamental operator for many important mining operations such as clustering and classification of trajectories. In this paper, we propose a distributed query framework to process trajectory similarity search over a large set of trajectories. We have implemented the proposed framework in Spark, a popular distributed data processing engine, by carefully considering different design choices. Our query framework supports both the Hausdorff distance the Frechet distance. Extensive experiments have demonstrated the excellent scalability and query efficiency achieved by our design, compared to other methods and design alternatives.

/pdf/distributed-trajectory-similarity-search-2ml9xptgua.pdf

Distributed trajectory similarity search

Today, enterprises collect large amounts of data and leverage the cloud to perform analytics over this data. Since the data is often sensitive, enterprises would prefer to keep it confidential and to hide it even from the cloud operator. Systems such as CryptDB and Monomi can accomplish this by operating mostly on encrypted data; however, these systems rely on expensive cryptographic techniques that limit performance in true "big data" scenarios that involve terabytes of data or more.This paper presents Seabed, a system that enables efficient analytics over large encrypted datasets. In contrast to previous systems, which rely on asymmetric encryption schemes, Seabed uses a novel, additively symmetric homomorphic encryption scheme (ASHE) to perform large-scale aggregations efficiently. Additionally, Seabed introduces a novel randomized encryption scheme called Splayed ASHE, or SPLASHE, that can, in certain cases, prevent frequency attacks based on auxiliary data.

/pdf/big-data-analytics-over-encrypted-datasets-with-seabed-3oe4qxh2pz.pdf

Big data analytics over encrypted datasets with seabed

Memory consumption of de Bruijn graphs is often prohibitive. Most de Bruijn graph-based assemblers reduce the complexity by compacting paths into single vertices, but this is challenging as it requires the uncompacted de Bruijn graph to be available in memory. We present a parallel and memory-efficient algorithm enabling the direct construction of the compacted de Bruijn graph without producing the intermediate uncompacted graph. Bifrost features a broad range of functions, such as indexing, editing, and querying the graph, and includes a graph coloring method that maps each k-mer of the graph to the genomes it occurs in. Availability
 https://github.com/pmelsted/bifrost

/pdf/bifrost-highly-parallel-construction-and-indexing-of-colored-3faljlmy8f.pdf

Bifrost: highly parallel construction and indexing of colored and compacted de Bruijn graphs

Bitmap indexes are commonly used in databases and search engines. By exploiting bit-level parallelism, they can significantly accelerate queries. However, they can use much memory, and thus, we might prefer compressed bitmap indexes. Following Oracle's lead, bitmaps are often compressed using run-length encoding RLE. Building on prior work, we introduce the Roaring compressed bitmap format: it uses packed arrays for compression instead of RLE. We compare it to two high-performance RLE-based bitmap encoding techniques: Word Aligned Hybrid compression scheme and Compressed 'n' Composable Integer Set. On synthetic and real data, we find that Roaring bitmaps 1 often compress significantly better e.g., 2× and 2 are faster than the compressed alternatives up to 900× faster for intersections. Our results challenge the view that RLE-based bitmap compression is best. Copyright © 2015 John Wiley & Sons, Ltd.

/pdf/better-bitmap-performance-with-roaring-bitmaps-3e42dxj5dv.pdf

Better bitmap performance with Roaring bitmaps

In the current Big Data era, systems for collecting, storing and efficiently exploiting huge amounts of data are continually introduced, such as Hadoop, Apache Spark, Dremel, etc. Druid is one of theses systems especially designed to manage such data quantities, and allows to perform detailed real-time analysis on terabytes of data within sub-second latencies. One of the important Druid's requirements is fast data filtering. To insure that, Druid makes an extensive use of bitmap indexes. Previously, we introduced a new compressed bitmap index scheme called Roaring bitmap that has shown interesting results when compared to the bitmap compression scheme adopted by Druid: Concise. Since, Roaring bitmap has been integrated to Druid as an indexing solution. In this work, we produce an extensive series of experiments in order to compare Roaring bitmap and Concise time-space performances when used to accelerate Druid's OLAP queries and other kinds of operations Druid realizes on bitmaps, like: retrieving set bits from bitmaps, computing bitmap complements, aggregating several bitmaps with logical ORs and ANDs operations. Roaring bitmap has shown to improve up to a 5× analytical queries response times under Druid compared to Concise.

/pdf/optimizing-druid-with-roaring-bitmaps-22nl5x1ozp.pdf

Optimizing Druid with Roaring bitmaps

Bitmap indexes are commonly used in databases and search engines. By exploiting bit-level parallelism, they can significantly accelerate queries. However, they can use much memory, and thus we might prefer compressed bitmap indexes. Following Oracle's lead, bitmaps are often compressed using run-length encoding (RLE). Building on prior work, we introduce the Roaring compressed bitmap format: it uses packed arrays for compression instead of RLE. We compare it to two high-performance RLE-based bitmap encoding techniques: WAH (Word Aligned Hybrid compression scheme) and Concise (Compressed `n' Composable Integer Set). On synthetic and real data, we find that Roaring bitmaps (1) often compress significantly better (e.g., 2 times) and (2) are faster than the compressed alternatives (up to 900 times faster for intersections). Our results challenge the view that RLE-based bitmap compression is best.

/pdf/better-bitmap-performance-with-roaring-bitmaps-2mu9yz2kk8.pdf

Les index bitmap sont tres utilises dans les entrepots de donnees et moteurs de recherche pour accelerer les requetes d'interrogation. Leurs principaux avantages sont leur forme compacte et leur capacite a tirer profit du traitement parallele de bits dans les CPU (bit-level parallelism). Dans l'ere actuelle du Big Data, les collections de donnees deviennent de plus en plus volumineuses. Les librairies d'index bitmap compresses disponibles a ce jour dans la litterature, telles que : Roaring bitmap, WAH ou Concise, ne supportent qu'au plus 4 milliards d'entrees et sont tres souvent impraticables dans de tels contextes. Apres avoir constate ce besoin tant dans le milieu industriel que scientifique, nous proposons trois nouveaux modeles d'index bitmap compresses, bases sur le format de notre precedente contribution Roaring bitmap et qui supportent jusqu'a 2^64 entrees. Des experiences sur des donnees synthetiques ont ete mises en oeuvre pour comparer les performances des trois nouvelles propositions avec la solution du moteur de recherche Apache Lucene : OpenBitSet, et d'autres collections Java. Les resultats ont montre que les trois nouvelles techniques ont ete pres de 300 millions de fois et 1800fois moins volumineuses en consommation memoire qu'OpenBitSet et les collections Java, respectivement. Aussi, les trois nouveaux modeles ont calcule des operations logiques, jusqu'a 6 millions de fois et jusqu'a 63 milles fois plus vite qu'OpenBitSet et les structures Java, respectivement.

Nouveaux modèles d’index bitmap compressés à 64 bits

Les index bitmap sont tres utilises dans les entrepots de donnees et moteurs de recherche. Leur capacite a executer efficacement des operations binaires entre bitmaps ameliore significativement les temps de reponse des requetes. Cependant, sur des attributs de hautes cardinalites, ils consomment un espace memoire important. Ainsi, plusieurs techniques de compression bitmap ont ete introduites pour reduire l'espace memoire occupe par ces index, et accelerer leurs temps de traitement. Ce papier introduit un nouveau modele de compression bitmap, appele Roaring bitmap. Une comparaison experimentale, sur des donnees reelles et synthetiques, avec deux autres solutions de compression bitmap connues dans la litterature : WAH (Word Aligned Hybrid compression scheme) et Concise (Compressed "n" Composable integer Set), a montre que Roaring bitmap n'utilise que 25% d'espace memoire compare a WAH et 50% par rapport a Concise, tout en accelerant significativement les temps de calcul des operations logiques entre bitmaps (jusqu'a 1100 fois pour les intersections).

Samy Chambi

Papers

Better bitmap performance with Roaring bitmaps

Optimizing Druid with Roaring bitmaps

Better bitmap performance with Roaring bitmaps

Nouveaux modèles d’index bitmap compressés à 64 bits

Roaring bitmap : nouveau modèle de compression bitmap