Open AccessPosted Content
Communication-Efficient Jaccard Similarity for High-Performance Distributed Genome Comparisons
Maciej Besta,Raghavendra Kanakagiri,Harun Mustafa,Mikhail Karasikov,Gunnar Rätsch,Torsten Hoefler,Edgar Solomonik +6 more
TLDR
The design and implementation of SimilarityAtScale is designed and implemented, the first communication-efficient distributed algorithm for computing the Jaccard similarity among pairs of large datasets, and the resulting scheme is the first to enable accurateJaccard distance derivations for massive datasets, using large-scale distributed-memory systems.Abstract:
The Jaccard similarity index is an important measure of the overlap of two sets, widely used in machine learning, computational genomics, information retrieval, and many other areas. We design and implement SimilarityAtScale, the first communication-efficient distributed algorithm for computing the Jaccard similarity among pairs of large datasets. Our algorithm provides an efficient encoding of this problem into a multiplication of sparse matrices. Both the encoding and sparse matrix product are performed in a way that minimizes data movement in terms of communication and synchronization costs. We apply our algorithm to obtain similarity among all pairs of a set of large samples of genomes. This task is a key part of modern metagenomics analysis and an evergrowing need due to the increasing availability of high-throughput DNA sequencing data. The resulting scheme is the first to enable accurate Jaccard distance derivations for massive datasets, using largescale distributed-memory systems. We package our routines in a tool, called GenomeAtScale, that combines the proposed algorithm with tools for processing input sequences. Our evaluation on real data illustrates that one can use GenomeAtScale to effectively employ tens of thousands of processors to reach new frontiers in large-scale genomic and metagenomic analysis. While GenomeAtScale can be used to foster DNA research, the more general underlying SimilarityAtScale algorithm may be used for high-performance distributed similarity computations in other data analytics application domains.read more
Citations
More filters
Posted Content
Slim Fly: A Cost Effective Low-Diameter Network Topology
Maciej Besta,Torsten Hoefler +1 more
TL;DR: This work proposes deadlock-free routing schemes and physical layouts for large computing centres as well as a detailed cost and power model for Slim Fly, a high-performance cost-effective network topology that approaches the theoretically optimal network diameter.
Proceedings ArticleDOI
Slim graph: practical lossy graph compression for approximate graph processing, storage, and analytics
Maciej Besta,Simon Weber,Lukas Gianinazzi,Robert Gerstenberger,Andrey Ivanov,Yishai Oltchik,Torsten Hoefler +6 more
TL;DR: Slim Graph is proposed, the first programming model and framework for practical lossy graph compression that facilitates high-performance approximate graph processing, storage, and analytics and may become the common ground for developing, executing, and analyzing emerging lossygraph compression schemes.
Posted ContentDOI
Practice of Streaming and Dynamic Graphs: Concepts, Models, Systems, and Parallelism.
TL;DR: This work provides the first analysis and taxonomy of dynamic and streaming graph processing, focusing on identifying the fundamental system designs and on understanding their support for concurrency and parallelism, and for different graph updates as well as analytics workloads.
Posted Content
Practice of Streaming Processing of Dynamic Graphs: Concepts, Models, and Systems
TL;DR: This work provides the first analysis and taxonomy of dynamic and streaming graph processing, focusing on identifying the fundamental system designs and on understanding their support for concurrency, and for different graph updates as well as analytics workloads.
Posted Content
High-Performance Parallel Graph Coloring with Strong Guarantees on Work, Depth, and Quality
Maciej Besta,Armon Carigiet,Zur Vonarburg-Shmaria,Kacper Janda,Lukas Gianinazzi,Torsten Hoefler +5 more
TL;DR: This work develops the first parallel graph coloring heuristics with strong theoretical guarantees on work and depth and coloring quality, and designs a relaxation of the vertex degeneracy order, a well-known graph theory concept, to introduce a tunable amount of parallelism into the degeneracy ordering.
References
More filters
Journal ArticleDOI
Basic Local Alignment Search Tool
TL;DR: A new approach to rapid sequence comparison, basic local alignment search tool (BLAST), directly approximates alignments that optimize a measure of local similarity, the maximal segment pair (MSP) score.
Journal ArticleDOI
The neighbor-joining method: a new method for reconstructing phylogenetic trees.
Naruya Saitou,Masatoshi Nei +1 more
TL;DR: The neighbor-joining method and Sattath and Tversky's method are shown to be generally better than the other methods for reconstructing phylogenetic trees from evolutionary distance data.
Journal ArticleDOI
MapReduce: simplified data processing on large clusters
Jeffrey Dean,Sanjay Ghemawat +1 more
TL;DR: This paper presents the implementation of MapReduce, a programming model and an associated implementation for processing and generating large data sets that runs on a large cluster of commodity machines and is highly scalable.
Journal ArticleDOI
MapReduce: simplified data processing on large clusters
Jeffrey Dean,Sanjay Ghemawat +1 more
TL;DR: This presentation explains how the underlying runtime system automatically parallelizes the computation across large-scale clusters of machines, handles machine failures, and schedules inter-machine communication to make efficient use of the network and disks.