Social big data

Traditionally distributed graph processing systems have largely focused on scalability through the optimizations of inter-node communication and load balance. However, they often deliver unsatisfactory overall processing efficiency compared with shared-memory graph computing frameworks. We analyze the behavior of several graph-parallel systems and find that the added overhead for achieving scalability becomes a major limiting factor for efficiency, especially with modern multi-core processors and high-speed interconnection networks.Based on our observations, we present Gemini, a distributed graph processing system that applies multiple optimizations targeting computation performance to build scalability on top of efficiency. Gemini adopts (1) a sparse-dense signal-slot abstraction to extend the hybrid push-pull computation model from shared-memory to distributed scenarios, (2) a chunk-based partitioning scheme enabling low-overhead scaling out designs and locality-preserving vertex accesses, (3) a dual representation scheme to compress accesses to vertex indices, (4) NUMA-aware sub-partitioning for efficient intra-node memory accesses, plus (5) locality-aware chunking and fine-grained work-stealing for improving both internode and intra-node load balance, respectively. Our evaluation on an 8-node high-performance cluster (using five widely used graph applications and five real-world graphs) shows that Gemini significantly outperforms all well-known existing distributed graph processing systems, delivering up to 39.8× (from 8.91×) improvement over the fastest among them.

/pdf/gemini-a-computation-centric-distributed-graph-processing-57yv5aorjj.pdf

Gemini: a computation-centric distributed graph processing system

Pregel [23] was recently introduced as a scalable graph mining system that can provide significant performance improvements over traditional MapReduce implementations. Existing implementations focus primarily on graph partitioning as a preprocessing step to balance computation across compute nodes. In this paper, we examine the runtime characteristics of a Pregel system. We show that graph partitioning alone is insufficient for minimizing end-to-end computation. Especially where data is very large or the runtime behavior of the algorithm is unknown, an adaptive approach is needed. To this end, we introduce Mizan, a Pregel system that achieves efficient load balancing to better adapt to changes in computing needs. Unlike known implementations of Pregel, Mizan does not assume any a priori knowledge of the structure of the graph or behavior of the algorithm. Instead, it monitors the runtime characteristics of the system. Mizan then performs efficient fine-grained vertex migration to balance computation and communication. We have fully implemented Mizan; using extensive evaluation we show that---especially for highly-dynamic workloads---Mizan provides up to 84% improvement over techniques leveraging static graph pre-partitioning.

/pdf/mizan-a-system-for-dynamic-load-balancing-in-large-scale-4mau486wr1.pdf

Mizan: a system for dynamic load balancing in large-scale graph processing

The vertex-centric programming model is an established computational paradigm recently incorporated into distributed processing frameworks to address challenges in large-scale graph processing. Billion-node graphs that exceed the memory capacity of commodity machines are not well supported by popular Big Data tools like MapReduce, which are notoriously poor performing for iterative graph algorithms such as PageRank. In response, a new type of framework challenges one to “think like a vertex” (TLAV) and implements user-defined programs from the perspective of a vertex rather than a graph. Such an approach improves locality, demonstrates linear scalability, and provides a natural way to express and compute many iterative graph algorithms. These frameworks are simple to program and widely applicable but, like an operating system, are composed of several intricate, interdependent components, of which a thorough understanding is necessary in order to elicit top performance at scale. To this end, the first comprehensive survey of TLAV frameworks is presented. In this survey, the vertex-centric approach to graph processing is overviewed, TLAV frameworks are deconstructed into four main components and respectively analyzed, and TLAV implementations are reviewed and categorized.

https://dl.acm.org/doi/pdf/10.1145/2818185

Thinking Like a Vertex: A Survey of Vertex-Centric Frameworks for Large-Scale Distributed Graph Processing

Over the last decade we have witnessed the relentless technological improvement in flash-based solid-state drives (SSDs) and they have many advantages over hard disk drives (HDDs) as a secondary storage such as performance and power consumption. However, the random write performance in SSDs still remains as a concern. Even in modern SSDs, the disparity between random and sequential write bandwidth is more than tenfold. Moreover, random writes can shorten the limited lifespan of SSDs because they incur more NAND block erases per write.In order to overcome these problems due to random writes, in this paper, we propose a new file system for SSDs, SFS. First, SFS exploits the maximum write bandwidth of SSD by taking a log-structured approach. SFS transforms all random writes at file system level to sequential ones at SSD level. Second, SFS takes a new data grouping strategy on writing, instead of the existing data separation strategy on segment cleaning. It puts the data blocks with similar update likelihood into the same segment. This minimizes the inevitable segment cleaning overhead in any log-structured file system by allowing the segments to form a sharp bimodal distribution of segment utilization.We have implemented a prototype SFS by modifying Linux-based NILFS2 and compared it with three state-of-the-art file systems using several realistic workloads. SFS outperforms the traditional LFS by up to 2.5 times in terms of throughput. Additionally, in comparison to modern file systems such as ext4 and btrfs, it drastically reduces the block erase count inside the SSD by up to 7.5 times.

/pdf/sfs-random-write-considered-harmful-in-solid-state-drives-kxvfv7josk.pdf

SFS: random write considered harmful in solid state drives

Various scientific computations have become so complex, and thus computation tools play an important role. In this paper, we explore the state-of-the-art framework providing high-level matrix computation primitives with MapReduce through the case study approach, and demonstrate these primitives with different computation engines to show the performance and scalability. We believe the opportunity for using MapReduce in scientific computation is even more promising than the success to date in the parallel systems literature.

/pdf/hama-an-efficient-matrix-computation-with-the-mapreduce-3u4f5ff3lt.pdf

HAMA: An Efficient Matrix Computation with the MapReduce Framework

MapReduce is a programming model that supports distributed and parallel processing for large-scale data-intensive applications such as machine learning, data mining, and scientific simulation. Hadoop is an open-source implementation of the MapReduce programming model. Hadoop is used by many companies including Yahoo!, Amazon, and Facebook to perform various data mining on large-scale data sets such as user search logs and visit logs. In these cases, it is very common to share the same computing resources by multiple users due to practical considerations about cost, system utilization, and manageability. However, Hadoop assumes that all cluster nodes are dedicated to a single user, failing to guarantee high performance in the shared MapReduce computation environment. In this paper, we propose two optimization schemes, prefetching and pre-shuffling, which improve the overall performance under the shared environment while retaining compatibility with the native Hadoop. The proposed schemes are implemented in the native Hadoop-0.18.3 as a plug-in component called HPMR (High Performance MapReduce Engine). Our evaluation on the Yahoo!Grid platform with three different workloads and seven types of test sets from Yahoo! shows that HPMR reduces the execution time by up to 73%.

HPMR: Prefetching and pre-shuffling in shared MapReduce computation environment

Solid state disks (SSDs) have many advantages over hard disk drives, including better reliability, performance, durability, and power efficiency. However, the characteristics of SSDs are completely different from those of hard disk drives with rotating disks. To achieve the full potential performance improvement with SSDs, operating systems or applications must understand the critical performance parameters of SSDs to fine-tune their accesses. However, the internal hardware and software organizations vary significantly among SSDs and, thus, each SSD exhibits different parameters which influence the overall performance. In this paper, we propose a methodology which can extract several essential parameters affecting the performance of SSDs, and apply the extracted parameters to SSD systems for performance improvement. The target parameters of SSDs considered in this paper are 1) the size of read/write unit, 2) the size of erase unit, 3) the size of read buffer, and 4) the size of write buffer. We modify two operating system components to optimize their operations with the SSD parameters. The experimental results show that such parameter-aware management leads to significant performance improvements for large file accesses by performing SSD-specific optimizations.

Parameter-Aware I/O Management for Solid State Disks (SSDs)

Chemical synthesis based on the skeletal variation has been prolifically utilized as an attractive approach for modification of molecular properties. Given the ubiquity of unstrained cyclic amines, the ability to directly alter such motifs would grant an efficient platform to access unique chemical space. Here, we report a highly efficient and practical strategy that enables the selective ring-opening functionalization of unstrained cyclic amines. The use of difluorocarbene leads to a wide variety of multifaceted acyclic architectures, which can be further diversified to a range of distinctive homologative cyclic scaffolds. The virtue of this deconstructive strategy is demonstrated by successful modification of several natural products and pharmaceutical analogues.

/pdf/ring-opening-functionalizations-of-unstrained-cyclic-amines-16rr86ao3q.pdf

Ring-opening functionalizations of unstrained cyclic amines enabled by difluorocarbene transfer.

The formal hydroamination/hydroamidation utilizing metal hydride is an appealing synthetic tool for the construction of valuable nitrogen-containing compounds from unsaturated hydrocarbons. While significant advances have been made for the functionalizations of alkenes in this realm, the direct hydroamidation of alkynes remains rather limited due to the high feasibility of the key metal-alkenyl intermediate to choose other reaction pathways. Herein, we report a NiH-catalyzed strategy for the hydroamidation of alkynes with dioxazolones, which allows convenient access to synthetically useful secondary enamides in (E)-anti-Markovnikov or Markovnikov selectivity. The reaction is viable for both terminal and internal alkynes and is also tolerant with a range of subtle functional groups. With H2O found as an essential component for high catalyst turnovers, the involvement of inner-sphere nitrenoid transfer is proposed that outcompetes an undesired semireduction process, thus representing the first example to show the competence of Ni catalysis for metal-nitrenoid formation from dioxazolones.

Sangwon Seo

Papers

HAMA: An Efficient Matrix Computation with the MapReduce Framework

HPMR: Prefetching and pre-shuffling in shared MapReduce computation environment

Parameter-Aware I/O Management for Solid State Disks (SSDs)

Ring-opening functionalizations of unstrained cyclic amines enabled by difluorocarbene transfer.

Merging NiH Catalysis and Inner-Sphere Metal-Nitrenoid Transfer for Hydroamidation of Alkynes.