Growing main memory capacity has fueled the development of in-memory big data management and processing. By eliminating disk I/O bottleneck, it is now possible to support interactive data analytics. However, in-memory systems are much more sensitive to other sources of overhead that do not matter in traditional I/O-bounded disk-based systems. Some issues such as fault-tolerance and consistency are also more challenging to handle in in-memory environment. We are witnessing a revolution in the design of database systems that exploits main memory as its data storage layer. Many of these researches have focused along several dimensions: modern CPU and memory hierarchy utilization, time/space efficiency, parallelism, and concurrency control. In this survey, we aim to provide a thorough review of a wide range of in-memory data management and processing proposals and systems, including both data storage systems and data processing frameworks. We also give a comprehensive presentation of important technology in memory management, and some key factors that need to be considered in order to achieve efficient in-memory data management and processing.

/pdf/in-memory-big-data-management-and-processing-a-survey-1k2b0h2jut.pdf

In-Memory Big Data Management and Processing: A Survey

https://cyberleninka.org/article/n/325149.pdf

Reference Architecture and Classification of Technologies, Products and Services for Big Data Systems

We live in the golden age of distributed computing. Public cloud platforms now offer virtually unlimited compute and storage resources on demand. At the same time, the Software-as-a-Service (SaaS) model brings enterprise-class systems to users who previously could not afford such systems due to their cost and complexity. Alas, traditional data warehousing systems are struggling to fit into this new environment. For one thing, they have been designed for fixed resources and are thus unable to leverage the cloud's elasticity. For another thing, their dependence on complex ETL pipelines and physical tuning is at odds with the flexibility and freshness requirements of the cloud's new types of semi-structured data and rapidly evolving workloads. We decided a fundamental redesign was in order. Our mission was to build an enterprise-ready data warehousing solution for the cloud. The result is the Snowflake Elastic Data Warehouse, or "Snowflake" for short. Snowflake is a multi-tenant, transactional, secure, highly scalable and elastic system with full SQL support and built-in extensions for semi-structured and schema-less data. The system is offered as a pay-as-you-go service in the Amazon cloud. Users upload their data to the cloud and can immediately manage and query it using familiar tools and interfaces. Implementation began in late 2012 and Snowflake has been generally available since June 2015. Today, Snowflake is used in production by a growing number of small and large organizations alike. The system runs several million queries per day over multiple petabytes of data. In this paper, we describe the design of Snowflake and its novel multi-cluster, shared-data architecture. The paper highlights some of the key features of Snowflake: extreme elasticity and availability, semi-structured and schema-less data, time travel, and end-to-end security. It concludes with lessons learned and an outlook on ongoing work.

https://dl.acm.org/doi/pdf/10.1145/2882903.2903741

The Snowflake Elastic Data Warehouse

Most "Big Data" systems are written in managed languages, such as Java, C#, or Scala. These systems suffer from severe memory problems due to the massive volume of objects created to process input data. Allocating and deallocating a sea of data objects puts a severe strain on existing garbage collectors (GC), leading to high memory management overheads and reduced performance.This paper describes the design and implementation of Yak, a "Big Data" friendly garbage collector that provides high throughput and low latency for all JVM-based languages. Yak divides the managed heap into a control space (CS) and a data space (DS), based on the observation that a typical data-intensive system has a clear distinction between a control path and a data path. Objects created in the control path are allocated in the CS and subject to regular tracing GC. The lifetimes of objects in the data path often align with epochs creating them. They are thus allocated in the DS and subject to region-based memory management. Our evaluation with three large systems shows very positive results.

/pdf/yak-a-high-performance-big-data-friendly-garbage-collector-3zgs6s4f2x.pdf

Yak: a high-performance big-data-friendly garbage collector

With the influence and social ripple effect of social media sites, diverse studies are in progress to analyze the contents generated by users. Numerous contents generated in real time contain information about social issues and events such as natural disasters. In particular, users show not only information about the events that occurred but also their sentiments. In this paper, we propose Polaris, a system for analyzing and predicting users’ sentimental trajectories for events analyzed in real time out of the massive social media contents, and show the results of preliminary validation work that we have done. We show both trajectory analysis and sentiment analysis so that users can obtain the insight at a glance. Also, we increased the accuracy in sentiment analysis and prediction by making use of the latest deep-learning technique.

Social media contents based sentiment analysis and prediction system

AsterixDB is a new, full-function BDMS (Big Data Management System) with a feature set that distinguishes it from other platforms in today's open source Big Data ecosystem. Its features make it well-suited to applications like web data warehousing, social data storage and analysis, and other use cases related to Big Data. AsterixDB has a flexible NoSQL style data model; a query language that supports a wide range of queries; a scalable runtime; partitioned, LSM-based data storage and indexing (including B+-tree, R-tree, and text indexes); support for external as well as natively stored data; a rich set of built-in types; support for fuzzy, spatial, and temporal types and queries; a built-in notion of data feeds for ingestion of data; and transaction support akin to that of a NoSQL store.Development of AsterixDB began in 2009 and led to a mid-2013 initial open source release. This paper is the first complete description of the resulting open source AsterixDB system. Covered herein are the system's data model, its query language, and its software architecture. Also included are a summary of the current status of the project and a first glimpse into how AsterixDB performs when compared to alternative technologies, including a parallel relational DBMS, a popular NoSQL store, and a popular Hadoop-based SQL data analytics platform, for things that both technologies can do. Also included is a brief description of some initial trials that the system has undergone and the lessons learned (and plans laid) based on those early "customer" engagements.

/pdf/asterixdb-a-scalable-open-source-bdms-45xsjf3ec5.pdf

AsterixDB: a scalable, open source BDMS

AsterixDB is a new, full-function BDMS (Big Data Management System) with a feature set that distinguishes it from other platforms in today's open source Big Data ecosystem. Its features make it well-suited to applications like web data warehousing, social data storage and analysis, and other use cases related to Big Data. AsterixDB has a flexible NoSQL style data model; a query language that supports a wide range of queries; a scalable runtime; partitioned, LSM-based data storage and indexing (including B+-tree, R-tree, and text indexes); support for external as well as natively stored data; a rich set of built-in types; support for fuzzy, spatial, and temporal types and queries; a built-in notion of data feeds for ingestion of data; and transaction support akin to that of a NoSQL store. 
Development of AsterixDB began in 2009 and led to a mid-2013 initial open source release. This paper is the first complete description of the resulting open source AsterixDB system. Covered herein are the system's data model, its query language, and its software architecture. Also included are a summary of the current status of the project and a first glimpse into how AsterixDB performs when compared to alternative technologies, including a parallel relational DBMS, a popular NoSQL store, and a popular Hadoop-based SQL data analytics platform, for things that both technologies can do. Also included is a brief description of some initial trials that the system has undergone and the lessons learned (and plans laid) based on those early "customer" engagements.

AsterixDB: A Scalable, Open Source BDMS

Memory disaggregation has replaced the landscape of dat-acenters by physically separating compute and memory nodes, achieving improved utilization. As early efforts, kernel paging-based approaches offer transparent virtual memory abstraction for remote memory with paging schemes but suffer from expensive page fault handling. This paper revisits the paging-based approaches and challenges their performance in paging schemes. We posit that the overhead of the paging-based approaches is not a fundamental limitation. We propose DiLOS, a new library operating system (LibOS) specialized for paging-based memory disaggregation. We have revamped the page fault handler to get away with the swap cache and incorporated known techniques in our prefetcher, page manager, and communication module for performance optimization. Furthermore, we provide APIs to augment the LibOS with application semantics. We present two app-aware guides, app-aware prefetching and bandwidth-reducing memory allocator in DiLOS. Through extensive evaluation of microbenchmarks and applications, we demonstrate that DiLOS outperforms the state-of-the-art kernel paging-based system (Fastswap) up to 2.24× and a recent user-level system (AIFM) 1.54× on a real-world data analytic workload.

DiLOS: Do Not Trade Compatibility for Performance in Memory Disaggregation

온라인 모바일 게임 운영에 있어서 이상 유저를 모니터링하는 것은 건전한 게임 생태계와 유저의 게임 이용 경험 향상을 위해 매우 중요하다. 그러나 이상 유저 탐지와 모니터링은 인적 자원의 직접적 투입을 필요로 하는 작업이므로, 이상 유저의 후보군을 좁히는 기법이 필요하다. 본 연구진은 모바일 게임 로그를 활용해, 한 국가에서 다른 국가로 짧은 시간 내에 이동한 것으로 나타나는 유저를 발견하고 이러한 현상에 대해 분석하였다. 유저의 갑작스런 이동을 설명할 수 있는 가장 유력한 시나리오는 VPN의 사용이다. VPN은 유저의 실제 IP 주소가 콘텐츠 제공자 서버와 같은 상대 종단 측에 드러나지 않게 해주며, 그렇기에 서버 측에서는 VPN 이용자와 그렇지 않은 이용자를 구분할 수 없다. 본 논문에서는 유저의 접근 기록과 위치 정보를 활용하여 이상 유저의 후보군 범위를 좁히는 기법을 제시하며, 로그 데이터 분석을 통해 약 2.01%의 유저가 모바일 게임에 VPN을 활용한 것으로 추론하였다. 

Ji Mahn Ok

Papers

AsterixDB: a scalable, open source BDMS

AsterixDB: A Scalable, Open Source BDMS

DiLOS: Do Not Trade Compatibility for Performance in Memory Disaggregation

Mining VPN Users from Network Logs of Commercial Games