Big﻿Data﻿emerged﻿after﻿a﻿big﻿explosion﻿of﻿data﻿from﻿the﻿Web﻿2.0,﻿digital﻿sensors,﻿and﻿social﻿media﻿ applications﻿ such﻿ as﻿ Facebook,﻿ Twitter,﻿ etc.﻿ In﻿ this﻿ constant﻿ growth﻿ of﻿ data,﻿ many﻿ domains﻿ are﻿ influenced,﻿ especially﻿ the﻿decisional﻿ support﻿ system﻿domain,﻿where﻿ the﻿ integration﻿of﻿ processes﻿ should﻿be﻿adapted﻿to﻿support﻿this﻿huge﻿amount﻿of﻿data﻿to﻿improve﻿analysis﻿goals.﻿The﻿basic﻿purpose﻿ of﻿this﻿research﻿article﻿is﻿to﻿adapt﻿extract-transform-load﻿processes﻿with﻿Big﻿Data﻿technologies,﻿in﻿ order﻿ to﻿support﻿not﻿only﻿this﻿evolution﻿of﻿data﻿but﻿also﻿the﻿knowledge﻿discovery.﻿In﻿this﻿article,﻿ a﻿new﻿approach﻿called﻿Big﻿Dimensional﻿ETL﻿(BigDimETL)﻿is﻿suggested﻿to﻿deal﻿with﻿ETL﻿basic﻿ operations﻿and﻿take﻿into﻿account﻿the﻿multidimensional﻿structure.﻿In﻿order﻿to﻿accelerate﻿data﻿handling,﻿ the﻿MapReduce﻿paradigm﻿is﻿used﻿to﻿enhance﻿data﻿warehousing﻿capabilities﻿and﻿HBase﻿as﻿a﻿distributed﻿ storage﻿mechanism.﻿Experimental﻿results﻿confirm﻿that﻿the﻿ETL﻿operation﻿performs﻿well﻿especially﻿ with﻿adapted﻿operations. KEywORDS Big Data, ETL, Hbase, Map Reduce, Multidimensional Structure

Towards Extract-Transform-Load Operations in a Big Data context

Recently, a wide range of Web applications (e.g. DBPedia, Uniprot, and Probase) are built on top of vast RDF knowledge bases and using the SPARQL query language. The continuous growth of these knowledge bases led to the investigation of new paradigms and technologies for storing, accessing, and querying RDF data. In practice, modern big data systems (e.g, Hadoop, Spark) can handle vast relational repositories, however, their application in the Semantic Web context is still limited. One possible reason is that such frameworks rely on distributed systems, which are good for relational data, however, their performance on dealing with graph data models like RDF have not been well-studied yet. In this paper, we present a systematic comparison of there relevant RDF relational schemas, i.e., Single Statement Table, Property Tables or Vertically-Partitioned Tables queried using Apache Spark. We evaluate the performance of Spark SQL querying engine for processing SPARQL queries using three different storage back-ends, namely, PostgreSQL, Hive, and HDFS. For the latter one, we compare four different data formats (CSV, ORC, Avro, and Parquet). We drove our experiment using a representative query workloads from the SPBench benchmark scenario. The results of our experiments show many interesting insights about the impact of the relational encoding scheme, storage backends and storage formats on the performance of the query execution process.

/pdf/benchmarking-spark-sql-under-alliterative-rdf-relational-3pwkbg227f.pdf

Benchmarking Spark-SQL under Alliterative RDF Relational Storage Backends.

Data is a precious resource in today’s society, and is generated at an unprecedented and constantly growing pace. The need to store, analyze, and make data promptly available to a multitude of users introduces formidable challenges in modern software platforms. These challenges radically impacted the research fields that gravitate around data management and processing, with the introduction of distributed data-intensive systems that offer innovative programming models and implementation strategies to handle data characteristics such as its volume, the rate at which it is produced, its heterogeneity, and its distribution. Each data-intensive system brings its specific choices in terms of data model, usage assumptions, synchronization, processing strategy, deployment, guarantees in terms of consistency, fault tolerance, ordering. Yet, the problems data-intensive systems face and the solutions they propose are frequently overlapping. This paper proposes a unifying model that dissects the core functionalities of data-intensive systems, and discusses alternative design and implementation strategies, pointing out their assumptions and implications. The model offers a common ground to understand and compare highly heterogeneous solutions, with the potential of fostering cross-fertilization across research communities. We apply our model by classifying tens of systems: an exercise that brings to interesting observations on the current trends in the domain of data-intensive systems and suggests open research directions.

https://dl.acm.org/doi/pdf/10.1145/3604801

A Model and Survey of Distributed Data-Intensive Systems

The article is devoted to the review of such modern phenomena in the field of data storage and processing as Big Data and FAIR data. For Big Data, you will find an overview of the technologies used to work with them. And for FAIR data, their definition is given, and the current state of their development is described, including the Internet of FAIR Data & Services (IFDS).

Chapter 6 Big Data and FAIR Data for Data Science

In critical situations, making quick and precise decisions requires a rapid execution of a large amount of concurrent navigational and exploratory queries over collected data stored in repositories such as data warehouses. To satisfy the decision-maker's requirement, a deep understanding of the properties of these queries is necessary. In addition to their large-scale, they are ad-hoc, dynamic and highly interacted. By a quick analysis of these properties, we figure out that the first three are factual whereas the last one is behavioral. The literature has widely reported that the interaction of analytical queries has a crucial impact on selecting optimization structures (e.g., materialized views) in data storage systems. By keeping these four properties in mind, it becomes a necessity to find scalable and efficient data structures to simultaneously model them for better optimization of large-scale queries. In this paper, we first show the crucial role of the interaction phenomenon in optimizing concurrent data and mining queries by identifying its limited capacity in considering all factual properties. Secondly, we propose a dynamic hypergraph as a data structure to manage the four above properties and we show its great contribution in selecting materialized views. Finally, intensive experiments are conducted to evaluate the efficiency of our proposal and its connectivity with a commercial DBMS.

HYRAQ: optimizing large-scale analytical queries through dynamic hypergraphs

Apache Hive is an open-source relational database system for analytic big-data workloads. In this paper we describe the key innovations on the journey from batch tool to fully fledged enterprise data warehousing system. We present a hybrid architecture that combines traditional MPP techniques with more recent big data and cloud concepts to achieve the scale and performance required by today's analytic applications. We explore the system by detailing enhancements along four main axis: Transactions, optimizer, runtime, and federation. We then provide experimental results to demonstrate the performance of the system for typical workloads and conclude with a look at the community roadmap.

Deepak Jaiswal

Papers

Apache Hive: From MapReduce to Enterprise-grade Big Data Warehousing

Apache Hive: From MapReduce to Enterprise-grade Big Data Warehousing