This chapter covers advanced techniques in Big Data analytics and query processing. As the data is getting bigger and, at the same time, workloads and analytics are getting more complex, the advances in big data applications are no longer hindered by their ability to collect or generate data. But instead, by their ability to efficiently and effectively manage the available data. Therefore, numerous scalable and distributed infrastructures have been proposed to manage big data. However, it is well known in literature that scalability and distributed processing alone are not enough to achieve high performance. Instead, the underlying infrastructure has to be highly optimized for various types of workloads and query classes. These optimizations typically start from the lowest layer of the data management stack, which is the storage layer. In this chapter, we will cover two well-known techniques for optimized storage and organization of data that have big influence on query performance, namely the indexing, and data layout techniques. However, in the cases of non-traditional workloads where queries have special execution and data-access characteristics, the standard indexing and layout techniques may fall short in providing the desired performance goals. Therefore, further optimizations specific to the workload characteristics can be applied. In this chapter, we will cover techniques addressing several of these non-traditional workloads in the context of big data. Some of these techniques rely on curating either the data or the workflows (or both) with useful metadata information. This curation information can be very valuable for both query optimization and the business logic. In this chapter, we will cover the curation and metadata management of big data in query optimization and different systems. In this chapter, we focus on the MapReduce-like infrastructures, more specifically its open-source implementation Hadoop. The chapter covers the state-of-art in big data indexing techniques, and the data layout and organization strategies to speedup queries. It will also cover advanced techniques for enabling non-traditional workloads in Hadoop. Hadoop is primarily designed for workloads that are characterized by being batch, offline, ad-hoc, and disk-based. Yet, this chapter will cover recent projects and techniques targeting non-traditional workloads such as continuous query evaluation, main-memory processing, and recurring workloads. In addition, the chapter covers recent techniques proposed for data curation and efficient metadata management in Hadoop. These techniques vary from being semantic specific, e.g., provenance tracking techniques, to generic frameworks for data curation and annotation.

Data Organization and Curation in Big Data

ETL (extract, transform, dan load) merupakan proses yang dilibatkan dalam pembuatan dan manajemen data warehouse. Desain ETL dibuat menyesuaikan struktur sumber data dan data warehouse. Dengan adanya ketergantungan tersebut maka perubahan di sumber data bisa berdampak besar terhadap desain ETL. Ketika perubahan tersebut terjadi, pengelola ETL akan berkomunikasi dengan pemilik data untuk mengetahui rincian perubahan struktur data dalam rangka memperbaiki desain ETL. Aliran komunikasi ini akan semakin meningkat sejalan dengan jumlah sumber data yang digunakan. Semakin banyak sumber data yang diproses maka komunikasi tersebut berpotensi menjadi bottleneck. Informasi perubahan struktur data ini dapat dikomunikasikan melalui anotasi yang dilekatkan pada sumber data. Anotasi tersebut kemudian dipropagasikan sehingga dapat digunakan untuk memperbaiki rancangan ETL. Dengan menggunakan anotasi, harapannya aliran komunikasi antara pengelola ETL dengan pemilik data dapat berkurang. Permasalahan tersebut menunjukkan seberapa penting dikembangkannya sistem propagasi anotasi. Sistem propagasi anotasi tersusun atas tiga komponen yaitu ekstraksi metadata, propagasi anotasi, dan adapter. Pengujian sistem dilakukan menggunakan teknik blackbox dan user acceptance testing bersama pengguna akhir. Pengujian blackbox menghasilkan 30 kasus uji yang hasilnya valid. Hasil evaluasi user acceptance testing menunjukkan bahwa rata-rata pengguna menyatakan sangat setuju dengan sistem yang dikembangkan. AbstractETL is a process of extracting, transforming, and loading data that is involved in creation and management of a data warehouse. Since ETL is deeply connected to the structure of the source data, if a small changes happens to that structure then the whole workflow might stop. Because one data source can be used by more than one ETL workflow, the impact of schema changes to the ETL design are enormous. When such incident happens, the ETL designer will ask the data owner for the details of the schema changes. The communication traffic between the ETL designer and the data owner will increase as the number of sources that are being used is increasing. Therefore potentially becoming a bottleneck. Information regarding schema changes of a data source can be attached as an annotation. This annotation will be then propagated so that the ETL manager can update the workflow according to the recorded changes.Using this technique, the communication traffic between the ETL designer and the data owner can be minimized. This problem highlights the need of an annotation propagation system. The system itself consists of three components: metadata extraction, adapter, and annotation propagation. To test the system, blackbox and user acceptance testing is used. The blackbox testing resulting with 30 test case which are all valid. The user acceptance testing is done with the end-user directly operating the system, and after analyzing the results it shows that on average the user is accepting the system. 

Sistem Propagasi Anotasi pada Metadata Lineage untuk Manajemen Data Warehouse

In recent years, tremendous efforts have been made on document image rectification, but existing advanced algorithms are limited to processing restricted document images, i.e., the input images must incorporate a complete document. Once the captured image merely involves a local text region, its rectification quality is degraded and unsatisfactory. Our previously proposed DocTr, a transformer-assisted network for document image rectification, also suffers from this limitation. In this work, we present DocTr++, a novel unified framework for document image rectification, without any restrictions on the input distorted images. Our major technical improvements can be concluded in three aspects. Firstly, we upgrade the original architecture by adopting a hierarchical encoder-decoder structure for multi-scale representation extraction and parsing. Secondly, we reformulate the pixel-wise mapping relationship between the unrestricted distorted document images and the distortion-free counterparts. The obtained data is used to train our DocTr++ for unrestricted document image rectification. Thirdly, we contribute a real-world test set and metrics applicable for evaluating the rectification quality. To our best knowledge, this is the first learning-based method for the rectification of unrestricted document images. Extensive experiments are conducted, and the results demonstrate the effectiveness and superiority of our method. We hope our DocTr++ will serve as a strong baseline for generic document image rectification, prompting the further advancement and application of learning-based algorithms. The source code and the proposed dataset are publicly available at https://github.com/fh2019ustc/DocTr-Plus.

Deep Unrestricted Document Image Rectification

Data curation and annotation are indispensable mechanisms to a wide range of applications for capturing various types of metadata information. This metadata not only increases the data's credibility and merit, and allows end users and applications to make more informed decisions, but also enables advanced processing over the data that is not feasible otherwise. That is why annotation management has been extensively studied in the context of scientific repositories, web documents, and relational database systems. In this paper, we make the case that cloud-based applications that rely on the emerging Hadoop infrastructure are also in need for data curation and annotation and that the presence of such mechanisms in Hadoop would bring value-added capabilities to these applications. We propose the "CloudNotes" system, a full-fledged MapReduce-based annotation management engine. CloudNotes addresses several new challenges to annotation management including: (1) scalable and distributed processing of annotations over large clusters, (2) propagation of annotations under the MapReduce's blackbox execution model, and (3) annotation-driven optimizations ranging from proactive prefetching and colocation of annotations, annotation-aware task scheduling, novel shared execution strategies among the annotation jobs, and concurrency control mechanisms for annotation management. These challenges have not been addressed or explored before by the state-of-art technologies. CloudNotes is built on top of the open-source Hadoop/HDFS infrastructure and experimentally evaluated to demonstrate the practicality and scalability of its features, and the effectiveness of its optimizations under large workloads.

Yue Lu

Papers

Decorating the cloud: enabling annotation management in MapReduce

End-to-end learning of representations for instance-level document image retrieval