Abstract In industrial production, customers’ requirements are rising regarding various aspects. Products have to be produced more economical, more flexible, faster, and with much higher quality requirements. Furthermore, especially for traditional mass production processes, shorter product cycles increase the demand in rapid production and process development. The inherent increased product and production complexity raises additional challenges not only in development but also in setup and operation. Lastly, upcoming requirements for sustainable production have to be incorporated. These conflicting aspects lead to increasing complexity for production development as well as production setup at each individual production step as well as along the complete value chain. To master these challenges, digitalization and data-driven models are fundamental tools, since these allow for the automation of many basic tasks as well as processing of large data sets to achieve process understanding and derive appropriate measures. This chapter illustrates requirements for digital systems to be created and benefits derived by different novel systems. Furthermore, because modern systems have to incorporate not only single processes but complex process chains, various production processes and assembly processes are taken into account. In the following chapters, Ruppel et al. 2023; Lockner et al. 2023; Idzik et al 2023; Kluge-Wilkes at al. 2023 digitalization and Industry 4.0 approaches are presented, which incorporate data-driven models for a wide variety of production processes and for different time scales. Many techniques are illustrated to generate benefits on various levels due to the use of data-driven, model-based systems, which are incorporated into a digital infrastructure. 

https://link.springer.com/content/pdf/10.1007/978-3-030-98062-7_26-1.pdf

Internet of Production: Challenges, Potentials, and Benefits for Production Processes due to Novel Methods in Digitalization

Thanks to continuously evolving data management solutions, data-driven strategies are considered the main success factor in many domains. These strategies consider data as the backbone, allowing advanced data analytics. However, in the agricultural field, and especially in fish farming, data-driven strategies have yet to be widely adopted. This research paper aims to demystify the situation of the fish farming domain in general by shedding light on big data generated in fish farms. The purpose is to propose a dedicated data lake functional architecture and extend it to a technical architecture to initiate a fish farming data-driven strategy. The research opted for an exploratory study to explore the existing big data technologies and to propose an architecture applicable to the fish farming data-driven strategy. The paper provides a review of how big data technologies offer multiple advantages for decision making and enabling prediction use cases. It also highlights different big data technologies and their use. Finally, the paper presents the proposed architecture to initiate a data-driven strategy in the fish farming domain.

https://www.mdpi.com/2571-5577/6/1/8/pdf?version=1673591993

Data Lake Architecture for Smart Fish Farming Data-Driven Strategy

The digital transformation of the utility sector has resulted in a flood of data incoming from diverse and dispersed data sources, which requires huge integration, storage, processing, and management efforts. In this work, we present a Big Data advanced analytics platform for utility data, that allows for easier data retrieval, processing, and visualization, with enhanced data security. The successful implementation of Data Lake at DEWA (Dubai Electricity and Water Authority) R&D Center increases valuable insight extraction from raw and processed data, that can be employed in informed decision-making and cross-utilization of data between different sectors in the utility company. 

DEWA R&amp;D Data Lake: Big Data Platform for Advanced Energy Data Analytics

Web server access log files are text files containing important data about server activities, client requests addressed to a server, server responses, etc. Large-scale analysis of these data can contribute to various improvements in different areas of interest. The main problem lies in storing these files in their raw form, over long time, to allow analysis processes to be run at any time enabling information to be extracted as foundation for high quality decisions. Our research focuses on offering an economical, secure, and high-performance solution for the storage of large amount of raw log files. Proposed system implements a Data Lake (DL) architecture in cloud using Azure Data Lake Storage Gen2 (ADLS Gen2) for extract–load–transform (ELT) pipelines. This architecture allows large volumes of data to be stored in their raw form. Afterwards they can be subjected to transformation and advanced analysis processes without the need of a structured writing scheme. The main contribution of this paper is to provide a solution that is affordable and more accessible to perform web server access log data ingestion, storage and transformation over the newest technology, Data Lake. As derivative contribution, we proposed the use of Azure Blob Trigger Function to implement the algorithm of transforming log files into parquet files leading to 90% reduction in storage space compared to their original size. That means much lower storage costs than if they had been stored as log files. A hierarchical data storage model has also been proposed for shared access to data over different layers in the DL architecture, on top of which Data Lifecycle Management (DLM) rules have been proposed for storage cost efficiency. We proposed ingesting log files into a Data Lake deployed in cloud due to ease of deployment and low storage costs. The aim is to maintain this data in the long term, to be used in future advanced analytics processes by cross-referencing with other organizational or external data. That could bring important benefits. While the proposed solution is explicitly based on ADLS Gen2, it represents an important benchmark in approaching a cloud DL solution offered by any other vendor. 

/pdf/data-lake-architecture-for-storing-and-transforming-web-2senoew7.pdf

Data Lake Architecture for Storing and Transforming Web Server Access Log Files

The digital transformation of the utility sector has resulted in a flood of data incoming from diverse and dispersed data sources, which requires huge integration, storage, processing, and management efforts. In this work, we present a Big Data advanced analytics platform for utility data, that allows for easier data retrieval, processing, and visualization, with enhanced data security. The successful implementation of Data Lake at DEWA (Dubai Electricity and Water Authority) R&D Center increases valuable insight extraction from raw and processed data, that can be employed in informed decision-making and cross-utilization of data between different sectors in the utility company.

DEWA R&D Data Lake: Big Data Platform for Advanced Energy Data Analytics

Data is the lifeblood of any organization. In today’s world, organizations recognize the vital role of data in modern business intelligence systems for making meaningful decisions and staying competitive in the field. Efficient and optimal data analytics provides a competitive edge to its performance and services. Major organizations generate, collect and process vast amounts of data, falling under the category of big data. Managing and analyzing the sheer volume and variety of big data is a cumbersome process. At the same time, proper utilization of the vast collection of an organization’s information can generate meaningful insights into business tactics. In this regard, two of the popular data management systems in the area of big data analytics (i.e., data warehouse and data lake) act as platforms to accumulate the big data generated and used by organizations. Although seemingly similar, both of them differ in terms of their characteristics and applications. This article presents a detailed overview of the roles of data warehouses and data lakes in modern enterprise data management. We detail the definitions, characteristics and related works for the respective data management frameworks. Furthermore, we explain the architecture and design considerations of the current state of the art. Finally, we provide a perspective on the challenges and promising research directions for the future.

/pdf/an-overview-of-data-warehouse-and-data-lake-in-modern-vhom5vdh.pdf

Divyansh Mundra

Papers

An Overview of Data Warehouse and Data Lake in Modern Enterprise Data Management