This is a thought piece on data-intensive science requirements for databases and science centers. It argues that peta-scale datasets will be housed by science centers that provide substantial storage and processing for scientists who access the data via smart notebooks. Next-generation science instruments and simulations will generate these peta-scale datasets. The need to publish and share data and the need for generic analysis and visualization tools will finally create a convergence on common metadata standards. Database systems will be judged by their support of these metadata standards and by their ability to manage and access peta-scale datasets. The procedural stream-of-bytes-file-centric approach to data analysis is both too cumbersome and too serial for such large datasets. Non-procedural query and analysis of schematized self-describing data is both easier to use and allows much more parallelism.

Scientific Data Management in the Coming Decade

Over the last two decades, water smart metering programs have been launched in a number of medium to large cities worldwide to nearly continuously monitor water consumption at the single household level. The availability of data at such very high spatial and temporal resolution advanced the ability in characterizing, modeling, and, ultimately, designing user-oriented residential water demand management strategies. Research to date has been focusing on one or more of these aspects but with limited integration between the specialized methodologies developed so far. This manuscript is the first comprehensive review of the literature in this quickly evolving water research domain. The paper contributes a general framework for the classification of residential water demand modeling studies, which allows revising consolidated approaches, describing emerging trends, and identifying potential future developments. In particular, the future challenges posed by growing population demands, constrained sources of water supply and climate change impacts are expected to require more and more integrated procedures for effectively supporting residential water demand modeling and management in several countries across the world. We review high resolution residential water demand modeling studies.We provide a classification of existing technologies and methodologies.We identify current trends, challenges and opportunities for future development.

/pdf/benefits-and-challenges-of-using-smart-meters-for-advancing-3nblgijm2n.pdf

Benefits and challenges of using smart meters for advancing residential water demand modeling and management

The growing interest in process mining is fueled by the increasing availability of event data. Process mining techniques use event logs to automatically discover process models, check conformance, identify bottlenecks and deviations, suggest improvements, and predict processing times. Lion's share of process mining research has been devoted to analysis techniques. However, the proper handling of problems and challenges arising in analyzing event logs used as input is critical for the success of any process mining effort. In this paper, we identify four categories of process characteristics issues that may manifest in an event log (e.g. process problems related to event granularity and case heterogeneity) and 27 classes of event log quality issues (e.g., problems related to timestamps in event logs, imprecise activity names, and missing events). The systematic identification and analysis of these issues calls for a consolidated effort from the process mining community. Five real-life event logs are analyzed to illustrate the omnipresence of process and event log issues. We hope that these findings will encourage systematic logging approaches (to prevent event log issues), repair techniques (to alleviate event log issues) and analysis techniques (to deal with the manifestation of process characteristics in event logs).

/pdf/wanna-improve-process-mining-results-3y9r0dkz6t.pdf

Wanna improve process mining results

Data is part of our everyday life and an essential asset in numerous businesses and organizations. The quality of the data, i.e., the degree to which the data characteristics fulfill requirements, can have a tremendous impact on the businesses themselves, the companies, or even in human lives. In fact, research and industry reports show that huge amounts of capital are spent to improve the quality of the data being used in many systems, sometimes even only to understand the quality of the information in use. Considering the variety of dimensions, characteristics, business views, or simply the specificities of the systems being evaluated, understanding how to measure data quality can be an extremely difficult task. In this paper we survey the state of the art in classification of poor data, including the definition of dimensions and specific data problems, we identify frequently used dimensions and map data quality problems to the identified dimensions. The huge variety of terms and definitions found suggests that further standardization efforts are required. Also, data quality research on Big Data appears to be in its initial steps, leaving open space for further research.

A Survey on Data Quality: Classifying Poor Data

The software processes that facilitate the original loading and the periodic refreshment of the data warehouse contents are commonly known as Extraction-Transformation-Loading (ETL) processes. The intention of this survey is to present the research work in the field of ETL technology in a structured way. To this end, we organize the coverage of the field as follows: (a) first, we cover the conceptual and logical modeling of ETL processes, along with some design methods, (b) we visit each stage of the E-T-L triplet, and examine problems that fall within each of these stages, (c) we discuss problems that pertain to the entirety of an ETL process, and, (d) we review some research prototypes of academic origin. [Article copies are available for purchase from InfoSci-on-Demand.com]

A Survey of Extract-Transform-Load Technology.

Data quality tools aim at detecting and correcting data problems that affect the accuracy and efficiency of data analysis applications. We propose a classification of the most relevant commercial and research data quality tools that can be used as a framework for comparing tools and understand their functionalities.

A Survey of Data Quality Tools.

Digital preservation aims to keep digital objects accessible over long periods of time, ensuring the authenticity and integrity of these digital objects. In such complex environments, Risk Management is a key factor in assuring the normal behaviour of systems over time. Currently, the digital preservation arena commonly uses Risk Management concepts to assess repositories. In this paper, we intend to go further and propose a perspective where Risk Management can be used not only to assess existing solutions, but also to conceive digital preservation environments. Thus, we propose a Risk Management-based approach to design and assess digital preservation environments, including: • the definition of context and identification of strategic objectives to determine specific requirements and characterize which consequences are acceptable within the identified context; • the identification, analysis and evaluation of threats and vulnerabilities that may affect the normal behaviour of a specific business or the achievement of the goals and conformance to the requirements identified in the context characterization; and, • definition of actions to deal with the risks associated with the identified threats and vulnerabilities. We generalize and survey the main requirements, threats, vulnerabilities and techniques that can be applied in the scope of digital preservation.

/pdf/designing-digital-preservation-solutions-a-risk-management-597k0r0e2r.pdf

Designing Digital Preservation Solutions: A Risk Management-Based Approach

Digital preservation aims at maintaining digital objects accessible over a long period of time, regardless of the challenges of organizational or technological changes or failures. In particular, data produced in e-Science domains could be reliably stored in today's data grids, taking advantage of the natural properties of this kind of infrastructure to support redundancy. However, to achieve reliability we must take into account failure interdependency. Taking into account the fact that correlated failures can affect multiple components and potentially cause complete loss of data, we propose a solution to evaluate redundancy strategies in the context of heterogeneous environments such as data grids. This solution is based on a simulation engine that can be used not only to support the process of designing the preservation environment and related policies, but also later on to observe and control the deployed system.

Preserving digital data in heterogeneous environments

Enterprise Architecture approaches are used to provide rigorous descriptions of the organization-wide environment, manage the alignment of deployed services to the organization's mission, end ensure a clear separation of the concerns addressed in an architecture. Thus, an effective Enterprise Architecture approach assists in the management of relations and dependencies of any components of the organization environment and supports the integration and evolution of the architecture. However, the quality of that approach is strongly influenced by the precision of the architecture context description, a fact which is not always recognized. This paper focuses on the architecture context description and addresses the gap between the stakeholders' concerns and the resulting architecture. Based on a combination of established references and standards, we show how an explicit integration of the architecture context into the architecture model improves the linking of concerns and key elements of the architecture vision. We apply our approach to a subject of increasing concern in the Information Systems area: longevity of information. Digital preservation is an interdisciplinary problem, but existent initiatives address it in a very domain-centric way, making it impossible to integrate documented knowledge into an overall organization architecture. We analyze several references and models and derive a description of the architecture context and a capability model that supports incremental development through an explicit distinction between systems and their capabilities. The presented approach allows not just any organization to assess their current digital preservation awareness and evolve their architectures to address this challenge, but in particular demonstrates the added value of an explicit architecture context model in an Enterprise Architecture approach.

/pdf/modeling-contextual-concerns-in-enterprise-architecture-6fb5cgzq8d.pdf

Modeling Contextual Concerns in Enterprise Architecture

The goal of Risk Management activities is to define prevention and control mechanisms to address the risks attached to specify activities and valuable assets. Many Risk Management efforts operate in silos with narrowly focused, functionally driven, and disjointed activities. That fact leads to a fragmented view of risks, where each activity uses its own language, customs and metrics. The lack of interconnection and holistic view of risks limits an organization-wide perception of risks, where interdependent risks are not anticipated, controlled or managed. In order to address the Risk Management interoperability and standardization issues, this paper proposes an alignment between Risk Management, Governance and Enterprise Architecture activities, providing a systematic support to map and trace identified risks to enterprise artifacts modeled within the Enterprise Architecture, supporting the overall strategy of any organization. We discuss the main relationships between Risk Management and Enterprise Architecture and propose an architecture to integrate risks concerns into the overall organization environment.

/pdf/manage-risks-through-the-enterprise-architecture-2xsa4tojsp.pdf

José Barateiro

Papers

A Survey of Data Quality Tools.

Designing Digital Preservation Solutions: A Risk Management-Based Approach

Preserving digital data in heterogeneous environments

Modeling Contextual Concerns in Enterprise Architecture

Manage Risks through the Enterprise Architecture