Provenance in the context of workflows, both for the data they derive and for their specification, is an essential component to allow for result reproducibility, sharing, and knowledge re-use in the scientific community. Several workshops have been held on the topic, and it has been the focus of many research projects and prototype systems. This tutorial provides an overview of research issues in provenance for scientific workflows, with a focus on recent literature and technology in this area. It is aimed at a general database research audience and at people who work with scientific data and workflows. We will (1) provide a general overview of scientific workflows, (2) describe research on provenance for scientific workflows and show in detail how provenance is supported in existing systems; (3) discuss emerging applications that are enabled by provenance; and (4) outline open problems and new directions for database-related research.

/pdf/provenance-and-scientific-workflows-challenges-and-27dsk8tmf9.pdf

Provenance and scientific workflows: challenges and opportunities

The problem of systematically capturing and managing provenance for computational tasks has recently received significant attention because of its relevance to a wide range of domains and applications. The authors give an overview of important concepts related to provenance management, so that potential users can make informed decisions when selecting or designing a provenance solution.

/pdf/provenance-for-computational-tasks-a-survey-1to2p7ylnb.pdf

Provenance for Computational Tasks: A Survey

Software traceability is a fundamentally important task in software engineering. The need for automated traceability increases as projects become more complex and as the number of artifacts increases. We propose an automated technique that combines traceability with a machine learning technique known as topic modeling. Our approach automatically records traceability links during the software development process and learns a probabilistic topic model over artifacts. The learned model allows for the semantic categorization of artifacts and the topical visualization of the software system. To test our approach, we have implemented several tools: an artifact search tool combining keyword-based search and topic modeling, a recording tool that performs prospective traceability, and a visualization tool that allows one to navigate the software architecture and view semantic topics associated with relevant artifacts and architectural components. We apply our approach to several data sets and discuss how topic modeling enhances software traceability, and vice versa.

/pdf/software-traceability-with-topic-modeling-47ykds5986.pdf

Software traceability with topic modeling

Provenance, i.e., the origin or source of something, is becoming an important concern, since it offers the means to verify data products, to infer their quality, to analyse the processes that led to them, and to decide whether they can be trusted. For instance, provenance enables the reproducibility of scientific results; provenance is necessary to track attribution and credit in curated databases; and, it is essential for reasoners to make trust judgements about the information they use over the Semantic Web.

As the Web allows information sharing, discovery, aggregation, filtering and flow in an unprecedented manner, it also becomes very difficult to identify, reliably, the original source that produced an information item on the Web. Since the emerging use of provenance in niche applications is undoubtedly demonstrating the benefits of provenance, this monograph contends that provenance can and should reliably be tracked and exploited on the Web, and investigates the necessary foundations to achieve such a vision.

Multiple data sources have been used to compile the largest bibliographical database on provenance so far. This large corpus permits the analysis of emerging trends in the research community. Specifically, the CiteSpace tool identifies clusters of papers that constitute research fronts, from which characteristics are extracted to structure a foundational framework for provenance on the Web. Such an endeavour requires a multi-disciplinary approach, since it requires contributions from many computer science sub-disciplines, but also other non-technical fields given the human challenge that is anticipated.

To develop such a vision, it is necessary to provide a definition of provenance that applies to the Web context. A conceptual definition of provenance is expressed in terms of processes, and is shown to generalise various definitions of provenance commonly encountered. Furthermore, by bringing realistic distributed systems assumptions, this definition is refined as a query over assertions made by applications.

Given that the majority of work on provenance has been undertaken by the database, workflow and e-science communities, some of their work is reviewed, contrasting approaches, and focusing on important topics believed to be crucial for bringing provenance to the Web, such as abstraction, collections, storage, queries, workflow evolution, semantics and activities involving human interactions.

However, provenance approaches developed in the context of databases and workflows essentially deal with closed systems. By that, it is meant that workflow or database management systems are in full control of the data they manage, and track their provenance within their own scope, but not beyond. In the context of the Web, a broader approach is required by which chunks of provenance representation can be brought together to describe the provenance of information flowing across multiple systems. For this purpose, this monograph puts forward the Open Provenance Vision, which is an approach that consists of controlled vocabulary, serialisation formats and interfaces to allow the provenance of individual systems to be expressed, connected in a coherent fashion, and queried seamlessly. In this context, the Open Provenance Model is an emerging community-driven representation of provenance, which has been actively used by some 20 teams to exchange provenance information, in line with the Open Provenance Vision.

After identifying an open approach and a model for provenance, techniques to expose provenance over the Web are investigated. In particular, Semantic Web technologies are discussed since they have been successfully exploited to express, query and reason over provenance. Symmetrically, Semantic Web technologies such as RDF, underpinning the Linked Data effort, are analysed since they offer their own difficulties with respect to provenance.

A powerful argument for provenance is that it can help make systems transparent, so that it becomes possible to determine whether a particular use of information is appropriate under a set of rules. Such capability helps make systems and information accountable. To offer accountability, provenance itself must be authentic, and rely on security approaches, which are described in the monograph. This is then followed by systems where provenance is the basis of an auditing mechanism to check past processes against rules or regulations. In practice, not all users want to check and audit provenance, instead, they may rely on measures of quality or trust; hence, emerging provenance-based approaches to compute trust and quality of data are reviewed.

/pdf/the-foundations-for-provenance-on-the-web-sktzvgz5mp.pdf

The Foundations for Provenance on the Web

Nowadays, more and more computer-based scientific experiments need to handle massive amounts of data. Their data processing consists of multiple computational steps and dependencies within them. A data-intensive scientific workflow is useful for modeling such process. Since the sequential execution of data-intensive scientific workflows may take much time, Scientific Workflow Management Systems (SWfMSs) should enable the parallel execution of data-intensive scientific workflows and exploit the resources distributed in different infrastructures such as grid and cloud. This paper provides a survey of data-intensive scientific workflow management in SWfMSs and their parallelization techniques. Based on a SWfMS functional architecture, we give a comparative analysis of the existing solutions. Finally, we identify research issues for improving the execution of data-intensive scientific workflows in a multisite cloud.

/pdf/a-survey-of-data-intensive-scientific-workflow-management-4fndqykbu1.pdf

A Survey of Data-Intensive Scientific Workflow Management

In many data-driven applications, analysis needs to be performed on scientific information obtained from several sources and generated by computations on distributed resources. Systematic analysis of this scientific information unleashes a growing need for automated data-driven applications that also can keep track of the provenance of the data and processes with little user interaction and overhead. Such data analysis can be facilitated by the recent advancements in scientific workflow systems. A major profit when using scientific workflow systems is the ability to make provenance collection a part of the workflow. Specifically, provenance should include not only the standard data lineage information but also information about the context in which the workflow was used, execution that processed the data, and the evolution of the workflow design. In this paper we describe a complete framework for data and process provenance in the Kepler Scientific Workflow System. We outline the requirements and issues related to data and workflow provenance in a multi-disciplinary workflow system and introduce how generic provenance capture can be facilitated in Kepler's actor-oriented workflow environment. We also describe the usage of the stored provenance information for efficient rerun of scientific workflows.

/pdf/provenance-collection-support-in-the-kepler-scientific-34sq47gmus.pdf

Provenance collection support in the kepler scientific workflow system

H. S. Kim, J R. Arrowsmith, C. J. Crosby, E. Jaeger-Frank, V. Nandigam, A. Memon, J. Conner, S. B. Baden, C. Baru Department of Computer Science and Engineering, University of California, San Diego School of Earth and Space Exploration, Arizona State University San Diego Super Computer Center, University of California, San Diego The GEON LiDAR Workflow (GLW) project (http://www.geongrid.org/science/lidar.html) has enabled geoscientific communities to process and download large LiDAR (Light Distance And Ranging), or ALSM (Airborne Laser Swath Mapping) datasets. The products--digital elevation models (DEMs)--are essential representations of the landscape that are imported to various offthe-shelf geoscientific tools. Initially, the GLW utilized the regularized spline with tension interpolation algorithm for the DEM generation from GRASS GIS successfully for small datasets (<1.6 million points). However, it is compute-intensive; in order to process 5 million points, it takes about 50 minutes on the Linux machine in a GEON cluster. As a solution to the problem, a local binning algorithm which quickly generates a DEM is employed. The algorithm utilizes the elevation information from only the points inside of a circular search area with user specified radius (Figure 1). For each node in a grid, five values are computed: the minimum, maximum, mean, and inverse distance weighted mean of the local points, and the number of points in the search area. The noble implementation technique can produce a grid containing those five values within O(N) time, where N denotes the size of the LiDAR point cloud. This implementation runs over 5 million points in 30 seconds in a single machine, which is about 100 times speedup relative to the O(N) spline or related methods. Along with the original implementation, an out-of-core (memory) version of the local binning algorithm has also been developed. This implementation exploits secondary storage for saving intermediate results when the size of a grid exceeds that of memory. The large grid is split into several small pieces. During the computation, each piece is fetched from secondary storage to main memory one at a time when needed. This version can compute a DEM from 150 million points (in about 20 minutes), which is impossible in the spline interpolation and also in the original local binning algorithm implementation. In addition, the local binning algorithm computes useful products. The maximum values track the highest elevation points, which are usually recorded at the top of vegetation areas. On the other hand, the minimum values may represent the information bounced from the ground, which is useful for meter-scale shrubbery removal for which there will not usually be laser penetration. Lastly, the density map of an area is useful when determining the accuracy of interpolation and assessing the appropriate grid resolution (Figure 2). This suite of DEM products provides geoscientists rich information complementary to those of the current spline algorithm. We have also ported this code to Windows (GEON Points2Grid Utility). Links: http://lidar.asu.edu/points2grid.html (GEON Points2Grid Utility page); http://lidar.asu.edu/knowledgebase.html (prototype and supporting documentation); http://lidar.asu.edu/ (ASU LiDAR research).

An Efficient Implementation of a Local Binning Algorithm for Digital Elevation Model Generation of LiDAR/ALSM Dataset

Emerging Grid technologies enable solving scientific problems that involve large datasets and complex analyses. Coordinating distributed Grid resources and computational processes requires adaptable interfaces and tools that provide a modularized and configurable environment for accessing Grid clusters and executing high performance computational tasks. In addition, it is beneficial to make these tools available to the community in a unified framework through a shared cyberinfrastructure, or a portal, so scientists can focus on their scientific work and not be concerned with the implementation of the underlying infrastructure. In this paper we describe a scientific workflow approach to coordinate various resources as data analysis pipelines. We present a three tier architecture for LiDAR interpolation and analysis, a high performance processing of point intensive datasets, utilizing a portal, a scientific workflow engine and Grid technologies. Our proposed solution is available through the GEON portal and, though focused on LiDAR processing, is applicable to other domains as well.

/pdf/a-three-tier-architecture-for-lidar-interpolation-and-44ibxbiaeb.pdf

A three tier architecture for LiDAR interpolation and analysis

This work explores the challenges of embedding long-term preservation processes into existing video production workflows. The challenges are many: size of video data, bulk and parallel transfer into the archive, automation of the processes, elimination of human steps, automatic triggering of the preservation workflows, replication of content, automatic extraction of metadata, and most important mechanisms to ldquohardenrdquo the automation by providing recovery mechanisms in case of errors and discrepancies. We present production-quality solutions that automate this process and make use of Kepler scientific workflows and Storage Resource Broker (SRB) data grid environments.

A scientific workflow solution to the archiving of digital media

Emerging Grid technologies enable solving scientific problems that involve large datasets and complex analyses, which in the past were often considered difficult to solve. Coordinating distributed Grid resources and computational processes requires adaptable interfaces and tools that provide modularized and configurable environments for accessing Grid clusters and executing high performance computational tasks. Computationally intensive processes are also subject to a high risk of component failures and thus require close monitoring. In this paper we describe a scientific workflow approach to coordinate various resources via data analysis pipelines. We present a three tier architecture for LiDAR interpolation and analysis, a high performance processing of point intensive datasets, utilizing a portal, a scientific workflow engine and Grid technologies. Our proposed solution is available to the community in a unified framework through a shared cyberinfrastructure, the GEON portal, enabling scientists to focus on their scientific work and not be concerned with the implementation of the underlying infrastructure.

/pdf/a-three-tier-architecture-applied-to-lidar-processing-and-3cgimmy8ja.pdf

Efrat Jaeger-Frank

Papers

Provenance collection support in the kepler scientific workflow system

An Efficient Implementation of a Local Binning Algorithm for Digital Elevation Model Generation of LiDAR/ALSM Dataset

A three tier architecture for LiDAR interpolation and analysis

A scientific workflow solution to the archiving of digital media

A three tier architecture applied to LiDAR processing and monitoring