Workflow

About: Workflow is a research topic. Over the lifetime, 31996 publications have been published within this topic receiving 498339 citations.

Sustainable data analysis with Snakemake.

Abstract: Data analysis often entails a multitude of heterogeneous steps, from the application of various command line tools to the usage of scripting languages like R or Python for the generation of plots and tables. It is widely recognized that data analyses should ideally be conducted in a reproducible way. Reproducibility enables technical validation and regeneration of results on the original or even new data. However, reproducibility alone is by no means sufficient to deliver an analysis that is of lasting impact (i.e., sustainable) for the field, or even just one research group. We postulate that it is equally important to ensure adaptability and transparency. The former describes the ability to modify the analysis to answer extended or slightly different research questions. The latter describes the ability to understand the analysis in order to judge whether it is not only technically, but methodologically valid. Here, we analyze the properties needed for a data analysis to become reproducible, adaptable, and transparent. We show how the popular workflow management system Snakemake can be used to guarantee this, and how it enables an ergonomic, combined, unified representation of all steps involved in data analysis, ranging from raw data processing, to quality control and fine-grained, interactive exploration and plotting of final results.

System and method for building multi-modal and multi-channel applications

Abstract: A system and method are provided for visually building multi-channel and multi-modal applications. The system includes a process design module for designing application workflow, an integration design module for integrating data sources into the application; a presentation design module for designing application views; a media library; and a componentization module, for packaging designed workflow into reusable components. The system further includes an interactive development/design environment (IDE). The IDE provides a graphical user interface for allowing a developer to visually interact with and operate modules. The system allows a developer to design a single application that can operate across multiple network standards, devices, browsers and languages, and that operate in one or more modes, such as real-time, off-line and asynchronous modes.

Mapping Abstract Complex Workflows onto Grid Environments

Abstract: In this paper we address the problem of automatically generating job workflows for the Grid. These workflows describe the execution of a complex application built from individual application components. In our work we have developed two workflow generators: the first (the Concrete Workflow Generator CWG) maps an abstract workflow defined in terms of application-level components to the set of available Grid resources. The second generator (Abstract and Concrete Workflow Generator, ACWG) takes a wider perspective and not only performs the abstract to concrete mapping but also enables the construction of the abstract workflow based on the available components. This system operates in the application domain and chooses application components based on the application metadata attributes. We describe our current ACWG based on AI planning technologies and outline how these technologies can play a crucial role in developing complex application workflows in Grid environments. Although our work is preliminary, CWG has already been used to map high energy physics applications onto the Grid. In one particular experiment, a set of production runs lasted 7 days and resulted in the generation of 167,500 events by 678 jobs. Additionally, ACWG was used to map gravitational physics workflows, with hundreds of nodes onto the available resources, resulting in 975 tasks, 1365 data transfers and 975 output files produced.

Analyzing process models using graph reduction techniques

Abstract: The foundation of a process model lies in its structural specifications. Using a generic process modeling language for workflows, we show how a structural specification may contain deadlock and lack of synchronization conflicts that could compromise the correct execution of workflows. In general, identification of such conflicts is a computationally complex problem and requires development of effective algorithms specific for the target modeling language. We present a visual verification approach and algorithm that employs a set of graph reduction rules to identify structural conflicts in process models for the given workflow modeling language. We also provide insights into the correctness and complexity of the reduction process. Finally, we show how the reduction algorithm may be used to count possible instance subgraphs of a correct process model. The main contribution of the paper is a new technique for satisfying well-defined correctness criteria in process models. © 2000 Published by Elsevier Science Ltd. All rights reserved.

Workflow resource patterns: identification, representation and tool support

Abstract: In the main, the attention of workflow researchers and workflow developers has focussed on the process perspective, i.e., control-flow. As a result, issues associated with the resource perspective, i.e., the people and machines actually doing the work, have been largely neglected. Although the process perspective is of most significance, appropriate consideration of the resource perspective is essential for successful implementation of workflow technology. Previous work has identified recurring, generic constructs in the control-flow and data perspectives, and presented them in the form of control-flow and data patterns. The next logical step is to describe workflow resource patterns that capture the various ways in which resources are represented and utilised in workflows. These patterns include a number of distinct groupings such as push patterns (“the system pushes work to a worker”) and pull patterns (“the worker pulls work from the system”) to describe the many ways in which work can be distributed. By delineating these patterns in a form that is independent of specific workflow technologies and modelling languages, we are able to provide a comprehensive treatment of the resource perspective and we subsequently use these patterns as the basis for a detailed comparison of a number of commercially available workflow management systems.