Showing papers by "Michael Wilde published in 2013"

Swift/T: Large-scale application composition via distributed-memory dataflow processing

Abstract: Many scientific applications are conceptually built up from independent component tasks as a parameter study, optimization, or other search. Large batches of these tasks may be executed on high-end computing systems; however, the coordination of the independent processes, their data, and their data dependencies is a significant scalability challenge. Many problems must be addressed, including load balancing, data distribution, notifications, concurrent programming, and linking to existing codes. In this work, we present Swift/T, a programming language and runtime that enables the rapid development of highly concurrent, task-parallel applications. Swift/T is composed of several enabling technologies to address scalability challenges, offers a high-level optimizing compiler for user programming and debugging, and provides tools for binding user code in C/C++/Fortran into a logical script. In this work, we describe the Swift/T solution and present scaling results from the IBM Blue Gene/P and Blue Gene/Q.

The parallel system for integrating impact models and sectors (pSIMS)

Abstract: We present a framework for massively parallel simulations of climate impact models in agriculture and forestry: the parallel System for Integrating Impact Models and Sectors (pSIMS). This framework comprises a) tools for ingesting large amounts of data from various sources and standardizing them to a versatile and compact data type; b) tools for translating this standard data type into the custom formats required for point-based impact models in agriculture and forestry; c) a scalable parallel framework for performing large ensemble simulations on various computer systems, from small local clusters to supercomputers and even distributed grids and clouds; d) tools and data standards for reformatting outputs for easy analysis and visualization; and d) a methodology and tools for aggregating simulated measures to arbitrary spatial scales such as administrative districts (counties, states, nations) or relevant environmental demarcations such as watersheds and river-basins. We present the technical elements of this framework and the results of an example climate impact assessment and validation exercise that involved large parallel computations on XSEDE.

Turbine: A Distributed-memory Dataflow Engine for High Performance Many-task Applications

Abstract: Efficiently utilizing the rapidly increasing concurrency of multi-petaflop computing systems is a significant programming challenge One approach is to structure applications with an upper layer of many loosely coupled coarse-grained tasks, each comprising a tightly-coupled parallel function or program “Many-task” programming models such as functional parallel dataflow may be used at the upper layer to generate massive numbers of tasks, each of which generates significant tightly coupled parallelism at the lower level through multithreading, message passing, and/or partitioned global address spaces At large scales, however, the management of task distribution, data dependencies, and intertask data movement is a significant performance challenge In this work, we describe Turbine, a new highly scalable and distributed many-task dataflow engine Turbine executes a generalized many-task intermediate representation with automated self-distribution and is scalable to multi-petaflop infrastructures We present here the architecture of Turbine and its performance on highly concurrent systems

Swift/T: scalable data flow programming for many-task applications

Abstract: Swift/T, a novel programming language implementation for highly scalable data flow programs, is presented.

MTC envelope: defining the capability of large scale computers in the context of parallel scripting applications

Abstract: Many scientific applications can be efficiently expressed with the parallel scripting (many-task computing, MTC) paradigm. These applications are typically composed of several stages of computation, with tasks in different stages coupled by a shared file system abstraction. However, we often see poor performance when running these applications on large scale computers due to the applications' frequency and volume of filesystem I/O and the absence of appropriate optimizations in the context of parallel scripting applications. In this paper, we show the capability of existing large scale computers to run parallel scripting applications by first defining the MTC envelope and then evaluating the envelope by benchmarking a suite of shared filesystem performance metrics. We also seek to determine the origin of the performance bottleneck by profiling the parallel scripting applications' I/O behavior and mapping the I/O operations to the MTC envelope. We show an example shared filesystem envelope and present a method to predict the I/O performance given the applications' level of I/O concurrency and I/O amount. This work is instrumental in guiding the development of parallel scripting applications to make efficient use of existing large scale computers, and to evaluate performance improvements in the hardware/software stack that will better facilitate parallel scripting applications.

Dataflow coordination of data-parallel tasks via MPI 3.0

Abstract: Scientific applications are often complex collections of many large-scale tasks. Mature tools exist for describing task-parallel workflows consisting of serial tasks, and a variety of tools exist for programming a single data-parallel operation. However, few tools cover the intersection of these two models. In this work, we extend the load balancing library ADLB to support parallel tasks. We demonstrate how applications can easily be composed of parallel tasks using Swift dataflow scripts, which are compiled to ADLB programs with performance comparable to hand-coded equivalents. By combining this framework with data-parallel analysis libraries, we are able to dynamically execute many instances of a parallel data analysis application in support of a parameter exploration workload.

Enabling multi-task computation on Galaxy-based gateways using swift

Abstract: The Galaxy science portal is a popular gateway to data analysis and computational tools for a broad range of life sciences communities. While Galaxy enables users to overcome the complexities of integrating diverse tools into unified workflows, it has only limited capabilities to execute those tools on the parallel and often distributed high-performance resources that the life sciences fields increasingly requires. We outline here an approach to meet this pressing requirement with the Swift parallel scripting language and its distributed runtime system. Swift's model of computation - implicitly parallel functional dataflow - is an elemental abstraction to which the core computing model of Galaxy maps very closely. We describe an integration between Galaxy and Swift that is transforming Galaxy into a much more powerful science gateway, retaining its user-friendly nature while extending its power to execute highly scalable workflows on diverse parallel environments.

JETS: Language and System Support for Many-Parallel-Task Workflows

Abstract: Many-task computing is a well-established paradigm for implementing loosely coupled applications (tasks) on large-scale computing systems. However, few of the model's existing implementations provide efficient, low-latency support for executing tasks that are tightly coupled multiprocessing applications. Thus, a vast array of parallel applications cannot readily be used effectively within many-task workloads. In this work, we present JETS, a middleware component that provides high performance support for many-parallel-task computing (MPTC). JETS is based on a highly concurrent approach to parallel task dispatch and on new capabilities now available in the MPICH2 MPI implementation and the ZeptoOS Linux operating system. JETS represents an advance over the few known examples of multilevel many-parallel-task scheduling systems: it more efficiently schedules and launches many short-duration parallel application invocations; it overcomes the challenges of coupling the user processes of each multiprocessing application invocation via the messaging fabric; and it concurrently manages many application executions in various stages. We report here on the JETS architecture and its performance on both synthetic benchmarks and an MPTC application in molecular dynamics.

Provenance traces of the swift parallel scripting system

Abstract: In this abstract, we describe provenance traces generated from executions of scientific workflows managed by the Swift parallel scripting system. They follow a provenance data model, used by MTCProv, the provenance management component of Swift. It is similar to PROV, representing most of its core concepts and including additional information about the scientific domain, computational resource consumption, and prospective provenance. We describe provenance queries that follow patterns commonly found in high performance computing and that are straightforward to support with MTCProv's built-in procedures. These queries often involve costly relational join operations and recursion, providing a relevant case for benchmarking.