Many scientific disciplines are now data and information driven, and new scientific knowledge is often gained by scientists putting together data analysis and knowledge discovery “pipelines”. A related trend is that more and more scientific communities realize the benefits of sharing their data and computational services, and are thus contributing to a distributed data and computational community infrastructure (a.k.a. “the Grid”). However, this infrastructure is only a means to an end and scientists ideally should be bothered little with its existence. The goal is for scientists to focus on development and use of what we call scientific workflows. These are networks of analytical steps that may involve, e.g., database access and querying steps, data analysis and mining steps, and many other steps including computationally intensive jobs on high performance cluster computers. In this paper we describe characteristics of and requirements for scientific workflows as identified in a number of our application projects. We then elaborate on Kepler, a particular scientific workflow system, currently under development across a number of scientific data management projects. We describe some key features of Kepler and its underlying Ptolemyii system, planned extensions, and areas of future research. Kepler is a communitydriven, open source project, and we always welcome related projects and new contributors to join.

/pdf/scientific-workflow-management-and-the-kepler-system-5a9uicngu7.pdf

Scientific Workflow Management and the Kepler System

1. Weinstein, J.N. et al. Nat. Genet. 45, 1113–1120 (2013). 2. Zhang, J. et al. Database. http://dx.doi.org/10.1093/ database/bar026 (2011) 3. Siva, N. Lancet 385, 103–104 (2015). 4. McKenna, A. et al. Genome Res. 20, 1297–1303 (2010). 5. UNC Bioinformatics. TCGA mRNA-seq pipeline for UNC data. https://webshare.bioinf.unc.edu/public/ mRNAseq_TCGA/UNC_mRNAseq_summary.pdf (2013). 6. Albrecht, M., Michael, A., Patrick, D., Peter, B. & Douglas, T. in Proceedings of the 1st ACM SIGMOD Workshop on Scalable Workflow Execution Engines and Technologies (SWEET ’12) 1. ACM (Association of Computing Machinery. http://dx.doi. org/10.1145/2443416.2443417 (2012). 7. Bernhardsson, E. & Frieder, E. Luigi. Github https:// github.com/spotify/luigi (2016). 8. Goecks, J., Nekrutenko, A. & Taylor, J. Genome Biol. 11, R86 (2010). 9. UCSC. Xena http://xena.ucsc.edu (2016). comprehensive analyses. Further, it means that results can be reproduced using the original computation’s set of tools and parameters. If we had run the original TCGA best-practices RNA-seq pipeline with one sample per node, it would have cost ~$800,000. Through the use of efficient algorithms (STAR and Kallisto) and Toil, we were able to reduce the final cost to $26,071 (Supplementary Note 9). We have demonstrated the utility of Toil by creating one of the single largest, consistently analyzed, public human RNA-seq expression repositories, which we hope the community will find useful.

Nextflow enables reproducible computational workflows

Modern embedded computing systems tend to be heterogeneous in the sense of being composed of subsystems with very different characteristics, which communicate and interact in a variety of ways-synchronous or asynchronous, buffered or unbuffered, etc. Obviously, when designing such systems, a modeling language needs to reflect this heterogeneity. Today's modeling environments usually offer a variant of what we call amorphous heterogeneity to address this problem. This paper argues that modeling systems in this manner leads to unexpected and hard-to-analyze interactions between the communication mechanisms and proposes a more structured approach to heterogeneity, called hierarchical heterogeneity, to solve this problem. It proposes a model structure and semantic framework that support this form of heterogeneity, and discusses the issues arising from heterogeneous component interaction and the desire for component reuse. It introduces the notion of domain polymorphism as a way to address these issues.

/pdf/taming-heterogeneity-the-ptolemy-approach-v4qpoppcbo.pdf

Taming heterogeneity - the Ptolemy approach

The most visible use of computers and software is processing information for human consumption. The vast majority of computers in use, however, are much less visible. They run the engine, brakes, seatbelts, airbag, and audio system in your car. They digitally encode your voice and construct a radio signal to send it from your cell phone to a base station. They command robots on a factory floor, power generation in a power plant, processes in a chemical plant, and traffic lights in a city. These less visible computers are called embedded systems, and the software they run is called embedded software. The principal challenges in designing and analyzing embedded systems stem from their interaction with physical processes. This book takes a cyber-physical approach to embedded systems, introducing the engineering concepts underlying embedded systems as a technology and as a subject of study. The focus is on modeling, design, and analysis of cyber-physical systems, which integrate computation, networking, and physical processes. The second edition offers two new chapters, several new exercises, and other improvements. The book can be used as a textbook at the advanced undergraduate or introductory graduate level and as a professional reference for practicing engineers and computer scientists. Readers should have some familiarity with machine structures, computer programming, basic discrete mathematics and algorithms, and signals and systems.

Introduction to Embedded Systems - A Cyber-Physical Systems Approach

We review a model of computation used in industrial practice in signal processing software environments and experimentally and other contexts. We give this model the name "dataflow process networks," and study its formal properties as well as its utility as a basis for programming language design. Variants of this model are used in commercial visual programming systems such as SPW from the Alta Group of Cadence (formerly Comdisco Systems), COSSAP from Synopsys (formerly Cadis), the DSP Station from Mentor Graphics, and Hypersignal from Hyperception. They are also used in research software such as Khoros from the University of New Mexico and Ptolemy from the University of California at Berkeley, among many others. Dataflow process networks are shown to be a special case of Kahn process networks, a model of computation where a number of concurrent processes communicate through unidirectional FIFO channels, where writes to the channel are nonblocking, and reads are blocking. In dataflow process networks, each process consists of repeated "firings" of a dataflow "actor." An actor defines a (often functional) quantum of computation. By dividing processes into actor firings, the considerable overhead of context switching incurred in most implementations of Kahn process networks is avoided. We relate dataflow process networks to other dataflow models, including those used in dataflow machines, such as static dataflow and the tagged-token model. We also relate dataflow process networks to functional languages such as Haskell, and show that modern language concepts such as higher-order functions and polymorphism can be used effectively in dataflow process networks. A number of programming examples using a visual syntax are given. >

/pdf/dataflow-process-networks-3mdwy0uhx2.pdf

Dataflow process networks

We present a scheduling policy for complete, bounded execution of Kahn process network programs. A program is a set of processes that communicate through a network of first-in first-out queues. In a complete execution, the program terminates if and only if all processes block attempting to consume data from empty communication channels. We are primarily interested in programs that operate on infinite streams of data and never terminate. In a bounded execution, the number of data elements buffered in each of the communication channels remains bounded.
The Kahn process network model of computation is powerful enough that the questions of termination and bounded buffering are undecidable. No finite-time algorithm can decide these questions for all Kahn process network programs. Fortunately, because we are interested in programs that never terminate, our scheduler has infinite time and can guarantee that programs execute forever with bounded buffering whenever possible. Our scheduling policy has been implemented using Ptolemy, an object-oriented simulation and prototyping environment.

Bounded scheduling of process networks

We compare synchronous dataflow (SDF) and cyclo-static dataflow (CSDF), which are each special cases of a model of computation we call dataflow process networks. In SDF actors have static firing rules: they consume and produce a fixed number of data tokens in each firing. This model is well suited to multirate signal processing applications and lends itself to efficient static scheduling, avoiding the run-time scheduling overhead incurred by general implementations of process networks. In CSDF which is a generalization of SDF actors have cyclically changing firing rules. In some situations, the added generality of CSDF can unnecessarily complicate the scheduling. We show how higher-order functions can be used to transform a CSDF graph into a SDF graph, simplifying the scheduling problem. In other situations, CSDF has a genuine advantage over SDF: simpler precedence constraints. We show how this makes it possible to eliminate unnecessary computations and expose additional parallelism. We use digital sample rate conversion as an example to illustrate these advantages of CSDF.

A comparison of synchronous and cycle-static dataflow

An analysis/synthesis system based on the sinusoidal speech model has been developed [1] In that system, the sine-wave amplitudes and frequencies are located by searching for the peaks of the magnitude of the short-time Fourier transform (STFT) of the input speech The phases are computed from the real and imaginary parts of the STFT at the measured frequencies The frequencies on successive frames are matched, used in a cubic phase interpolator and applied to a sine-wave generator Each sine wave is amplitude-modulated by the linear interpolation of the matched sine-wave amplitudes At a 10 ms frame rate, this system produces speech that is perceptually indistinguishable from the original [1] Since it is not possible to code all of the sine-wave parameters at low data rates, a system has been developed that codes the sine-wave frequencies by fitting a harmonic set of sine waves to the input waveform using a modified mean-squared error criterion [2], and codes the phase information implicitly using a voicing adaptive transition frequency to provide for a mixed voiced/unvoiced phase excitation model [3] Provided a postfilter is used at the synthesizer to attenuate the noise in the formant nulls, the speech synthesized by this system is of quite high quality having achieved a DAM score of 630 in the uncoded mode Since the fundamental frequency can be coded using ≈ 7 bits and the voicing measure can be coded using ≈ 3 bits, then the possibility exists for good speech quality at low data rates provided the sine-wave amplitudes can be coded efficiently In this paper the zero-phase, harmonic analysis/synthesis system and the post-filter design methodology will be described and then the various techniques that have been examined for coding the sine-wave amplitudes will be discussed

Sine-Wave Amplitude Coding at Low Data Rates

This paper describes the use of Ptolemy to automatically generate code for heterogeneous multiprocessor systems. The framework presented lets the designer migrate from simulation to code generation while developing an application that is specified by constructing a dataflow graph. From primitive send and receive actors, the framework can automatically construct three classes of interprocessor communication (IPC) interfaces. The first type of interface uses a synchronous dataflow (SDF) parallel scheduler to partition and schedule the graph across the available processors. The second type of interface allows hierarchical use of cooperating schedulers within an application. Finally, the third interface uses the send and receive actors as an interface between code generation and simulation systems in Ptolemy. To illustrate the framework, we present an example of a heterogeneous architecture consisting of a workstation with multiple Motorola 56001 processors. We present the relative strengths and weaknesses of each type of interface. >

T.M. Parks

Papers

Dataflow process networks

Bounded scheduling of process networks

A comparison of synchronous and cycle-static dataflow

Sine-Wave Amplitude Coding at Low Data Rates

Automatic code generation for heterogeneous multiprocessors