Workflow applications are gaining popularity in recent years because of the prevalence of cluster environments. Many algorithms have been developed since, however most static algorithms are designed in the problem domain of scheduling single workflow applications, thus not applicable to a common cluster environment where multiple workflow applications and other independent jobs compete for resources. Dynamic scheduling approaches can handle the mixed workload practically by nature but their performance has yet to optimize as they do not have a global view of workflow applications. Recent research efforts suggest merging multiple workflows into one workflow before execution, but fail to address an important issue that multiple workflow applications may be submitted at different times by different users. In this paper, we propose a planner-guided dynamic scheduling strategy for multiple workflow applications, leveraging job dependence information and execution time estimation.Our approach schedules individual jobs dynamically without requiring merging the workflow applications a priori. The simulation results show that the proposed algorithm significantly outperforms two other algorithms by 43.6% and 36.7% with respect to workflow makespan and turnaround time respectively, and it performs even better when the number of concurrent workflow applications increases and the resources are scarce.

https://www2.computer.org/portal/web/csdl/doi/10.1109/ICPP-W.2008.10

A Planner-Guided Scheduling Strategy for Multiple Workflow Applications

It is often difficult to perform efficiently a collection of jobs with complex job dependencies due to temporal unpredictability of the Grid. One way to mitigate the unpredictability is to schedule job execution in a manner that constantly maximizes the number of jobs that can be sent to workers. A recently developed scheduling theory provides a basis to meet that optimization goal. Intuitively, when the number of such jobs is always large, high parallelism can be maintained, even if the number of workers changes over time in an unpredictable manner. In this paper we present the design, implementation, and evaluation of a practical scheduling tool inspired by the theory. Given a DAGMan input file with interdependent jobs, the tool prioritizes the jobs. The resulting schedule significantly outperforms currently used schedules under a wide range of system parameters, as shown by simulation studies. For example, a scientific data analysis application, AIRSN, was executed at least 13% faster with 95% confidence. An implementation of the tool was integrated with the Condor high-throughput computing system.

/pdf/a-tool-for-prioritizing-dagman-jobs-and-its-evaluation-4el03b1w4b.pdf

A Tool for Prioritizing DAGMan Jobs and its Evaluation

Conceptual and algorithmic tools are developed as a foundation for a theory of scheduling complex computation-dags for Internet-based computing. The goal of the schedules produced is to render tasks eligible for allocation to remote clients (hence, for execution) at the maximum possible rate. This allows one to utilize remote clients well, as well as to lessen the likelihood of the "gridlock" that ensues when a computation stalls for lack of eligible tasks. Earlier work has introduced a formalism for studying this optimization problem and has identified optimal schedules for several significant families of structurally uniform dags. The current paper extends this work via a methodology for devising optimal schedules for a much broader class of complex dags, which are obtained via composition from a prespecified collection of simple building-block dags. The paper provides a suite of algorithms that decompose a given dag Gscr to expose its building blocks and an execution-priority relation xutri on building blocks. When the building blocks are appropriately interrelated under xutri the algorithms specify an optimal schedule for Gscr

Toward a theory for scheduling dags in Internet-based computing

It is often difficult to perform efficiently a collection of jobs with complex job dependencies due to temporal unpredictability of the grid. One way to mitigate the unpredictability is to schedule job execution in a manner that constantly maximizes the number of jobs that can be sent to workers. A recently developed scheduling theory provides a basis to meet that optimization goal. Intuitively, when the number of such jobs is always large, high parallelism can be maintained, even if the number of workers changes over time in an unpredictable manner. In this paper we present the design, implementation, and evaluation of a practical scheduling tool inspired by the theory. Given a DAGMan input file with interdependent jobs, the tool prioritizes the jobs. The resulting schedule significantly outperforms currently used schedules under a wide range of system parameters, as shown by simulation studies. For example, a scientific data analysis application, AIRSN, was executed at least 13% faster with 95% confidence. An implementation of the tool was integrated with the Condor high-throughput computing system

/pdf/a-tool-for-prioritizing-dagman-jobs-and-its-evaluation-2kj2dss0bo.pdf

A Tool for Prioritizing DAGMan Jobs and Its Evaluation

Task scheduling is one of the core steps to effectively exploit the capabilities of resources in the Grid. The task scheduling problem is an NP-complete problem. This paper studied on the task scheduling problem in grid environment and proposed a task scheduling mechanism, which expressed each possible task scheduling scheme as a task-resource assignment graph (T-RAG) and thus mapped the task scheduling problem into a graph optimal selection problem. Moreover, in order to find the optimal solution quickly and accurately, a task scheduling algorithm based on particle swarm optimization (PSO) was proposed. This algorithm regards the longest path of the task-resource assignment graph as fitness value and encodes every task-resource assignment as a particle. Finally, the experimentation shows that the approach proposed in this paper is effective to solve task scheduling problem.

Task Scheduling in Grid Based on Particle Swarm Optimization

Conceptual tools are developed to aid in crafting a theory of scheduling complex computation-dags for Internet-based computing. The goal of the schedules produced is to render tasks eligible for allocation to remote clients (hence for execution) at the maximum possible rate. This allows one to utilize remote clients well, and also lessen the likelihood of the "gridlock" that ensues when a computation stalls for lack of eligible tasks. Earlier work has introduced formalism for studying this optimization problem and has identified optimal schedules for several significant families of structurally uniform dags. The current paper extends this work via a methodology for devising optimal schedules for a much broader class of complex dags. These dags are obtained via composition from a prespecified collection of simple building-block dags. The paper introduces a suite of algorithms that decompose a given dag to expose its building-blocks, and a priority relation on building-blocks. When the building-blocks are appropriately interrelated, the dag can be scheduled optimally.

M. Yurkewych

Papers

Toward a theory for scheduling dags in Internet-based computing

On scheduling complex dags for Internet-based computing