The resource management system is the central component of distributed network computing systems. There have been many projects focused on network computing that have designed and implemented resource management systems with a variety of architectures and services. In this paper, an abstract model and a comprehensive taxonomy for describing resource management architectures is developed. The taxonomy is used to identify approaches followed in the implementation of existing resource management systems for very large-scale network computing systems known as Grids. The taxonomy and the survey results are used to identify architectural approaches and issues that have not been fully explored in the research. Copyright © 2001 John Wiley & Sons, Ltd.

A taxonomy and survey of grid resource management systems for distributed computing

The accelerated development in peer-to-peer and Grid computing has positioned them as promising next-generation computing platforms. They enable the creation of virtual enterprises for sharing resources distributed across the world. However, resource management, application development and usage models in these environments is a complex undertaking. This is due to the geographic distribution of resources that are owned by different organizations or peers. The resource owners of each of these resources have different usage or access policies and cost models, and varying loads and availability. In order to address complex resource management issues, we have proposed a computational economy framework for resource allocation and for regulating supply and demand in Grid computing environments. This framework provides mechanisms for optimizing resource provider and consumer objective functions through trading and brokering services. In a real world market, there exist various economic models for setting the price of services based on supply-and-demand and their value to the user. They include commodity market, posted price, tender and auction models. In this paper, we discuss the use of these models for interaction between Grid components to decide resource service value, and the necessary infrastructure to realize each model. In addition to usual services offered by Grid computing systems, we need an infrastructure to support interaction protocols, allocation mechanisms, currency, secure banking and enforcement services. We briefly discuss existing technologies that provide some of these services and show their usage in developing the Nimrod-G grid resource broker. Furthermore, we demonstrate the effectiveness of some of the economic models in resource trading and scheduling using the Nimrod/G resource broker, with deadline and cost constrained scheduling for two different optimization strategies, on the World-Wide Grid testbed that has resources distributed across five continents.

Economic models for resource management and scheduling in Grid computing

Computational Grids, emerging as an infrastructure for next generation computing, enable the sharing, selection, and aggregation of geographically distributed resources for solving large-scale problems in science, engineering, and commerce. As the resources in the Grid are heterogeneous and geographically distributed with varying availability and a variety of usage and cost policies for diverse users at different times and, priorities as well as goals that vary with time. The management of resources and application scheduling in such a large and distributed environment is a complex task. This thesis proposes a distributed computational economy as an effective metaphor for the management of resources and application scheduling. It proposes an architectural framework that supports resource trading and quality of services based scheduling. It enables the regulation of supply and demand for resources and provides an incentive for resource owners for participating in the Grid and motives the users to trade-off between the deadline, budget, and the required level of quality of service. The thesis demonstrates the capability of economic-based systems for peer-to-peer distributed computing by developing users' quality-of-service requirements driven scheduling strategies and algorithms. It demonstrates their effectiveness by performing scheduling experiments on the World-Wide Grid for solving parameter sweep applications.

Economic-based Distributed Resource Management and Scheduling for Grid Computing

With the ever-growing complexity and dynamicity of computer systems, proactive fault management is an effective approach to enhancing availability. Online failure prediction is the key to such techniques. In contrast to classical reliability methods, online failure prediction is based on runtime monitoring and a variety of models and methods that use the current state of a system and, frequently, the past experience as well. This survey describes these methods. To capture the wide spectrum of approaches concerning this area, a taxonomy has been developed, whose different approaches are explained and major concepts are described in detail.

A survey of online failure prediction methods

This work identifies challenges in managing resources in a Grid computing environment and proposes computational economy as a metaphor for effective management of resources and application scheduling. It identifies distributed resource management challenges and requirements of economy-based Grid systems, and discusses various representative economy-based systems, both historical and emerging, for cooperative and competitive trading of resources such as CPU cycles, storage, and network bandwidth. It presents an extensive, service-oriented Grid architecture driven by Grid economy and an approach for its realization by leveraging various existing Grid technologies. It also presents commodity and auction models for resource allocation. The use of commodity economy model for resource management and application scheduling in both computational and data grids is also presented.

The Grid Economy

This paper describes and evaluates the application of three local learning algorithms-nearest-neighbor, weighted-average, and locally-weighted polynomial regression-for the prediction of run-specific resource-usage on the basis of run-time input parameters supplied to tools. A two-level knowledge base allows the learning algorithms to track short-term fluctuations in the performances of computing systems, and the use of instance editing techniques improves the scalability of the performance-modeling system. The learning algorithms assist PUNCH, a network-computing system at Purdue University, in emulating an ideal user in terms of its resource management and usage policies.

Predictive application-performance modeling in a computational grid environment

This paper addresses the architectural issues that arise in the design of a universally accessible wide-area network-computing system that is capable of making automatic cost/performance tradeoff decisions at run-time. The core system is designed around a three-level hierarchically distributed architecture, a choice driven by the dynamic, incremental, and distributed nature of the information associated with run-time cost/performance tradeoff decisions. Support for independent replication of each component in the hierarchy contributes to the overall scalability and reliability of the architecture. Meta-information is managed in a scalable manner by employing self-encoded resource identifiers that allow O(1) access to all managed information. Security and access control across administrative domains are provided by partitioning the infrastructure into independently-managed cells, and by giving administrators the ability to customize user-views directly at the location at which the request is processed. Demand-driven resource management is achieved by predicting the run-specific resource usage characteristics of tools via machine learning techniques. The concepts described in this paper are embodied in the Purdue University Network-Computing Hubs (PUNCH), a demand-based network-computing system that allows users to access and run unmodified tools via standard World Wide Web browsers. Tools do not have to be written in any particular language, and access to the source or object code is not required. The PUNCH infrastructure can be distributed in a manner that allows tools to be (user-transparently) executed wherever they reside. Currently, PUNCH contains over thirty tools developed by eight universities and four vendors, and serves more than five hundred users. During the past three years, PUNCH users have logged more than one million hits and have performed over seventy thousand simulations.

PUNCH: An architecture for Web-enabled wide-area network-computing

This distributed network computer lets users access and run engineering tools anytime, anywhere via web browsers. It is the enabling technology for Netcare, an Internet resource that is now freely available for computer architecture education and research.

Punch: web portal for running tools

Describes a virtual file system that allows data to be transferred on demand between storage and computational servers for the duration of a computing session. The solution works with unmodified applications (even commercial ones) running on standard operating systems and hardware. The virtual file system employs software proxies to broker transactions between standard NFS (Network File System) clients and servers; the proxies are dynamically configured and controlled by computational grid middleware. The approach has been implemented and extensively exercised in the context of PUNCH (Purdue University Network Computing Hubs), an operational computing portal that has more than 1,500 users across 24 countries. The results show that the virtual file system performs well in comparison to native NFS: performance analyses show that the proxy incurs mean overheads of 1% and 18% with respect to native NFS for a single-client execution of the Andrew benchmark in two representative computing environments, and that the average overhead for eight clients can be reduced to within 1% of native NFS with concurrent proxies.

The PUNCH virtual file system: seamless access to decentralized storage services in a computational grid

Many of the systems that currently allow computing on the World Wide Web target specific tools. Such solutions tend to be non-reusable in spite of the fact that they involve a significant amount of duplicated effort. This paper describes the issues involved in the design of a demand-based network-computing system, and presents an operational prototype (the Purdue University Network-Computing Hubs, or PUNCH) that allows users to access and run existing software tools via standard Web browsers. The tools do not have to be written in any particular language, and access to source code is not required. The PUNCH infrastructure can be distributed in a manner that allows tools to be (user-transparently) executed wherever they reside. Currently, PUNCH contains over 30 tools from eight universities and four vendors, and serves more than 500 users. During the past three years, PUNCH users have logged more than 860,000 hits and have performed over 54,000 simulations.

N.H. Kapadia

Papers

Predictive application-performance modeling in a computational grid environment

PUNCH: An architecture for Web-enabled wide-area network-computing

Punch: web portal for running tools

The PUNCH virtual file system: seamless access to decentralized storage services in a computational grid

On the design of a demand-based network-computing system: the Purdue University Network-Computing Hubs