Secure, Efficient Data Transport and Replica Management for High-Performance Data-Intensive Computing
Bill Allcock,Joe Bester,John Bresnahan,Ann L. Chervenak,Carl Kesselman,Sam Meder,Veronika Nefedova,Darcy Quesnel,Steven Tuecke,Ian Foster +9 more
- pp 13-13
Reads0
Chats0
TLDR
The high-speed transport service, GridFTP, extends the popular FTP protocol with new features required for Data Grid applications, such as striping and partial file access, and the replica management service integrates a replica catalog with gridFTP transfers to provide for the creation, registration, location, and management of dataset replicas.Abstract:
An emerging class of data-intensive applications involve the geographically dispersed extraction of complex scientific information from very large collections of measured or computed data. Such applications arise, for example, in experimental physics, where the data in question is generated by accelerators, and in simulation science, where the data is generated by supercomputers. So-called Data Grids provide essential infrastructure for such applications, much as the Internet provides essential services for applications such as e-mail and the Web. We describe here two services that we believe are fundamental to any Data Grid: reliable, high-speed transport and replica management. Our high-speed transport service, GridFTP, extends the popular FTP protocol with new features required for Data Grid applications, such as striping and partial file access. Our replica management service integrates a replica catalog with GridFTP transfers to provide for the creation, registration, location, and management of dataset replicas. We present the design of both services and also preliminary performance results. Our implementations exploit security and other services provided by the Globus Toolkit.read more
Citations
More filters
Journal ArticleDOI
The Anatomy of the Grid: Enabling Scalable Virtual Organizations
TL;DR: The authors present an extensible and open Grid architecture, in which protocols, services, application programming interfaces, and software development kits are categorized according to their roles in enabling resource sharing.
Posted Content
The Anatomy of the Grid - Enabling Scalable Virtual Organizations
TL;DR: This article reviews the "Grid problem," and presents an extensible and open Grid architecture, in which protocols, services, application programming interfaces, and software development kits are categorized according to their roles in enabling resource sharing.
Journal ArticleDOI
Pegasus: A framework for mapping complex scientific workflows onto distributed systems
Ewa Deelman,Gurmeet Singh,Mei-Hui Su,Jim Blythe,Yolanda Gil,Carl Kesselman,Gaurang Mehta,Karan Vahi,G. Bruce Berriman,John C. Good,Anastasia C. Laity,Joseph C. Jacob,Daniel S. Katz +12 more
TL;DR: The results of improving application performance through workflow restructuring which clusters multiple tasks in a workflow into single entities are presented.
Posted Content
A Community Authorization Service for Group Collaboration
TL;DR: In this paper, the authors propose an approach to the representation, maintenance, and enforcement of fine-grained access control policies in distributed communities of resource providers and resource consumers, within which often complex and dynamic policies govern who can use which resources for which purpose.
Proceedings ArticleDOI
Chimera: a virtual data system for representing, querying, and automating data derivation
TL;DR: The Chimera virtual data system is developed, which combines avirtual data catalog for representing data derivation procedures and derived data, with a virtual data language interpreter that translates user requests into data definition and query operations on the database.
References
More filters
Journal ArticleDOI
The GRID: Blueprint for a New Computing Infrastructure
TL;DR: The main purpose is to update the designers and users of parallel numerical algorithms with the latest research in the field and present the novel ideas, results and work in progress and advancing state-of-the-art techniques in the area of parallel and distributed computing for numerical and computational optimization problems in scientific and engineering application.
Posted Content
The Anatomy of the Grid - Enabling Scalable Virtual Organizations
TL;DR: This article reviews the "Grid problem," and presents an extensible and open Grid architecture, in which protocols, services, application programming interfaces, and software development kits are categorized according to their roles in enabling resource sharing.
Journal ArticleDOI
Globus: a Metacomputing Infrastructure Toolkit
Ian Foster,Carl Kesselman +1 more
TL;DR: The Globus system is intended to achieve a vertically integrated treatment of application, middleware, and net work, an integrated set of higher level services that enable applications to adapt to heteroge neous and dynamically changing metacomputing environ ments.
Journal ArticleDOI
The data grid
TL;DR: In this paper, the authors introduce design principles for a data management architecture called the data grid, and describe two basic services that are fundamental to the design of a data grid: storage systems and metadata management.
Proceedings ArticleDOI
The SDSC storage resource broker
TL;DR: The architecture and various features of the SDSC SRB are described, which provides applications a uniform API to access heterogeneous distributed storage resources including, filesystems, database systems, and archival storage systems.