Grid technologies enable large-scale sharing of resources within formal or informal consortia of individuals and/or institutions: what are sometimes called virtual organizations. In these settings, the discovery, characterization, and monitoring of resources, services, and computations are challenging problems due to the considerable diversity; large numbers, dynamic behavior, and geographical distribution of the entities in which a user might be interested. Consequently, information services are a vital part of any Grid software infrastructure, providing fundamental mechanisms for discovery and monitoring, and hence for planning and adapting application behavior. We present an information services architecture that addresses performance, security, scalability, and robustness requirements. Our architecture defines simple low-level enquiry and registration protocols that make it easy to incorporate individual entities into various information structures, such as aggregate directories that support a variety of different query languages and discovery strategies. These protocols can also be combined with other Grid protocols to construct additional higher-level services and capabilities such as brokering, monitoring, fault detection, and troubleshooting. Our architecture has been implemented as MDS-2, which forms part of the Globus Grid toolkit and has been widely deployed and applied.

/pdf/grid-information-services-for-distributed-resource-sharing-5cbg5d9rra.pdf

Grid information services for distributed resource sharing

An emerging class of data-intensive applications involve the geographically dispersed extraction of complex scientific information from very large collections of measured or computed data. Such applications arise, for example, in experimental physics, where the data in question is generated by accelerators, and in simulation science, where the data is generated by supercomputers. So-called Data Grids provide essential infrastructure for such applications, much as the Internet provides essential services for applications such as e-mail and the Web. We describe here two services that we believe are fundamental to any Data Grid: reliable, high-speed transport and replica management. Our high-speed transport service, GridFTP, extends the popular FTP protocol with new features required for Data Grid applications, such as striping and partial file access. Our replica management service integrates a replica catalog with GridFTP transfers to provide for the creation, registration, location, and management of dataset replicas. We present the design of both services and also preliminary performance results. Our implementations exploit security and other services provided by the Globus Toolkit.

https://www.cct.lsu.edu/~kosar/csc7700-fall06/papers/Allcock02-2.pdf

Data management and transfer in high-performance computational grid environments

Advances in hardware and software technologies have made it possible to deploy parallel applications over increasingly large sets of distributed resources. Consequently, the study of scheduling algorithms for such applications has been an active area of research. Given the nature of most scheduling problems one must resort to simulation to effectively evaluate and compare their efficacy over a wide range of scenarios. It has thus become necessary to simulate those algorithms for increasingly complex distributed dynamic, heterogeneous environments. We present Simgrid a simulation toolkit for the study of scheduling algorithms for distributed application. We give the main concepts and models behind Simgrid, describe its API and highlight current implementation issues. We also give some experimental results and describe work that builds on Simgrid's functionalities.

Simgrid: a toolkit for the simulation of application scheduling

SLA monitoring is difficult to automate as it would need precise and unambiguous specification and a customizable engine that collects the right measurement, models the data and evaluates the SLA at certain times or when certain events happen Also most of the SLA neglect client side measurement or restrict SLAs to measurements based only on server side In a cross-enterprise scenario like web services it will be important to obtain measurements at multiple sites and to guarantee SLAs on them In this article we propose an automated and distributed SLA monitoring engine

/pdf/automated-sla-monitoring-for-web-services-11pfxugb29.pdf

Automated SLA Monitoring for Web Services

/pdf/secure-efficient-data-transport-and-replica-management-for-4edj637bmd.pdf

Secure, Efficient Data Transport and Replica Management for High-Performance Data-Intensive Computing

Large distributed systems, such as computational grids, require a large amount of monitoring data be collected for a variety of tasks, such as fault detection, performance analysis, performance tuning, performance prediction and scheduling. Ensuring that all necessary monitoring is turned on and that the data is being collected can be a very tedious and error-prone task. We have developed an agent-based system to automate the execution of monitoring sensors and the collection of event data.

/pdf/a-monitoring-sensor-management-system-for-grid-environments-iky5ldfod4.pdf

A monitoring sensor management system for grid environments

Diagnosis and debugging of performance problems on complex distributed systems requires end-to-end performance information at both the application and system level. We describe a methodology, called NetLogger, that enables real-time diagnosis of performance problems in such systems. The methodology includes tools for generating precision event logs, an interface to a system event-monitoring framework, and tools for visualizing the log data and real-time state of the distributed system. Low overhead is an important requirement for such tools, therefore we evaluate efficiency of the monitoring itself. The approach is novel in that it combines network, host, and application-level monitoring, providing a complete view of the entire system.

/pdf/netlogger-a-toolkit-for-distributed-system-performance-1bs5ifz0zd.pdf

NetLogger: a toolkit for distributed system performance analysis

Modern scientific computing involves organizing, moving, visualizing, and analyzing massive amounts of data at multiple sites around the world. The technologies, the middleware services, and the architectures that are used to build useful high-speed, wide area distributed systems, constitute the field of data intensive computing. We describe an architecture for data intensive applications where we use a high-speed distributed data cache as a common element for all of the sources and sinks of data. This cache-based approach provides standard interfaces to a large, application-oriented, distributed, on-line, transient storage system. We describe our implementation of this cache, how we have made it "network aware ", and how we do dynamic load balancing based on the current network conditions. We also show large increases in application throughput by access to knowledge of the network conditions.

/pdf/a-network-aware-distributed-storage-cache-for-data-intensive-4qi3d4xx4o.pdf

A network-aware distributed storage cache for data intensive environments

Modern scientific computing involves organizing, moving, visualizing, and analyzing massive amounts of data at multiple sites around the world. The technologies, the middleware services, and the architectures that are used to build useful high-speed, wide area distributed systems, constitute the field of data intensive computing. In this paper the authors describe an architecture for data intensive applications where they use a high-speed distributed data cache as a common element for all of the sources and sinks of data. This cache-based approach provides standard interfaces to a large, application-oriented, distributed, on-line, transient storage system. They describe their implementation of this cache, how they have made it network aware, and how they do dynamic load balancing based on the current network conditions. They also show large increases in application throughput by access to knowledge of the network conditions.

/pdf/a-network-aware-distributed-storage-cache-for-data-intensive-i9iea91i4e.pdf

M. Holding

Papers

A monitoring sensor management system for grid environments

NetLogger: a toolkit for distributed system performance analysis

A network-aware distributed storage cache for data intensive environments

A Network-Aware Distributed Storage Cache for Data Intensive Environments