Data access

About: Data access is a research topic. Over the lifetime, 13141 publications have been published within this topic receiving 172859 citations. The topic is also known as: Data access.

System and Methodology Providing Multiple Heterogeneous Buffer Caches

Abstract: A method for temporarily storing data objects in memory of a distributed system comprising a plurality of servers sharing access to data comprises steps of: reserving memory at each of the plurality of servers as a default data cache for storing data objects; in response to user input, allocating memory of at least one of the plurality of servers as a named cache reserved for storing a specified type of data object; in response to an operation at a particular server requesting a data object, determining whether the requested data object is of the specified type corresponding to the named cache at the particular server; if the data object is determined to be of the specified type corresponding to the named cache, storing the requested data object in the named cache at the particular server; and otherwise, using the default data cache for storing the requested data object.

The Dark Energy Survey Data Management System

Abstract: The Dark Energy Survey collaboration will study cosmic acceleration with a 5000 deg2 griZY survey in the southern sky over 525 nights from 2011-2016. The DES data management (DESDM) system will be used to process and archive these data and the resulting science ready data products. The DESDM system consists of an integrated archive, a processing framework, an ensemble of astronomy codes and a data access framework. We are developing the DESDM system for operation in the high performance computing (HPC) environments at NCSA and Fermilab. Operating the DESDM system in an HPC environment offers both speed and flexibility. We will employ it for our regular nightly processing needs, and for more compute-intensive tasks such as large scale image coaddition campaigns, extraction of weak lensing shear from the full survey dataset, and massive seasonal reprocessing of the DES data. Data products will be available to the Collaboration and later to the public through a virtual-observatory compatible web portal. Our approach leverages investments in publicly available HPC systems, greatly reducing hardware and maintenance costs to the project, which must deploy and maintain only the storage, database platforms and orchestration and web portal nodes that are specific to DESDM. In Fall 2007, we tested the current DESDM system on both simulated and real survey data. We used Teragrid to process 10 simulated DES nights (3TB of raw data), ingesting and calibrating approximately 250 million objects into the DES Archive database. We also used DESDM to process and calibrate over 50 nights of survey data acquired with the Mosaic2 camera. Comparison to truth tables in the case of the simulated data and internal crosschecks in the case of the real data indicate that astrometric and photometric data quality is excellent.

Heterogeneous query processing through SQL table functions

Abstract: In today's IT infrastructures, data is stored in SQL databases, non-SQL, databases, and host databases like ISAM/VSAM files. Non-SQL databases are specialized data stores controlled by applications like spreadsheets, mail, directory and index services. Developing applications accessing a variety of different data sources is challenging for application developers due to different environments, APIs, bindings, etc. 20 years ago, SQL was created to ease the life of database application developers and provide a uniform way for accessing data which is stored in SQL databases. The paper describes an implementation of table functions and its usage for accessing data stored outside SQL databases in diverse external data stores. Table functions are compliant with the relational data model, and therefore fit into the well established SQL language. The table-function architecture is open, and allows the deployment of generic data access infrastructures such as Microsoft's OLE DB or Java's JDBC (G. Hamilton et al., 1997). The paper describes a prototype implementation of OLE DB table functions with advanced query optimization techniques. The prototype is based on IBM DB2 UDB relational database system.

Role-based access control for grid database services using the community authorization service

Abstract: In this paper, we propose a role-based access control (RBAC) method for grid database services in open grid services architecture-data access and integration (OGSA-DAI). OGSA-DAI is an efficient grid-enabled middleware implementation of interfaces and services to access and control data sources and sinks. However, in OGSA-DAI, access control causes substantial administration overhead for resource providers in virtual organizations (VOs) because each of them has to manage a role-map file containing authorization information for individual grid users. To solve this problem, we used the community authorization service (CAS) provided by the globus toolkit to support the RBAC within the OGSA-DAI framework. The CAS grants the membership on VO roles to users. The resource providers then need to maintain only the mapping information from VO roles to local database roles in the role-map files, so that the number of entries in the role-map file is reduced dramatically. Furthermore, the resource providers control the granting of access privileges to the local roles. Thus, our access control method provides increased manageability for a large number of users and reduces day-to-day administration tasks of the resource providers, while they maintain the ultimate authority over their resources. Performance analysis shows that our method adds very little overhead to the existing security infrastructure of OGSA-DAI

Reality-based 3D modeling, segmentation and web-based visualization

Abstract: One of the most significant consequences of the introduction of digital 3D modeling in the Cultural Heritage field is the possibility to use 3D models as highly effective and intuitive means of communication as well as interface to share and visualize information collected in databases Due to the usual complexity of architectural and archaeological artifacts or sites, their digital models need be subdivided in sub-components and organized following semantic definitions in order to facilitate data retrieval This paper outlines a methodology to semantically segment complex reality-based 3D models, annotate information and share the results within online open-source tools The segmentation stage to subdivide and organize 3D models is based on both automated and manual methods, giving priority to automated procedures that can ease and speed these steps but not neglecting the user intervention to achieve accurate results The segmented 3D models are then visualized in web-based systems to allow data access to a wider range of users and enlarge knowledge The methodology is presented and tested on large and complex reality-based 3D models of architectural and archaeological heritage structures