A method and system is disclosed for providing a cloud storage system supporting existing APIs and protocols. The method of storing cloud storage system (CSS) object metadata separates object metadata that describes each CSS object as a collection of named chunks with chunk locations specified as a separate part of the metadata. Chunks are identified using globally unique permanent identifiers that are never re-used to identify different chunk payload. While avoiding the bottleneck of a single metadata server, the disclosed system provides ordering guarantees to clients such as guaranteeing access to the most recent version of an object. The disclosed system also provides end-to-end data integrity protection, inline data deduplication, configurable replication, hierarchical storage management and location-aware optimization of chunk storage.

Cloud storage system with distributed metadata

A method and system for providing unified local storage support for file and cloud access is disclosed. The method comprises writing a chunk on a storage server, and replicating the chunk to other selected storage servers when necessary. The method and system further comprise writing a version manifest on the storage server; replicating the version manifest to other selected storage servers when necessary. Object puts or appends are implemented by first chunking the object, determining if the chunks are new, transferring the chunks if required, followed by creation of a new version manifest referencing the chunks. Finally, the method and system include providing concurrent file-oriented read and write access consistent with the stored version manifests and chunks.

Unified local storage supporting file and cloud object access

This book Databases and Transaction Processing constitutes a standard database textbook for advanced undergraduate and graduate courses— albeit with a somewhat different focus compared to the established books. As the subtitle An Application-Oriented Approach indicates, the authors put an emphasis on teaching the systematic usage of database systems—rather than concentrating on an in-depth coverage of implementation techniques for building database systems. The rationale for this focus is that many more students will be implementing applications than will actually be implementing database systems. Therefore, the authors set out to present the conceptual and engineering principles underlying the design and implementation of database and transaction processing applications. In order to provide a solid foundation for these principles, the book thoroughly presents the concepts and the theory underlying relational databases, conceptual/relational data modelling, and relational query languages. This book covers the material traditionally taught in an introductory course in database systems: the relational model, conceptual design, query languages (including SQL, QBE, relational algebra, and relational calculus), normalization theory, and query processing. In addition to these core database topics, the book also contains advanced database material often not covered in introductory texts and courses. Object databases, XML, and document processing on the web are given extensive coverage. Together with additional material on distributed databases, CORBA, OLAP, and data mining, the latter part of the book provides the basis for an advanced course on databases. The text also covers transaction processing in depth such that an entire course can be devised based on this book—maybe with some supplementary material from dedicated textbooks on this subject (e.g., Weikum and Vossen’s book which was reviewed by Marc Scholl in the preceding SIGMOD record). In addition to discussing issues related to building transactions and ACID properties, the book describes TP monitors, the architecture of centralized and distributed transaction processing systems, and the transactional issues related to building correct and efficient database applications.

Databases and Transaction Processing: An Application-Oriented Approach - Book Review.

A distributed file system and a file processing method thereof are disclosed. The distributed file system comprises at least two file nodes ( 1, 2 , . . . , n), and each file node comprises a metadata server ( 12, 22 , . . . , n 2 ), a file access client ( 11, 21 , . . . , n 1 ), a file access server ( 13, 23 , . . . , n 3 ) and a storage medium ( 14, 24 , . . . , n 4 ), and file nodes can access each other. The file processing method comprises the steps of: storing different files in each file node in a scattered way; for each file node, users of the current file node firstly accessing metadata/valid data of a required file in the file node through the file access client of the file node; if the metadata/valid data of the required file are not found in the file node, the users access other nodes through the file access clients. The present invention effectively solves the problem that a metadata server is easy to become a performance bottleneck when a great number of users access hot data concurrently, and can furthest release the storage space of some file nodes by the file processing methods, such as data block dispatching and data block aging, thereby is greatly increasing the utilization ratio.

Distributed file system supporting data block dispatching and file processing method thereof

Recent research has shown that applications often incorrectly implement crash consistency. We present the Crash-Consistent File System (ccfs), a file system that improves the correctness of application-level crash consistency protocols while maintaining high performance. A key idea in ccfs is the abstraction of a stream. Within a stream, updates are committed in program order, improving correctness; across streams, there are no ordering restrictions, enabling scheduling flexibility and high performance. We empirically demonstrate that applications running atop ccfs achieve high levels of crash consistency. Further, we show that ccfs performance under standard file-system benchmarks is excellent, in the worst case on par with the highest performing modes of Linux ext4, and in some cases notably better. Overall, we demonstrate that both application correctness and high performance can be realized in a modern file system.

Application Crash Consistency and Performance with CCFS.

Provided are an apparatus and a method which can be easily implemented with flexibility enabling distributing all metadata of trees and files in an asymmetric distributed file system. The apparatus includes: a metadata storage unit storing metadata corresponding to a part of partitions of a virtual metadata address space storing metadata for directories and/or files for each of the partitions; and a metadata storage management unit controlling the metadata so that the metadata are stored in the metadata storage unit and manages a master map including information on the part of the partitions. Since all directories and files can be distributed to a plurality of metadata servers without a limitation, it is possible to prevent a load from being concentrated on a predetermined metadata server. Metadata roles of the metadata servers are very simply readjusted and as a result, the load can be easily distributed in a partition level.

Apparatus and method of managing metadata in asymmetric distributed file system

There are provided a system and method for providing a virtual desktop service using a cache server. A system for providing a virtual desktop service according to the invention includes a host server configured to provide a virtual desktop service to a client terminal using a virtual machine, a distributed file system configured to store data for the virtual machine, and a cache server that is provided for each host server group having at least one host server, and performs a read process or a write process of data using physically separate caches when the read process or write process of the data is requested from the virtual machine in the host server.

System and method for providing virtual desktop service using cache server

Disclosed herein is a distributed file system using a torus network. The distributed file system may include multiple servers. The multiple servers are connected with each other through an n-dimensional torus network, and each of the multiple servers may be arranged along n-dimensional axes. Among the servers in the distributed file system, a server group of an (n−1)-dimensional torus network may be connected to a public network. Different loads of the distributed file system may be distributed based on the axes. Through the distribution, the processing performance of the distributed file system may be improved.

Distributed file system using torus network and method for configuring and operating distributed file system using torus network

This paper presents an effective method of metadata rebalance in exascale distributed file systems. Exponential data growth has led to the need for an adaptive and robust distributed file system whose typical architecture is composed of a large cluster of metadata servers and data servers. Though each metadata server can have an equally divided subset from the entire metadata set at first, there will eventually be a global imbalance in the placement of metadata among metadata servers, and this imbalance worsens over time. To ensure that disproportionate metadata placement will not have a negative effect on the intrinsic performance of a metadata server cluster, it is necessary to recover the balanced performance of the cluster periodically. However, this cannot be easily done because rebalancing seriously hampers the normal operation of a file system. This situation continues to get worse with both an ever-present heavy workload on the file system and frequent failures of server components at exascale. As one of the primary reasons for such a degraded performance, file system clients frequently fail to look up metadata from the metadata server cluster during the period of metadata rebalance; thus, metadata operations cannot proceed at their normal speed. We propose a metadata rebalance model that minimizes failures of metadata operations during the metadata rebalance period and validate the proposed model through a cost analysis. The analysis results demonstrate that our model supports the feasibility of online metadata rebalance without the normal operation obstruction and increases the chances of maintaining balance in a huge cluster of metadata servers.

Adaptive metadata rebalance in exascale file system

Disclosed are a method and apparatus for reading data in a distributed file system in which a client and a server are separated. In the method and apparatus, a prefetching operation is performed to provide a continuous read function with high performance even in the distributed file system so that an optimum continuous read function of a local file system within the client may be effectively supported when an application program of the client requests continuous reading on a file (or chunk).

Hong-Yeon Kim

Papers

Apparatus and method of managing metadata in asymmetric distributed file system

System and method for providing virtual desktop service using cache server

Distributed file system using torus network and method for configuring and operating distributed file system using torus network

Adaptive metadata rebalance in exascale file system

Method and apparatus for reading data in distributed file system