The major challenges today’s libraries face are the long-term preservation and utilization of the rapidly growing digital resources. Based on the current storage requirements of libraries, this paper proposes an open source cloud storage system based on GlusterFS. The system is optimized to provide low cost and high availability. Some tests and practical experience can be provided to libraries with reference for the long-term preservation of digital resources.1

https://www.dpi-proceedings.com/index.php/dtcse/article/download/18872/18368

The Construction of Open Source Cloud Storage System for Digital Resources

Facing the complex tasks of dynamic and timely adjustment according to the storage demands in Cloud environments and Big-data environments, we have already developed a web-based system for GlusterFS named WebGlusterFS. WebGlusterFS2.0 is based on WebGlusterFS and have been reengineered and upgraded while maintaining the ability to address the challenges of heterogeneous complex Cloud environments and Big-data environments. WebGlusterFS2.0 has two new features: unattended deployment function and automatic configuration of Xlator. WebGlusterFS2.0 enables us to conduct unattended deployment for multiple server nodes and allows us to add translator (Xlator) for GlusterFS without having to manually modify the GlusterFS volume configuration file each time.

WebGlusterFS2.0: A Web-based Administration Tool for GlusterFS with Unattended Deployment and Automatic Configuration of Xlator

The working principle of DFC compiler is introduced in this article. DFC is a grammatical extension of standard C language, with special DF function which describe the dependence of computing DAG. DFC compiler, dfcc, is used to convert the DFC source codes to standard C codes with the assistance of multi-threaded library. The lexical rules and grammatical rules are used to setup the AST of DFC codes, and then it is converted to the AST of standard C without DF function nodes. The derived AST is printed into a text file, a normal C file, and finally is processed by GCC to obtain the executable file. The experiment gives some demonstration of that converting details, and the memory footprint is studied to show an ideal scalability.

The Compiler of DFC: A Source Code Converter that Transform the Dataflow Code to the Multi-threaded C Code

A Hadoop application will produce lots of intermediate results in the map/reduce process that requires disk I/O and network transmission, which can take a lot of time. By compressing large scale data of intermediate results, it will greatly improve disk access efficiently and reduce program run time. Therefore, Hadoop provides a compression option. Hardware-accelerated solutions have become more desirable. This paper proposes a way that applies a hardware compression accelerator on the Hadoop platform, and the system performance analysis compared with a software-only solution that needs CPU participation. The testing programs are modified the zpipe, TestDFSIO, and Terasort. In contrast with the software solution. The speedup ratio of the zpipe is 6.42X, the speedup ratio of TestDFSIO is 6.28X, the speedup ratio of the Terasort application is up to 3.25X.

Haixin Du

Papers

WebGlusterFS2.0: A Web-based Administration Tool for GlusterFS with Unattended Deployment and Automatic Configuration of Xlator

The Compiler of DFC: A Source Code Converter that Transform the Dataflow Code to the Multi-threaded C Code

The Library for Hadoop deflate compression based on FPGA accelerator