In this paper, we define and explore proofs of retrievability (PORs). A POR scheme enables an archive or back-up service (prover) to produce a concise proof that a user (verifier) can retrieve a target file F, that is, that the archive retains and reliably transmits file data sufficient for the user to recover F in its entirety.A POR may be viewed as a kind of cryptographic proof of knowledge (POK), but one specially designed to handle a large file (or bitstring) F. We explore POR protocols here in which the communication costs, number of memory accesses for the prover, and storage requirements of the user (verifier) are small parameters essentially independent of the length of F. In addition to proposing new, practical POR constructions, we explore implementation considerations and optimizations that bear on previously explored, related schemes.In a POR, unlike a POK, neither the prover nor the verifier need actually have knowledge of F. PORs give rise to a new and unusual security definition whose formulation is another contribution of our work.We view PORs as an important tool for semi-trusted online archives. Existing cryptographic techniques help users ensure the privacy and integrity of files they retrieve. It is also natural, however, for users to want to verify that archives do not delete or modify files prior to retrieval. The goal of a POR is to accomplish these checks without users having to download the files themselves. A POR can also provide quality-of-service guarantees, i.e., show that a file is retrievable within a certain time bound.

PORs: Proofs of Retrievability for Large Files

１ インフラを構築する（ＡＷＳにおけるインフラ；ＶＰＣを構成する；ＶＰＣとオンプレミス環境とを接続する） ２ ファイルオブジェクトを保存・共有・公開する（オブジェクトストレージＳ３の機能；ファイルストレージとして利用する；Ｗｅｂサーバーを構築する；信頼性とコストのバランスをとりたい） ３ アプリケーションサーバーを構築する（Ａｍａｚｏｎ ＥＣ２とＡＷＳ Ｌａｍｂｄａ；スケーラビリティーを高める；サーバーレスでプログラムを動かす；データベースサービスを活用する） ４ ＡＷＳシステムを管理する（リソース監視と異常検知・通報；耐障害性を高める仕組みとバックアップ＆リカバリー；構成管理）

Amazon Web Services完全ソリューションガイド

Modern data appliances face severe bandwidth bottlenecks when moving vast amounts of data from storage to the query processing nodes. A possible solution to mitigate these bottlenecks is query off-loading to an intelligent storage engine, where partial or whole queries are pushed down to the storage engine. In this paper, we present Ibex, a prototype of an intelligent storage engine that supports off-loading of complex query operators. Besides increasing performance, Ibex also reduces energy consumption, as it uses an FPGA rather than conventional CPUs to implement the off-load engine. Ibex is a hybrid engine, with dedicated hardware that evaluates SQL expressions at line-rate and a software fallback for tasks that the hardware engine cannot handle. Ibex supports GROUP BY aggregation, as well as projection- and selection- based filtering. GROUP BY aggregation has a higher impact on performance but is also a more challenging operator to implement on an FPGA.

/pdf/ibex-an-intelligent-storage-engine-with-support-for-advanced-3ltyzbqqmu.pdf

Ibex: an intelligent storage engine with support for advanced SQL offloading

After nearly a decade of anticipation, scalable nonvolatile memory DIMMs are finally commercially available with the release of Intel's 3D XPoint DIMM. This new nonvolatile DIMM supports byte-granularity accesses with access times on the order of DRAM, while also providing data storage that survives power outages. Researchers have not idly waited for real nonvolatile DIMMs (NVDIMMs) to arrive. Over the past decade, they have written a slew of papers proposing new programming models, file systems, libraries, and applications built to exploit the performance and flexibility that NVDIMMs promised to deliver. Those papers drew conclusions and made design decisions without detailed knowledge of how real NVDIMMs would behave or how industry would integrate them into computer architectures. Now that 3D XPoint NVDIMMs are actually here, we can provide detailed performance numbers, concrete guidance for programmers on these systems, reevaluate prior art for performance, and reoptimize persistent memory software for the real 3D XPoint DIMM. In this paper, we explore the performance properties and characteristics of Intel's new 3D XPoint DIMM at the micro and macro level. First, we investigate the basic characteristics of the device, taking special note of the particular ways in which its performance is peculiar relative to traditional DRAM or other past methods used to emulate NVM. From these observations, we recommend a set of best practices to maximize the performance of the device. With our improved understanding, we then explore the performance of prior art in application-level software for persistent memory, taking note of where their performance was influenced by our guidelines.

https://www.usenix.org/system/files/fast20-yang.pdf

An Empirical Guide to the Behavior and Use of Scalable Persistent Memory

Mit seiner Publikation “One Size Fits All: An Idea Whose Time Has Come and Gone?“ [St05] hat Michael Stonebraker 2005 die Datenbankszene provoziert. Er vertritt dort die These, dass die eingeführten und in vielen Kontexten eingesetzten „traditionellen“ Datenbanksysteme zwar für viele datenzentrische Anwendungen eingesetzt werden, aber keineswegs die optimalen Systeme für viele dieser Anwendungen (OLAP, Stream Processing, XML Datenbanken, wissenschaftliche Datenbanken, Textsuche) sind. In einer späteren Publikation [St07] stellte er sogar die These auf, dass es keine Anwendungen gibt, für die die traditionellen Datenbanksysteme die beste Alternative sind. Er sagte voraus, dass sich in der kommerziellen Welt eine Anzahl unabhängiger spezialisierter „neuer“ Datenbanksysteme aufsplitten werde, die bestenfalls durch ein gemeinsames Frontend verbunden sind.

"One Size Fits All": An Idea Whose Time Has Come and Gone?

Web service providers have been using NoSQL datastores to provide scalability and availability for globally distributed data at the cost of sacrificing transactional guarantees. Recently, major web service providers like Google have moved towards building storage systems that provide ACID transactional guarantees for globally distributed data. For example, the newly published system, Spanner, uses Two-Phase Commit and Two-Phase Locking to provide atomicity and isolation for globally distributed data, running on top of Paxos to provide fault-tolerant log replication. We show in this paper that it is possible to provide the same ACID transactional guarantees for multi-datacenter databases with fewer cross-datacenter communication trips, compared to replicated logging. Instead of replicating the transactional log, we replicate the commit operation itself, by running Two-Phase Commit multiple times in different datacenters and using Paxos to reach consensus among datacenters as to whether the transaction should commit. Doing so not only replaces several inter-datacenter communication trips with intra-datacenter communication trips, but also allows us to integrate atomic commitment and isolation protocols with consistent replication protocols to further reduce the number of cross-datacenter communication trips needed for consistent replication; for example, by eliminating the need for an election phase in Paxos. We analyze our approach in terms of communication trips to compare it against the log replication approach, then we conduct an extensive experimental study to compare the performance and scalability of both approaches under various multi-datacenter setups.

http://www.nawab.me/Uploads/Replicated_Commit_VLDB_2013.pdf

Low-latency multi-datacenter databases using replicated commit

We present a framework for concurrency control and availability in multi-datacenter datastores. While we consider Google's Megastore as our motivating example, we define general abstractions for key components, making our solution extensible to any system that satisfies the abstraction properties. We first develop and analyze a transaction management and replication protocol based on a straightforward implementation of the Paxos algorithm. Our investigation reveals that this protocol acts as a concurrency prevention mechanism rather than a concurrency control mechanism. We then propose an enhanced protocol called Paxos with Combination and Promotion (Paxos-CP) that provides true transaction concurrency while requiring the same per instance message complexity as the basic Paxos protocol. Finally, we compare the performance of Paxos and Paxos-CP in a multi-datacenter experimental study, and we demonstrate that Paxos-CP results in significantly fewer aborted transactions than basic Paxos.

Serializability, not Serial: Concurrency Control and Availability in Multi-Datacenter Datastores

Technology trends suggest that byte-addressable nonvolatile memory (NVM) will supplant many uses of DRAM over the coming decade, raising the prospect of inexpensive recovery from power failures and similar faults. Ensuring the consistency of persistent state remains nontrivial, however, in the presence of volatile caches; cached values can "leak" back to persistent memory in arbitrary order. To ensure consistency, existing persistent memory algorithms use expensive, explicit write-back instructions to force each value back to memory before performing a dependent write, thereby incurring significant run-time overhead.

To reduce this overhead, we present a new design paradigm that we call periodic persistence. In a periodically persistent data structure, updates are made "in place," but can safely leak back to memory in any order, because only those updates that are known to be valid will be heeded during recovery. To guarantee forward progress, we periodically force a write-back of all dirty data in the cache, ensuring that all "sufficiently old" updates have indeed become persistent, at which point they become semantically visible to the recovery process.

As an example of periodic persistence, we present a transactional hash map, Dali, together with an informal proof of safety (buffered durable linearizability). Experiments with a prototype implementation suggest that periodic persistence can offer substantially better performance than either file-based or incrementally persistent (per-access write-back) alternatives.

https://drops.dagstuhl.de/opus/volltexte/2017/8014/pdf/LIPIcs-DISC-2017-37.pdf/

Dalí: A Periodically Persistent Hash Map.

https://vldb.org/pvldb/vol5/p1459_stacypatterson_vldb2012.pdf

Serializability, not serial: concurrency control and availability in multi-datacenter datastores

The efficient processing of video streams is a key component in many emerging Internet of Things (IoT) and edge applications, such as Virtual and Augmented Reality (V/AR) and self-driving cars These applications require real-time high-throughput video processing This can be attained via a collaborative processing model between the edge and the cloud---called an Edge-Cloud model To this end, many approaches were proposed to optimize the latency and bandwidth consumption of Edge-Cloud video processing, especially for Neural Networks (NN)-based methods In this demonstration We investigate the efficiency of these NN techniques, how they can be combined, and whether combining them leads to better performance Our demonstration invites participants to experiment with the various NN techniques, combine them, and observe how the underlying NN changes with different techniques and how these changes affect accuracy, latency and bandwidth consumption

/pdf/collaborative-edge-and-cloud-neural-networks-for-real-time-2zjmo7o0jm.pdf

Faisal Nawab

Papers

Low-latency multi-datacenter databases using replicated commit

Serializability, not Serial: Concurrency Control and Availability in Multi-Datacenter Datastores

Dalí: A Periodically Persistent Hash Map.

Serializability, not serial: concurrency control and availability in multi-datacenter datastores

Collaborative edge and cloud neural networks for real-time video processing