A Secure Sharding Protocol For Open Blockchains
24 Oct 2016-pp 17-30
TL;DR: ELASTICO is the first candidate for a secure sharding protocol with presence of byzantine adversaries, and scalability experiments on Amazon EC2 with up to $1, 600$ nodes confirm ELASTICO's theoretical scaling properties.
Abstract: Cryptocurrencies, such as Bitcoin and 250 similar alt-coins, embody at their core a blockchain protocol --- a mechanism for a distributed network of computational nodes to periodically agree on a set of new transactions. Designing a secure blockchain protocol relies on an open challenge in security, that of designing a highly-scalable agreement protocol open to manipulation by byzantine or arbitrarily malicious nodes. Bitcoin's blockchain agreement protocol exhibits security, but does not scale: it processes 3--7 transactions per second at present, irrespective of the available computation capacity at hand. In this paper, we propose a new distributed agreement protocol for permission-less blockchains called ELASTICO. ELASTICO scales transaction rates almost linearly with available computation for mining: the more the computation power in the network, the higher the number of transaction blocks selected per unit time. ELASTICO is efficient in its network messages and tolerates byzantine adversaries of up to one-fourth of the total computational power. Technically, ELASTICO uniformly partitions or parallelizes the mining network (securely) into smaller committees, each of which processes a disjoint set of transactions (or "shards"). While sharding is common in non-byzantine settings, ELASTICO is the first candidate for a secure sharding protocol with presence of byzantine adversaries. Our scalability experiments on Amazon EC2 with up to $1, 600$ nodes confirm ELASTICO's theoretical scaling properties.
Citations
More filters
09 May 2017
TL;DR: Blockbench as mentioned in this paper is an evaluation framework for analyzing private blockchains, which can be used to assess blockchains' viability as another distributed data processing platform, while helping developers to identify bottlenecks and accordingly improve their platforms.
Abstract: Blockchain technologies are taking the world by storm. Public blockchains, such as Bitcoin and Ethereum, enable secure peer-to-peer applications like crypto-currency or smart contracts. Their security and performance are well studied. This paper concerns recent private blockchain systems designed with stronger security (trust) assumption and performance requirement. These systems target and aim to disrupt applications which have so far been implemented on top of database systems, for example banking, finance and trading applications. Multiple platforms for private blockchains are being actively developed and fine tuned. However, there is a clear lack of a systematic framework with which different systems can be analyzed and compared against each other. Such a framework can be used to assess blockchains' viability as another distributed data processing platform, while helping developers to identify bottlenecks and accordingly improve their platforms. In this paper, we first describe BLOCKBENCH, the first evaluation framework for analyzing private blockchains. It serves as a fair means of comparison for different platforms and enables deeper understanding of different system design choices. Any private blockchain can be integrated to BLOCKBENCH via simple APIs and benchmarked against workloads that are based on real and synthetic smart contracts. BLOCKBENCH measures overall and component-wise performance in terms of throughput, latency, scalability and fault-tolerance. Next, we use BLOCKBENCH to conduct comprehensive evaluation of three major private blockchains: Ethereum, Parity and Hyperledger Fabric. The results demonstrate that these systems are still far from displacing current database systems in traditional data processing workloads. Furthermore, there are gaps in performance among the three systems which are attributed to the design choices at different layers of the blockchain's software stack. We have released BLOCKBENCH for public use.
731 citations
15 Oct 2018
TL;DR: RapidChain is proposed, the first sharding-based public blockchain protocol that is resilient to Byzantine faults from up to a 1/3 fraction of its participants, and achieves complete sharding of the communication, computation, and storage overhead of processing transactions without assuming any trusted setup.
Abstract: A major approach to overcoming the performance and scalability limitations of current blockchain protocols is to use sharding which is to split the overheads of processing transactions among multiple, smaller groups of nodes. These groups work in parallel to maximize performance while requiring significantly smaller communication, computation, and storage per node, allowing the system to scale to large networks. However, existing sharding-based blockchain protocols still require a linear amount of communication (in the number of participants) per transaction, and hence, attain only partially the potential benefits of sharding. We show that this introduces a major bottleneck to the throughput and latency of these protocols. Aside from the limited scalability, these protocols achieve weak security guarantees due to either a small fault resiliency (e.g., 1/8 and 1/4) or high failure probability, or they rely on strong assumptions (e.g., trusted setup) that limit their applicability to mainstream payment systems. We propose RapidChain, the first sharding-based public blockchain protocol that is resilient to Byzantine faults from up to a 1/3 fraction of its participants, and achieves complete sharding of the communication, computation, and storage overhead of processing transactions without assuming any trusted setup. RapidChain employs an optimal intra-committee consensus algorithm that can achieve very high throughputs via block pipelining, a novel gossiping protocol for large blocks, and a provably-secure reconfiguration mechanism to ensure robustness. Using an efficient cross-shard transaction verification technique, our protocol avoids gossiping transactions to the entire network. Our empirical evaluations suggest that RapidChain can process (and confirm) more than 7,300 tx/sec with an expected confirmation latency of roughly 8.7 seconds in a network of 4,000 nodes with an overwhelming time-to-failure of more than 4,500 years.
695 citations
Cites background from "A Secure Sharding Protocol For Open..."
...Protocol # Nodes Resiliency Complexity1 Throughput Latency Storage2 Shard Size Time to Fail Elastico [47] n= 1,600 t < n/4 Ω(m2/b+n) 40 tx/sec 800 sec 1x m= 100 1 hour OmniLedger [42] n= 1,800 t < n/4 Ω(m2/b+n) 500 tx/sec 14 sec 1/3x m= 600 230 years OmniLedger [42] n= 1,800 t < n/4 Ω(m2/b+n) 3,500 tx/sec 63 sec 1/3x m= 600 230 years RapidChain n= 1,800 t < n/3 O (m2/b+m logn) 4,220 tx/sec 8....
[...]
...Recently, several protocols have been proposed to mitigate the performance and scalability issues of Bitcoin’s blockchain [23, 29, 41, 52, 56, 47, 32, 4, 42] using hybrid architectures that combine the open-membership nature of Bitcoin with traditional Byzantine fault tolerance [57, 19]....
[...]
...Building on [5, 60], we reduce the communication overhead and latency of P2P consensus on large blocks gossiped in each committee by roughly 3-10 times compared to previous solutions [47, 32, 4, 42]....
[...]
...Such a partitioning of operations and/or data among multiple groups of nodes is often referred to as sharding [21] and has been recently studied in the context of blockchain protocols [47, 42]....
[...]
...Moreover, the protocol cannot scale out its transaction processing capacity with the number of participants joining the protocol [47, 42]....
[...]
TL;DR: A critical assessment of the often exaggerated benefits of blockchain technology found in the literature is presented and a shift from a technology-driven to need-driven approach in which blockchain applications are customized to ensure a fit with requirements of administrative processes is pleaded.
686 citations
Cites background or methods from "A Secure Sharding Protocol For Open..."
...A smart contract is a program that runs on the BC and has its correct execution enforced by the consensus protocol (Luu et al., 2016a)....
[...]
...A Peer-to-Peer Electronic Cash System” by a mysterious Satoshi Nakamoto (2008) was circulated among an email list of cryptographers (Popper, 2015)....
[...]
...A smart contract contains information about a deal and will only be executed if the conditions are validated by all nodes in the network (Luu et al., 2016a)....
[...]
...On the 31st of October 2008 the white paper “Bitcoin – A Peer-to-Peer Electronic Cash System” by a mysterious Satoshi Nakamoto (2008) was circulated among an email list of cryptographers (Popper, 2015)....
[...]
...Promising innovations like sidechains (Back et al, 2014) and drivechains, off-chain payment channels [e.g. Lightning Network] (Luu et al, 2016b), smart contracts (Bartoletti &Pompianu, 2017), colored coins (Rosenfeld, 2012), IOTA utilizing a blockless 'tangle' (IOTA, 2017) and more are aimed at…...
[...]
TL;DR: This paper provides a systematic vision of the organization of the blockchain networks, a comprehensive survey of the emerging applications of blockchain networks in a broad area of telecommunication, and discusses several open issues in the protocol design for blockchain consensus.
Abstract: The past decade has witnessed the rapid evolution in blockchain technologies, which has attracted tremendous interests from both the research communities and industries. The blockchain network was originated from the Internet financial sector as a decentralized, immutable ledger system for transactional data ordering. Nowadays, it is envisioned as a powerful backbone/framework for decentralized data processing and data-driven self-organization in flat, open-access networks. In particular, the plausible characteristics of decentralization, immutability, and self-organization are primarily owing to the unique decentralized consensus mechanisms introduced by blockchain networks. This survey is motivated by the lack of a comprehensive literature review on the development of decentralized consensus mechanisms in blockchain networks. In this paper, we provide a systematic vision of the organization of blockchain networks. By emphasizing the unique characteristics of decentralized consensus in blockchain networks, our in-depth review of the state-of-the-art consensus protocols is focused on both the perspective of distributed consensus system design and the perspective of incentive mechanism design. From a game-theoretic point of view, we also provide a thorough review of the strategy adopted for self-organization by the individual nodes in the blockchain backbone networks. Consequently, we provide a comprehensive survey of the emerging applications of blockchain networks in a broad area of telecommunication. We highlight our special interest in how the consensus mechanisms impact these applications. Finally, we discuss several open issues in the protocol design for blockchain consensus and the related potential research directions.
680 citations
Cites background or methods from "A Secure Sharding Protocol For Open..."
...The study in [172] proposes a mechanism of generating distributive...
[...]
...As in [171] and [172], full sharding also partitions the storage of the blockchain state into multiple shards (e....
[...]
...In the network level, RapidChain requires a reference BFTcommittee to run a distributed randomness generation protocol similar to [172] and generate a public random string to initialize the formation of shard-level committees....
[...]
...As in [172], OmniLedger is built upon two levels of epochbased Byzantine agreement processes, with the network level being responsible for epoch randomness generation and the shard level for intra-committee consensus....
[...]
...In [172], a different sharding protocol named ‘‘Elastico’’ is proposed with the emphasis on the process of validator sharding through dynamically forming multiple BFTcommittees....
[...]
TL;DR: An in-depth survey of BCoT is presented and the insights of this new paradigm are discussed and the open research directions in this promising area are outlined.
Abstract: Internet of Things (IoT) is reshaping the incumbent industry to smart industry featured with data-driven decision-making. However, intrinsic features of IoT result in a number of challenges, such as decentralization, poor interoperability, privacy, and security vulnerabilities. Blockchain technology brings the opportunities in addressing the challenges of IoT. In this paper, we investigate the integration of blockchain technology with IoT. We name such synthesis of blockchain and IoT as blockchain of things (BCoT). This paper presents an in-depth survey of BCoT and discusses the insights of this new paradigm. In particular, we first briefly introduce IoT and discuss the challenges of IoT. Then, we give an overview of blockchain technology. We next concentrate on introducing the convergence of blockchain and IoT and presenting the proposal of BCoT architecture. We further discuss the issues about using blockchain for fifth generation beyond in IoT as well as industrial applications of BCoT. Finally, we outline the open research directions in this promising area.
654 citations
Cites background from "A Secure Sharding Protocol For Open..."
...[134], in which less computational-extensive puzzle is first solved in PoW and consensus is then reached in multiple small groups....
[...]
References
More filters
TL;DR: The Albanian Generals Problem as mentioned in this paper is a generalization of Dijkstra's dining philosophers problem, where two generals have to come to a common agreement on whether to attack or retreat, but can communicate only by sending messengers who might never arrive.
Abstract: I have long felt that, because it was posed as a cute problem about philosophers seated around a table, Dijkstra’s dining philosopher’s problem received much more attention than it deserves. (For example, it has probably received more attention in the theory community than the readers/writers problem, which illustrates the same principles and has much more practical importance.) I believed that the problem introduced in [41] was very important and deserved the attention of computer scientists. The popularity of the dining philosophers problem taught me that the best way to attract attention to a problem is to present it in terms of a story. There is a problem in distributed computing that is sometimes called the Chinese Generals Problem, in which two generals have to come to a common agreement on whether to attack or retreat, but can communicate only by sending messengers who might never arrive. I stole the idea of the generals and posed the problem in terms of a group of generals, some of whom may be traitors, who have to reach a common decision. I wanted to assign the generals a nationality that would not offend any readers. At the time, Albania was a completely closed society, and I felt it unlikely that there would be any Albanians around to object, so the original title of this paper was The Albanian Generals Problem. Jack Goldberg was smart enough to realize that there were Albanians in the world outside Albania, and Albania might not always be a black hole, so he suggested that I find another name. The obviously more appropriate Byzantine generals then occurred to me. The main reason for writing this paper was to assign the new name to the problem. But a new paper needed new results as well. I came up with a simpler way to describe the general 3n+1-processor algorithm. (Shostak’s 4-processor algorithm was subtle but easy to understand; Pease’s generalization was a remarkable tour de force.) We also added a generalization to networks that were not completely connected. (I don’t remember whose work that was.) I also added some discussion of practical implementation details.
5,208 citations
TL;DR: In this article, a group of generals of the Byzantine army camped with their troops around an enemy city are shown to agree upon a common battle plan using only oral messages, if and only if more than two-thirds of the generals are loyal; so a single traitor can confound two loyal generals.
Abstract: Reliable computer systems must handle malfunctioning components that give conflicting information to different parts of the system. This situation can be expressed abstractly in terms of a group of generals of the Byzantine army camped with their troops around an enemy city. Communicating only by messenger, the generals must agree upon a common battle plan. However, one or more of them may be traitors who will try to confuse the others. The problem is to find an algorithm to ensure that the loyal generals will reach agreement. It is shown that, using only oral messages, this problem is solvable if and only if more than two-thirds of the generals are loyal; so a single traitor can confound two loyal generals. With unforgeable written messages, the problem is solvable for any number of generals and possible traitors. Applications of the solutions to reliable computer systems are then discussed.
4,901 citations
[...]
TL;DR: It is shown that, without a logically centralized authority, Sybil attacks are always possible except under extreme and unrealistic assumptions of resource parity and coordination among entities.
Abstract: Large-scale peer-to-peer systems face security threats from faulty or hostile remote computing elements. To resist these threats, many such systems employ redundancy. However, if a single faulty entity can present multiple identities, it can control a substantial fraction of the system, thereby undermining this redundancy. One approach to preventing these "Sybil attacks" is to have a trusted agency certify identities. This paper shows that, without a logically centralized authority, Sybil attacks are always possible except under extreme and unrealistic assumptions of resource parity and coordination among entities.
4,816 citations
"A Secure Sharding Protocol For Open..." refers background in this paper
...In the worst case, we can assume that each committee has 3c/2 members, of which at most 1/3 of them are malicious....
[...]
Book•
01 Jan 1996
TL;DR: This book familiarizes readers with important problems, algorithms, and impossibility results in the area, and teaches readers how to reason carefully about distributed algorithms-to model them formally, devise precise specifications for their required behavior, prove their correctness, and evaluate their performance with realistic measures.
Abstract: In Distributed Algorithms, Nancy Lynch provides a blueprint for designing, implementing, and analyzing distributed algorithms. She directs her book at a wide audience, including students, programmers, system designers, and researchers.
Distributed Algorithms contains the most significant algorithms and impossibility results in the area, all in a simple automata-theoretic setting. The algorithms are proved correct, and their complexity is analyzed according to precisely defined complexity measures. The problems covered include resource allocation, communication, consensus among distributed processes, data consistency, deadlock detection, leader election, global snapshots, and many others.
The material is organized according to the system model-first by the timing model and then by the interprocess communication mechanism. The material on system models is isolated in separate chapters for easy reference.
The presentation is completely rigorous, yet is intuitive enough for immediate comprehension. This book familiarizes readers with important problems, algorithms, and impossibility results in the area: readers can then recognize the problems when they arise in practice, apply the algorithms to solve them, and use the impossibility results to determine whether problems are unsolvable. The book also provides readers with the basic mathematical tools for designing new algorithms and proving new impossibility results. In addition, it teaches readers how to reason carefully about distributed algorithms-to model them formally, devise precise specifications for their required behavior, prove their correctness, and evaluate their performance with realistic measures.
Table of Contents
1 Introduction
2 Modelling I; Synchronous Network Model
3 Leader Election in a Synchronous Ring
4 Algorithms in General Synchronous Networks
5 Distributed Consensus with Link Failures
6 Distributed Consensus with Process Failures
7 More Consensus Problems
8 Modelling II: Asynchronous System Model
9 Modelling III: Asynchronous Shared Memory Model
10 Mutual Exclusion
11 Resource Allocation
12 Consensus
13 Atomic Objects
14 Modelling IV: Asynchronous Network Model
15 Basic Asynchronous Network Algorithms
16 Synchronizers
17 Shared Memory versus Networks
18 Logical Time
19 Global Snapshots and Stable Properties
20 Network Resource Allocation
21 Asynchronous Networks with Process Failures
22 Data Link Protocols
23 Partially Synchronous System Models
24 Mutual Exclusion with Partial Synchrony
25 Consensus with Partial Synchrony
4,340 citations
22 Feb 1999
TL;DR: A new replication algorithm that is able to tolerate Byzantine faults that works in asynchronous environments like the Internet and incorporates several important optimizations that improve the response time of previous algorithms by more than an order of magnitude.
Abstract: This paper describes a new replication algorithm that is able to tolerate Byzantine faults. We believe that Byzantinefault-tolerant algorithms will be increasingly important in the future because malicious attacks and software errors are increasingly common and can cause faulty nodes to exhibit arbitrary behavior. Whereas previous algorithms assumed a synchronous system or were too slow to be used in practice, the algorithm described in this paper is practical: it works in asynchronous environments like the Internet and incorporates several important optimizations that improve the response time of previous algorithms by more than an order of magnitude. We implemented a Byzantine-fault-tolerant NFS service using our algorithm and measured its performance. The results show that our service is only 3% slower than a standard unreplicated NFS.
3,562 citations
"A Secure Sharding Protocol For Open..." refers background or methods in this paper
...Here we provide an intuition why this works....
[...]
...The second goal is to compare ELASTICO to other related consensus protocols including Bitcoin [1], Bitcoin-NG [9] and PBFT [13]....
[...]
...Our scalability experiments on Amazon EC2 with up to 1, 600 nodes confirm ELASTICO’s theoretical scaling properties....
[...]
...At its core, ELASTICO scales up the agreement throughput near linearly with the computational power of the network and tolerates byzantine adversaries which controls up to one-forth computation capacity, in the partially synchronous network....
[...]