There is, I think, something ethereal about i —the square root of minus one. I remember first hearing about it at school. It seemed an odd beast at that time—an intruder hovering on the edge of reality.

Usually familiarity dulls this sense of the bizarre, but in the case of i it was the reverse: over the years the sense of its surreal nature intensified. It seemed that it was impossible to write mathematics that described the real world in …

I and i

This paper describes a new replication algorithm that is able to tolerate Byzantine faults. We believe that Byzantinefault-tolerant algorithms will be increasingly important in the future because malicious attacks and software errors are increasingly common and can cause faulty nodes to exhibit arbitrary behavior. Whereas previous algorithms assumed a synchronous system or were too slow to be used in practice, the algorithm described in this paper is practical: it works in asynchronous environments like the Internet and incorporates several important optimizations that improve the response time of previous algorithms by more than an order of magnitude. We implemented a Byzantine-fault-tolerant NFS service using our algorithm and measured its performance. The results show that our service is only 3% slower than a standard unreplicated NFS.

/pdf/practical-byzantine-fault-tolerance-58gb0de0hd.pdf

Practical Byzantine fault tolerance

Fabric is a modular and extensible open-source system for deploying and operating permissioned blockchains and one of the Hyperledger projects hosted by the Linux Foundation (www.hyperledger.org). Fabric is the first truly extensible blockchain system for running distributed applications. It supports modular consensus protocols, which allows the system to be tailored to particular use cases and trust models. Fabric is also the first blockchain system that runs distributed applications written in standard, general-purpose programming languages, without systemic dependency on a native cryptocurrency. This stands in sharp contrast to existing block-chain platforms that require "smart-contracts" to be written in domain-specific languages or rely on a cryptocurrency. Fabric realizes the permissioned model using a portable notion of membership, which may be integrated with industry-standard identity management. To support such flexibility, Fabric introduces an entirely novel blockchain design and revamps the way blockchains cope with non-determinism, resource exhaustion, and performance attacks. This paper describes Fabric, its architecture, the rationale behind various design decisions, its most prominent implementation aspects, as well as its distributed application programming model. We further evaluate Fabric by implementing and benchmarking a Bitcoin-inspired digital currency. We show that Fabric achieves end-to-end throughput of more than 3500 transactions per second in certain popular deployment configurations, with sub-second latency, scaling well to over 100 peers.

https://dl.acm.org/doi/pdf/10.1145/3190508.3190538

Hyperledger fabric: a distributed operating system for permissioned blockchains

Our growing reliance on online services accessible on the Internet demands highly available systems that provide correct service without interruptions. Software bugs, operator mistakes, and malicious attacks are a major cause of service interruptions and they can cause arbitrary behavior, that is, Byzantine faults. This article describes a new replication algorithm, BFT, that can be used to build highly available systems that tolerate Byzantine faults. BFT can be used in practice to implement real services: it performs well, it is safe in asynchronous environments such as the Internet, it incorporates mechanisms to defend against Byzantine-faulty clients, and it recovers replicas proactively. The recovery mechanism allows the algorithm to tolerate any number of faults over the lifetime of the system provided fewer than 1/3 of the replicas become faulty within a small window of vulnerability. BFT has been implemented as a generic program library with a simple interface. We used the library to implement the first Byzantine-fault-tolerant NFS file system, BFS. The BFT library and BFS perform well because the library incorporates several important optimizations, the most important of which is the use of symmetric cryptography to authenticate messages. The performance results show that BFS performs 2p faster to 24p slower than production implementations of the NFS protocol that are not replicated. This supports our claim that the BFT library can be used to build practical systems that tolerate Byzantine faults.

/pdf/practical-byzantine-fault-tolerance-and-proactive-recovery-41bx19pxe1.pdf

Practical byzantine fault tolerance and proactive recovery

Wireless networks are fundamentally limited by the intensity of the received signals and by their interference. Since both of these quantities depend on the spatial location of the nodes, mathematical techniques have been developed in the last decade to provide communication-theoretic results accounting for the networks geometrical configuration. Often, the location of the nodes in the network can be modeled as random, following for example a Poisson point process. In this case, different techniques based on stochastic geometry and the theory of random geometric graphs -including point process theory, percolation theory, and probabilistic combinatorics-have led to results on the connectivity, the capacity, the outage probability, and other fundamental limits of wireless networks. This tutorial article surveys some of these techniques, discusses their application to model wireless networks, and presents some of the main results that have appeared in the literature. It also serves as an introduction to the field for the other papers in this special issue.

/pdf/stochastic-geometry-and-random-graphs-for-the-analysis-and-1e671n4af3.pdf

Stochastic geometry and random graphs for the analysis and design of wireless networks

Many atomic broadcast algorithms have been published in the last 20 years. Token-based algorithms represent a large class of these algorithms. Interestingly, all the token-based atomic broadcast algorithms rely on a group membership service and none of them uses unreliable failure detectors directly. This paper presents the first token-based atomic broadcast algorithm that uses an unreliable failure detector instead of a group membership service. It requires a system size that is quadratic in the number of supported failures. The special case of a single supported failure (f=1) requires n=3 processes. We experimentally evaluate the performance of this algorithm in local and wide area networks, in order to emphasize that atomic broadcast is efficiently implemented by combining a failure detector and a token-based mechanism. The evaluation shows that the new token-based algorithm surpasses the performance of the other algorithms in most small-system settings.

/pdf/a-fault-tolerant-token-based-atomic-broadcast-algorithm-25qbawisnu.pdf

A Fault-Tolerant Token-Based Atomic Broadcast Algorithm

Peter Urban and Andre Schiper Comparing Distributed Consensus Algorithms Protocols that solve agreement problems are essential building blocks for fault tolerant distributed systems. While many protocols have been published, little has been done to analyze their performance, especially the performance of their fault tolerance mechanisms. In this paper, we compare two consensus algorithms with different communication schemes: one is centralized and the other decentralized. The elements of the simulation study form a generic methodology for evaluating consensus algorithms. The results show that the centralized algorithm performs better in some environments, in spite of the fact that the decentralized algorithm finishes in fewer communication steps. The reason is that it generates less contention. Keywords: simulation tools and techniques, Java-based simulation, distributed consensus, benchmarks

Comparing Distributed Consensus Algorithms

While group communication systems have been proposed for some time, they are still not used much in actual systems. We believe that one reason for this is the lack of standardisation of group communication system interfaces. The paper proposes an architecture, using the standard decomposition into services, where services are based on standard interfaces: both interactions between services and interactions with the application use existing, open standards. A decomposition of the group communication into services is presented, along with a description of applicable standards. As an example, a group membership service based on the LDAP standard is discussed.

Group communication based on standard interfaces

Practical Impact of Group Communication Theory.- On the Impact of Academic Distributed Systems Research on Industrial Practice.- Using Error-Correcting Codes to Solve Distributed Agreement Problems: A Future Direction in Distributed Computing?.- Lower Bounds for Asynchronous Consensus.- Designing Algorithms for Dependent Process Failures.- Comparing the Atomic Commitment and Consensus Problems.- Open Questions on Consensus Performance inWell-Behaved Runs.- Challenges in Evaluating Distributed Algorithms.- Towards Robust Optimistic Approaches.- Towards a Practical Approach to Confidential Byzantine Fault Tolerance.- Modeling Complexity in Secure Distributed Computing.- Communication and Data Sharing for Dynamic Distributed Systems.- Dissecting Distributed Computations.- Ordering vs Timeliness: Two Facets of Consistency?.- WAIF:Web of Asynchronous Information Filters.- The Importance of Aggregation.- Dynamic Lookup Networks.- The Surprising Power of Epidemic Communication.- Topology-Aware Routing in Structured Peer-to-Peer Overlay Networks.- Uncertainty and Predictability: Can They Be Reconciled?.- Fuzzy Group Membership.- Toward Self-Organizing, Self-Repairing and Resilient Distributed Systems.- Dynamically Provisioning Distributed Systems to Meet Target Levels of Performance, Availability, and Data Quality.- Database Replication Based on Group Communication: Implementation Issues.- The Evolution of Publish/Subscribe Communication Systems.- Naming and Integrity: Self-Verifying Data in Peer-to-Peer Systems.- Spread Spectrum Storage with Mnemosyne.- Replication Strategies for Highly Available Peer-to-Peer Storage.- A Data-Centric Approach for Scalable State Machine Replication.- Scaling Optimistic Replication.- Building a Bridge between Distributed Systems Theory and Commercial Practice.- Holistic Operations in Large-Scale Sensor Network Systems: A Probabilistic Peer-to-Peer Approach.- Challenges in Making Pervasive Systems Dependable.- Towards Dependable Networks of Mobile Arbitrary Devices - Diagnosis and Scalability.- Technology Challenges for the Global Real-Time Enterprise.- Middleware for Supporting Inter-organizational Interactions.- Hosting of Libre Software Projects: A Distributed Peer-to-Peer Approach.- System Support for Pervasive Applications.- Technology Challenges for the Global Real-Time Enterprise.- Middleware for Supporting Inter-organizational Interactions.- Hosting of Libre Software Projects: A Distributed Peer-to-Peer Approach.- System Support for Pervasive Applications.

Future Directions in Distributed Computing: Research and Position Papers

This paper focuses on the design of an asynchronous broadcast primitive on the Intel SCC. Our solution is based on OC-Bcast, a state-of-the-art k-ary tree synchronous broadcast algorithm that leverages the parallelism provided by on-chip Remote Memory Accesses to Message Passing Buffers. In the paper, we study the use of parallel inter-core interrupts as a means to implement an efficient asynchronous group communication primitive, and present the userspace library we designed to be able to use interrupts in OC-Bcast and make it work asynchronously. Our experimental evaluation shows that our algorithm allows parallel broadcast operations to efficiently progress concurrently and provides low latency for a single broadcast operation. It highlights that parallel interrupts can help implementing efficient group communication primitives on many-core systems.

https://hal.archives-ouvertes.fr/hal-00719022/document

André Schiper

Papers

A Fault-Tolerant Token-Based Atomic Broadcast Algorithm

Comparing Distributed Consensus Algorithms

Group communication based on standard interfaces

Future Directions in Distributed Computing: Research and Position Papers

Asynchronous Broadcast on the Intel SCC using Interrupts