Minimizing coordination, or blocking communication between concurrently executing operations, is key to maximizing scalability, availability, and high performance in database systems. However, uninhibited coordination-free execution can compromise application correctness, or consistency. When is coordination necessary for correctness? The classic use of serializable transactions is sufficient to maintain correctness but is not necessary for all applications, sacrificing potential scalability. In this paper, we develop a formal framework, invariant confluence, that determines whether an application requires coordination for correct execution. By operating on application-level invariants over database states (e.g., integrity constraints), invariant confluence analysis provides a necessary and sufficient condition for safe, coordination-free execution. When programmers specify their application invariants, this analysis allows databases to coordinate only when anomalies that might violate invariants are possible. We analyze the invariant confluence of common invariants and operations from real-world database systems (i.e., integrity constraints) and applications and show that many are invariant confluent and therefore achievable without coordination. We apply these results to a proof-of-concept coordination-avoiding database prototype and demonstrate sizable performance gains compared to serializable execution, notably a 25-fold improvement over prior TPC-C New-Order performance on a 200 server cluster.

/pdf/coordination-avoidance-in-database-systems-4rob8ajs58.pdf

Coordination avoidance in database systems

The AOP community has successfully promoted and illustrated the power and elegance of aspect-oriented programming. One of the main problems of Aspect-oriented programming is, however, the aspect interference problem. When multiple aspects are superimposed on the same join point, undesired or incorrect behavior may emerge due to the side effects of behavior of the aspects at the join point. In this paper we present a technique and a tool to detect and correct the semantic conflicts among aspects that are superimposed on the same join point.

/pdf/reasoning-about-semantic-conflicts-between-aspects-2ozs5h6qh2.pdf

Reasoning About Semantic Conflicts Between Aspects

The rise of Internet-scale geo-replicated services has led to upheaval in the design of modern data management systems. Given the availability, latency, and throughput penalties associated with classic mechanisms such as serializable transactions, a broad class of systems (e.g., "NoSQL") has sought weaker alternatives that reduce the use of expensive coordination during system operation, often at the cost of application integrity. When can we safely forego the cost of this expensive coordination, and when must we pay the price?In this thesis, we investigate the potential for coordination avoidance---the use of as little coordination as possible while ensuring application integrity---in several modern data-intensive domains. We demonstrate how to leverage the semantic requirements of applications in data serving, transaction processing, and web services to enable more efficient distributed algorithms and system designs. The resulting prototype systems demonstrate regular order-of-magnitude speedups compared to their traditional, coordinated counterparts on a variety of tasks, including referential integrity and index maintenance, transaction execution under common isolation models, and database constraint enforcement. A range of open source applications and systems exhibit similar results.

/pdf/coordination-avoidance-in-distributed-databases-15e5dtsszx.pdf

Coordination Avoidance in Distributed Databases

In crowdsensing to mobile phones, each user needs incentives to participate. Mobile devices with sensing capabilities have enabled a new paradigm of mobile crowdsensing with a broad range of applications. A major challenge in achieving stable crowdsensing on a large scale is the incentive issue. Proper incentive mechanisms are necessary to keep the crowdsensing working. However, most existing incentive mechanisms for crowdsensing assume the system admit continuous strategies like sensing time in opposite of the fact that many digital devices and crowdsensing models only admit discretized strategies. In this paper, we show that discretization, like rounding method, can make the existing crowdsensing incentive mechanisms invalid. To address this problem, we design the incentive mechanism for discrete crowdsensing in which each user has a uniform sensing subtask length. We rigorously show that our mechanism can achieve perfect Bayesian equilibrium (PBE) and maximize the platform utility. Our algorithm is efficient since its complexity is linear to the number of users. We also consider the cases in which the users have diverse subtask lengths, and propose another two incentive mechanisms to achieve PBEs and maximize platform utility. Extensive simulations verify our mechanisms are efficient, individual-rational, and system-optimal.

Incentive Mechanisms for Discretized Mobile Crowdsensings

This paper presents TMProf, a transactional memory (TM) profiler, based on three visualization principles. These principles are (i) the precise graphical representation of transaction interactions including cross-correlated information and source code, (ii) visualized soft real-time playback of concurrently executing transactions, and (iii) dynamic visualizations of multiple executions. We describe how these principles break new ground and create new challenges for TM profilers. We discuss our experience using TMProf with InvalSTM, a state-of-the-art software TM, and show how TMProf's feedback led to the design of two new contention managers (CMs). We demonstrate the performance benefits of these CMs, which generally led to improved performance as the amount of work and threads increase per benchmark. Our experimental results show that iBalanced, one of our newly designed CMs, can increase transaction throughput by nearly 10× over iFair, InvalSTM's previously best performing CM.

Visualizing transactional memory

From the Publisher:
This book develops a theory for transactions that provides practical solutions for system developers, focusing on the interface between the user and the database that executes transactions. Atomic transactions are a useful abstraction for programming concurrent and distributed data processing systems. Presents many important algorithms which provide maximum concurrency for transaction processing without sacrificing data integrity. The authors include a well-developed data processing case study to help readers understand transaction processing algorithms more clearly. The book offers conceptual tools for the design of new algorithms, and for devising variations on the familiar algorithms presented in the discussions. Whether your background is in the development of practical systems or formal methods, this book will offer you a new way to view distributed systems.

Ronald R. Yager

Papers

Atomic Transactions: In Concurrent and Distributed Systems