CompCertTSO: A Verified Compiler for Relaxed-Memory Concurrency
TL;DR: The semantic design and verified compilation of a C-like programming language for concurrent shared-memory computation on x86 multiprocessors is considered, and some verified fence-elimination optimizations, integrated into CompCertTSO are described.
Abstract: In this article, we consider the semantic design and verified compilation of a C-like programming language for concurrent shared-memory computation on x86 multiprocessors. The design of such a language is made surprisingly subtle by several factors: the relaxed-memory behavior of the hardware, the effects of compiler optimization on concurrent code, the need to support high-performance concurrent algorithms, and the desire for a reasonably simple programming model. In turn, this complexity makes verified compilation both essential and challenging.We describe ClightTSO, a concurrent extension of CompCert’s Clight in which the TSO-based memory model of x86 multiprocessors is exposed for high-performance code, and CompCertTSO, a formally verified compiler from ClightTSO to x86 assembly language, building on CompCert. CompCertTSO is verified in Coq: for any well-behaved and successfully compiled ClightTSO source program, any permitted observable behavior of the generated assembly code (if it does not run out of memory) is also possible in the source semantics. We also describe some verified fence-elimination optimizations, integrated into CompCertTSO.
Citations
More filters
14 Jan 2015
TL;DR: This paper presents a new layer calculus showing how to formally specify, program, verify, and compose abstraction layers and shows that they correspond to a strong form of abstraction over a particularly rich class of specifications which they call deep specifications.
Abstract: Modern computer systems consist of a multitude of abstraction layers (e.g., OS kernels, hypervisors, device drivers, network protocols), each of which defines an interface that hides the implementation details of a particular set of functionality. Client programs built on top of each layer can be understood solely based on the interface, independent of the layer implementation. Despite their obvious importance, abstraction layers have mostly been treated as a system concept; they have almost never been formally specified or verified. This makes it difficult to establish strong correctness properties, and to scale program verification across multiple layers. In this paper, we present a novel language-based account of abstraction layers and show that they correspond to a strong form of abstraction over a particularly rich class of specifications which we call deep specifications. Just as data abstraction in typed functional languages leads to the important representation independence property, abstraction over deep specification is characterized by an important implementation independence property: any two implementations of the same deep specification must have contextually equivalent behaviors. We present a new layer calculus showing how to formally specify, program, verify, and compose abstraction layers. We show how to instantiate the layer calculus in realistic programming languages such as C and assembly, and how to adapt the CompCert verified compiler to compile certified C layers such that they can be linked with assembly layers. Using these new languages and tools, we have successfully developed multiple certified OS kernels in the Coq proof assistant, the most realistic of which consists of 37 abstraction layers, took less than one person year to develop, and can boot a version of Linux as a guest.
168 citations
Cites background from "CompCertTSO: A Verified Compiler fo..."
...We leave the investigation of nondeterministic deep specifications as future work....
[...]
11 Jan 2016
TL;DR: This paper develops a concurrency model with a microarchitectural flavour, abstracting from many hardware implementation concerns but still close to hardware-designer intuition, and builds a tool from the combined semantics that lets one explore the full range of architecturally allowed behaviour, for litmus tests and (small) ELF executables.
Abstract: In this paper we develop semantics for key aspects of the ARMv8 multiprocessor architecture: the concurrency model and much of the 64-bit application-level instruction set (ISA). Our goal is to clarify what the range of architecturally allowable behaviour is, and thereby to support future work on formal verification, analysis, and testing of concurrent ARM software and hardware. Establishing such models with high confidence is intrinsically difficult: it involves capturing the vendor's architectural intent, aspects of which (especially for concurrency) have not previously been precisely defined. We therefore first develop a concurrency model with a microarchitectural flavour, abstracting from many hardware implementation concerns but still close to hardware-designer intuition. This means it can be discussed in detail with ARM architects. We then develop a more abstract model, better suited for use as an architectural specification, which we prove sound w.r.t.~the first. The instruction semantics involves further difficulties, handling the mass of detail and the subtle intensional information required to interface to the concurrency model. We have a novel ISA description language, with a lightweight dependent type system, letting us do both with a rather direct representation of the ARM reference manual instruction descriptions. We build a tool from the combined semantics that lets one explore, either interactively or exhaustively, the full range of architecturally allowed behaviour, for litmus tests and (small) ELF executables. We prove correctness of some optimisations needed for tool performance. We validate the models by discussion with ARM staff, and by comparison against ARM hardware behaviour, for ISA single- instruction tests and concurrent litmus tests.
145 citations
Cites background from "CompCertTSO: A Verified Compiler fo..."
...[5, 13–15, 18, 26] With the exception of CompCertTSO [26], which was w....
[...]
01 Jan 2017
TL;DR: The first relaxed memory model that accounts for a broad spectrum of features from the C++11 concurrency model, is implementable, and defines the semantics of racy programs without relying on undefined behaviors, which is a prerequisite for applicability to type-safe languages like Java is proposed.
Abstract: Despite many years of research, it has proven very difficult to develop a memory model for concurrent programming languages that adequately balances the conflicting desiderata of programmers, compilers, and hardware. In this paper, we propose the first relaxed memory model that (1) accounts for a broad spectrum of features from the C++11 concurrency model, (2) is implementable, in the sense that it provably validates many standard compiler optimizations and reorderings, as well as standard compilation schemes to x86-TSO and Power, (3) justifies simple invariant-based reasoning, thus demonstrating the absence of bad "out-of-thin-air" behaviors, (4) supports "DRF" guarantees, ensuring that programmers who use sufficient synchronization need not understand the full complexities of relaxed-memory semantics, and (5) defines the semantics of racy programs without relying on undefined behaviors, which is a prerequisite for applicability to type-safe languages like Java. The key novel idea behind our model is the notion of *promises*: a thread may promise to execute a write in the future, thus enabling other threads to read from that write out of order. Crucially, to prevent out-of-thin-air behaviors, a promise step requires a thread-local certification that it will be possible to execute the promised write even in the absence of the promise. To establish confidence in our model, we have formalized most of our key results in Coq.
134 citations
14 Jan 2015
TL;DR: It is shown that the weak memory model introduced by the 2011 C and C++ standards does not permit many common source-to-source program transformations that modern compilers perform and that are deemed to be correct.
Abstract: We show that the weak memory model introduced by the 2011 C and C++ standards does not permit many common source-to-source program transformations (such as expression linearisation and "roach motel" reorderings) that modern compilers perform and that are deemed to be correct. As such it cannot be used to define the semantics of intermediate languages of compilers, as, for instance, LLVM aimed to. We consider a number of possible local fixes, some strengthening and some weakening the model. We evaluate the proposed fixes by determining which program transformations are valid with respect to each of the patched models. We provide formal Coq proofs of their correctness or counterexamples as appropriate.
115 citations
Additional excerpts
...The certified compilers CompCert [9] and CompCertTSO [20] (the latter extending an earlier version of the former to concurrent shared memory programming with a TSO-based memory semantics) share the same memory model for all the intermediate languages....
[...]
References
More filters
Book•
01 Jan 1989TL;DR: This chapter discusses Bisimulation and Observation Equivalence as a Modelling Communication, a Programming Language, and its application to Equational laws.
Abstract: Foreword. 1. Modelling Communication. 2. Basic Definitions. 3. Equational laws and Their Application. 4. Strong Bisimulation and Strong Equivalence. 5. Bisimulation and Observation Equivalence. 6. Further Examples. 7. The Theory of Observation Congruence. 8. Defining a Programming Language. 9. Operators and Calculi. 10. Specifications and Logic. 11. Determinancy and Confluence. 12. Sources and Related Work. Bibliography. Index.
8,625 citations
"CompCertTSO: A Verified Compiler fo..." refers background in this paper
...These transitions are in the style of the early transition system for value-passing CCS [Milner 1989]: a thread doing a memory read will have a transition for each possible value of the right type....
[...]
Book•
01 Jan 1985TL;DR: Bjarne Stroustrup makes C even more accessible to those new to the language, while adding advanced information and techniques that even expert C programmers will find invaluable.
Abstract: From the Publisher:
Written by Bjarne Stroustrup, the creator of C, this is the world's most trusted and widely read book on C.
For this special hardcover edition, two new appendixes on locales and standard library exception safety have been added. The result is complete, authoritative coverage of the C language, its standard library, and key design techniques. Based on the ANSI/ISO C standard, The C Programming Language provides current and comprehensive coverage of all C language features and standard library components.
For example:
abstract classes as interfaces class hierarchies for object-oriented programming templates as the basis for type-safe generic software exceptions for regular error handling namespaces for modularity in large-scale software run-time type identification for loosely coupled systems the C subset of C for C compatibility and system-level work standard containers and algorithms standard strings, I/O streams, and numerics C compatibility, internationalization, and exception safety
Bjarne Stroustrup makes C even more accessible to those new to the language, while adding advanced information and techniques that even expert C programmers will find invaluable.
6,795 citations
TL;DR: Many large sequential computers execute operations in a different order than is specified by the program, and a correct execution by each processor does not guarantee the correct execution of the entire program.
Abstract: Many large sequential computers execute operations in a different order than is specified by the program. A correct execution is achieved if the results produced are the same as would be produced by executing the program steps in order. For a multiprocessor computer, such a correct execution by each processor does not guarantee the correct execution of the entire program. Additional conditions are given which do guarantee that a computer correctly executes multiprocess programs.
2,301 citations
"CompCertTSO: A Verified Compiler fo..." refers background in this paper
...…x86, Sparc, Power, Itanium, and ARM) provide only relaxed shared-memory abstractions, substantially weaker than sequentially consistent (SC) memory [Lamport 1979]: some of the hardware optimizations they rely on, while unobservable to sequential code, can observably affect the behavior of…...
[...]
TL;DR: This paper reports on the development and formal verification of CompCert, a compiler from Clight (a large subset of the C programming language) to PowerPC assembly code, using the Coq proof assistant both for programming the compiler and for proving its correctness.
Abstract: This paper reports on the development and formal verification (proof of semantic preservation) of CompCert, a compiler from Clight (a large subset of the C programming language) to PowerPC assembly code, using the Coq proof assistant both for programming the compiler and for proving its correctness. Such a verified compiler is useful in the context of critical software and its formal verification: the verification of the compiler guarantees that the safety properties proved on the source code hold for the executable compiled code as well.
1,124 citations
"CompCertTSO: A Verified Compiler fo..." refers background in this paper
...Syntactically, ClightTSO is a straightforward extension of the CompCert Clight language [Blazy and Leroy 2009], adding thread creation and some atomic read-modifywrite primitives that are directly implementable by x86 LOCK d instructions....
[...]
...In the sequential setting, veri.ed compilation has recently been shown to be feasible by Leroy et al. s CompCert [Blazy and Leroy 2009; Leroy 2009b, 2009c; Leroy and Blazy 2008]....
[...]
...CompCert 1.5, our starting point, is a veri.ed compiler from a sequential C-like language, Clight, to PowerPC and ARM assembly language [Leroy 2009a].1 In this article, we consider veri.ed compilation in the setting of concurrent programs with a realistic relaxed memory model....
[...]
30 Sep 2008
TL;DR: This paper introduces the Stanford Transactional Application for Multi-Processing (STAMP), a comprehensive benchmark suite for evaluating TM systems and uses the suite to evaluate six different TM systems, identify their shortcomings, and motivate further research on their performance characteristics.
Abstract: Transactional Memory (TM) is emerging as a promising technology to simplify parallel programming. While several TM systems have been proposed in the research literature, we are still missing the tools and workloads necessary to analyze and compare the proposals. Most TM systems have been evaluated using microbenchmarks, which may not be representative of any real-world behavior, or individual applications, which do not stress a wide range of execution scenarios. We introduce the Stanford Transactional Application for Multi-Processing (STAMP), a comprehensive benchmark suite for evaluating TM systems. STAMP includes eight applications and thirty variants of input parameters and data sets in order to represent several application domains and cover a wide range of transactional execution cases (frequent or rare use of transactions, large or small transactions, high or low contention, etc.). Moreover, STAMP is portable across many types of TM systems, including hardware, software, and hybrid systems. In this paper, we provide descriptions and a detailed characterization of the applications in STAMP. We also use the suite to evaluate six different TM systems, identify their shortcomings, and motivate further research on their performance characteristics.
934 citations
"CompCertTSO: A Verified Compiler fo..." refers background in this paper
...…1986], the TL2 lock-based STM [Dice et al. 2006], Fraser s lockfree skiplist implementation [Fraser 2003], and several of the STAMP benchmarks [Cao Minh et al. 2008]; for each the table reports the total numbers of fences in the generated assembler .les, following the br and aw…...
[...]