An empirical study of the reliability of UNIX utilities
TL;DR: The following section describes the tools built to test the utilities, including the fuzz (random character) generator, ptyjig (to test interactive utilities), and scripts to automate the testing process.
Abstract: The following section describes the tools we built to test the utilities. These tools include the fuzz (random character) generator, ptyjig (to test interactive utilities), and scripts to automate the testing process. Next, we will describe the tests we performed, giving the types of input we presented to the utilities. Results from the tests will follow along with an analysis of the results, including identification and classification of the program bugs that caused the crashes. The final section presents concluding remarks, including suggestions for avoiding the types of problems detected by our study and some commentary on the bugs we found. We include an Appendix with the user manual pages for fuzz and ptyjig.
Citations
More filters
Proceedings Article•
26 Jan 1998TL;DR: StackGuard is described: a simple compiler technique that virtually eliminates buffer overflow vulnerabilities with only modest performance penalties, and a set of variations on the technique that trade-off between penetration resistance and performance.
Abstract: This paper presents a systematic solution to the persistent problem of buffer overflow attacks. Buffer overflow attacks gained notoriety in 1988 as part of the Morris Worm incident on the Internet. While it is fairly simple to fix individual buffer overflow vulnerabilities, buffer overflow attacks continue to this day. Hundreds of attacks have been discovered, and while most of the obvious vulnerabilities have now been patched, more sophisticated buffer overflow attacks continue to emerge.
We describe StackGuard: a simple compiler technique that virtually eliminates buffer overflow vulnerabilities with only modest performance penalties. Privileged programs that are recompiled with the StackGuard compiler extension no longer yield control to the attacker, but rather enter a fail-safe state. These programs require no source code changes at all, and are binary-compatible with existing operating systems and libraries. We describe the compiler technique (a simple patch to gcc), as well as a set of variations on the technique that trade-off between penetration resistance and performance. We present experimental results of both the penetration resistance and the performance impact of this technique.
1,536 citations
Cites background from "An empirical study of the reliabili..."
...Many C programs have buffer overflow vulnerabilities, both because the C language lacks array bounds checking, and because the culture of C programmers encourages a performance-oriented style that avoids error checking where possible [14, 13]....
[...]
TL;DR: The delta debugging algorithm generalizes and simplifies the failing test case to a minimal test case that still produces the failure, and isolates the difference between a passing and a failingTest case.
Abstract: Given some test case, a program fails. Which circumstances of the test case are responsible for the particular failure? The delta debugging algorithm generalizes and simplifies the failing test case to a minimal test case that still produces the failure. It also isolates the difference between a passing and a failing test case. In a case study, the Mozilla Web browser crashed after 95 user actions. Our prototype implementation automatically simplified the input to three relevant user actions. Likewise, it simplified 896 lines of HTML to the single line that caused the failure. The case study required 139 automated test runs or 35 minutes on a 500 MHz PC.
980 citations
Cites methods from "An empirical study of the reliabili..."
...In a classical experiment [6], [7], Miller et al....
[...]
TL;DR: This paper describes GenProg, an automated method for repairing defects in off-the-shelf, legacy programs without formal specifications, program annotations, or special coding practices, and analyzes the generated repairs qualitatively and quantitatively to demonstrate the process efficiently produces evolved programs that repair the defect.
Abstract: This paper describes GenProg, an automated method for repairing defects in off-the-shelf, legacy programs without formal specifications, program annotations, or special coding practices. GenProg uses an extended form of genetic programming to evolve a program variant that retains required functionality but is not susceptible to a given defect, using existing test suites to encode both the defect and required functionality. Structural differencing algorithms and delta debugging reduce the difference between this variant and the original program to a minimal repair. We describe the algorithm and report experimental results of its success on 16 programs totaling 1.25 M lines of C code and 120K lines of module code, spanning eight classes of defects, in 357 seconds, on average. We analyze the generated repairs qualitatively and quantitatively to demonstrate that the process efficiently produces evolved programs that repair the defect, are not fragile input memorizations, and do not lead to serious degradation in functionality.
930 citations
Cites background from "An empirical study of the reliabili..."
...’s work on fuzz testing, in which programs crash when given random inputs [34]....
[...]
TL;DR: This article presents EXE, an effective bug-finding tool that automatically generates inputs that crash real code by solving the current path constraints to find concrete values using its own co-designed constraint solver, STP.
Abstract: This article presents EXE, an effective bug-finding tool that automatically generates inputs that crash real code. Instead of running code on manually or randomly constructed input, EXE runs it on symbolic input initially allowed to be anything. As checked code runs, EXE tracks the constraints on each symbolic (i.e., input-derived) memory location. If a statement uses a symbolic value, EXE does not run it, but instead adds it as an input-constraint; all other statements run as usual. If code conditionally checks a symbolic expression, EXE forks execution, constraining the expression to be true on the true branch and false on the other. Because EXE reasons about all possible values on a path, it has much more power than a traditional runtime tool: (1) it can force execution down any feasible program path and (2) at dangerous operations (e.g., a pointer dereference), it detects if the current path constraints allow any value that causes a bug. When a path terminates or hits a bug, EXE automatically generates a test case by solving the current path constraints to find concrete values using its own co-designed constraint solver, STP. Because EXE’s constraints have no approximations, feeding this concrete input to an uninstrumented version of the checked code will cause it to follow the same path and hit the same bug (assuming deterministic code).EXE works well on real code, finding bugs along with inputs that trigger them in: the BSD and Linux packet filter implementations, the dhcpd DHCP server, the pcre regular expression library, and three Linux file systems.
912 citations
Cites methods from "An empirical study of the reliabili..."
...When a path terminates or hits a bug, EXE automatically generates a test case by solving the current pathconstraintsto .nd concretevaluesusingitsownco-designed constraint solver,STP....
[...]
24 May 2007
TL;DR: Experimental results indicate that feedback-directed random test generation can outperform systematic and undirectedrandom test generation, in terms of coverage and error detection.
Abstract: We present a technique that improves random test generation by incorporating feedback obtained from executing test inputs as they are created. Our technique builds inputs incrementally by randomly selecting a method call to apply and finding arguments from among previously-constructed inputs. As soon as an input is built, it is executed and checked against a set of contracts and filters. The result of the execution determines whether the input is redundant, illegal, contract-violating, or useful for generating more inputs. The technique outputs a test suite consisting of unit tests for the classes under test. Passing tests can be used to ensure that code contracts are preserved across program changes; failing tests (that violate one or more contract) point to potential errors that should be corrected. Our experimental results indicate that feedback-directed random test generation can outperform systematic and undirected random test generation, in terms of coverage and error detection. On four small but nontrivial data structures (used previously in the literature), our technique achieves higher or equal block and predicate coverage than model checking (with and without abstraction) and undirected random generation. On 14 large, widely-used libraries (comprising 780KLOC), feedback-directed random test generation finds many previously-unknown errors, not found by either model checking or undirected random generation.
815 citations
Cites methods from "An empirical study of the reliabili..."
...Random testing [14] has been used to find errors in many applications; a partial list includes Unix utilities [19], Windows GUI applications [10], Haskell programs [2], and Java programs [3, 23, 22]....
[...]
References
More filters
TL;DR: In this paper, the authors discuss modularization as a mechanism for improving the flexibility and comprehensibility of a system while allowing the shortening of its development time, and the effectiveness of modularization is dependent upon the criteria used in dividing the system into modules.
Abstract: This paper discusses modularization as a mechanism for improving the flexibility and comprehensibility of a system while allowing the shortening of its development time. The effectiveness of a “modularization” is dependent upon the criteria used in dividing the system into modules. A system design problem is presented and both a conventional and unconventional decomposition are described. It is shown that the unconventional decompositions have distinct advantages for the goals outlined. The criteria used in arriving at the decompositions are discussed. The unconventional decomposition, if implemented with the conventional assumption that a module consists of one or more subroutines, will be less efficient in most cases. An alternative approach to implementation which does not have this effect is sketched.
5,028 citations
TL;DR: My considerations are that, although the programmer's activity ends when he has constructed a correct program, the process taking place under control of his program is the true subject matter of his activity, and that his intellectual powers are rather geared to master static relations and his powers to visualize processes evolving in time are relatively poorly developed.
Abstract: For a number of years I have been familiar with the observation that the quality of programmers is a decreasing function of the density of go to statements in the programs they produce. More recently I discovered why the use of the go to statement has such disastrous effects, and I became convinced that the go to statement should be abolished from all "higher level" programming languages (i.e. everything except, perhaps, plain machine Code). At'that time I did not attach too much importance to this discovery ; I now submit my considerations for publication because in very recent discussions in which the subject turned up, I have been urged to do so. My first remark is that, although the programmer's activity ends when he has constructed a correct program, the process taking place under control of his program is the true subject matter of his activity, for it is this process that has to accomplish the desired effect; it is this process that in its dynamic behavior has to satisfy the desired specifications. Yet, once the program has been made, the "making" of the corresponding process is delegated to the machine. My second remark is that our intellectual powers are rather geared to master static relations and that our powers to visualize processes evolving in time are relatively poorly developed. For that reason we should do (as wise programmers aware of our limitations) our utmost to shorten the conceptual gap between the static program and the dynamic process, to make the correspondence between the program (spread out in text space) and the process (spread out in time) as trivial as possible. Let us now consider how we can characterize the progress of a process. (You may think about this question in a very concrete manner: suppose that a process, considered as a time succession of actions, is stopped after an arbitrary action, what data do we have to fix in order that we can redo the process until the very same point?) If the program text is a pure concatenation of, say, assignment statements (for the purpose of this discussion regarded as the descriptions of single actions) it is sufficient to point in the program text to a point between two successive action descriptions. (In the absence of go to statements I can permit myself the syntactic ambiguity in the last three words of the previous sentence: if we parse …
911 citations
Book•
01 Jan 1979
TL;DR: In form and content, Dijkstra's letter is similar to his 1965 paper, and the last few paragraphs underscore once again why the subject of structured programming stayed out of the mainstream of the data processing industry for so long.
Abstract: To many people, Dijkstra's letter to the Editor of Communications of the A CM, published in March 1968, marks the true beginning of structured programming. That it influenced the industry is clear, if for no other reason than for the articles it spawned, ranging from "IF-THEN-ELSE Considered Harmful," to "The Else Must Go, Too," to "Programming Considered Harmful."
In form and content, Dijkstra's letter is similar to his 1965 paper, which appears first in this collection. Description of the inverse relationship between a programmer's ability and the density of goto statements in his program is repeated, as is the emphasis on the limited ability of the human brain. Much of the discussion is somewhat theoretical in nature, and the typical COBOL programmer will hunger for some coding examples so that he can see why goto statements make program logic harder to understand.
Echoing his 1965 paper, the last few paragraphs underscore once again why the subject of structured programming stayed out of the mainstream of the data processing industry for so long. As Dijkstra points out, goto statements were a subject of discussion among academicians as far back as 1959. But even today, people whom Dijkstra acknowledges --- names like Wirth, Hoare, Strachey, and Landin --- are not well known to business-oriented or scientificoriented programmers, so it should be no surprise that their ideas have languished for so many years.
432 citations
TL;DR: It is shown that the class of reversible context-free grammars can be identified in the limit frompositive samples of structural descriptions and there exists an efficient algorithm to identify them from positive samples ofStructural descriptions, where a structural description of a context- free grammar is an unlabelled derivation tree of the grammar.
Abstract: In this paper, we introduce a new normal form for context-free grammars, called reversible context-free grammars, for the problem of learning context-free grammars from positive-only examples. A context-free grammar G = (N, @S, P, S) is said to be reversible if (1) A -> @a and B -> @a in P implies A = B and (2) A -> @[email protected] and A -> @[email protected] in P implies B = C. We show that the class of reversible context-free grammars can be identified in the limit from positive samples of structural descriptions and there exists an efficient algorithm to identify them from positive samples of structural descriptions, where a structural description of a context-free grammar is an unlabelled derivation tree of the grammar. This implies that if positive structural examples of a reversible context-free grammar for the target language are available to the learning algorithm, the full class of context-free languages can be learned efficiently from positive samples.
208 citations
[...]
TL;DR: Last November the Internet was infected with a worm program that eventually spread to thousands of machines, disrupting normal activities and Internet connectivity for many days.
Abstract: Last November the Internet was infected with a worm program that eventually spread to thousands of machines, disrupting normal activities and Internet connectivity for many days. The following article examines just how this worm operated.
149 citations