Test suite

Abstract: We present a new symbolic execution tool, KLEE, capable of automatically generating tests that achieve high coverage on a diverse set of complex and environmentally-intensive programs. We used KLEE to thoroughly check all 89 stand-alone programs in the GNU COREUTILS utility suite, which form the core user-level environment installed on millions of Unix systems, and arguably are the single most heavily tested set of open-source programs in existence. KLEE-generated tests achieve high line coverage -- on average over 90% per tool (median: over 94%) -- and significantly beat the coverage of the developers' own hand-written test suite. When we did the same for 75 equivalent tools in the BUSYBOX embedded system suite, results were even better, including 100% coverage on 31 of them.We also used KLEE as a bug finding tool, applying it to 452 applications (over 430K total lines of code), where it found 56 serious bugs, including three in COREUTILS that had been missed for over 15 years. Finally, we used KLEE to crosscheck purportedly identical BUSYBOX and COREUTILS utilities, finding functional correctness errors and a myriad of inconsistencies.

Abstract: : This research organizes, presents, and analyzes contemporary Multiobjective Evolutionary Algorithm (MOEA) research and associated Multiobjective Optimization Problems (MOPs). Using a consistent MOEA terminology and notation, each cited MOEAs' key factors are presented in tabular form for ease of MOEA identification and selection. A detailed quantitative and qualitative MOEA analysis is presented, providing a basis for conclusions about various MOEA-related issues. The traditional notion of building blocks is extended to the MOP domain in an effort to develop more effective and efficient MOEAs. Additionally, the MOEA community's limited test suites contain various functions whose origins and rationale for use are often unknown. Thus, using general test suite guidelines appropriate MOEA test function suites are substantiated and generated. An experimental methodology incorporating a solution database and appropriate metrics is offered as a proposed evaluation framework allowing absolute comparisons of specific MOEA approaches. Taken together, this document's classifications, analyses, and new innovations present a complete, contemporary view of current MOEA "state of the art" and possible future research. Researchers with basic EA knowledge may also use part of it as a largely self-contained introduction to MOEAs.

Abstract: Regression testing is a testing activity that is performed to provide confidence that changes do not harm the existing behaviour of the software. Test suites tend to grow in size as software evolves, often making it too costly to execute entire test suites. A number of different approaches have been studied to maximize the value of the accrued test suite: minimization, selection and prioritization. Test suite minimization seeks to eliminate redundant test cases in order to reduce the number of tests to run. Test case selection seeks to identify the test cases that are relevant to some set of recent changes. Test case prioritization seeks to order test cases in such a way that early fault detection is maximized. This paper surveys each area of minimization, selection and prioritization technique and discusses open problems and potential directions for future research. Copyright © 2010 John Wiley & Sons, Ltd.

Abstract: Objective measurement of test quality is one of the key issues in software testing. It has been a major research focus for the last two decades. Many test criteria have been proposed and studied for this purpose. Various kinds of rationales have been presented in support of one criterion or another. We survey the research work in this area. The notion of adequacy criteria is examined together with its role in software dynamic testing. A review of criteria classification is followed by a summary of the methods for comparison and assessment of criteria.

Abstract: List of Figures. List of Tables. List of Procedures. Foreword. Preface. Acknowledgments. I. PRELIMINARIES. 1. A Small Challenge. 2. How to Use This Book. Reader Guidance. Conventions. FAQs for Object-oriented Testing. Test Process. 3. Testing: A Brief Introduction. What Is Software Testing? Definitions. The Limits of Testing. What Can Testing Accomplish? Bibliographic Notes. 4. With the Necessary Changes: Testing and Object-oriented Software. The Dismal Science of Software Testing. Side Effects of the Paradigm. Language-specific Hazards. Coverage Models for Object-oriented Testing. An OO Testing Manifesto. Bibliographic Notes. II. MODELS. 5. Test Models. Test Design and Test Models. Bibliographic Notes. 6. Combinational Models. How Combinational Models Support Testing. How to Develop a Decision Table. Deriving the Logic Function. Decision Table Validation. Test Generation. Choosing a Combinational Test Strategy. Bibliographic Notes. 7. State Machines. Motivation. The Basic Model. The FREE State Model. State-based Test Design. Bibliographic Notes. 8. A Tester's Guide to the UML. Introduction. General-purpose Elements. Use Case Diagram. Class Diagram. Sequence Diagram. Activity Diagram. Statechart Diagram. Collaboration Diagram. Component Diagram. Deployment Diagram. Graphs, Relations, and Testing. Bibliographic Notes. III. PATTERNS. 9. Results-oriented Test Strategy. Results-oriented Testing. Test Design Patterns. Test Design Template. Documenting Test Cases, Suites, and Plans. Bibliographic Notes. 10. Classes. Class Test and Integration. Preliminaries. Method Scope Test Design Patterns. Category-Partition. Combinational Function Test. Recursive Function Test. Polymorphic Message Test. Class Scope Test Design Patterns. Invariant Boundaries. Nonmodal Class Test. Quasi-modal Class Test. Modal Class Test. Flattened Class Scope Test Design Patterns. Polymorphic Server Test. Modal Hierarchy Test. Bibliographic Notes. 11. Reusable Components. Testing and Reuse. Test Design Patterns. Abstract Class Test. Generic Class Test. New Framework Test. Popular Framework Test. Bibliographic Notes. 12. Subsystems. Subsystems. Subsystem Test Design Patterns. Class Association Test. Round-trip Scenario Test. Controlled Exception Test. Mode Machine Test. Bibliographic Notes. 13. Integration. Integration in Object-oriented Development. Integration Patterns. Subsystem/System Scope. Big Bang Integration. Bottom-up Integration. Top-down Integration. Collaboration Integration. Backbone Integration. Layer Integration. Client/Server Integration. Distributed Services Integration. High-frequency Integration. Bibliographic Notes. 14. Application Systems. Testing Application Systems. Test Design Patterns. Extended Use Case Test. Covered in CRUD. Allocate Tests by Profile. Implementation-specific Capabilities. Post-development Testing. Note on Testing Performance Objectives. Bibliographic Notes. 15. Regression Testing. Preliminaries. Test Patterns. Retest All. Retest Risky Use Cases. Retest by Profile. Retest Changed Code. Retest Within Firewall. Bibliographic Notes. IV. TOOLS. 16. Test Automation. Why Testing Must Be Automated. Limitations and Caveats. 17. Assertions. Introduction. Implementation-based Assertions. Responsibility-based Assertions. Implementation. The Percolation Pattern. Deployment. Limitations and Caveats. Some Assertion Tools. Bibliographic Notes. 18. Oracles. Introduction. Oracle Patterns. Comparators. Bibliographic Notes. 19. Test Harness Design. How to Develop a Test Harness. Test Case Patterns. Test Case/Test Suite Method. Test Case/Test Suite Class. Catch All Exceptions. Test Control Patterns. Server Stub. Server Proxy. Driver Patterns. TestDriver Superclass. Percolate the Object Under Test. Symmetric Driver. Subclass Driver. Private Access Driver. Test Control Interface. Drone. Built-in Test Driver. Test Execution Patterns. Command Line Test Bundle. Incremental Testing Framework. Fresh Objects. A Test Implementation Syntax. Bibliographic Notes. Appendix. BigFoot's Tootsie: A Case Study. Requirements. OOA/D for Capability-driven Testing. Implementation. Glossary. References. Index. 0201809389T04062001