scispace - formally typeset
Search or ask a question
Journal ArticleDOI

An orchestrated survey of methodologies for automated software test case generation

TL;DR: An orchestrated survey of the most prominent techniques for automatic generation of software test cases, reviewed in self-standing sections, aimed at giving an introductory, up-to-date and (relatively) short overview of research in automatic test case generation.
About: This article is published in Journal of Systems and Software.The article was published on 2013-08-01. It has received 599 citations till now. The article focuses on the topics: Software reliability testing & Test strategy.
Citations
More filters
Proceedings ArticleDOI
27 May 2018
TL;DR: DeepTest is a systematic testing tool for automatically detecting erroneous behaviors of DNN-driven vehicles that can potentially lead to fatal crashes and systematically explore different parts of the DNN logic by generating test inputs that maximize the numbers of activated neurons.
Abstract: Recent advances in Deep Neural Networks (DNNs) have led to the development of DNN-driven autonomous cars that, using sensors like camera, LiDAR, etc., can drive without any human intervention. Most major manufacturers including Tesla, GM, Ford, BMW, and Waymo/Google are working on building and testing different types of autonomous vehicles. The lawmakers of several US states including California, Texas, and New York have passed new legislation to fast-track the process of testing and deployment of autonomous vehicles on their roads. However, despite their spectacular progress, DNNs, just like traditional software, often demonstrate incorrect or unexpected corner-case behaviors that can lead to potentially fatal collisions. Several such real-world accidents involving autonomous cars have already happened including one which resulted in a fatality. Most existing testing techniques for DNN-driven vehicles are heavily dependent on the manual collection of test data under different driving conditions which become prohibitively expensive as the number of test conditions increases. In this paper, we design, implement, and evaluate DeepTest, a systematic testing tool for automatically detecting erroneous behaviors of DNN-driven vehicles that can potentially lead to fatal crashes. First, our tool is designed to automatically generated test cases leveraging real-world changes in driving conditions like rain, fog, lighting conditions, etc. DeepTest systematically explore different parts of the DNN logic by generating test inputs that maximize the numbers of activated neurons. DeepTest found thousands of erroneous behaviors under different realistic driving conditions (e.g., blurring, rain, fog, etc.) many of which lead to potentially fatal crashes in three top performing DNNs in the Udacity self-driving car challenge.

963 citations


Additional excerpts

  • ...[27], McMinn et al....

    [...]

Journal ArticleDOI
TL;DR: This article provides a comprehensive survey on metamorphic testing, which summarises the research results and application areas, and analyses common practice in empirical studies of metamorphIC testing as well as the main open challenges.
Abstract: A test oracle determines whether a test execution reveals a fault, often by comparing the observed program output to the expected output. This is not always practical, for example when a program's input-output relation is complex and difficult to capture formally. Metamorphic testing provides an alternative, where correctness is not determined by checking an individual concrete output, but by applying a transformation to a test input and observing how the program output “morphs” into a different one as a result. Since the introduction of such metamorphic relations in 1998, many contributions on metamorphic testing have been made, and the technique has seen successful applications in a variety of domains, ranging from web services to computer graphics. This article provides a comprehensive survey on metamorphic testing: It summarises the research results and application areas, and analyses common practice in empirical studies of metamorphic testing as well as the main open challenges.

362 citations


Cites background or methods from "An orchestrated survey of methodolo..."

  • ...To report the results, we also took inspiration from recent surveys on related topics such as the oracle problem [3], search–based testing [24], automated test case generation [2] and mutation analysis [25]....

    [...]

  • ...This article provides a comprehensive survey on metamorphic testing: It summarises the research results and application areas, and analyses common practice in empirical studies of metamorphic testing as well as the main open challenges....

    [...]

Book ChapterDOI
01 Jan 2019
TL;DR: This chapter presents a survey of recent advances, over the past decade, related to the fundamental problems of mutation testing and sets out the challenges and open problems for the future development of the method.
Abstract: Mutation testing realizes the idea of using artificial defects to support testing activities. Mutation is typically used as a way to evaluate the adequacy of test suites, to guide the generation of test cases, and to support experimentation. Mutation has reached a maturity phase and gradually gains popularity both in academia and in industry. This chapter presents a survey of recent advances, over the past decade, related to the fundamental problems of mutation testing and sets out the challenges and open problems for the future development of the method. It also collects advices on best practices related to the use of mutation in empirical studies of software testing. Thus, giving the reader a “mini-handbook”-style roadmap for the application of mutation testing as experimental methodology.

317 citations

BookDOI
01 Jan 2016
TL;DR: This book is the definitive guide to KeY that lets you explore the full potential of deductive software verification in practice and contains the complete theory behind KeY for active researchers who want to understand it in depth or use it in their own work.
Abstract: Static analysis of software with deductive methods is a highly dynamic field of research on the verge of becoming a mainstream technology in software engineering. It consists of a large portfolio of - mostly fully automated - analyses: formal verification, test generation, security analysis, visualization, and debugging. All of them are realized in the state-of-art deductive verification framework KeY. This book is the definitive guide to KeY that lets you explore the full potential of deductive software verification in practice. It contains the complete theory behind KeY for active researchers who want to understand it in depth or use it in their own work. But the book also features fully self-contained chapters on the Java Modeling Language and on Using KeY that require nothing else than familiarity with Java. All other chapters are accessible for graduate students (M.Sc. level and beyond). The KeY framework is free and open software, downloadable from the book companion website which contains also all code examples mentioned in this book.

241 citations

Proceedings ArticleDOI
31 May 2014
TL;DR: The goal of this paper is to provide an accounting of some of the most successful research performed in software testing since the year 2000, and to present what appear to be the most significant challenges and opportunities in this area.
Abstract: Despite decades of work by researchers and practitioners on numerous software quality assurance techniques, testing remains one of the most widely practiced and studied approaches for assessing and improving software quality. Our goal, in this paper, is to provide an accounting of some of the most successful research performed in software testing since the year 2000, and to present what appear to be some of the most significant challenges and opportunities in this area. To be more inclusive in this effort, and to go beyond our own personal opinions and biases, we began by contacting over 50 of our colleagues who are active in the testing research area, and asked them what they believed were (1) the most significant contributions to software testing since 2000 and (2) the greatest open challenges and opportunities for future research in this area. While our colleagues’ input (consisting of about 30 responses) helped guide our choice of topics to cover and ultimately the writing of this paper, we by no means claim that our paper represents all the relevant and noteworthy research performed in the area of software testing in the time period considered—a task that would require far more space and time than we have available. Nevertheless, we hope that the approach we followed helps this paper better reflect not only our views, but also those of the software testing community in general.

196 citations


Cites background or methods from "An orchestrated survey of methodolo..."

  • ...More recently, more sophisticated approaches (e.g., meta-heuristic approaches such as genetic and ant colony algorithms [175], simulated annealing [69], and tabu search [137]) have been utilized (see References [6, 136] for many additional examples.)...

    [...]

  • ...In Reference [6], Harman and colleagues describe what they consider to be the greatest open challenges and opportunities for SBST techniques; we summarize that discussion here....

    [...]

  • ...(Several other surveys are also available, including [2, 3, 6, 127, 128]....

    [...]

  • ...Anand and colleagues [6] summarize three tools that have been available for nearly 10 years: Conformiq Designer [45, 93]), Smartesting CertifyIt [113, 177]), and Microsoft Spec Explorer [75, 180])....

    [...]

  • ...Model-based testing (MBT) involves deriving test suites from models of software systems (see the brief survey by Anand and colleagues [6] and the taxonomy by Utting and colleagues [193])....

    [...]

References
More filters
Book ChapterDOI
29 Mar 2008
TL;DR: Z3 is a new and efficient SMT Solver freely available from Microsoft Research that is used in various software verification and analysis applications.
Abstract: Satisfiability Modulo Theories (SMT) problem is a decision problem for logical first order formulas with respect to combinations of background theories such as: arithmetic, bit-vectors, arrays, and uninterpreted functions. Z3 is a new and efficient SMT Solver freely available from Microsoft Research. It is used in various software verification and analysis applications.

6,859 citations

Book
01 Jan 1950

5,820 citations

Book
01 Jan 1935

4,510 citations

Book
01 Aug 2001
TL;DR: The Three Essential Activities: Core Asset Development, Software Engineering Practice Areas, and Single-System Development with Reuse - All Three Together.
Abstract: Foreword. Preface. Acknowledgements. Dedication. Reader's Guide. I. SOFTWARE PRODUCT LINE FUNDAMENTALS. 1. Basic Ideas and Terms. What Is a Software Product Line? What Software Product Lines Are Not. Fortuitous Small-Grained Reuse. Single-System Development with Reuse. Just Component-Based Development. Just a Reconfigurable Architecture. Releases and Versions of Single Products. Just a Set of Technical Standards. A Note on Terminology. For Further Reading. Discussion Questions. 2. Benefits. Organizational Benefits. Individual Benefits. Benefits versus Costs. For Further Reading. Discussion Questions. 3. The Three Essential Activities. What Are the Essential Activities? Core Asset Development. Product Development. Management. All Three Together. For Further Reading. Discussion Questions. II. SOFTWARE PRODUCT LINE PRACTICE AREAS. Describing the Practice Areas. Starting versus Running a Product Line. Organizing the Practice Areas. 4. Software Engineering Practice Areas. Architecture Definition. Aspects Peculiar to Product Lines. Application to Core Asset Development. Application to Product Development. Specific Practices. Practice Risks. For Further Reading. Discussion Questions. Architecture Evaluation. Aspects Peculiar to Product Lines. Application to Core Asset Development. Application to Product Development. Specific Practices. Practice Risks. For Further Reading. Discussion Questions. Component Development. Aspects Peculiar to Product Lines. Application to Core Asset Development. Application to Product Development. Specific Practices. Practice Risks. For Further Reading. Discussion Questions. COTS Utilization. Aspects Peculiar to Product Lines. Application to Core Asset Development. Application to Product Development. Specific Practices. Practice Risks. For Further Reading. Discussion Questions. Mining Existing Assets. Aspects Peculiar to Product Lines. Application to Core Asset Development. Application to Product Development. Specific Practices. Practice Risks. Discussion Questions. Requirements Engineering. Aspects Peculiar to Product Lines. Application to Core Asset Development. Application to Product Development. Specific Practices. Practice Risks. For Further Reading. Discussion Questions. Software System Integration. Aspects Peculiar to Product Lines. Application to Core Asset Development. Application to Product Development. Specific Practices. Practice Risks. For Further Reading. Discussion Questions. Testing. Aspects Peculiar to Product Lines. Application to Core Asset Development. Application to Product Development. Specific Practices. Practice Risks. For Further Reading. Discussion Questions. Understanding Relevant Domains. Aspects Peculiar to Product Lines. Application to Core Asset Development. Application to Product Development. Specific Practices. Practice Risks. For Further Reading. Discussion Questions. 5. Technical Management Practice Areas. Configuration Management. Aspects Peculiar to Product Lines. Application to Core Asset Development. Application to Product Development. Specific Practices. Practice Risks. For Further Reading. Discussion Questions. Data Collection, Metrics, and Tracking. Aspects Peculiar to Product Lines. Application to Core Asset Development. Application to Product Development. Specific Practices. Practice Risks. For Further Reading. Discussion Questions. Make/Buy/Mine/Commission Analysis. Aspects Peculiar to Product Lines. Application to Core Asset Development. Application to Product Development. Specific Practices. Practice Risks. For Further Reading. Discussion Questions. Process Definition. Aspects Peculiar to Product Lines. Application to Core Asset Development. Application to Product Development. Specific Practices. Practice Risks. For Further Reading. Discussion Questions. Scoping. Aspects Peculiar to Product Lines. Application to Core Asset Development. Application to Product Development. Specific Practices. Practice Risks. For Further Reading. Discussion Questions. Technical Planning. Aspects Peculiar to Product Lines. Application to Core Asset Development. Application to Product Development. Specific Practices. Practice Risks. Discussion Questions. Technical Risk Management. Aspects Peculiar to Product Lines. Application to Core Asset Development. Application to Product Development. Specific Practices. Practice Risks. For Further Reading. Discussion Questions. Tool Support. Aspects Peculiar to Product Lines. Application to Core Asset Development. Application to Product Development. Specific Practices. Practice Risks. For Further Reading. Discussion Questions. 6. Organizational Management Practice Areas. Building a Business Case. Aspects Peculiar to Product Lines. Application to Core Asset Development. Application to Product Development. Specific Practices. Practice Risks. For Further Reading. Discussion Questions. Customer Interface Management. Aspects Peculiar to Product Lines. Application to Core Asset Development. Application to Product Development. Specific Practices. Practice Risks. Discussion Questions. Developing an Acquisition Strategy. Aspects Peculiar to Product Lines. Application to Core Asset Development. Application to Product Development. Specific Practices. Practice Risks. For Further Reading. Discussion Questions. Funding. Aspects Peculiar to Product Lines. Application to Core Asset Development. Application to Product Development. Specific Practices. Practice Risks. Discussion Questions. Launching and Institutionalizing. Aspects Peculiar to Product Lines. Application to Core Asset Development. Application to Product Development. Specific Practices. Practice Risks. Discussion Questions. Market Analysis. Aspects Peculiar to Product Lines. Application to Core Asset Development. Application to Product Development. Specific Practices. Practice Risks. For Further Reading. Discussion Questions. Operations. Aspects Peculiar to Product Lines. Application to Core Asset Development. Application to Product Development. Specific Practices. Practice Risks. For Further Reading. Discussion Questions. Organizational Planning. Aspects Peculiar to Product Lines. Application to Core Asset Development. Application to Product Development. Specific Practices. Practice Risks. Discussion Questions. Organizational Risk Management. Aspects Peculiar to Product Lines. Application to Core Asset Development. Application to Product Development. Specific Practices. Practice Risks. For Further Reading. Discussion Questions. Structuring the Organization. Aspects Peculiar to Product Lines. Application to Core Asset Development. Application to Product Development. Specific Practices. Practice Risks. Discussion Questions. Technology Forecasting. Aspects Peculiar to Product Lines. Application to Core Asset Development. Application to Product Development. Specific Practices. Practice Risks. For Further Reading. Discussion Questions. Training. Aspects Peculiar to Product Lines. Application to Core Asset Development. Application to Product Development. Specific Practices. Practice Risks. For Further Reading. Discussion Questions. III. PUTTING THE PRACTICE AREAS INTO ACTION. 7. Software Product Line Practice Patterns. The Value of Patterns. Software Product Line Practice Pattern Descriptions. The Curriculum Pattern. The Essentials Coverage Pattern. Each Asset Pattern. What to Build Pattern. Product Parts Pattern. Assembly Line Pattern. Monitor Pattern. Product Builder Pattern. Cold Start Pattern. In Motion Pattern. Process Pattern. Factory Pattern. Other Patterns. Practice Area Coverage. Discussion Questions. 8. Product Line Technical Probe. What Is the Product Line Technical Probe? Probe Interview Questions. Probe Participants. Probe Process. Using the Probe Results. Conducting a Mini Self-Probe. Discussion Questions. 9. Cummins Engine Company: Embracing the Future. Prologue. Company History. A Product Line of Engine Software. Getting off the Ground. An Organization Structured for Cooperation. Running the Product Line. Results. Lessons Learned. Epilogue. Practice Area Compendium. For Further Reading. Discussion Questions. 10. Control Channel Toolkit: A Software Product Line that Controls Satellites. Contextual Background. Organizational Profiles. Project History. Control Channels. Launching CCT. Developing a Business Case for CCT. Developing the Acquisition Strategy and Funding CCT. Structuring the CCT Organization. Organizational and Technical Planning. Operations. Engineering the CCT Core Assets. Domain Analysis. Architecture. Component Engineering. Testing: Application and Test Engineering. Sustainment Engineering: Product Line Evolution. Documentation. Managing the CCT Effort. Early Benefits from CCT. First CCT Product. Benefits beyond CCT Products. Lessons and Issues. Tool Support Is Inadequate. Domain Analysis Documentation Is Important. An Early Architecture Focus Is Best. Product Builders Need More Support. CCT Users Need Reuse Metrics. It Pays to Be Flexible, and Cross-Unit Teams Work. A Real Product Is a Benefit. Summary. For Further Reading. Discussion Questions. 11. Successful Software product Line Development in Small Organization. Introduction. The Early Years. The MERGER Software Product Line. Market Maker Software Product Line Practices. Architecture Definition. Component Development. Structuring (and Staffing) the Organization. Testing. Data Collection and Metrics. Launching and Institutionalizing the Product Line. Understanding the Market. Technology Forecasting. A Few Observations. Effects of Company Culture. Cost Issues. The Customer Paradox. Tool Support. Lessons Learned. Drawbacks. Conclusions: Software Product Lines in Small Organizations. For Further Reading. Discussion Questions. 12. Conclusions: Practices, Patterns and Payoffs. The Practices. The Patterns. The Success Factors. The Payoff. Finale. Glossary. Bibliography. Index.

3,502 citations


"An orchestrated survey of methodolo..." refers background in this paper

  • ...Software product lines are sysems of program families that have a well managed asset base and eature model; from which one can derive a CIT model (Cohen t al., 2006; Clements and Northrop, 2001)....

    [...]

  • ...There is a long history of research on the mathematics of covering arrays which we do not attempt to cover, but instead refer the reader to two excellent surveys, one by Colbourn (2004) and another by Hartman and Raskin (2004). Mathematical techniques may be probabilistic (i....

    [...]

  • ...There is a long history of research on the mathematics of covering arrays which we do not attempt to cover, but instead refer the reader to two excellent surveys, one by Colbourn (2004) and another by Hartman and Raskin (2004)....

    [...]

Proceedings ArticleDOI
08 Dec 2008
TL;DR: A new symbolic execution tool, KLEE, capable of automatically generating tests that achieve high coverage on a diverse set of complex and environmentally-intensive programs, and significantly beat the coverage of the developers' own hand-written test suite is presented.
Abstract: We present a new symbolic execution tool, KLEE, capable of automatically generating tests that achieve high coverage on a diverse set of complex and environmentally-intensive programs. We used KLEE to thoroughly check all 89 stand-alone programs in the GNU COREUTILS utility suite, which form the core user-level environment installed on millions of Unix systems, and arguably are the single most heavily tested set of open-source programs in existence. KLEE-generated tests achieve high line coverage -- on average over 90% per tool (median: over 94%) -- and significantly beat the coverage of the developers' own hand-written test suite. When we did the same for 75 equivalent tools in the BUSYBOX embedded system suite, results were even better, including 100% coverage on 31 of them.We also used KLEE as a bug finding tool, applying it to 452 applications (over 430K total lines of code), where it found 56 serious bugs, including three in COREUTILS that had been missed for over 15 years. Finally, we used KLEE to crosscheck purportedly identical BUSYBOX and COREUTILS utilities, finding functional correctness errors and a myriad of inconsistencies.

2,896 citations