Supporting Controlled Experimentation with Testing Techniques: An Infrastructure and its Potential Impact
Reads0
Chats0
TLDR
The infrastructure that is being designed and constructed to support controlled experimentation with testing and regression testing techniques is described and the impact that this infrastructure has had and can be expected to have.Abstract:
Where the creation, understanding, and assessment of software testing and regression testing techniques are concerned, controlled experimentation is an indispensable research methodology. Obtaining the infrastructure necessary to support such experimentation, however, is difficult and expensive. As a result, progress in experimentation with testing techniques has been slow, and empirical data on the costs and effectiveness of techniques remains relatively scarce. To help address this problem, we have been designing and constructing infrastructure to support controlled experimentation with testing and regression testing techniques. This paper reports on the challenges faced by researchers experimenting with testing techniques, including those that inform the design of our infrastructure. The paper then describes the infrastructure that we are creating in response to these challenges, and that we are now making available to other researchers, and discusses the impact that this infrastructure has had and can be expected to have.read more
Citations
More filters
Posted Content
Large-Scale Manual Validation of Bug Fixing Commits: A Fine-grained Analysis of Tangling.
Steffen Herbold,Alexander Trautsch,Benjamin Ledel,Alireza Aghamohammadi,Taher Ahmed Ghaleb,Kuljit Kaur Chahal,Tim Bossenmaier,Bhaveet Nagaria,Philip Makedonski,Matin Nili Ahmadabadi,Kristóf Szabados,Helge Spieker,Matej Madeja,Nathaniel Hoy,Valentina Lenarduzzi,Shangwen Wang,Gema Rodríguez-Pérez,Ricardo Colomo Palacios,Roberto Verdecchia,Paramvir Singh,Yihao Qin,Debasish Chakroborti,Davis Willard A,Vijay Walunj,Hongjun Wu,Diego Marcilio,Omar Alam,Abdullah Aldaeej,Idan Amit,Burak Turhan,Simon Eismann,Anna-Katharina Wickert,Ivano Malavolta,Matúš Sulír,Fatemeh Hendijani Fard,Austin Z. Henley,Stratos Kourtzanidis,Eray Tüzün,Christoph Treude,Simin Maleki Shamasbi,Ivan Pashchenko,Marvin Wyrich,James C. Davis,Alexander Serebrenik,Ella Albrecht,Ethem Utku Aktas,Daniel Strüber,Johannes Erbel +47 more
TL;DR: A crowd sourcing approach for manual labeling to validate which changes contribute to bug fixes for each line in bug fixing commits finds that about 11% of lines are hard to label leading to active disagreements between participants.
Journal ArticleDOI
Search-based testing using constraint-based mutation
Jan Malburg,Gordon Fraser +1 more
TL;DR: This paper describes a method that integrates both techniques and delivers the best of both worlds, but the twist is that during evolution, a constraint solver is used to ensure that mutated offspring efficiently explores different control flow.
Proceedings ArticleDOI
Building test oracles by clustering failures
Rafig Almaghairbe,Marc Roper +1 more
TL;DR: The findings reveal that failing outputs do indeed tend to congregate in small clusters, suggesting that the application of anomaly detection techniques to automatically build an oracle using a system's input/output pairs is feasible and has the potential to reduce by an order of magnitude the numbers of outputs that would need to be manually examined following a test run.
A firewall model for testing user-configurable software systems
Lee J. White,Brian P. Robinson +1 more
TL;DR: Empirical studies show that this incremental testing method is effective at detecting latent defects which are exposed by customer configuration changes in user-configurable systems.
Dissertation
Oracle assessment, improvement and placement
TL;DR: A technique for assessing and improving test oracles is proposed, which necessarily places the human tester in the loop and is based on reducing the incidence of both false positives and false negatives and results in an increase in the mutual information between the actual and perfect oracles.
References
More filters
Book
Case Study Research: Design and Methods
TL;DR: In this article, buku ini mencakup lebih dari 50 studi kasus, memberikan perhatian untuk analisis kuantitatif, membahas lebah lengkap penggunaan desain metode campuran penelitian, and termasuk wawasan metodologi baru.
Book Chapter
Case study research
TL;DR: The comprehensive and accessible nature of this collection will make it an essential and lasting handbook for researchers and students studying organizations.
Book
Research methods knowledge base
TL;DR: The Research Methods Knowledge Base is a comprehensive web-based textbook that addresses all of the topics in a typical introductory undergraduate or graduate course in social research methods and uses an informal, conversational style to engage both the newcomer and the more experienced student of research.
Book
Experimentation in Software Engineering: An Introduction
TL;DR: The purpose of Experimentation in Software Engineering: An Introduction is to introduce students, teachers, researchers, and practitioners to experimentation and experimental evaluation with a focus on software engineering, and to provide guidelines for performing experiments evaluating methods, techniques and tools in software engineering.
Related Papers (5)
Empirical evaluation of the tarantula automatic fault-localization technique
James A. Jones,Mary Jean Harrold +1 more