Showing papers on "White-box testing published in 2008"

Combining unit-level symbolic execution and system-level concrete execution for testing nasa software

Abstract: We describe an approach to testing complex safety critical software that combines unit-level symbolic execution and system-level concrete execution for generating test cases that satisfy user-specified testing criteria. We have developed Symbolic Java PathFinder, a symbolic execution framework that implements a non-standard bytecode interpreter on top of the Java PathFinder model checking tool. The framework propagates the symbolic information via attributes associated with the program data. Furthermore, we use two techniques that leverage system-level concrete program executions to gather information about a unit's input to improve the precision of the unit-level test case generation. We applied our approach to testing a prototype NASA flight software component. Our analysis helped discover a serious bug that resulted in design changes to the software. Although we give our presentation in the context of a NASA project, we believe that our work is relevant for other critical systems that require thorough testing.

ARTOO: adaptive random testing for object-oriented software

Abstract: Intuition is often not a good guide to know which testing strategies will work best. There is no substitute for experimental analysis based on objective criteria: how many faults a strategy finds, and how fast. "Random" testing is an example of an idea that intuitively seems simplistic or even dumb, but when assessed through such criteria can yield better results than seemingly smarter strategies. The efficiency of random testing is improved if the generated inputs are evenly spread across the input domain. This is the idea of Adaptive Random Testing (ART). ART was initially proposed for numerical inputs, on which a notion of distance is immediately available. To extend the ideas to the testing of object-oriented software, we have developed a notion of distance between objects and a new testing strategy called ARTOO, which selects as inputs objects that have the highest average distance to those already used as test inputs. ARTOO has been implemented as part of a tool for automated testing of object-oriented software. We present the ARTOO concepts, their implementation, and a set of experimental results of its application. Analysis of the results shows in particular that, compared to a directed random strategy, ARTOO reduces the number of tests generated until the first fault is found, in some cases by as much as two orders of magnitude. ARTOO also uncovers faults that the random strategy does not find in the time allotted, and its performance is more predictable.

Configuration-aware regression testing: an empirical study of sampling and prioritization

Abstract: Configurable software lets users customize applications in many ways, and is becoming increasingly prevalent. Researchers have created techniques for testing configurable software, but to date, only a little research has addressed the problems of regression testing configurable systems as they evolve. Whereas problems such as selective retesting and test prioritization at the test case level have been extensively researched, these problems have rarely been considered at the configuration level. In this paper we address the problem of providing configuration-aware regression testing for evolving software systems. We use combinatorial interaction testing techniques to model and generate configuration samples for use in regression testing. We conduct an empirical study on a non-trivial evolving software system to measure the impact of configurations on testing effectiveness, and to compare the effectiveness of different configuration prioritization techniques on early fault detection during regression testing. Our results show that configurations can have a large impact on fault detection and that prioritization of configurations can be effective.

Error Moderation in Low-Cost Machine-Learning-Based Analog/RF Testing

Abstract: Machine-learning-based test methods for analog/RF devices have been the subject of intense investigation over the last decade. However, despite the significant cost benefits that these methods promise, they have seen a limited success in replacing the traditional specification testing, mainly due to the incurred test error which, albeit small, cannot meet industrial standards. To address this problem, we introduce a neural system that is trained not only to predict the pass/fail labels of devices based on a set of low-cost measurements, as aimed by the previous machine-learning-based test methods, but also to assess the confidence in this prediction. Devices for which this confidence is insufficient are then retested through the more expensive specification testing in order to reach an accurate test decision. Thus, this two-tier test approach sustains the high accuracy of specification testing while leveraging the low cost of machine-learning-based testing. In addition, by varying the desired level of confidence, it enables the exploration of the tradeoff between test cost and test accuracy and facilitates the development of cost-effective test plans. We discuss the structure and the training algorithm of an ontogenic neural network which is embodied in the neural system in the first tier, as well as the extraction of appropriate measurements such that only a small fraction of devices are funneled to the second tier. The proposed test-error-moderation method is demonstrated on a switched-capacitor filter and an ultrahigh-frequency receiver front end.

On the automation of fixing software bugs

Abstract: Software Testing can take up to half of the resources of the development of new software. Although there has been a lot of work on automating the testing phase, fixing a bug after its presence has been discovered is still a duty of the programmers. Techniques to help the software developers for locating bugs exist though, and they take name of Automated Debugging. However, to our best knowledge, there has been only little attempt in the past to completely automate the actual changing of the software for fixing the bugs. Therefore, in this paper we propose an evolutionary approach to automate the task of fixing bugs. The basic idea is to evolve the programs (e.g., by using Genetic Programming) with a fitness function that is based on how many unit tests they are able to pass. If a formal specification of the buggy software is given, more sophisticated fitness functions can be designed. Moreover, by using the formal specification as an oracle, we can generate as many unit tests as we want. Hence, a co-evolution between programs and unit tests might take place to give even better results. It is important to know that, to fix the bugs in a program with this novel approach, a user needs only to provide either a formal specification or a set of unit tests. No other information is required.

An upper bound on software testing effectiveness

Abstract: Failure patterns describe typical ways in which inputs revealing program failure are distributed across the input domain—in many cases, clustered together in contiguous regions. Based on these observations several debug testing methods have been developed. We examine the upper bound of debug testing effectiveness improvements possible through making assumptions about the shape, size and orientation of failure patterns. We consider the bounds for testing strategies with respect to minimizing the F-measure, maximizing the P-measure, and maximizing the E-measure. Surprisingly, we find that the empirically measured effectiveness of some existing methods that are not based on these assumptions is close to the theoretical upper bound of these strategies. The assumptions made to obtain the upper bound, and its further implications, are also examined.

Privacy oracle: a system for finding application leaks with black box differential testing

Abstract: We describe the design and implementation of Privacy Oracle, a system that reports on application leaks of user information via the network traffic that they send. Privacy Oracle treats each application as a black box, without access to either its internal structure or communication protocols. This means that it can be used over a broad range of applications and information leaks (i.e., not only Web traffic or credit card numbers). To accomplish this, we develop a differential testing technique in which perturbations in the application inputs are mapped to perturbations in the application outputs to discover likely leaks; we leverage alignment algorithms from computational biology to find high quality mappings between different byte-sequences efficiently. Privacy Oracle includes this technique and a virtual machine-based testing system. To evaluate it, we tested 26 popular applications, including system and file utilities, media players, and IM clients. We found that Privacy Oracle discovered many small and previously undisclosed information leaks. In several cases, these are leaks of directly identifying information that are regularly sent in the clear (without end-to-end encryption) and which could make users vulnerable to tracking by third parties or providers.

Finding errors in .net with feedback-directed random testing

Abstract: We present a case study in which a team of test engineers at Microsoft applied a feedback-directed random testing tool to a critical component of the .NET architecture. Due to its complexity and high reliability requirements, the component had already been tested by 40 test engineers over five years, using manual testing and many automated testing techniques.Nevertheless, the feedback-directed random testing tool found errors in the component that eluded previous testing, and did so two orders of magnitude faster than a typical test engineer (including time spent inspecting the results of the tool). The tool also led the test team to discover errors in other testing and analysis tools, and deficiencies in previous best-practice guidelines for manual testing. Finally, we identify challenges that random testing faces for continued effectiveness, including an observed decrease in the technique's error detection rate over time.

Software Testing and Quality Assurance: Theory and Practice

Abstract: Preface. List of Figures. List of Tables. CHAPTER 1: BASIC CONCEPTS AND PRELIMINARIES. 1.1 Quality Revolution. 1.2 Software Quality. 1.3 Role of Testing. 1.4 Verification and Validation. 1.5 Failure, Error, Fault, and Defect. 1.6 Notion of Software Reliability. 1.7 Objectives of Testing. 1.8 What Is a Test Case? 1.9 Expected Outcome. 1.10 Concept of Complete Testing. 1.11 Central Issue in Testing. 1.12 Testing Activities. 1.13 Test Levels. 1.14 Sources of Information for Test Case Studies. 1.15 White-Box and Black-Box Testing. 1.16 Test Planning and Design. 1.17 Monitoring and Measuring Test Execution. 1.18 Test Tools and Automation. 1.19 Test Team Organization and Management. 1.20 Outline of Book. References. Exercises. CHAPTER 2: THEORY OF PROGRAM TESTING. 2.1 Basic Concepts in Testing Theory. 2.2 Theory of Goodenough and Gerhart. 2.3 Theory of Weyuker and Ostrand. 2.4 Theory of Gourlay. 2.5 Adequacy of Testing. 2.6 Limitations of Testing. 2.7 Summary. Literature Review. References. Exercises. CHAPTER 3: UNIT TESTING. 3.1 Concept of Unit Testing. 3.2 Static Unit Testing. 3.3 Defect Prevention. 3.4 Dynamic Unit Testing. 3.5 Mutation Testing. 3.6 Debugging. 3.7 Unit Testing in extreme Programming. 3.8 JUnit: Framework for Unit Testing. 3.9 Tools for Unit Testing. 3.10 Summary. Literature Review. References. Exercises. CHAPTER 4: CONTROL FLOW TESTING. 4.1 Basic Idea. 4.2 Outline of Control Flow Testing. 4.3 Control Flow Graph. 4.4 Paths in a Control Flow Chart. 4.5 Path Selection Criteria. 4.6 Generating Test Input. 4.7 Examples of Test Data Selection. 4.8 Containing Infeasible Paths. 4.9 Summary. Literature Review. References. Exercises. CHAPTER 5: DATA FLOW TESTING. 5.1 General Idea. 5.2 Data flow Anomaly. 5.3 Overview of Dynamic Data flow Testing. 5.4 Data Flow Graph. 5.5 Data Flow Terms. 5.6 Data Flow Testing Criteria. 5.7 Comparison of Data Flow Test Selection Criteria. 5.8 Feasible Paths and Test Selection Criteria. 5.9 Comparison of Testing Techniques. 5.10 Summary. Literature Review. References. Exercises. CHAPTER 6: DOMAIN TESTING. 6.1 Domain Error. 6.2 Testing for Domain Errors. 6.3 Sources of Domain. 6.4 Types of Domain Errors. 6.5 ON and OFF Points. 6.6 Test Selection Criterion. 6.7 Summary. Literature Review. References. Exercises. CHAPTER 7: SYSTEM INTEGRATION. 7.1 Concept of Integration Testing. 7.2 Different Types of Interfaces and Interface Errors. 7.3 Granularity of System Integration Testing. 7.4 System Integration Techniques. 7.5 Software and Hardware Integration. 7.6 Test Plan for System Integration. 7.7 Off-the Shelf Component Integration. 7.8 Summary. Literature Review. References. Exercises. CHAPTER 8: SYSTEM TEST CATEGORIES. 8.1 Taxonomy of System Tests. 8.2 Basic Tests. 8.3 Functionality Tests. 8.4 Robustness Tests. 8.5 Interoperability Tests. 8.6 Performance Tests. 8.7 Scalability Tests. 8.8 Stress Tests. 8.9 Load and Stability Tests. 8.10 Reliability Tests. 8.11 Regression Tests. 8.12 Documentation Tests. 8.13 Regulatory Tests. 8.14 Summary. Literature Review. References. Exercises. CHAPTER 9: FUNCTIONAL TESTING. 9.1 Functional Testing Concepts of Howden. 9.2 Complexity of Applying Functional Testing. 9.3 Pairwise Testing. 9.4 Equivalence Class Partitioning. 9.5 Boundary Value Analysis. 9.6 Decision Tables. 9.7 Random Testing. 9.8 Error Guessing. 9.9 Category Partition. 9.10 Summary. Literature review. References. Exercises. CHAPTER 10: TEST GENERATION FROM FSM MODELS. 10.1 State-Oriented Model. 10.2 Points of Control and Observation. 10.3 Finite-State Machine. 10.4 Test Generation from an FSM. 10.5 Transition Tour Method. 10.6 Testing with State Verification. 10.7 Unique Input-Output Sequence. 10.8 Distinguishing Sequence. 10.9 Characterizing Sequence. 10.10 Test Architecture. 10.11 Testing and Test Control Notation Version 3 (TTCN-3). 10.12 Extended FSMs. 10.13 Test Generation from EFSM Models. 10.14 Additional Coverage Criteria for System Testing. 10.15 Summary. Literature review. References. Exercises. CHAPTER 11: SYSTEM TEST DESIGN. 11.1 Test Design Factors. 11.2 Requirement Identification. 11.3 Characteristics of Testable Requirements. 11.4 Test Objective Identification. 11.5 Example. 11.6 Modeling a Test Design Process. 11.7 Modeling Test Results. 11.8 Test Design Preparedness Metrics. 11.9 Test Case Design Effectiveness. 11.10 Summary. Literature Review. References. Exercises. CHAPTER 12: SYSTEM TEST PLANNING AND AUTOMATION. 12.1 Structure of a System Test Plan. 12.2 Introduction and Feature Description. 12.3 Assumption. 12.4 Test Approach. 12.5 Test Suite Structure. 12.6 Test Environment. 12.7 Test Execution Strategy. 12.8 Test Effort Estimation. 12.9 Scheduling and Test Milestones. 12.10 System Test Automation. 12.11 Evaluation and Selection of Test Automation Tools. 12.12 Test Selection Guidelines for Automation. 12.13 Characteristics of Automated Test Cases. 12.14 Structure of an Automated Test Case. 12.15 Test Automation Infrastructure. 12.16 Summary. Literature Review. References. Exercises. CHAPTER 13: SYSTEM TEST EXECUTION. 13.1 Basic Ideas. 13.2 Modeling Defects. 13.3 Preparedness to Start System Testing. 13.4 Metrics for Tracking System Test. 13.5 Orthogonal Defect Classification. 13.6 Defect Causal Analysis. 13.7 Beta Testing. 13.8 First Customer Shipment. 13.9 System Test Report. 13.10 Product Sustaining. 13.11 Measuring Test Effectiveness. 13.12 Summary. Literature Review. References. Exercises. CHAPTER 14: ACCEPTANCE TESTING. 14.1 Types of Acceptance Testing. 14.2 Acceptance Criteria. 14.3 Selection of Acceptance Criteria. 14.4 Acceptance Test Plan. 14.5 Acceptance Test Execution. 14.6 Acceptance Test Report. 14.7 Acceptance Testing in extreme Programming. 14.8 Summary. Literature Review. References. Exercises. CHAPTER 15: SOFTWARE RELIABILTY. 15.1 What is Reliability? 15.2 Definitions of Software Reliability. 15.3 Factors Influencing Software Reliability. 15.4 Applications of Software Reliability. 15.5 Operational Profiles. 15.6 Reliability Models. 15.7 Summary. Literature Review. References. Exercises. CHAPTER 16: TEST TEAM ORGANIZATION. 16.1 Test Groups. 16.2 Software Quality Assurance Group. 16.3 System Test Team Hierarchy. 16.4 Effective Staffing of Test Engineers. 16.5 Recruiting Test Engineers. 16.6 Retaining Test Engineers. 16.7 Team Building. 16.8 Summary. Literature Review. References. Exercises. CHAPTER 17: SOFTWARE QUALITY. 17.1 Five Views of Software Quality. 17.2 McCall's Quality Factors and Criteria. 17.3 ISO 9126 Quality Characteristics. 17.4 ISO 9000:2000 Software Quality Standard. 17.5 Summary. Literature Review. References. Exercises. CHAPTER 18: MATURITY MODELS. 18.1 Basic Idea in Software Process. 18.2 Capability Maturity Models. 18.3 Test Process Improvement. 18.4 Testing Maturity Model. 18.5 Summary. Literature Review. References. Exercises. GLOSSARY. INDEX.

Model-Based Tests for Access Control Policies

Abstract: We present a model-based approach to testing access control requirements. By using combinatorial testing, we first automatically generate test cases from and without access control policies-i.e., the model- and assess the effectiveness of the test suites by means of mutation testing. We also compare them to purely random tests. For some of the investigated strategies, non-random tests kill considerably more mutants than the same number of random tests. Since we rely on policies only, no information on the application is required at this stage. As a consequence, our methodology applies to arbitrary implementations of the policy decision points.

Historical Value-Based Approach for Cost-Cognizant Test Case Prioritization to Improve the Effectiveness of Regression Testing

Abstract: Regression testing has been used to support software testing activities and assure the acquirement of appropriate quality through several versions of a software program. Regression testing, however, is too expensive because it requires many test case executions, and the number of test cases increases sharply as the software evolves. In this paper, we propose the Historical Value-Based Approach, which is based on the use of historical information, to estimate the current cost and fault severity for cost-cognizant test case prioritization. We also conducted a controlled experiment to validate the proposed approach, the results of which proved the proposed approachpsilas usefulness. As a result of the proposed approach, software testers who perform regression testing are able to prioritize their test cases so that their effectiveness can be improved in terms of average percentage of fault detected per cost.

System and method for distributed software testing

Abstract: In general, in one aspect, a method for distributed testing includes providing a testing framework for the distributed testing of software to a number of clients at least some of which have a different operating environment than other of the clients, transmitting to each testing framework a component for testing, receiving client environment data and testing results from each testing framework, storing the testing results and environment data in a database; and reporting the results of running the tests on the clients.

An automated technique for risk-based test case generation and prioritization

Abstract: In practice, available testing budgets limit the number of test cases that can be executed. Thus, a representative subset of all possible test cases must be chosen to guarantee adequate coverage of a test object. In risk-based testing, the probability of a fault and the damage that this fault can cause when leading to a failure is considered for test case prioritization. Existing approaches for risk-based testing provide guidelines for deriving test cases. However, those guidelines lack the level of detail and precision needed for automation. In this contribution, we introduce the risk-based testing technique RiteDAP, which automatically generates system test cases from activity diagrams and prioritizes those test cases based on risk. The results of applying the technique to a practical example are presented and the ability of different prioritization strategies to uncover faults is evaluated.

Randomized directed testing (REDIRECT) for Simulink/Stateflow models

Abstract: The Simulink/Stateflow (SL/SF) environment from Math-works is becoming the de facto standard in industry for model based development of embedded control systems. Many commercial tools are available in the market for test case generation from SL/SF designs; however, we have observed that these tools do not achieve satisfactory coverage in cases when designs involve nonlinear blocks and Stateflow blocks occur deep inside the Simulink blocks.The recent past has seen the emergence of several novel techniques for testing large C, C++ and Java programs; prominent among them are directed automated random testing (DART), hybrid concolic testing and feedback-directed random testing. We believe that some of these techniques could be lifted to testing of SL/SF based designs; REDIRECT (RandomizEd DIRECted Testing), the proposed testing method of this paper, is an attempt towards this direction. Specifically, REDIRECT uses a careful combination of the above techniques, and in addition, the method uses a set of pattern-guided heuristics for tackling nonlinear blocks. A prototype tool has been developed and the tool has been applied to many industrial strength case studies. Our experiments indicate that a careful choice of heuristics and certain combinations of random and directed testing achieve better coverages as compared to the existing commercial tools. 1

FShell: Systematic Test Case Generation for Dynamic Analysis and Measurement

Abstract: Although the principal analogy between counterexample generation and white box testing has been repeatedly addressed, the usage patterns and performance requirements for software testing are quite different from formal verification. Our tool FS hell provides a versatile testing environment for C programs which supports both interactive explorative use and a rich scripting language. More than a frontend for software model checkers, FS hell is designed as a database engine which dispatches queries about the program to program analysis tools. We report on the integration of CBMC into FS hell and describe architectural modifications which support efficient test case generation.

Efficient Software Test Case Generation Using Genetic Algorithm Based Graph Theory

Abstract: In orthodox software testing approach we generally use modeling based testing approach for generating the test cases of a given problem. This leads to confusion of the test input and the expected output for a given test case. More over we also can miss some of the test cases due to lack of clarity in the test paths. To over come such sort of predictive modeling we propose graph theory based genetic approach to generate test cases for software testing. At first we will create a directed graph of all the intermediate state of the system for the expected behavior of the system. Then we will create a population of all the nodes of the graph as the base population of genetic algorithm. From this population we can find a pair of node the parents and perform genetic crossover and mutation on them for the getting the optimum child nodes as the out put. We should continue this process of genetic operation until all the nodes are covered or any of the nodes, which are visited more than once, should be discarded form the population. Then follow the same process for the generation of test case in the real time system. This technique will be more concrete in case of network testing or any of the system testing where the predictive model based tests are not optimized to produced the out put.

FSHELL: Systematic Test Case Generation for Dynamic Analysis and Measurement Tool Paper

Abstract: Although the principal analogy between counterexample generation and white box testing has been repeatedly addressed, the usage patterns and per- formance requirements for software testing are quite different from formal verifi- cation. Our tool FSHELL provides a versatile testing environment for C programs which supports both interactive explorative use and a rich scripting language. More than a frontend for software model checkers, FSHELL is designed as a database engine which dispatches queries about the program to program analysis tools. We report on the integration of CBMC into FSHELL and describe architec- tural modifications which support efficient test case generation.

Applying Particle Swarm Optimization to Prioritizing Test Cases for Embedded Real Time Software Retesting

Abstract: In recent years, complex embedded systems are used in every device that is infiltrating our daily lives. Since most of the embedded systems are multi-tasking real time systems, the task interleaving issues, dead lines and other factors needs software units retesting to follow the subsequence changes. Regression testing is used for the software maintenance that revalidates the old functionality of the software unit. Testing is one of the most complex and time-consuming activities, in which running of all combination of test cases in test suite may require a large amount of efforts. Test case prioritization techniques can take advantage that orders test cases, which attempts to increase effectiveness in regression testing. This paper proposes to use particle swarm optimization (PSO) algorithm to prioritize the test cases automatically based on the modified software units. Regarding to the recent investigations, PSO is a multi-object optimization technique that can find out the best positions of the objects. The goal is to prioritize the test cases to the new best order, based on modified software components, so that test cases, which have new higher priority, can be selected in the regression testing process. The empirical results show that by using the PSO algorithm, the test cases can be prioritized in the test suites with their new best positions effectively and efficiently.

Automatic Timed Test Case Generation for Web Services Composition

Abstract: In order to specify the composition of Web services, WSBPEL was defined as an orchestrating language by an international standards consortium. In this paper, we propose a method to test composite Web service described in BPEL. As a first step, the BPEL specification is transformed into an Intermediate Format (IF) model that is based on timed automata, which enables modeling of timing constraints.We defined a conformance relation between two timed automata(of implementation and specification) and then proposed an algorithm to generate test cases. Test case generation is based on simulation where the exploration is guided by test purposes. The proposed method was implemented in a set of tools which were applied to a common Web service as a case study.

Early fault detection with model-based testing

Abstract: Current and future trends for software include increasingly complex requirements on interaction between systems. As a result, the difficulty of system testing increases. Model-based testing is a test technique where test cases are generated from a model of the system. In this study we explore model-based testing on the system level, starting from early development. We apply model-based testing to a subsystem of a message gateway product in order to improve early fault detection. The results are compared to another subsystem that is tested with hand-crafted test cases.Based on our experiences, we present a set of challenges and recommendations for system-level, model-based testing. Our results indicate that model-based testing, starting from early development, significantly increases the number of faults detected during system testing.

Using Genetic Algorithm for Unit Testing of Object Oriented Software

Abstract: Genetic algorithms have been successfully applied in the area of software testing. The demand for automation of test case generation in object oriented software testing is increasing. Genetic algorithms are well applied in procedural software testing but a little has been done in testing of object oriented software. In this paper, we propose a method to generate test cases for classes in object oriented software using a genetic programming approach. This method uses tree representation of statements in test cases. Strategies for encoding the test cases and using the objective function to evolve them as suitable test case are proposed.

TestQ: Exploring Structural and Maintenance Characteristics of Unit Test Suites

Abstract: The increasing interest in unit testing in recent years has resulted in lots of persistent test code that has to co-evolve with production code in order to remain eective. Moreover, poor test design decisions, and complexity introduced during the course of evolution harm the maintenance of these test suites ‐ making test cases harder to understand and modify. Literature about xUnit ‐ the de facto family of unit testing frameworks ‐ has a fairly clear set of anti-patterns (called test smells). In this paper we present TestQ which allows developers to (i) visually explore test suites and (ii) quantify test smelliness. We present the feature set of this tool as well as demonstrate its use on test suites for both C++ and Java systems.

Testing Java Components based on Algebraic Specifications

Abstract: This paper presents a method of component testing based on algebraic specifications. An algorithm for generating checkable test cases is proposed. A prototype testing tool called CASCAT for testing Java Enterprise Beans is developed. It has the advantages of high degree of automation, which include test case generation, test harness construction and test result checking. It achieves scalability by allowing incremental integration. It also allows testing to focus on a subset of used functions and key properties, thus suitable for component testing. The paper also reports an experimental evaluation of the method and the tool.

How We Test Software at Microsoft

Abstract: Discover how Microsoft implements and manages a company-wide software-testing process straight from testers at Microsoft. Software testing requires people, processes, and tools. Organizing a testing program is resource intensive, even when necessary tradeoffs are made no development team can test every scenario. And using a company-wide process is a challenging issue for most organizations. This book explains how a worldwide leader in software, services, and solutions with 8,000 testers implements and manages its testing process. It includes expert insights and effective testing techniques and methodologies including the pros and cons of various approaches. To provide interesting context, it also provides facts about Microsoft, including the number of test machines, how the company uses automated test cases, and the number of bugs entered for products. Ideal for both testers and test managers, this book answers key testing questions, such as who tests what, when, and with what tools. It covers how test teams are organized, when and how testing is automated, tools used for testing, and feedback details with insights for all software-development groups.Key Book Benefits:Delivers a consolidated list of highly effective testing techniques and methodologies including pros and cons of different approachesProvides expert insights into testing at MicrosoftIncludes interesting facts about Microsoft, such as number of test machines, use of automated test cases, and numbers of bugs entered in a product or across all productsFeatures information about future thoughts on testing and verification methods

Effective and scalable software compatibility testing

Abstract: Today's software systems are typically composed of multiple components, each with different versions. Software compatibility testing is a quality assurance task aimed at ensuring that multi-component based systems build and/or execute correctly across all their versions' combinations, or configurations. Because there are complex and changing interdependencies between components and their versions, and because there are such a large number of configurations, it is generally infeasible to test all potential configurations. Consequently, in practice, compatibility testing examines only a handful of default or popular configurations to detect problems; as a result costly errors can and do escape to the field.This paper presents a new approach to compatibility testing, called Rachet. We formally model the entire configuration space for software systems and use the model to generate test plans to sample a portion of the space. In this paper, we test all direct dependencies between components and execute the test plan efficiently in parallel. We present empirical results obtained by applying our approach to two large-scale scientific middleware systems. The results show that for these systems Rachet scaled well and discovered incompatibilities between components, and that testing only direct dependences did not compromise test quality.