Machine Learning is the study of methods for programming computers to learn. Computers are applied to a wide range of tasks, and for most of these it is relatively easy for programmers to design and implement the necessary software. However, there are many tasks for which this is difficult or impossible. These can be divided into four general categories. First, there are problems for which there exist no human experts. For example, in modern automated manufacturing facilities, there is a need to predict machine failures before they occur by analyzing sensor readings. Because the machines are new, there are no human experts who can be interviewed by a programmer to provide the knowledge necessary to build a computer system. A machine learning system can study recorded data and subsequent machine failures and learn prediction rules. Second, there are problems where human experts exist, but where they are unable to explain their expertise. This is the case in many perceptual tasks, such as speech recognition, hand-writing recognition, and natural language understanding. Virtually all humans exhibit expert-level abilities on these tasks, but none of them can describe the detailed steps that they follow as they perform them. Fortunately, humans can provide machines with examples of the inputs and correct outputs for these tasks, so machine learning algorithms can learn to map the inputs to the outputs. Third, there are problems where phenomena are changing rapidly. In finance, for example, people would like to predict the future behavior of the stock market, of consumer purchases, or of exchange rates. These behaviors change frequently, so that even if a programmer could construct a good predictive computer program, it would need to be rewritten frequently. A learning program can relieve the programmer of this burden by constantly modifying and tuning a set of learned prediction rules. Fourth, there are applications that need to be customized for each computer user separately. Consider, for example, a program to filter unwanted electronic mail messages. Different users will need different filters. It is unreasonable to expect each user to program his or her own rules, and it is infeasible to provide every user with a software engineer to keep the rules up-to-date. A machine learning system can learn which mail messages the user rejects and maintain the filtering rules automatically. Machine learning addresses many of the same research questions as the fields of statistics, data mining, and psychology, but with differences of emphasis. Statistics focuses on understanding the phenomena that have generated the data, often with the goal of testing different hypotheses about those phenomena. Data mining seeks to find patterns in the data that are understandable by people. Psychological studies of human learning aspire to understand the mechanisms underlying the various learning behaviors exhibited by people (concept learning, skill acquisition, strategy change, etc.).

Machine learning

Data Mining Practical Machine Learning Tools and Techniques

Mutation Testing is a fault-based software testing technique that has been widely studied for over three decades The literature on Mutation Testing has contributed a set of approaches, tools, developments, and empirical results This paper provides a comprehensive analysis and survey of Mutation Testing The paper also presents the results of several development trend analyses These analyses provide evidence that Mutation Testing techniques and tools are reaching a state of maturity and applicability, while the topic of Mutation Testing itself is the subject of increasing interest

An Analysis and Survey of the Development of Mutation Testing

There is increasing concern that most current published research findings are false. The probability that a research claim is true may depend on study power and bias, the number of other studies on the same question, and, importantly, the ratio of true to no relationships among the relationships probed in each scientific field. In this framework, a research finding is less likely to be true when the studies conducted in a field are smaller; when effect sizes are smaller; when there is a greater number and lesser preselection of tested relationships; where there is greater flexibility in designs, definitions, outcomes, and analytical modes; when there is greater financial and other interest and prejudice; and when more teams are involved in a scientific field in chase of statistical significance. Simulations show that for most study designs and settings, it is more likely for a research claim to be false than true. Moreover, for many current scientific fields, claimed research findings may often be simply accurate measures of the prevailing bias. In this essay, I discuss the implications of these problems for the conduct and interpretation of research.

Why most published research findings are false

Regression testing is a testing activity that is performed to provide confidence that changes do not harm the existing behaviour of the software. Test suites tend to grow in size as software evolves, often making it too costly to execute entire test suites. A number of different approaches have been studied to maximize the value of the accrued test suite: minimization, selection and prioritization. Test suite minimization seeks to eliminate redundant test cases in order to reduce the number of tests to run. Test case selection seeks to identify the test cases that are relevant to some set of recent changes. Test case prioritization seeks to order test cases in such a way that early fault detection is maximized. This paper surveys each area of minimization, selection and prioritization technique and discusses open problems and potential directions for future research. Copyright © 2010 John Wiley & Sons, Ltd.

Regression testing minimization, selection and prioritization: a survey

To find defects in software, one needs test cases that execute the software systematically, and oracles that assess the correctness of the observed behavior when running these test cases. This paper presents EvoSuite, a tool that automatically generates test cases with assertions for classes written in Java code. To achieve this, EvoSuite applies a novel hybrid approach that generates and optimizes whole test suites towards satisfying a coverage criterion. For the produced test suites, EvoSuite suggests possible oracles by adding small and effective sets of assertions that concisely summarize the current behavior; these assertions allow the developer to detect deviations from expected behavior, and to capture the current behavior in order to protect against future defects breaking this behavior.

/pdf/evosuite-automatic-test-suite-generation-for-object-oriented-4tj5vvqvqg.pdf

EvoSuite: automatic test suite generation for object-oriented software

A good test suite is one that detects real faults Because the set of faults in a program is usually unknowable, this definition is not useful to practitioners who are creating test suites, nor to researchers who are creating and evaluating tools that generate test suites In place of real faults, testing research often uses mutants, which are artificial faults -- each one a simple syntactic variation -- that are systematically seeded throughout the program under test Mutation analysis is appealing because large numbers of mutants can be automatically-generated and used to compensate for low quantities or the absence of known real faults Unfortunately, there is little experimental evidence to support the use of mutants as a replacement for real faults This paper investigates whether mutants are indeed a valid substitute for real faults, ie, whether a test suite’s ability to detect mutants is correlated with its ability to detect real faults that developers have fixed Unlike prior studies, these investigations also explicitly consider the conflating effects of code coverage on the mutant detection rate Our experiments used 357 real faults in 5 open-source applications that comprise a total of 321,000 lines of code Furthermore, our experiments used both developer-written and automatically-generated test suites The results show a statistically significant correlation between mutant detection and real fault detection, independently of code coverage The results also give concrete suggestions on how to improve mutation analysis and reveal some inherent limitations

/pdf/are-mutants-a-valid-substitute-for-real-faults-in-software-2w3vlgaztq.pdf

Are mutants a valid substitute for real faults in software testing

Not all bugs lead to program crashes, and not always is there a formal specification to check the correctness of a software test's outcome. A common scenario in software testing is therefore that test data are generated, and a tester manually adds test oracles. As this is a difficult task, it is important to produce small yet representative test sets, and this representativeness is typically measured using code coverage. There is, however, a fundamental problem with the common approach of targeting one coverage goal at a time: Coverage goals are not independent, not equally difficult, and sometimes infeasible-the result of test generation is therefore dependent on the order of coverage goals and how many of them are feasible. To overcome this problem, we propose a novel paradigm in which whole test suites are evolved with the aim of covering all coverage goals at the same time while keeping the total size as small as possible. This approach has several advantages, as for example, its effectiveness is not affected by the number of infeasible targets in the code. We have implemented this novel approach in the EvoSuite tool, and compared it to the common approach of addressing one goal at a time. Evaluated on open source libraries and an industrial case study for a total of 1,741 classes, we show that EvoSuite achieved up to 188 times the branch coverage of a traditional approach targeting single branches, with up to 62 percent smaller test suites.

/pdf/whole-test-suite-generation-261oywj832.pdf

Whole Test Suite Generation

A test oracle determines whether a test execution reveals a fault, often by comparing the observed program output to the expected output. This is not always practical, for example when a program's input-output relation is complex and difficult to capture formally. Metamorphic testing provides an alternative, where correctness is not determined by checking an individual concrete output, but by applying a transformation to a test input and observing how the program output “morphs” into a different one as a result. Since the introduction of such metamorphic relations in 1998, many contributions on metamorphic testing have been made, and the technique has seen successful applications in a variety of domains, ranging from web services to computer graphics. This article provides a comprehensive survey on metamorphic testing: It summarises the research results and application areas, and analyses common practice in empirical studies of metamorphic testing as well as the main open challenges.

/pdf/a-survey-on-metamorphic-testing-2diycuud9b.pdf

A Survey on Metamorphic Testing

Most fault localization techniques take as input a faulty program, and produce as output a ranked list of suspicious code locations at which the program may be defective When researchers propose a new fault localization technique, they typically evaluate it on programs with known faults The technique is scored based on where in its output list the defective code appears This enables the comparison of multiple fault localization techniques to determine which one is better Previous research has evaluated fault localization techniques using artificial faults, generated either by mutation tools or manually In other words, previous research has determined which fault localization techniques are best at finding artificial faults However, it is not known which fault localization techniques are best at finding real faults It is not obvious that the answer is the same, given previous work showing that artificial faults have both similarities to and differences from real faults We performed a replication study to evaluate 10 claims in the literature that compared fault localization techniques (from the spectrum-based and mutation-based families) We used 2995 artificial faults in 6 real-world programs Our results support 7 of the previous claims as statistically significant, but only 3 as having non-negligible effect sizes Then, we evaluated the same 10 claims, using 310 real faults from the 6 programs Every previous result was refuted or was statistically and practically insignificant Our experiments show that artificial faults are not useful for predicting which fault localization techniques perform best on real faults In light of these results, we identified a design space that includes many previously-studied fault localization techniques as well as hundreds of new techniques We experimentally determined which factors in the design space are most important, using an overall set of 395 real faults Then, we extended this design space with new techniques Several of our novel techniques outperform all existing techniques, notably in terms of ranking defective code in the top-5 or top-10 reports

Gordon Fraser

Papers

EvoSuite: automatic test suite generation for object-oriented software

Are mutants a valid substitute for real faults in software testing

Whole Test Suite Generation

A Survey on Metamorphic Testing

Evaluating and improving fault localization