Data Mining Practical Machine Learning Tools and Techniques

This paper summarizes the current state of the art and recent trends in software engineering economics. It provides an overview of economic analysis techniques and their applicability to software engineering and management. It surveys the field of software cost estimation, including the major estimation techniques available, the state of the art in algorithmic cost models, and the outstanding research issues in software cost estimation.

Software engineering economics

A variety of research methods and techniques are available to SE researchers, and while several overviews exist, there is consistency neither in the research methods covered nor in the terminology used. Furthermore, research is sometimes critically reviewed for characteristics inherent to the methods. We adopt a taxonomy from the social sciences, termed here the ABC framework for SE research, which offers a holistic view of eight archetypal research strategies. ABC refers to the research goal that strives for generalizability over Actors (A) and precise measurement of their Behavior (B), in a realistic Context (C). The ABC framework uses two dimensions widely considered to be key in research design: the level of obtrusiveness of the research and the generalizability of research findings. We discuss metaphors for each strategy and their inherent limitations and potential strengths. We illustrate these research strategies in two key SE domains, global software engineering and requirements engineering, and apply the framework on a sample of 75 articles. Finally, we discuss six ways in which the framework can advance SE research.

https://dl.acm.org/doi/pdf/10.1145/3241743

The ABC of Software Engineering Research

/pdf/introduction-to-web-accessibility-1bl4xpivqu.pdf

Introduction to Web Accessibility

Context Concerns have been raised from many quarters regarding the reliability of empirical research findings and this includes software engineering. Replication has been proposed as an important means of increasing confidence. Objective We aim to better understand the value of replication studies, the level of confirmation between replication and original studies, what confirmation means in a statistical sense and what factors modify this relationship. Method We perform a systematic review to identify relevant replication experimental studies in the areas of (i) software project effort prediction and (ii) pair programming. Where sufficient details are provided we compute prediction intervals. Results Our review locates 28 unique articles that describe replications of 35 original studies that address 75 research questions. Of these 10 are external, 15 internal and 3 internal-same-article replications. The odds ratio of internal to external (conducted by independent researchers) replications of obtaining a ‘confirmatory’ result is 8.64. We also found incomplete reporting hampered our ability to extract estimates of effect sizes. Where we are able to compute replication prediction intervals these were surprisingly large. Conclusion We show that there is substantial evidence to suggest that current approaches to empirical replications are highly problematic. There is a consensus that replications are important, but there is a need for better reporting of both original and replicated studies. Given the low power and incomplete reporting of many original studies, it can be unclear the extent to which a replication is confirmatory and to what extent it yields additional knowledge to the software engineering community. We recommend attention is switched from replication research to meta-analysis.

/pdf/the-role-and-value-of-replication-in-empirical-software-47b9swvtyh.pdf

The role and value of replication in empirical software engineering results

Several prediction models have been proposed in the literature using different techniques obtaining different results in different contexts. The need for accurate effort predictions for projects is one of the most critical and complex issues in the software industry. The automated selection and the combination of techniques in alternative ways could improve the overall accuracy of the prediction models. In this study, we validate an automated genetic framework, and then conduct a sensitivity analysis across different genetic configurations. Following is the comparison of the framework with a baseline random guessing and an exhaustive framework. Lastly, we investigate the performance results of the best learning schemes. In total, six hundred learning schemes that include the combination of eight data preprocessors, five attribute selectors and fifteen modeling techniques represent our search space. The genetic framework, through the elitism technique, selects the best learning schemes automatically. The best learning scheme in this context means the combination of data preprocessing + attribute selection + learning algorithm with the highest coefficient correlation possible. The selected learning schemes are applied to eight datasets extracted from the ISBSG R12 Dataset. The genetic framework performs as good as an exhaustive framework. The analysis of the standardized accuracy (SA) measure revealed that all best learning schemes selected by the genetic framework outperforms the baseline random guessing by 45–80%. The sensitivity analysis confirms the stability between different genetic configurations. The genetic framework is stable, performs better than a random guessing approach, and is as good as an exhaustive framework. Our results confirm previous ones in the field, simple regression techniques with transformations could perform as well as nonlinear techniques, and ensembles of learning machines techniques such as SMO, M5P or M5R could optimize effort predictions.

/pdf/a-genetic-algorithm-based-framework-for-software-effort-61g0rtamnx.pdf

A genetic algorithm based framework for software effort prediction

Identifying security requirements early on can lay the foundation for secure software development. Security requirements are often implied by existing functional requirements but are mostly left unspecified. The Security Discoverer (SD) process automatically identifies security implications of individual requirements sentences and suggests applicable security requirements templates. The objective of this research is to support requirements analysts in identifying security requirements by automating the suggestion of security requirements templates that are implied by existing functional requirements. We conducted a controlled experiment in a graduate-level security class at North Carolina State University (NCSU) to evaluate the SD process in eliciting implied security requirements in 2014. We have subsequently conducted three differentiated replications to evaluate the generalizability and applicability of the initial findings. The replications were conducted across three countries at the University of Trento, NCSU, and the University of Costa Rica. We evaluated the responses of the 205 total participants in terms of quality, coverage, relevance and efficiency. We also develop shared insights regarding the impact of context factors such as time, motivation and support, on the study outcomes and provide lessons learned in conducting the replications. Treatment group, using the SD process, performed significantly better than the control group (at p-value <0.05) in terms of the coverage of the identified security requirements and efficiency of the requirements elicitation process in two of the three replications, supporting the findings of the original study. Participants in the treatment group identified 84 % more security requirements in the oracle as compared to the control group on average. Overall, 80 % of the 111 participants in the treatment group were favorable towards the use of templates in identifying security requirements. Our qualitative findings indicate that participants may be able to differentiate between relevant and extraneous templates suggestions and be more inclined to fill in the templates with additional support. Security requirements templates capture the security knowledge of multiple experts and can support the security requirements elicitation process when automatically suggested, making the implied security requirements more evident. However, individual participants may still miss out on identifying a number of security requirements due to empirical constraints as well as potential limitations on knowledge and security expertise.

/pdf/identifying-the-implied-findings-from-three-differentiated-28gl82bagw.pdf

Identifying the implied: Findings from three differentiated replications on the use of security requirements templates

Studies in software effort estimation (SEE) have explored the use of hyper-parameter tuning for machine learning algorithms (MLA) to improve the accuracy of effort estimates. In other contexts random search (RS) has shown similar results to grid search, while being less computationally-expensive. In this paper, we investigate to what extent the random search hyper-parameter tuning approach affects the accuracy and stability of support vector regression (SVR) in SEE. Results were compared to those obtained from ridge regression models and grid search-tuned models. A case study with four data sets extracted from the ISBSG 2018 repository shows that random search exhibits similar performance to grid search, rendering it an attractive alternative technique for hyper-parameter tuning. RS-tuned SVR achieved an increase of 0.227 standardized accuracy (SA) with respect to default hyper-parameters. In addition, random search improved prediction stability of SVR models to a minimum ratio of 0.840. The analysis showed that RS-tuned SVR attained performance equivalent to GS-tuned SVR. Future work includes extending this research to cover other hyper-parameter tuning approaches and machine learning algorithms, as well as using additional data sets.

Evaluating hyper-parameter tuning using random search in support vector machines for software effort estimation

Recently, defect prediction software is an important research topic in the software engineering field. The demand for development of good quality software has seen a rapid growth in the last few years. The software measurement data collected during the software development process include valuable information about software projects status, progress, quality, performance, and evolution. The software fault prediction in the early phases of software development can help and guide software practitioners to focus the available testing resources on the weaker areas during the software development. OBJECTIVE: This paper presents an approach that combines three phases: data preprocessing, attribute selector and learning algorithms using a genetic approach and select the best combination. METHOD: The framework is comprised of 1) scheme learning generator. This component evaluates performance of the learning schemes and suggests the best option according to each data set analyzed, 2) defect predictor component builds models according to the evaluated learning schemes and predicts software defects with new data agreed to the constructed model. CONCLUSIONS: The framework has considered more combinations of learning schemes than other proposals which select the model configuration manually, which means that there are more possibilities to find better learning schemes for each data set. The computational processing of the genetic approach was less costly than Song approach. Finally, The Genetic approach presented an improvement of 0.032 equivalent to 3.2% more than Song approach.

/pdf/a-software-defect-proneness-prediction-framework-a-new-3ucnfvzmo2.pdf

A Software Defect-Proneness Prediction Framework: A new approach using genetic algorithms to generate learning schemes.

Statistical Process Control (SPC) has become of great significance for software engineering organizations as more of them decide to implement quality improvement initiatives. The Capability Maturity Model Integration (CMMI-DEV 1.3) for example, proposes the use of statistical techniques at maturity level 4 to ensure some degree of process predictability. However, the nature of software products and processes poses many challenges to the application of SPC, mainly regarding the design of control charts, a key tool. These challenges have led to opposing views on the applicability of SPC to software processes. This article presents an industry experience report on the application of SPC in a Software Verification and Validation Unit at an Information Technology Division from a financial institution. We present the steps followed to implement SPC in this organization, describe the theoretical assumptions involved in selecting the appropriate control charts, and show a process improvement analysis of using SPC in the organization.

Marcelo Jenkins

Papers

A genetic algorithm based framework for software effort prediction

Identifying the implied: Findings from three differentiated replications on the use of security requirements templates

Evaluating hyper-parameter tuning using random search in support vector machines for software effort estimation

A Software Defect-Proneness Prediction Framework: A new approach using genetic algorithms to generate learning schemes.

Application of Statistical Process Control to Software Defect Metrics: An Industry Experience Report