There is no general consensus on which classifier performance metrics are better to use as compared to others. While some studies investigate a handful of such metrics in a comparative fashion, an evaluation of specific relationships among a large set of commonly-used performance metrics is much needed in the data mining and machine learning community. This study provides a unique insight into the underlying relationships among classifier performance metrics. We do so with a large case study involving 35 datasets from various domains and the C4.5 decision tree algorithm. A common property of the 35 datasets is that they suffer from the class imbalance problem. Our approach is based on applying factor analysis to the classifier performance space which is characterized by 22 performance metrics. It is shown that such a large number of performance metrics can be grouped into two-to-four relationship-based groups extracted by factor analysis. This work is a step in the direction of providing the analyst with an improved understanding about the different relationships and groupings among the performance metrics, thus facilitating the selection of performance metrics that capture relatively independent aspects of a classifier’s performance.

A Study on the Relationships of Classifier Performance Metrics

An enhanced technique for risk categorization is presented. This technique, PCA-ANN, provides an improved capability to discriminate high-risk software. The approach draws on the combined strengths of pattern recognition, multivariate statistics and neural networks. Principal component analysis is utilized to provide a means of normalizing and orthogonalizing the input data, thus eliminating the ill effects of multicollinearity. A neural network is used for risk determination/classification. A significant feature of this approach is a procedure, herein termed cross-normalization. This procedure provides the technique with capability to discriminate data sets that include disproportionately large numbers of high-risk software modules.

An enhanced neural network technique for software risk analysis

A critique of three metrics

A new approach to complexity metrics is described based not on empirical analysis of the final product, viz. the code, but on an understanding of the cognitive processes of the analyst or programmer as they approach and undertake the challenges of program development, modification and debugging. The resulting metric, the Cognitive Complexity Model, involves quantification of a number of cognitive processes, focused on descriptions of comprehension resulting from the twin processes of ‘chunking’ and ‘tracing’ used by software developers in an attempt to reach a cognition of a software system at the code level. A conceptual framework is given as well as some illustrative indicators of likely component measures together with areas needing further research.

A conceptual model of cognitive complexity of elements of the programming process

The main focus of program understanding and reverse engineering research has been on modeling the structure of a program by examining its code. This has been the result of the nature of the systems investigated and the perceived goals of the reverse engineering activities. The types of systems under investigation have changed, however, and the maintenance objectives have evolved. Many legacy systems today are object-oriented and component-based. One of the most prominent maintenance objectives is system migration to distributed environments, most notably the World Wide Web, for interoperation with other systems. This new maintenance objective has a great impact on the types of models expected as products of reverse engineering. As the traditional static software analysis techniques keep their valuable role in program comprehension, additional techniques, especially those focusing on run-time analysis of the subject systems, become equally important. In this paper, we focus on the analysis of the system's dynamic behavior, as it pertains to understanding the system's processes and uses. We give an overview of currently used dynamic reverse engineering techniques and identify some challenges yet to be tackled.

/pdf/dynamic-analysis-for-reverse-engineering-and-program-us7x9l0c0u.pdf

Dynamic analysis for reverse engineering and program understanding

A relative complexity technique that combines the features of many complexity metrics to predict performance and reliability of a computer program is presented. Relative complexity aggregates many similar metrics into a linear compound metric that describes a program. Since relative complexity is a static measure, it is expanded by measuring relative complexity over time to find a program's functional complexity. It is shown that relative complexity gives feedback on the same complexity domains that many other metrics do. Thus, developers can save time by choosing one metric to do the work of many. >

J.C. Munson

Papers

Measuring dynamic program complexity