While methods for comparing two learning algorithms on a single data set have been scrutinized for quite some time already, the issue of statistical tests for comparisons of more algorithms on multiple data sets, which is even more essential to typical machine learning studies, has been all but ignored. This article reviews the current practice and then theoretically and empirically examines several suitable tests. Based on that, we recommend a set of simple, yet safe and robust non-parametric tests for statistical comparisons of classifiers: the Wilcoxon signed ranks test for comparison of two classifiers and the Friedman test with the corresponding post-hoc tests for comparison of more classifiers over multiple data sets. Results of the latter can also be neatly presented with the newly introduced CD (critical difference) diagrams.

/pdf/statistical-comparisons-of-classifiers-over-multiple-data-18bm1e3eph.pdf

Statistical Comparisons of Classifiers over Multiple Data Sets

a b s t r a c t The interest in nonparametric statistical analysis has grown recently in the field of computational intelligence. In many experimental studies, the lack of the required properties for a proper application of parametric procedures - independence, normality, and homoscedasticity - yields to nonparametric ones the task of performing a rigorous comparison among algorithms. In this paper, we will discuss the basics and give a survey of a complete set of nonparametric procedures developed to perform both pairwise and multiple comparisons, for multi-problem analysis. The test problems of the CEC'2005 special session on real parameter optimization will help to illustrate the use of the tests throughout this tutorial, analyzing the results of a set of well-known evolutionary and swarm intelligence algorithms. This tutorial is concluded with a compilation of considerations and recommendations, which will guide practitioners when using these tests to contrast their experimental results.

https://sci2s.ugr.es/sites/default/files/ficherosPublicaciones/1374_2011-Derrac-SWEVO.pdf

A practical tutorial on the use of nonparametric statistical tests as a methodology for comparing evolutionary and swarm intelligence algorithms

Many of the more useful and powerful nonparametric procedures may be presented in a unified manner by treating them as rank transformation procedures. Rank transformation procedures are ones in which the usual parametric procedure is applied to the ranks of the data instead of to the data themselves. This technique should be viewed as a useful tool for developing nonparametric procedures to solve new problems.

Rank Transformations as a Bridge between Parametric and Nonparametric Statistics

(Knowledge Extraction based onEvolutionary Learning) tool, an open source software that supports datamanagement and a designer of experiments. KEEL pays special attentionto the implementation of evolutionary learning and soft computing basedtechniques for Data Mining problems including regression, classiﬁcation,clustering, pattern mining and so on.The aim of this paper is to present three new aspects of KEEL: KEEL-dataset, a data set repository which includes the data set partitions in theKEELformatandshowssomeresultsofalgorithmsinthesedatasets; someguidelines for including new algorithms in KEEL, helping the researcherstomaketheirmethodseasilyaccessibletootherauthorsandtocomparetheresults of many approaches already included within the KEEL software;and a module of statistical procedures developed in order to provide to theresearcher a suitable tool to contrast the results obtained in any experimen-talstudy.Acaseofstudyisgiventoillustrateacompletecaseofapplicationwithin this experimental analysis framework.

/pdf/keel-data-mining-software-tool-data-set-repository-1jctj7itm5.pdf

KEEL Data-Mining Software Tool: Data Set Repository, Integration of Algorithms and Experimental Analysis Framework

http://www-personal.umich.edu/~simeonem/USpaper.pdf

A Methodology For Performing Global Uncertainty And Sensitivity Analysis In Systems Biology

The Friedman (1937) test for the randomized complete block design is used to test the hypothesis of no treatment effect among k treatments with b blocks. Difficulty in determination of the size of the critical region for this hypothesis is com¬pounded by the facts that (1) the most recent extension of exact tables for the distribution of the test statistic by Odeh (1977) go up only to the case with k6 and b6, and (2) the usual chi-square approximation is grossly inaccurate for most commonly used combinations of (k,b). The purpose of this paper 2 is to compare two new approximations with the usual x2 and F large sample approximations. This work represents an extension to the two-way layout of work done earlier by the authors for the one-way Kruskal-Wallis test statistic.

Approximations of the critical region of the fbietkan statistic

This article focuses on the relationship between true Types I and II error probabilities and the effects of departures from independence assumptions on hypothesis testing in the oneway analysis of variance. A method for constructing a useful class of nonidentity error correlation matrices suitable for studying this relationship is offered and explored. Special emphasis is placed on the numerical features of this relationship that can be easily exploited in the classroom. The perspective is adaptable to more complicated designs including regression models.

https://www.jstor.org/stable/info/2684223

The effects of violations of independence assumptions in the one-way ANOVA

A method for inducing a desired rank correlation matrix on multivariate input vectors for simulation studies has recently been developed by Iman and Conover (1982). The primary intention of this procedure is to produce correlated input variables for use with computer models. Since this procedure is distribution free and allows the exact marginal distributions to remain intact it can be used with any marginal distributions for which it is reasonable to think in terms of correlation. In this paper we present a series of rank correlation plots based on this procedure when the marginal distributions are normal, lognormal, uniform and loguniform. These plots provide a convenient tool both for aiding the modeler in determining the degree of dependence among input variables (rather than guessing) and for communicating with the modeler the effect of different correlation assumptions. In addition this procedure can be used with sample multivariate data by sampling directly from the respective marginal empirical di...

Rank correlation plots for use with correlated input variables

The well known pure error-lack of fit test which can be used to assess the adequacy of a proposed linear regression model requires replication and assumes that the error structure is . This procedure is generalized to provide a test for lack of fit for the 2 case of nonreplication and error structure for certain known positive definite correlation matrices V. Included in the class of applicable correlation matrices are the cases of intraclass correlation and equicorrelation. The critical points of the F distribution can be used to provide a test of the exact desired size.

Testing regression function adequacy with correlation and without replication

Fixed sample size approximately similar tests for the Behrens-Fisher problem are studied and compared with various other tests suggested in current sttistical methodelogy texts. Several fourmoment approxiamtely similar tests are developed and offered as alternatives. These tests are shown to be good practical solutions which are easily implemented in practice.

James M. Davenport

Papers

Approximations of the critical region of the fbietkan statistic

The effects of violations of independence assumptions in the one-way ANOVA

Rank correlation plots for use with correlated input variables

Testing regression function adequacy with correlation and without replication

A four-moment approach and other practical solutions to the behrens-f1sher problem