Fidelity Estimation for a Hierarchical Classifier

TLDR: A simple classifier is investigated and a direct fidelity estimation is proposed, which shows good or bad fidelity estimations but, unfortunately, vice versa a big or small confidence interval for the estimation.

Abstract: To estimate the correct classification rate of a classifier, many different methods exist (test sample, bootstrap, cross validation). The test sample is a method with very small expense. Sometimes, only a small number of objects is available (seldom diseases, high costs for experiments). When we split the sample in training set and test set, we get good or bad fidelity estimations but, unfortunately, vice versa a big or small confidence interval for the estimation. Overcoming this dilemma is only possible for simple classifiers. Such a simple classifier is investigated and a direct fidelity estimation is proposed.