scispace - formally typeset
Search or ask a question
Proceedings ArticleDOI

Improving classifier fusion via Pool Adjacent Violators normalization

TL;DR: This research explores an alternative method to combine classifiers at the score level and proposes the PAV algorithm for classifier fusion on publicly available NIST multi-modal biometrics score dataset, finding that it provides several advantages over existing techniques and is able to further improve the results obtained by other approaches.
Abstract: Classifier fusion is a well-studied problem in which decisions from multiple classifiers are combined at the score, rank, or decision level to obtain better results than a single classifier. Subsequently, various techniques for combining classifiers at each of these levels have been proposed in the literature. Many popular methods entail scaling and normalizing the scores obtained by each classifier to a common numerical range before combining the normalized scores using the sum rule or another classifier. In this research, we explore an alternative method to combine classifiers at the score level. The Pool Adjacent Violators (PAV) algorithm has traditionally been utilized to convert classifier match scores to confidence values that model posterior probabilities for test data. The PAV algorithm and other score normalization techniques have studied the same problem without being aware of each other. In this first ever study to combine the two, we propose the PAV algorithm for classifier fusion on publicly available NIST multi-modal biometrics score dataset. We observe that it provides several advantages over existing techniques and find that the interpretation learned by the PAV algorithm is more robust than the scaling learned by other popular normalization algorithms such as min-max. Moreover, the PAV algorithm enables the combined score to be interpreted as confidence and is able to further improve the results obtained by other approaches. We also observe that utilizing traditional normalization techniques first for individual classifiers and then normalizing the fused score using PAV offers a performance boost compared to only using the PAV algorithm.
Citations
More filters
Journal ArticleDOI
TL;DR: In this article, a deep learning-based multimodal fusion architecture for classification tasks is proposed, which guarantees compatibility with any kind of learning models, deals with cross-modal information carefully, and prevents performance degradation due to partial absence of data.

82 citations

Journal ArticleDOI
TL;DR: A novel deep learning-based multimodal fusion architecture for classification tasks, which guarantees compatibility with any kind of learning models, deals with cross-modal information carefully, and prevents performance degradation due to partial absence of data.
Abstract: Classification using multimodal data arises in many machine learning applications. It is crucial not only to model cross-modal relationship effectively but also to ensure robustness against loss of part of data or modalities. In this paper, we propose a novel deep learning-based multimodal fusion architecture for classification tasks, which guarantees compatibility with any kind of learning models, deals with cross-modal information carefully, and prevents performance degradation due to partial absence of data. We employ two datasets for multimodal classification tasks, build models based on our architecture and other state-of-the-art models, and analyze their performance on various situations. The results show that our architecture outperforms the other multimodal fusion architectures when some parts of data are not available.

41 citations


Cites methods from "Improving classifier fusion via Poo..."

  • ...[34] employed the pool adjacent violators algorithm for combining multiple classifiers, which calibrates the outputs of the classifiers with respect to their confidence values....

    [...]

Journal ArticleDOI
TL;DR: This report shows how differences in the variance of features lead to differences inThe strength of the influence of each feature on the similarity scores produced from all the features, and compares six variance normalization methods in terms of how well they reduce the impact of the variance differences.
Abstract: The importance of normalizing biometric features or matching scores is understood in the multimodal biometric case, but there is less attention to the unimodal case. Prior reports assess the effectiveness of normalization directly on biometric performance. We propose that this process is logically comprised of two independent steps: (1) methods to equalize the effect of each biometric feature on the similarity scores calculated from all the features together and (2) methods of weighting the normalized features to optimize biometric performance. In this report, we address step 1 only and focus exclusively on normally distributed features. We show how differences in the variance of features lead to differences in the strength of the influence of each feature on the similarity scores produced from all the features. Since these differences in variance have nothing to do with importance in the biometric sense, it makes no sense to allow them to have greater weight in the assessment of biometric performance. We employed two types of features: (1) real eye-movement features and (2) synthetic features. We compare six variance normalization methods (histogram equalization, L1-normalization, median normalization, z-score normalization, min–max normalization, and L-infinite normalization) and one distance metric (Mahalanobis distance) in terms of how well they reduce the impact of the variance differences. The effectiveness of different techniques on real data depended on the strength of the inter-correlation of the features. For weakly correlated real features and synthetic features, histogram equalization was the best method followed by L1 normalization.

29 citations

References
More filters
Journal ArticleDOI
TL;DR: A common theoretical framework for combining classifiers which use distinct pattern representations is developed and it is shown that many existing schemes can be considered as special cases of compound classification where all the pattern representations are used jointly to make a decision.
Abstract: We develop a common theoretical framework for combining classifiers which use distinct pattern representations and show that many existing schemes can be considered as special cases of compound classification where all the pattern representations are used jointly to make a decision. An experimental comparison of various classifier combination schemes demonstrates that the combination rule developed under the most restrictive assumptions-the sum rule-outperforms other classifier combinations schemes. A sensitivity analysis of the various schemes to estimation errors is carried out to show that this finding can be justified theoretically.

5,670 citations


"Improving classifier fusion via Poo..." refers background in this paper

  • ...Combining multiple classifiers to improve results has been studied extensively in literature [1], [2], [3]....

    [...]

Journal ArticleDOI
TL;DR: Study of the performance of different normalization techniques and fusion rules in the context of a multimodal biometric system based on the face, fingerprint and hand-geometry traits of a user found that the application of min-max, z-score, and tanh normalization schemes followed by a simple sum of scores fusion method results in better recognition performance compared to other methods.

2,021 citations


"Improving classifier fusion via Poo..." refers background or methods in this paper

  • ...[5] where the authors discuss the merits and demerits of popular score normalization techniques such as min-max, z-score, and tan-h normalization [6]....

    [...]

  • ...A number of different methods have been proposed in the literature to address the problem of score normalization, an analytical review of which is presented in [5]....

    [...]

Journal ArticleDOI
TL;DR: It is shown that it is possible to build a hybrid classifier that will perform at least as well as the best available classifier for any target conditions, and in some cases, the performance of the hybrid actually can surpass that of the best known classifier.
Abstract: In real-world environments it usually is difficult to specify target operating conditions precisely, for example, target misclassification costs. This uncertainty makes building robust classification systems problematic. We show that it is possible to build a hybrid classifier that will perform at least as well as the best available classifier for any target conditions. In some cases, the performance of the hybrid actually can surpass that of the best known classifier. This robust performance extends across a wide variety of comparison frameworks, including the optimization of metrics such as accuracy, expected cost, lift, precision, recall, and workforce utilization. The hybrid also is efficient to build, to store, and to update. The hybrid is based on a method for the comparison of classifier performance that is robust to imprecise class distributions and misclassification costs. The ROC convex hull (ROCCH) method combines techniques from ROC analysis, decision analysis and computational geometry, and adapts them to the particulars of analyzing learned classifiers. The method is efficient and incremental, minimizes the management of classifier performance data, and allows for clear visual comparisons and sensitivity analyses. Finally, we point to empirical evidence that a robust hybrid classifier indeed is needed for many real-world problems.

1,134 citations


"Improving classifier fusion via Poo..." refers background or methods in this paper

  • ...It has been shown in [16] that only a classifier which lies on the convex hull of the...

    [...]

  • ...Since the PAV algorithm has been shown to be functionally equivalent to the ROCCH algorithm [10], it transforms the scores such that they are both interpretable and optimal as per the ROCCH algorithm [16]....

    [...]

Posted Content
TL;DR: The ROC convex hull (ROCCH) method as mentioned in this paper combines techniques from ROC analysis, decision analysis and computational geometry, and adapts them to the particulars of analyzing learned classifiers.
Abstract: In real-world environments it usually is difficult to specify target operating conditions precisely, for example, target misclassification costs. This uncertainty makes building robust classification systems problematic. We show that it is possible to build a hybrid classifier that will perform at least as well as the best available classifier for any target conditions. In some cases, the performance of the hybrid actually can surpass that of the best known classifier. This robust performance extends across a wide variety of comparison frameworks, including the optimization of metrics such as accuracy, expected cost, lift, precision, recall, and workforce utilization. The hybrid also is efficient to build, to store, and to update. The hybrid is based on a method for the comparison of classifier performance that is robust to imprecise class distributions and misclassification costs. The ROC convex hull (ROCCH) method combines techniques from ROC analysis, decision analysis and computational geometry, and adapts them to the particulars of analyzing learned classifiers. The method is efficient and incremental, minimizes the management of classifier performance data, and allows for clear visual comparisons and sensitivity analyses. Finally, we point to empirical evidence that a robust hybrid classifier indeed is needed for many real-world problems.

1,114 citations

Proceedings ArticleDOI
23 Jul 2002
TL;DR: This work shows how to obtain accurate probability estimates for multiclass problems by combining calibrated binary probability estimates, and proposes a new method for obtaining calibrated two-class probability estimates that can be applied to any classifier that produces a ranking of examples.
Abstract: Class membership probability estimates are important for many applications of data mining in which classification outputs are combined with other sources of information for decision-making, such as example-dependent misclassification costs, the outputs of other classifiers, or domain knowledge. Previous calibration methods apply only to two-class problems. Here, we show how to obtain accurate probability estimates for multiclass problems by combining calibrated binary probability estimates. We also propose a new method for obtaining calibrated two-class probability estimates that can be applied to any classifier that produces a ranking of examples. Using naive Bayes and support vector machine classifiers, we give experimental results from a variety of two-class and multiclass domains, including direct marketing, text categorization and digit recognition.

1,091 citations


"Improving classifier fusion via Poo..." refers methods in this paper

  • ...The PAV algorithm [10] has traditionally been utilized for calibrating classifier scores into probability estimates [11]....

    [...]

  • ...The PAV algorithm is a non-parametric isotonic regression based method to obtain the desired mapping f , proposed by Zadrozny and Elkan [11]....

    [...]