Open AccessProceedings Article
Consistency Analysis for Binary Classification Revisited
Krzysztof Dembczyński,Wojciech Kotłowski,Oluwasanmi Koyejo,Nagarajan Natarajan +3 more
- pp 961-969
TLDR
This manuscript analyzes non-decomposable metrics such as the F-measure and the Jaccard measure from statistical and algorithmic points of view, and provides guidance to the theory and practice of binary classification with complex metrics.Abstract:
Statistical learning theory is at an inflection point enabled by recent advances in understanding and optimizing a wide range of metrics. Of particular interest are non-decomposable metrics such as the F-measure and the Jaccard measure which cannot be represented as a simple average over examples. Non-decomposability is the primary source of difficulty in theoretical analysis, and interestingly has led to two distinct settings and notions of consistency. In this manuscript we analyze both settings, from statistical and algorithmic points of view, to explore the connections and to highlight differences between them for a wide range of metrics. The analysis complements previous results on this topic, clarifies common confusions around both settings, and provides guidance to the theory and practice of binary classification with complex metrics.read more
Citations
More filters
Proceedings Article
A no-regret generalization of hierarchical softmax to extreme multi-label classification
TL;DR: It is shown that PLTs are a no-regret multi-label generalization of HSM when precision@$k$ is used as a model evaluation metric, and it is proved that pick-one-label heuristic---a reduction technique from multi- label to multi-class that is routinely used along with HSM---is not consistent in general.
Proceedings Article
A Generalized Neyman-Pearson Criterion for Optimal Domain Adaptation
TL;DR: In this paper, a new Neyman-Pearson-like criterion was proposed for domain adaptation for binary classification, and stronger domain adaptation results are possible than what has previously been established.
Journal ArticleDOI
Threshold optimization for F measure of macro-averaged precision and recall
Anna Berger,Sergey A. Guda +1 more
TL;DR: The problem of selecting the optimal threshold for each class is reduced to the problem of obtaining a fixed point of a specifically introduced transformation of a unit square and the suggested algorithm lets us localize all possible coordinate-wise maximums and detect the optimal among them.
Book ChapterDOI
Deep F-Measure Maximization in Multi-label Classification: A Comparative Study
TL;DR: Extensions of utility maximization and decision-theoretic methods that can optimize the F\(_\beta \)-measure with (convolutional) neural networks are introduced and results illustrate that decision- theoretic inference algorithms are worth the investment.
Proceedings Article
Calibrated Surrogate Maximization of Linear-fractional Utility in Binary Classification
Han Bao,Masashi Sugiyama +1 more
TL;DR: This paper considers linear-fractional metrics, which are a family of classification performance metrics that encompasses many standard ones such as the F${}_\beta$-measure and Jaccard index, and proposes methods to directly maximize performances under those metrics.
References
More filters
Book
Foundations of Machine Learning
TL;DR: This graduate-level textbook introduces fundamental concepts and methods in machine learning, and provides the theoretical underpinnings of these algorithms, and illustrates key aspects for their application.
Proceedings ArticleDOI
A support vector method for multivariate performance measures
TL;DR: An algorithm with which such multivariate SVMs can be trained in polynomial time for large classes of potentially non-linear performance measures, in particular ROCArea and all measures that can be computed from the contingency table are given.
Journal Article
A Survey of Binary Similarity and Distance Measures
TL;DR: This work has collected 76 binary similarity and distance measures used over the last century and reveals their correlations through the hierarchical clustering technique.
Proceedings ArticleDOI
Evaluating and optimizing autonomous text classification systems
TL;DR: This work shows how to define what constitutes good effectiveness for binary text classification systems, tune the systems to achieve the highest possible effectiveness, and estimate how the effectiveness changes as new data is processed.
Journal ArticleDOI
On label dependence and loss minimization in multi-label classification
TL;DR: It is claimed that two types of label dependence should be distinguished, namely conditional and marginal dependence, and three scenarios in which the exploitation of one of these types of dependence may boost the predictive performance of a classifier are presented.