Image annotation using metric learning in semantic neighbourhoods
Yashaswi Verma,C. V. Jawahar +1 more
- pp 836-849
Reads0
Chats0
TLDR
2PKNN, a two-step variant of the classical K-nearest neighbour algorithm, is proposed that performs comparable to the current state-of-the-art on three challenging image annotation datasets, and shows significant improvements after metric learning.Abstract:
Automatic image annotation aims at predicting a set of textual labels for an image that describe its semantics. These are usually taken from an annotation vocabulary of few hundred labels. Because of the large vocabulary, there is a high variance in the number of images corresponding to different labels ("class-imbalance"). Additionally, due to the limitations of manual annotation, a significant number of available images are not annotated with all the relevant labels ("weak-labelling"). These two issues badly affect the performance of most of the existing image annotation models. In this work, we propose 2PKNN, a two-step variant of the classical K-nearest neighbour algorithm, that addresses these two issues in the image annotation task. The first step of 2PKNN uses "image-to-label" similarities, while the second step uses "image-to-image" similarities; thus combining the benefits of both. Since the performance of nearest-neighbour based methods greatly depends on how features are compared, we also propose a metric learning framework over 2PKNN that learns weights for multiple features as well as distances together. This is done in a large margin set-up by generalizing a well-known (single-label) classification metric learning algorithm for multi-label prediction. For scalability, we implement it by alternating between stochastic sub-gradient descent and projection steps.
Extensive experiments demonstrate that, though conceptually simple, 2PKNN alone performs comparable to the current state-of-the-art on three challenging image annotation datasets, and shows significant improvements after metric learning.read more
Citations
More filters
Journal ArticleDOI
Partial label metric learning by collapsing classes
TL;DR: Experimental results on six UCI data sets and four real-world PLL data sets show that the proposed algorithm can obviously improve the accuracy of the existing PLL algorithms.
Proceedings Article
Fixed-rank supervised metric learning on riemannian manifold
TL;DR: This paper harnesses the Riemannian manifold geometry of the collection of fixed-rank matrices and devise a novel second-order RiemANNian retraction operator that ensures that W always resides on the manifold and can be applied in generic rank-constrained machine learning algorithms.
Proceedings ArticleDOI
Multi-label annotation of music
TL;DR: A simple feature augmentation technique based on non-negative matrix factorization (NMF) with an intuition to decompose a music piece into its constituent components is proposed.
Proceedings Article
VSE-ens: Visual-Semantic Embeddings with Efficient Negative Sampling
TL;DR: In this paper, a fast adaptive negative sampler is proposed to choose the negative examples that are most likely to meet the requirements of violation according to the latent factors of images, which can linearly scale up to large datasets.
Book ChapterDOI
Robust Mapping Learning for Multi-view Multi-label Classification with Missing Labels
TL;DR: This paper proposes a new robust method that incorporates multiple views into a mixed feature matrix, and augments the initial label matrix with label correlation matrix to estimate authentic label assignments.
References
More filters
Proceedings Article
Distance Metric Learning for Large Margin Nearest Neighbor Classification
TL;DR: In this article, a Mahanalobis distance metric for k-NN classification is trained with the goal that the k-nearest neighbors always belong to the same class while examples from different classes are separated by a large margin.
Journal ArticleDOI
Distance Metric Learning for Large Margin Nearest Neighbor Classification
TL;DR: This paper shows how to learn a Mahalanobis distance metric for kNN classification from labeled examples in a globally integrated manner and finds that metrics trained in this way lead to significant improvements in kNN Classification.
Proceedings ArticleDOI
Labeling images with a computer game
Luis von Ahn,Laura Dabbish +1 more
TL;DR: A new interactive system: a game that is fun and can be used to create valuable output that addresses the image-labeling problem and encourages people to do the work by taking advantage of their desire to be entertained.
Journal ArticleDOI
Pegasos: primal estimated sub-gradient solver for SVM
TL;DR: A simple and effective stochastic sub-gradient descent algorithm for solving the optimization problem cast by Support Vector Machines, which is particularly well suited for large text classification problems, and demonstrates an order-of-magnitude speedup over previous SVM learning methods.
Book ChapterDOI
Object Recognition as Machine Translation: Learning a Lexicon for a Fixed Image Vocabulary
TL;DR: This work shows how to cluster words that individually are difficult to predict into clusters that can be predicted well, and cannot predict the distinction between train and locomotive using the current set of features, but can predict the underlying concept.