Image annotation using metric learning in semantic neighbourhoods
Yashaswi Verma,C. V. Jawahar +1 more
- pp 836-849
Reads0
Chats0
TLDR
2PKNN, a two-step variant of the classical K-nearest neighbour algorithm, is proposed that performs comparable to the current state-of-the-art on three challenging image annotation datasets, and shows significant improvements after metric learning.Abstract:
Automatic image annotation aims at predicting a set of textual labels for an image that describe its semantics. These are usually taken from an annotation vocabulary of few hundred labels. Because of the large vocabulary, there is a high variance in the number of images corresponding to different labels ("class-imbalance"). Additionally, due to the limitations of manual annotation, a significant number of available images are not annotated with all the relevant labels ("weak-labelling"). These two issues badly affect the performance of most of the existing image annotation models. In this work, we propose 2PKNN, a two-step variant of the classical K-nearest neighbour algorithm, that addresses these two issues in the image annotation task. The first step of 2PKNN uses "image-to-label" similarities, while the second step uses "image-to-image" similarities; thus combining the benefits of both. Since the performance of nearest-neighbour based methods greatly depends on how features are compared, we also propose a metric learning framework over 2PKNN that learns weights for multiple features as well as distances together. This is done in a large margin set-up by generalizing a well-known (single-label) classification metric learning algorithm for multi-label prediction. For scalability, we implement it by alternating between stochastic sub-gradient descent and projection steps.
Extensive experiments demonstrate that, though conceptually simple, 2PKNN alone performs comparable to the current state-of-the-art on three challenging image annotation datasets, and shows significant improvements after metric learning.read more
Citations
More filters
Posted Content
Deep Context-Aware Kernel Networks.
Mingyuan Jiu,Hichem Sahbi +1 more
TL;DR: A novel deep network architecture that learns context in kernel design is introduced, fully determined by the solution of an objective function mixing a content term that captures the intrinsic similarity between data, a context criterion which models their structure and a regularization term that helps designing smooth kernel network representations.
Posted Content
Learning Deep Context-Network Architectures for Image Annotation
Mingyuan Jiu,Hichem Sahbi +1 more
TL;DR: This paper introduces a novel context-aware kernel design framework based on deep learning that discriminatively learns spatial geometric context as the weights of a deep network (DN) while the parameters of this network determine the most relevant parts of the learned context.
Book ChapterDOI
Image Annotation Based on Multi-view Learning
TL;DR: A novel multi-view semi-supervised learning scheme to improve the performance of image annotation by using multiple views of an image and leveraging the information contained in pseudo-labeled images is proposed.
Proceedings ArticleDOI
Tag propagation by using multi-view NMF consistent matrix for image annotation
TL;DR: A novel propagation based on multi-view Negative-Matrix-Factorization (NMF) clustering and sparse tag sensing, where a ranked consistent matrix is created from multiview NMF observations to estimate the clustering structures, while sparse tags of a query image are approximately reconstructed with consistent matrix sensing.
Book ChapterDOI
Multiple Kernel Learning Based on Weak Learner for Automatic Image Annotation
TL;DR: This work proposes a multiple kernel learning method based on weak learner for image annotation, which can acquire the semantic correlations to predict tags of a given image, and employs the convolutional neural network to extract the semantic features of images.
References
More filters
Proceedings Article
Distance Metric Learning for Large Margin Nearest Neighbor Classification
TL;DR: In this article, a Mahanalobis distance metric for k-NN classification is trained with the goal that the k-nearest neighbors always belong to the same class while examples from different classes are separated by a large margin.
Journal ArticleDOI
Distance Metric Learning for Large Margin Nearest Neighbor Classification
TL;DR: This paper shows how to learn a Mahalanobis distance metric for kNN classification from labeled examples in a globally integrated manner and finds that metrics trained in this way lead to significant improvements in kNN Classification.
Proceedings ArticleDOI
Labeling images with a computer game
Luis von Ahn,Laura Dabbish +1 more
TL;DR: A new interactive system: a game that is fun and can be used to create valuable output that addresses the image-labeling problem and encourages people to do the work by taking advantage of their desire to be entertained.
Journal ArticleDOI
Pegasos: primal estimated sub-gradient solver for SVM
TL;DR: A simple and effective stochastic sub-gradient descent algorithm for solving the optimization problem cast by Support Vector Machines, which is particularly well suited for large text classification problems, and demonstrates an order-of-magnitude speedup over previous SVM learning methods.
Book ChapterDOI
Object Recognition as Machine Translation: Learning a Lexicon for a Fixed Image Vocabulary
TL;DR: This work shows how to cluster words that individually are difficult to predict into clusters that can be predicted well, and cannot predict the distinction between train and locomotive using the current set of features, but can predict the underlying concept.