Image annotation using metric learning in semantic neighbourhoods
Yashaswi Verma,C. V. Jawahar +1 more
- pp 836-849
Reads0
Chats0
TLDR
2PKNN, a two-step variant of the classical K-nearest neighbour algorithm, is proposed that performs comparable to the current state-of-the-art on three challenging image annotation datasets, and shows significant improvements after metric learning.Abstract:
Automatic image annotation aims at predicting a set of textual labels for an image that describe its semantics. These are usually taken from an annotation vocabulary of few hundred labels. Because of the large vocabulary, there is a high variance in the number of images corresponding to different labels ("class-imbalance"). Additionally, due to the limitations of manual annotation, a significant number of available images are not annotated with all the relevant labels ("weak-labelling"). These two issues badly affect the performance of most of the existing image annotation models. In this work, we propose 2PKNN, a two-step variant of the classical K-nearest neighbour algorithm, that addresses these two issues in the image annotation task. The first step of 2PKNN uses "image-to-label" similarities, while the second step uses "image-to-image" similarities; thus combining the benefits of both. Since the performance of nearest-neighbour based methods greatly depends on how features are compared, we also propose a metric learning framework over 2PKNN that learns weights for multiple features as well as distances together. This is done in a large margin set-up by generalizing a well-known (single-label) classification metric learning algorithm for multi-label prediction. For scalability, we implement it by alternating between stochastic sub-gradient descent and projection steps.
Extensive experiments demonstrate that, though conceptually simple, 2PKNN alone performs comparable to the current state-of-the-art on three challenging image annotation datasets, and shows significant improvements after metric learning.read more
Citations
More filters
Journal ArticleDOI
A review on visual content-based and users’ tags-based image annotation: methods and techniques
TL;DR: Various image annotation methods, namely: Visual Content-based and Users’ Tags-based Image Annotation Methods are analyzed since they are one of the dynamic research fields nowadays.
Journal ArticleDOI
Nonparametric label propagation using mutual local similarity in nearest neighbors
TL;DR: An automatic label propagation approach that transfers labels from a small set of manually labeled images to a large set of unlabeled items by means of nearest-neighbor search operating on HoG image descriptors, and introduces the concept of mutual local similarity between the labeled query image and its nearest neighbors as the condition to be verified for propagating labels.
Book ChapterDOI
Image annotation by learning label-specific distance metrics
TL;DR: A novel label specific prediction model is proposed, in which the weight of each label is determined by its specific distance value rather than previous global distance value, which is able to exactly discriminate each label in each neighbor, and efficiently reduce the prediction of false positive and false negative labels.
Journal ArticleDOI
Architecture to improve the accuracy of automatic image annotation systems
TL;DR: The authors designed a more detailed architecture for AIA and suggested new algorithms for its main parts and designed a novel learning method using machine learning and probability bases that resulted in new accuracy milestones in F1-score on most commonly used datasets.
Active learning in social context for image classification
TL;DR: In this paper, a probabilistic approach for jointly maximizing the two aforementioned quantities with a view to automate the process of active learning is proposed. But, the authors do not consider the noisy nature of user-contributed tags.
References
More filters
Proceedings Article
Distance Metric Learning for Large Margin Nearest Neighbor Classification
TL;DR: In this article, a Mahanalobis distance metric for k-NN classification is trained with the goal that the k-nearest neighbors always belong to the same class while examples from different classes are separated by a large margin.
Journal ArticleDOI
Distance Metric Learning for Large Margin Nearest Neighbor Classification
TL;DR: This paper shows how to learn a Mahalanobis distance metric for kNN classification from labeled examples in a globally integrated manner and finds that metrics trained in this way lead to significant improvements in kNN Classification.
Proceedings ArticleDOI
Labeling images with a computer game
Luis von Ahn,Laura Dabbish +1 more
TL;DR: A new interactive system: a game that is fun and can be used to create valuable output that addresses the image-labeling problem and encourages people to do the work by taking advantage of their desire to be entertained.
Journal ArticleDOI
Pegasos: primal estimated sub-gradient solver for SVM
TL;DR: A simple and effective stochastic sub-gradient descent algorithm for solving the optimization problem cast by Support Vector Machines, which is particularly well suited for large text classification problems, and demonstrates an order-of-magnitude speedup over previous SVM learning methods.
Book ChapterDOI
Object Recognition as Machine Translation: Learning a Lexicon for a Fixed Image Vocabulary
TL;DR: This work shows how to cluster words that individually are difficult to predict into clusters that can be predicted well, and cannot predict the distinction between train and locomotive using the current set of features, but can predict the underlying concept.