scispace - formally typeset
Search or ask a question
Book Chapter•DOI•

Image annotation using metric learning in semantic neighbourhoods

Yashaswi Verma1, C. V. Jawahar1•
International Institute of Information Technology, Hyderabad1
07 Oct 2012-pp 836-849
TL;DR: 2PKNN, a two-step variant of the classical K-nearest neighbour algorithm, is proposed that performs comparable to the current state-of-the-art on three challenging image annotation datasets, and shows significant improvements after metric learning.
Abstract: Automatic image annotation aims at predicting a set of textual labels for an image that describe its semantics. These are usually taken from an annotation vocabulary of few hundred labels. Because of the large vocabulary, there is a high variance in the number of images corresponding to different labels ("class-imbalance"). Additionally, due to the limitations of manual annotation, a significant number of available images are not annotated with all the relevant labels ("weak-labelling"). These two issues badly affect the performance of most of the existing image annotation models. In this work, we propose 2PKNN, a two-step variant of the classical K-nearest neighbour algorithm, that addresses these two issues in the image annotation task. The first step of 2PKNN uses "image-to-label" similarities, while the second step uses "image-to-image" similarities; thus combining the benefits of both. Since the performance of nearest-neighbour based methods greatly depends on how features are compared, we also propose a metric learning framework over 2PKNN that learns weights for multiple features as well as distances together. This is done in a large margin set-up by generalizing a well-known (single-label) classification metric learning algorithm for multi-label prediction. For scalability, we implement it by alternating between stochastic sub-gradient descent and projection steps. Extensive experiments demonstrate that, though conceptually simple, 2PKNN alone performs comparable to the current state-of-the-art on three challenging image annotation datasets, and shows significant improvements after metric learning.

Content maybe subject to copyright    Report

Citations
More filters
Book Chapter•DOI•
Recurrent Image Annotation with Explicit Inter-label Dependencies

[...]

Ayushi Dutta1, Yashaswi Verma2, C. V. Jawahar3•
Target Corporation1, Indian Institute of Technology, Jodhpur2, International Institute of Information Technology, Hyderabad3
23 Aug 2020
TL;DR: This paper proposes a novel approach in which the RNN is explicitly forced to learn multiple relevant inter-label dependencies, without the need of feeding the ground-truth in any particular order, and outperforms several state-of-the-art techniques on two popular datasets.
Abstract: Inspired by the success of the CNN-RNN framework in the image captioning task, several works have explored this in multi-label image annotation with the hope that the RNN followed by a CNN would encode inter-label dependencies better than using a CNN alone. To do so, for each training sample, the earlier methods converted the ground-truth label-set into a sequence of labels based on their frequencies (e.g., rare-to-frequent) for training the RNN. However, since the ground-truth is an unordered set of labels, imposing a fixed and predefined sequence on them does not naturally align with this task. To address this, some of the recent papers have proposed techniques that are capable to train the RNN without feeding the ground-truth labels in a particular sequence/order. However, most of these techniques leave it to the RNN to implicitly choose one sequence for the ground-truth labels corresponding to each sample at the time of training, thus making it inherently biased. In this paper, we address this limitation and propose a novel approach in which the RNN is explicitly forced to learn multiple relevant inter-label dependencies, without the need of feeding the ground-truth in any particular order. Using thorough empirical comparisons, we demonstrate that our approach outperforms several state-of-the-art techniques on two popular datasets (MS-COCO and NUS-WIDE). Additionally, it provides a new perspecitve of looking at an unordered set of labels as equivalent to a collection of different permutations (sequences) of those labels, thus naturally aligning with the image annotation task. Our code is available at: https://github.com/ayushidutta/multi-order-rnn.

5 citations

Proceedings Article•DOI•
Image annotation using multi-view non-negative matrix factorization and semantic co-occurrence

[...]

Fuping Zhong1, Lihong Ma1•
South China University of Technology1
01 Nov 2016
TL;DR: A relevance feedback algorithm based on Multi-view non-negative matrix factorization (MultiNMF) is presented to improve the retrieval performance during the process of querying the nearest neighbors and a semantic co-occurrence (SC) based strategy is derived to effectively adjust the order of the annotated keywords.
Abstract: Image annotation aims to automatically predict a set of relevant keywords for an image that describe its semantics. Nearest Neighbor (NN) based methods have been successfully applied to address image annotation problems. In this paper,a novel method is introduced to improve the performance of annotating images. Firstly, we present a relevance feedback algorithm based on Multi-view non-negative matrix factorization (MultiNMF) to improve the retrieval performance during the process of querying the nearest neighbors. Secondly, a semantic co-occurrence (SC) based strategy is derived to effectively adjust the order of the annotated keywords. Experiment results on Corel5K dataset demonstrate that the proposed method outperforms those previous similar methods.

5 citations

Cites background or methods or result from "Image annotation using metric learn..."

  • ...On the other line of research,Nearest Neighbor (NN) based methods [1,2,3] have been successfully applied in solving image annotation problems,giving some of the best results despite their simplicity....

    [...]

  • ...In 2PKNN [3],the authors proposed a two-step algorithm that uses image-to-tag and image-to-image similarities and a metric learning framework to learn weights for multiple features....

    [...]

  • ...Some known methods including Metric Learning [2,3] (ML) based methods and Data structure [10,11] based methods are selected to address this problem....

    [...]

  • ...MultiNMF Relevance Feedback Algorithm Given a test image,we first find its N nearest neighbors by utilizing traditional distance based methods [2,3],and then choose K samples from these nearest neighbors to generate basis matrix U and coefficient matrix V through MultiNMF;A new latent factors space is formed by the technique,then these N nearest neighbors are mapped into the new space;Finally,we utilize the features in the new space to retrieve nearest neighbors again;We repeat these steps until the result meet our demands....

    [...]

  • ...A common consensus has been reached from [1,2,3] that the existing image annotation approaches can be roughly divided into three groups:Generative models [4,5],Discriminative models [6,7],and Nearest Neighbor based models [1,2,3]....

    [...]

Posted Content•
Sampled Image Tagging and Retrieval Methods on User Generated Content

[...]

Karl Ni, Kyle Zaragoza, Charles Foster, Carmen J. Carrano, Barry Chen, Yonas Tesfaye, Alex Gude 
21 Nov 2016-arXiv: Computer Vision and Pattern Recognition
TL;DR: This work trains a deep learning image tagging and retrieval system on large scale, user generated content (UGC) using sampling methods and joint optimization of word embeddings to enable new capability to train algorithms on large, scale unstructured text in the YFCC100M dataset and outperform cited work in zero-shot capability.
Abstract: Traditional image tagging and retrieval algorithms have limited value as a result of being trained with heavily curated datasets. These limitations are most evident when arbitrary search words are used that do not intersect with training set labels. Weak labels from user generated content (UGC) found in the wild (e.g., Google Photos, FlickR, etc.) have an almost unlimited number of unique words in the metadata tags. Prior work on word embeddings successfully leveraged unstructured text with large vocabularies, and our proposed method seeks to apply similar cost functions to open source imagery. Specifically, we train a deep learning image tagging and retrieval system on large scale, user generated content (UGC) using sampling methods and joint optimization of word embeddings. By using the Yahoo! FlickR Creative Commons (YFCC100M) dataset, such an approach builds robustness to common unstructured data issues that include but are not limited to irrelevant tags, misspellings, multiple languages, polysemy, and tag imbalance. As a result, the final proposed algorithm will not only yield comparable results to state of the art in conventional image tagging, but will enable new capability to train algorithms on large, scale unstructured text in the YFCC100M dataset and outperform cited work in zero-shot capability.

5 citations

Cites background from "Image annotation using metric learn..."

  • ...Methods for automatic image annotation have ranged from generative [1, 6, 34] and discriminative models [10] for image tags to nearest neighbor search-based approaches [18, 8, 32]....

    [...]

  • ...Feature selection [32] on manual features through metric learning provided an adequate analysis of which manual features provided the most amount of information....

    [...]

Journal Article•DOI•
Personalized Image Annotation Using Deep Architecture

[...]

Fangxiang Feng1, Ruishan Liu, Xiaojie Wang1, Xueming Li1, Song Bi2 •
Beijing University of Posts and Telecommunications1, North China University of Technology2
20 Oct 2017-IEEE Access
TL;DR: A deep architecture for personalized image annotation is proposed by leveraging the wealth of information in user’s tagging history by combining the two learned features to predict the tags.
Abstract: In image-centric social networks, such as Instagram and Pinterest, users tend to share photos with several tags. These tags describe the content of the image or provide additional contextual information, and therefore may not be necessarily tied to image content and usually carry personal preference. Annotating images in social networks in a personalized manner is in demand. However, the existing image annotation models, which rely only on image content information, cannot capture the user’s tagging preference. In this paper, we propose a deep architecture for personalized image annotation by leveraging the wealth of information in user’s tagging history. The proposed architecture consists of three components: two components for learning features of the image content and user’s history tags and the other one for combining the two learned features to predict the tags. We also explore two ways to model user’s history tags: 1) simply average the embeddings of user’s history tags and 2) model user’s history tags with a sequence model by long short-term memory recurrent neural network. We evaluate our proposed deep architecture on a large-scale and realistic data set, consisting of ~22.8 million public images uploaded by ~4.69 million users. Experimental results show that our proposed deep architecture is effective on a personalized image annotation task.

5 citations

Cites methods from "Image annotation using metric learn..."

  • ...fer labels between visually similar images [19], [20] and the later used learnable metrics and weighted voting schemes [21], [22]....

    [...]

Journal Article•DOI•
A Review of Methods for The Image Automatic Annotation

[...]

Myasar Mundher Adnan1, Mohd Shafry Mohd Rahim, Mohammed Hasan Ali, Karrar Al-Jawaheri1, Karrar Neamah •
Islamic University1
01 Apr 2021

5 citations

References
More filters
Proceedings Article•
Distance Metric Learning for Large Margin Nearest Neighbor Classification

[...]

Kilian Q. Weinberger1, John Blitzer1, Lawrence K. Saul1•
University of Pennsylvania1
05 Dec 2005
TL;DR: In this article, a Mahanalobis distance metric for k-NN classification is trained with the goal that the k-nearest neighbors always belong to the same class while examples from different classes are separated by a large margin.
Abstract: We show how to learn a Mahanalobis distance metric for k-nearest neighbor (kNN) classification by semidefinite programming. The metric is trained with the goal that the k-nearest neighbors always belong to the same class while examples from different classes are separated by a large margin. On seven data sets of varying size and difficulty, we find that metrics trained in this way lead to significant improvements in kNN classification—for example, achieving a test error rate of 1.3% on the MNIST handwritten digits. As in support vector machines (SVMs), the learning problem reduces to a convex optimization based on the hinge loss. Unlike learning in SVMs, however, our framework requires no modification or extension for problems in multiway (as opposed to binary) classification.

4,433 citations

Journal Article•DOI•
Distance Metric Learning for Large Margin Nearest Neighbor Classification

[...]

Kilian Q. Weinberger, Lawrence K. Saul
01 Dec 2009-Journal of Machine Learning Research
TL;DR: This paper shows how to learn a Mahalanobis distance metric for kNN classification from labeled examples in a globally integrated manner and finds that metrics trained in this way lead to significant improvements in kNN Classification.
Abstract: The accuracy of k-nearest neighbor (kNN) classification depends significantly on the metric used to compute distances between different examples. In this paper, we show how to learn a Mahalanobis distance metric for kNN classification from labeled examples. The Mahalanobis metric can equivalently be viewed as a global linear transformation of the input space that precedes kNN classification using Euclidean distances. In our approach, the metric is trained with the goal that the k-nearest neighbors always belong to the same class while examples from different classes are separated by a large margin. As in support vector machines (SVMs), the margin criterion leads to a convex optimization based on the hinge loss. Unlike learning in SVMs, however, our approach requires no modification or extension for problems in multiway (as opposed to binary) classification. In our framework, the Mahalanobis distance metric is obtained as the solution to a semidefinite program. On several data sets of varying size and difficulty, we find that metrics trained in this way lead to significant improvements in kNN classification. Sometimes these results can be further improved by clustering the training examples and learning an individual metric within each cluster. We show how to learn and combine these local metrics in a globally integrated manner.

4,157 citations

"Image annotation using metric learn..." refers background or methods in this paper

  • ...With this goal, we perform metric learning over 2PKNN by generalizing the LMNN [11] algorithm for multi-label prediction....

    [...]

  • ...In such a scenario, (i) since each base distance contributes differently, we can learn appropriate weights to combine them in the distance space [2, 3]; and (ii) since every feature (such as SIFT or colour histogram) itself is represented as a multidimensional vector, its individual elements can also be weighted in the feature space [11]....

    [...]

  • ...Our extension of LMNN conceptually differs from its previous extensions such as [21] in at least two significant ways: (i) we adapt LMNN in its choice of target/impostors to learn metrics for multi-label prediction problems, whereas [21] uses the same definition of target/impostors as in LMNN to address classification problem in multi-task setting, and (ii) in our formulation, the amount of push applied on an impostor varies depending on its conceptual similarity w.r.t. a given sample, which makes it suitable for multi-label prediction tasks....

    [...]

  • ...Our metric learning framework extends LMNN in two major ways: (i) LMNN is meant for single-label classification (or simply classification) problems, while we adapt it for images annotation which is a multi-label classification task; and (ii) LMNN learns a single Mahalanobis metric in the feature space, while we extend it to learn linear metrics for multi- Image Annotation Using Metric Learning in Semantic Neighbourhoods 3 ple features as well as distances together....

    [...]

  • ...For this purpose, we extend the classical LMNN [11] algorithm for multi-label prediction....

    [...]

Proceedings Article•DOI•
Labeling images with a computer game

[...]

Luis von Ahn1, Laura Dabbish1•
Carnegie Mellon University1
25 Apr 2004
TL;DR: A new interactive system: a game that is fun and can be used to create valuable output that addresses the image-labeling problem and encourages people to do the work by taking advantage of their desire to be entertained.
Abstract: We introduce a new interactive system: a game that is fun and can be used to create valuable output. When people play the game they help determine the contents of images by providing meaningful labels for them. If the game is played as much as popular online games, we estimate that most images on the Web can be labeled in a few months. Having proper labels associated with each image on the Web would allow for more accurate image search, improve the accessibility of sites (by providing descriptions of images to visually impaired individuals), and help users block inappropriate images. Our system makes a significant contribution because of its valuable output and because of the way it addresses the image-labeling problem. Rather than using computer vision techniques, which don't work well enough, we encourage people to do the work by taking advantage of their desire to be entertained.

2,365 citations

"Image annotation using metric learn..." refers background in this paper

  • ...ESP Game contains images annotated using an on-line game, where two (mutually unknown) players are randomly given an image for which they have to predict same keyword(s) to score points [22]....

    [...]

Journal Article•DOI•
Pegasos: primal estimated sub-gradient solver for SVM

[...]

Shai Shalev-Shwartz1, Yoram Singer2, Nathan Srebro3, Andrew Cotter3•
Hebrew University of Jerusalem1, Google2, Toyota Technological Institute at Chicago3
01 Mar 2011-Mathematical Programming
TL;DR: A simple and effective stochastic sub-gradient descent algorithm for solving the optimization problem cast by Support Vector Machines, which is particularly well suited for large text classification problems, and demonstrates an order-of-magnitude speedup over previous SVM learning methods.
Abstract: We describe and analyze a simple and effective stochastic sub-gradient descent algorithm for solving the optimization problem cast by Support Vector Machines (SVM). We prove that the number of iterations required to obtain a solution of accuracy $${\epsilon}$$ is $${\tilde{O}(1 / \epsilon)}$$, where each iteration operates on a single training example. In contrast, previous analyses of stochastic gradient descent methods for SVMs require $${\Omega(1 / \epsilon^2)}$$ iterations. As in previously devised SVM solvers, the number of iterations also scales linearly with 1/λ, where λ is the regularization parameter of SVM. For a linear kernel, the total run-time of our method is $${\tilde{O}(d/(\lambda \epsilon))}$$, where d is a bound on the number of non-zero features in each example. Since the run-time does not depend directly on the size of the training set, the resulting algorithm is especially suited for learning from large datasets. Our approach also extends to non-linear kernels while working solely on the primal objective function, though in this case the runtime does depend linearly on the training set size. Our algorithm is particularly well suited for large text classification problems, where we demonstrate an order-of-magnitude speedup over previous SVM learning methods.

2,037 citations

"Image annotation using metric learn..." refers methods in this paper

  • ...To overcome this issue, we solve it by alternatively using stochastic sub-gradient descent and projection steps (similar to Pegasos [12])....

    [...]

  • ...To address this, we implement metric learning by alternating between stochastic sub-gradient descent and projection steps (similar to Pegasos [12])....

    [...]

Book Chapter•DOI•
Object Recognition as Machine Translation: Learning a Lexicon for a Fixed Image Vocabulary

[...]

Pinar Duygulu1, Kobus Barnard1, J. F. G. de Freitas2, David Forsyth1•
University of California, Berkeley1, University of British Columbia2
28 May 2002
TL;DR: This work shows how to cluster words that individually are difficult to predict into clusters that can be predicted well, and cannot predict the distinction between train and locomotive using the current set of features, but can predict the underlying concept.
Abstract: We describe a model of object recognition as machine translation. In this model, recognition is a process of annotating image regions with words. Firstly, images are segmented into regions, which are classified into region types using a variety of features. A mapping between region types and keywords supplied with the images, is then learned, using a method based around EM. This process is analogous with learning a lexicon from an aligned bitext. For the implementation we describe, these words are nouns taken from a large vocabulary. On a large test set, the method can predict numerous words with high accuracy. Simple methods identify words that cannot be predicted well. We show how to cluster words that individually are difficult to predict into clusters that can be predicted well -- for example, we cannot predict the distinction between train and locomotive using the current set of features, but we can predict the underlying concept. The method is trained on a substantial collection of images. Extensive experimental results illustrate the strengths and weaknesses of the approach.

1,765 citations

"Image annotation using metric learn..." refers background in this paper

  • ...translation models [13, 14] and nearest-neighbour based relevance models [1, 8]....

    [...]

  • ...Corel 5K was first used in [14], and since then it has become a benchmark for comparing annotation performance....

    [...]

Related Papers (5)
TagProp: Discriminative metric learning in nearest neighbor models for image auto-annotation

[...]

01 Sep 2009
Matthieu Guillaumin, Thomas Mensink, Jakob Verbeek, Cordelia Schmid 
Multiple Bernoulli relevance models for image and video annotation

[...]

27 Jun 2004
Shaolei Feng, R. Manmatha, Victor Lavrenko
Object Recognition as Machine Translation: Learning a Lexicon for a Fixed Image Vocabulary

[...]

28 May 2002
Pinar Duygulu, Kobus Barnard, J. F. G. de Freitas, David Forsyth 
A Model for Learning the Semantics of Pictures

[...]

09 Dec 2003
Victor Lavrenko, R. Manmatha, Jiwoon Jeon
Supervised Learning of Semantic Classes for Image Annotation and Retrieval

[...]

01 Mar 2007-IEEE Transactions on Pattern Analysis and Machine Intelligence
Gustavo Carneiro, Antoni B. Chan, Pedro J. Moreno, Nuno Vasconcelos