Annotating the increasing amounts of user-contributed images in a personalized manner is in great demand. However, this demand is largely ignored by the mainstream of automated image annotation research. In this paper we aim for personalizing automated image annotation by jointly exploiting personalized tag statistics and content-based image annotation. We propose a cross-entropy based learning algorithm which personalizes a generic annotation model by learning from a user's multimedia tagging history. Using cross-entropy-minimization based Monte Carlo sampling, the proposed algorithm optimizes the personalization process in terms of a performance measurement which can be flexibly chosen. Automatic image annotation experiments with 5,315 realistic users in the social web show that the proposed method compares favorably to a generic image annotation method and a method using personalized tag statistics only. For 4,442 users the performance improves, where for 1,088 users the absolute performance gain is at least 0.05 in terms of average precision. The results show the value of the proposed method.

Personalizing automated image annotation using cross-entropy

Large Vision&Language models pretrained on web-scale data provide representations that are invaluable for numerous V&L problems. However, it is unclear how they can be used for reasoning about user-specific visual concepts in unstructured language. This problem arises in multiple domains, from personalized image retrieval to personalized interaction with smart devices. We introduce a new learning setup called Personalized Vision&Language (PerVL) with two new benchmark datasets for retrieving and segmenting user-specific"personalized"concepts"in the wild". In PerVL, one should learn personalized concepts (1) independently of the downstream task (2) allowing a pretrained model to reason about them with free language, and (3) does not require personalized negative examples. We propose an architecture for solving PerVL that operates by extending the input vocabulary of a pretrained model with new word embeddings for the new personalized concepts. The model can then reason about them by simply using them in a sentence. We demonstrate that our approach learns personalized visual concepts from a few examples and can effectively apply them in image retrieval and semantic segmentation using rich textual queries.

"This is my unicorn, Fluffy": Personalizing frozen vision-language representations

Thanks to the evolution of technology, we find a very large number of internet users who use social networks to react and share things with each other. These networks are exploited in the study of several domains. In fact, an Internet user can easily share his images with a simple click on his/her Smartphone. However, we get a large amount of published images on the Internet. Such an amount requires specific access techniques to be used by the search engines to provide searchers with the desired results. An effective way to access the target images is their keywords (or tags). Nevertheless, tags, which people manually attach to images, are of low quality and negatively affect the search engines. Consequently, automatic annotation of images by tags has become an active topic of research in recent years. In this paper, we introduce an automatic annotation method in social networks named MDL-STag (Multimodal features Deep Learning approach for Social image Tagging). This method provides high-quality features using the visual content of the image as well as the textual content of the annotation tag history to personalize the annotation, and that’s with the help of some deep learning models. Then, it merges these features and makes multimodal features for the images of the annotator’s contacts who share the same interests to provide more useful tags through the propagation of images tags that are similar to the target image. In fact, we find that tests give good results on real social networks as the well known Instagram and Flickr.

Image annotation in social networks using graph and multimodal deep learning features

Large Vision & Language models pretrained on web-scale data provide representations that are invaluable for numerous V &L problems. However, it is unclear how they can be extended to reason about user-specific visual concepts in unstructured language. This problem arises in multiple domains, from personalized image retrieval to personalized interaction with smart devices. We introduce a new learning setup called Personalized Vision & Language (PerVL) with two new benchmark datasets for retrieving and segmenting user-specific (“personalized”) concepts “in the wild”. In PerVL, one should learn personalized concepts (1) independently of the downstream task (2) allowing a pretrained model to reason about them with free language, and (3) without providing personalized negative examples. We propose an architecture for solving PerVL that operates by expanding the input vocabulary of a pretrained model with new word embeddings for the personalized concepts. The model can then simply employ them as part of a sentence. We demonstrate that our approach learns personalized visual concepts from a few examples and effectively applies them in image retrieval and semantic segmentation using rich textual queries. For example the model improves MRR by 51.1% (28.4% vs 18.8%) compared to the strongest baseline. The code and benchmark are available on github under NVlabs/PALAVRA ( https://github.com/NVlabs/PALAVRA ) and NVlabs/PerVLBenchmark ( https://github.com/NVlabs/PerVLBenchmark ). 

/pdf/this-is-my-unicorn-fluffy-personalizing-frozen-vision-1f6h00xc.pdf

“This Is My Unicorn, Fluffy”: Personalizing Frozen Vision-Language Representations

Automatic image annotation (AIA) has been adopted in different applications such as image retrieval and classification. Deep Learning is used in AIA to extract image features and then convert these features into text descriptions and labels. However, conventional AIA models that employ deep learning methods suffer from various shortcomings, such as poor annotation performance. This work proposes an AIA model based on convolutional neural networks (CNNs), generative adversarial networks (GANs), and transfer learning. GANs have attracted a lot of interest because of its ability to generate data without explicitly using probability density. Thus, it has proven its usefulness in image annotation and image augmentation. In this work, an Auxiliary classifier-GAN (ACGAN) has been used, where the discriminator predicts the class of an image rather than taking it as a given input; therefore, the stabilization of the training stage is ensured, and the generation of high-quality images is provided. Transfer learning is also used to enhance the performance of the classification. The proposed model outperforms the best state-of-the-art models in terms of MiAP, F-measure and error rate using ImageClef, ESPGame and IAPR-TC12 datasets.

https://ieeexplore.ieee.org/ielx7/6287639/6514899/09749241.pdf

DeepAIA: An Automatic Image Annotation Model Based on Generative Adversarial Networks and Transfer Learning

In image-centric social networks, such as Instagram and Pinterest, users tend to share photos with several tags. These tags describe the content of the image or provide additional contextual information, and therefore may not be necessarily tied to image content and usually carry personal preference. Annotating images in social networks in a personalized manner is in demand. However, the existing image annotation models, which rely only on image content information, cannot capture the user’s tagging preference. In this paper, we propose a deep architecture for personalized image annotation by leveraging the wealth of information in user’s tagging history. The proposed architecture consists of three components: two components for learning features of the image content and user’s history tags and the other one for combining the two learned features to predict the tags. We also explore two ways to model user’s history tags: 1) simply average the embeddings of user’s history tags and 2) model user’s history tags with a sequence model by long short-term memory recurrent neural network. We evaluate our proposed deep architecture on a large-scale and realistic data set, consisting of ~22.8 million public images uploaded by ~4.69 million users. Experimental results show that our proposed deep architecture is effective on a personalized image annotation task.

Song Bi

Papers

Personalized Image Annotation Using Deep Architecture