Combining Multiple Cues for Visual Madlibs Question Answering
Tatiana Tommasi,Arun Mallya,Bryan A. Plummer,Svetlana Lazebnik,Alexander C. Berg,Tamara L. Berg +5 more
TLDR
In this paper, the authors present an approach for answering fill-in-the-blank multiple choice questions from the Visual Madlibs dataset, which employs a combination of networks trained for specialized tasks such as scene recognition, person activity classification and attribute prediction.Abstract:
This paper presents an approach for answering fill-in-the-blank multiple choice questions from the Visual Madlibs dataset. Instead of generic and commonly used representations trained on the ImageNet classification task, our approach employs a combination of networks trained for specialized tasks such as scene recognition, person activity classification, and attribute prediction. We also present a method for localizing phrases from candidate answers in order to provide spatial support for feature extraction. We map each of these features, together with candidate answers, to a joint embedding space through normalized canonical correlation analysis (nCCA). Finally, we solve an optimization problem to learn to combine scores from nCCA models trained on multiple cues to select the best answer. Extensive experimental results show a significant improvement over the previous state of the art and confirm that answering questions from a wide range of types benefits from examining a variety of image cues and carefully choosing the spatial support for feature extraction.read more
Citations
More filters
Journal ArticleDOI
Visual question answering: a state-of-the-art review
Sruthy Manmadhan,Binsu C. Kovoor +1 more
TL;DR: This review extensively and critically examines the current status of VQA research in terms of step by step solution methodologies, datasets and evaluation metrics and discusses future research directions for all the above-mentioned aspects of V QA separately.
Proceedings ArticleDOI
VQD: Visual Query Detection In Natural Scenes
TL;DR: In this paper, a new visual grounding task called Visual Query Detection (VQD) is proposed, where the task is to localize a variable number of objects in an image where the objects are specified in natural language.
Posted Content
VQD: Visual Query Detection in Natural Scenes
TL;DR: The first algorithms for VQD are proposed, and they are evaluated on both visual referring expression datasets and the authors' new V QDv1 dataset.
Journal ArticleDOI
A survey of methods, datasets and evaluation metrics for visual question answering
TL;DR: This paper has discussed some of the core concepts used in VQA systems and presented a comprehensive survey of efforts in the past to address this problem, and discussed some new datasets developed in 2019 and 2020.
Journal ArticleDOI
New ideas and trends in deep multimodal content understanding : a review
TL;DR: A survey of multimodal deep learning can be found in this paper, where the authors examine recent multimodAL deep models and structures, including auto-encoders, generative adversarial nets and their variants.
References
More filters
Proceedings ArticleDOI
Deep Residual Learning for Image Recognition
TL;DR: In this article, the authors proposed a residual learning framework to ease the training of networks that are substantially deeper than those used previously, which won the 1st place on the ILSVRC 2015 classification task.
Proceedings Article
Very Deep Convolutional Networks for Large-Scale Image Recognition
Karen Simonyan,Andrew Zisserman +1 more
TL;DR: This work investigates the effect of the convolutional network depth on its accuracy in the large-scale image recognition setting using an architecture with very small convolution filters, which shows that a significant improvement on the prior-art configurations can be achieved by pushing the depth to 16-19 weight layers.
Proceedings ArticleDOI
Going deeper with convolutions
Christian Szegedy,Wei Liu,Yangqing Jia,Pierre Sermanet,Scott Reed,Dragomir Anguelov,Dumitru Erhan,Vincent Vanhoucke,Andrew Rabinovich +8 more
TL;DR: Inception as mentioned in this paper is a deep convolutional neural network architecture that achieves the new state of the art for classification and detection in the ImageNet Large-Scale Visual Recognition Challenge 2014 (ILSVRC14).
Journal ArticleDOI
ImageNet Large Scale Visual Recognition Challenge
Olga Russakovsky,Jia Deng,Hao Su,Jonathan Krause,Sanjeev Satheesh,Sean Ma,Zhiheng Huang,Andrej Karpathy,Aditya Khosla,Michael S. Bernstein,Alexander C. Berg,Li Fei-Fei +11 more
TL;DR: The ImageNet Large Scale Visual Recognition Challenge (ILSVRC) as mentioned in this paper is a benchmark in object category classification and detection on hundreds of object categories and millions of images, which has been run annually from 2010 to present, attracting participation from more than fifty institutions.
Book ChapterDOI
Microsoft COCO: Common Objects in Context
Tsung-Yi Lin,Michael Maire,Serge Belongie,James Hays,Pietro Perona,Deva Ramanan,Piotr Dollár,C. Lawrence Zitnick +7 more
TL;DR: A new dataset with the goal of advancing the state-of-the-art in object recognition by placing the question of object recognition in the context of the broader question of scene understanding by gathering images of complex everyday scenes containing common objects in their natural context.