We propose the task of free-form and open-ended Visual Question Answering (VQA). Given an image and a natural language question about the image, the task is to provide an accurate natural language answer. Mirroring real-world scenarios, such as helping the visually impaired, both the questions and answers are open-ended. Visual questions selectively target different areas of an image, including background details and underlying context. As a result, a system that succeeds at VQA typically needs a more detailed understanding of the image and complex reasoning than a system producing generic image captions. Moreover, VQA is amenable to automatic evaluation, since many open-ended answers contain only a few words or a closed set of answers that can be provided in a multiple-choice format. We provide a dataset containing ~0.25M images, ~0.76M questions, and ~10M answers (www.visualqa.org), and discuss the information it provides. Numerous baselines for VQA are provided and compared with human performance.

/pdf/vqa-visual-question-answering-3vu8bkyw7k.pdf

VQA: Visual Question Answering

We introduce a system for sensing complex social systems with data collected from 100 mobile phones over the course of 9 months. We demonstrate the ability to use standard Bluetooth-enabled mobile telephones to measure information access and use in different contexts, recognize social patterns in daily user activity, infer relationships, identify socially significant locations, and model organizational rhythms.

/pdf/reality-mining-sensing-complex-social-systems-2gf3rdnw3x.pdf

Reality mining: sensing complex social systems

We propose the task of free-form and open-ended Visual Question Answering (VQA). Given an image and a natural language question about the image, the task is to provide an accurate natural language answer. Mirroring real-world scenarios, such as helping the visually impaired, both the questions and answers are open-ended. Visual questions selectively target different areas of an image, including background details and underlying context. As a result, a system that succeeds at VQA typically needs a more detailed understanding of the image and complex reasoning than a system producing generic image captions. Moreover, VQA is amenable to automatic evaluation, since many open-ended answers contain only a few words or a closed set of answers that can be provided in a multiple-choice format. We provide a dataset containing ~0.25M images, ~0.76M questions, and ~10M answers (this http URL), and discuss the information it provides. Numerous baselines and methods for VQA are provided and compared with human performance. Our VQA demo is available on CloudCV (this http URL).

Human linguistic annotation is crucial for many natural language processing tasks but can be expensive and time-consuming. We explore the use of Amazon's Mechanical Turk system, a significantly cheaper and faster method for collecting annotations from a broad base of paid non-expert contributors over the Web. We investigate five tasks: affect recognition, word similarity, recognizing textual entailment, event temporal ordering, and word sense disambiguation. For all five, we show high agreement between Mechanical Turk non-expert annotations and existing gold standard labels provided by expert labelers. For the task of affect recognition, we also show that using non-expert labels for training machine learning algorithms can be as effective as using gold standard annotations from experts. We propose a technique for bias correction that significantly improves annotation quality on two tasks. We conclude that many large labeling tasks can be effectively designed and carried out in this method at a fraction of the usual expense.

/pdf/cheap-and-fast-but-is-it-good-evaluating-non-expert-4kvurc68of.pdf

Cheap and Fast -- But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks

ConceptNet is a freely available commonsense knowledge base and natural-language-processing tool-kit which supports many practical textual-reasoning tasks over real-world documents including topic-gisting, analogy-making, and other context oriented inferences. The knowledge base is a semantic network presently consisting of over 1.6 million assertions of commonsense knowledge encompassing the spatial, physical, social, temporal, and psychological aspects of everyday life. ConceptNet is generated automatically from the 700 000 sentences of the Open Mind Common Sense Project — a World Wide Web based collaboration with over 14 000 authors.

/pdf/conceptnet-a-practical-commonsense-reasoning-tool-kit-3723gsymn1.pdf

ConceptNet — A Practical Commonsense Reasoning Tool-Kit

Open Mind Common Sense is a knowledge acquisition system designed to acquire commonsense knowledge from the general public over the web. We describe and evaluate our first fielded system, which enabled the construction of a 450,000 assertion commonsense knowledge base. We then discuss how our second-generation system addresses weaknesses discovered in the first. The new system acquires facts, descriptions, and stories by allowing participants to construct and fill in natural language templates. It employs word-sense disambiguation and methods of clarifying entered knowledge, analogical inference to provide feedback, and allows participants to validate knowledge and in turn each other.

/pdf/open-mind-common-sense-knowledge-acquisition-from-the-1zssg2ka8k.pdf

Open Mind Common Sense: Knowledge Acquisition from the General Public

The Open Mind Common Sense project is an attempt to construct a database of commonsense knowledge through the collaboration of a distributed community of thousands of non-expert netizens. We give an overview of the project, describe our knowledge acquisition and representation strategy of using natural language rather than formal logic, and demonstrate this strategy with a search engine application that employs simple commonsense reasoning to reformulate problem queries into more effective solution queries.

/pdf/the-public-acquisition-of-commonsense-knowledge-lnvpeiz2cp.pdf

The Public Acquisition of Commonsense Knowledge

ConceptNet is a very large semantic network of commonsense knowledge suitable for making various kinds of practical inferences over text. ConceptNet captures a wide range of commonsense concepts and relations like those in Cyc, while its simple semantic network structure lends it an ease-of-use comparable to WordNet. To meet the dual challenge of having to encode complex higher-order concepts, and maintaining ease-of-use, we introduce a novel use of semi-structured natural language fragments as the knowledge representation of commonsense concepts. In this paper, we present a methodology for reasoning flexibly about these semi-structured natural language fragments. We also examine the tradeoffs associated with representing commonsense knowledge in formal logic versus in natural language. We conclude that the flexibility of natural language makes it a highly suitable representation for achieving practical inferences over text, such as context finding, inference chaining, and conceptual analogy.

/pdf/commonsense-reasoning-in-and-over-natural-language-22fi2jbf56.pdf

Commonsense Reasoning in and Over Natural Language

The paper discusses Aria, an interface agent designed to assist users by preactively looking for opportunities for image annotation and retrieval. While it does not completely automate the image annotation and retrieval process, Aria dramatically reduces user interface overhead, which can lead to better annotated image libraries and fewer missed opportunities for image use.

Push Singh

Papers

ConceptNet — A Practical Commonsense Reasoning Tool-Kit

Open Mind Common Sense: Knowledge Acquisition from the General Public

The Public Acquisition of Commonsense Knowledge

Commonsense Reasoning in and Over Natural Language

Aria: an agent for annotating and retrieving images