Annotation

About: Annotation is a research topic. Over the lifetime, 6719 publications have been published within this topic receiving 203463 citations. The topic is also known as: note & markup.

Lean Crowdsourcing: Combining Humans and Machines in an Online System

Abstract: We introduce a method to greatly reduce the amount of redundant annotations required when crowdsourcing annotations such as bounding boxes, parts, and class labels. For example, if two Mechanical Turkers happen to click on the same pixel location when annotating a part in a given image–an event that is very unlikely to occur by random chance–, it is a strong indication that the location is correct. A similar type of confidence can be obtained if a single Turker happened to agree with a computer vision estimate. We thus incrementally collect a variable number of worker annotations per image based on online estimates of confidence. This is done using a sequential estimation of risk over a probabilistic model that combines worker skill, image difficulty, and an incrementally trained computer vision model. We develop specialized models and algorithms for binary annotation, part keypoint annotation, and sets of bounding box annotations. We show that our method can reduce annotation time by a factor of 4-11 for binary filtering of websearch results, 2-4 for annotation of boxes of pedestrians in images, while in many cases also reducing annotation error. We will make an end-to-end version of our system publicly available.

MyMiner: a web application for computer-assisted biocuration and text annotation

Abstract: Motivation: The exponential growth of scientific literature has resulted in a massive amount of unstructured natural language data that cannot be directly handled by means of bioinformatics tools. Such tools generally require structured data, often generated through a cumbersome process of manual literature curation. Here, we present MyMiner, a free and user-friendly text annotation tool aimed to assist in carrying out the main biocuration tasks and to provide labelled data for the development of text mining systems. MyMiner allows easy classification and labelling of textual data according to user-specified classes as well as predefined biological entities. The usefulness and efficiency of this application has been tested for a range of real-life annotation scenarios of various research topics. Availability: http://myminer.armi.monash.edu.au. Supplementary information: Supplementary data are available at Bioinformatics online.

Game-powered machine learning

Abstract: Searching for relevant content in a massive amount of multimedia information is facilitated by accurately annotating each image, video, or song with a large number of relevant semantic keywords, or tags We introduce game-powered machine learning, an integrated approach to annotating multimedia content that combines the effectiveness of human computation, through online games, with the scalability of machine learning We investigate this framework for labeling music First, a socially-oriented music annotation game called Herd It collects reliable music annotations based on the “wisdom of the crowds” Second, these annotated examples are used to train a supervised machine learning system Third, the machine learning system actively directs the annotation games to collect new data that will most benefit future model iterations Once trained, the system can automatically annotate a corpus of music much larger than what could be labeled using human computation alone Automatically annotated songs can be retrieved based on their semantic relevance to text-based queries (eg, “funky jazz with saxophone,” “spooky electronica,” etc) Based on the results presented in this paper, we find that actively coupling annotation games with machine learning provides a reliable and scalable approach to making searchable massive amounts of multimedia data

Information extraction methods and apparatus including a computer-user interface

Abstract: Disclosed is an information extraction system and method. The method comprises receiving a document and annotation data, the annotation data comprising instances of entities which have been identified in the document, the annotation entity data comprising identifiers of instances of one or more entities which have been identified in the document and data specifying the location of the identified instances of entities within the document, wherein the identifiers of instances of entities comprise references to ontology data; displaying the document to a user, with annotations dependent on the annotation data, highlighting one or more of the instances of entities whose location is specified in the annotation entity data at the location within the document specified by the annotation entity data; preparing revised annotation data from a user and outputting output data derived from the amended annotation data. The output data is typically used to populate a database.

Interactive multimedia annotations: enriching and extending content

Abstract: This paper discusses an approach to the problem of annotating multimedia content. Our approach provides annotation as metadata for indexing retrieval and semantic processing as well as content enrichment. We use an underlying model for structured multimedia descriptions and annotations allowing the establishment of spatial temporal and linking relationships. We discuss aspects related with documents and annotations used to guide the design of an application that allows annotations to be made with pen-based interaction with Tablet PCs. As a result a video stream can be annotated during the capture. The annotation can be further edited extended or played back synchronously.