In this paper we propose to use lexical semantic networks to extend the state-of-the-art object recognition techniques. We use the semantics of image labels to integrate prior knowledge about inter-class relationships into the visual appearance learning. We show how to build and train a semantic hierarchy of discriminative classifiers and how to use it to perform object detection. We evaluate how our approach influences the classification accuracy and speed on the Pascal VOC challenge 2006 dataset, a set of challenging real-world images. We also demonstrate additional features that become available to object recognition due to the extension with semantic inference tools- we can classify high-level categories, such as animals, and we can train part detectors, for example a window detector, by pure inference in the semantic network.

/pdf/semantic-hierarchies-for-visual-object-recognition-thtx0sx5mj.pdf

Semantic Hierarchies for Visual Object Recognition

Learning visual classifiers for object recognition from weakly labeled data requires determining correspondence between image regions and semantic object classes. Most approaches use co-occurrence of "nouns" and image features over large datasets to determine the correspondence, but many correspondence ambiguities remain. We further constrain the correspondence problem by exploiting additional language constructs to improve the learning process from weakly labeled data. We consider both "prepositions" and "comparative adjectives" which are used to express relationships between objects. If the models of such relationships can be determined, they help resolve correspondence ambiguities. However, learning models of these relationships requires solving the correspondence problem. We simultaneously learn the visual features defining "nouns" and the differential visual features defining such "binary-relationships" using an EM-based approach.

/pdf/beyond-nouns-exploiting-prepositions-and-comparative-24w1ul7nu8.pdf

Beyond Nouns: Exploiting Prepositions and Comparative Adjectives for Learning Visual Classifiers

Geographic Object-Based Image Analysis (GEOBIA) represents the most innovative new trend for processing remote sensing images that has appeared during the last decade. However, its application is mainly based on expert knowledge, which consequently highlights important scientific issues with respect to the robustness of the methods applied in GEOBIA. In this paper, we argue that GEOBIA would benefit from another technical enhancement involving knowledge representation techniques such as ontologies. Although the role of ontologies in Geographical Information Sciences (GISciences) is not a new topic, few works have discussed how ontologies, considered from the perspective of a remote sensing specialist, can contribute to advancing remote sensing science. We summarize the main applications of ontologies in GEOBIA, especially for data discovery, automatic image interpretation, data interoperability, workflow management and data publication. Finally, we discuss the major issues related to the construction of ontologies suitable for remote sensing applications and outline long-term future advances that can be expected for the remote sensing community.

Advances in Geographic Object-Based Image Analysis with ontologies: A review of main contributions and limitations from a remote sensing perspective

http://certis.enpc.fr/~audibert/Mes%20articles/AMT10.pdf

Semantic hierarchies for image annotation: A survey

An approach using many intermediate semantic concepts is proposed with the potential to bridge the semantic gap between what a color, shape, and texture-based ldquolow-levelrdquo image analysis can extract from video and what users really want to find, most likely using text descriptions of their information needs. Semantic concepts such as cars, planes, roads, people, animals, and different types of scenes (outdoor, night time, etc.) can be automatically detected in the video with reasonable accuracy. This leads us to ask how can they be used automatically and how does a user (or a retrieval system) translate the user's information need into a selection of related concepts that would help find the relevant video clips, from the large list of available concepts. We illustrate how semantic concept retrieval can be automatically exploited by mapping queries into query classes and through pseudo-relevance feedback. We also provide evidence how a semantic concept can be utilized by users in interactive retrieval, through interfaces that provide affordances of explicit concept selection and search, concept filtering, and relevance feedback. How many concepts we actually need and how accurately they need to be detected and linked through various relationships is specified in the ontology structure.

Video Retrieval Based on Semantic Concepts

Automatic image annotation is the task of automatically assigning words to an image that describe the content of the image. Machine learning approaches have been explored to model the association between words and images from an annotated set of images and generate annotations for a test image. The paper proposes methods to use a hierarchy defined on the annotation words derived from a text ontology to improve automatic image annotation and retrieval. Specifically, the hierarchy is used in the context of generating a visual vocabulary for representing images and as a framework for the proposed hierarchical classification approach for automatic image annotation. The effect of using the hierarchy in generating the visual vocabulary is demonstrated by improvements in the annotation performance of translation models. In addition to performance improvements, hierarchical classification approaches yield well to constructing multimedia ontologies.

Joshua Varner

Papers

Exploiting ontologies for automatic image annotation