The explosion of image data on the Internet has the potential to foster more sophisticated and robust models and algorithms to index, retrieve, organize and interact with images and multimedia data. But exactly how such data can be harnessed and organized remains a critical problem. We introduce here a new database called “ImageNet”, a large-scale ontology of images built upon the backbone of the WordNet structure. ImageNet aims to populate the majority of the 80,000 synsets of WordNet with an average of 500-1000 clean and full resolution images. This will result in tens of millions of annotated images organized by the semantic hierarchy of WordNet. This paper offers a detailed analysis of ImageNet in its current state: 12 subtrees with 5247 synsets and 3.2 million images in total. We show that ImageNet is much larger in scale and diversity and much more accurate than the current image datasets. Constructing such a large-scale database is a challenging task. We describe the data collection scheme with Amazon Mechanical Turk. Lastly, we illustrate the usefulness of ImageNet through three simple applications in object recognition, image classification and automatic object clustering. We hope that the scale, accuracy, diversity and hierarchical structure of ImageNet can offer unparalleled opportunities to researchers in the computer vision community and beyond.

/pdf/imagenet-a-large-scale-hierarchical-image-database-1e8f663s1h.pdf

ImageNet: A large-scale hierarchical image database

This paper presents a method for extracting distinctive invariant features from images that can be used to perform reliable matching between different views of an object or scene. The features are invariant to image scale and rotation, and are shown to provide robust matching across a substantial range of affine distortion, change in 3D viewpoint, addition of noise, and change in illumination. The features are highly distinctive, in the sense that a single feature can be correctly matched with high probability against a large database of features from many images. This paper also describes an approach to using these features for object recognition. The recognition proceeds by matching individual features to a database of features from known objects using a fast nearest-neighbor algorithm, followed by a Hough transform to identify clusters belonging to a single object, and finally performing verification through least-squares solution for consistent pose parameters. This approach to recognition can robustly identify objects among clutter and occlusion while achieving near real-time performance.

/pdf/distinctive-image-features-from-scale-invariant-keypoints-3waurjqzke.pdf

Distinctive Image Features from Scale-Invariant Keypoints

Fiji is a distribution of the popular open-source software ImageJ focused on biological-image analysis. Fiji uses modern software engineering practices to combine powerful software libraries with a broad range of scripting languages to enable rapid prototyping of image-processing algorithms. Fiji facilitates the transformation of new algorithms into ImageJ plugins that can be shared with end users through an integrated update system. We propose Fiji as a platform for productive collaboration between computer science and biology research communities.

/pdf/fiji-an-open-source-platform-for-biological-image-analysis-1v529iu3cx.pdf

Fiji: an open-source platform for biological-image analysis

We study the question of feature sets for robust visual object recognition; adopting linear SVM based human detection as a test case. After reviewing existing edge and gradient based descriptors, we show experimentally that grids of histograms of oriented gradient (HOG) descriptors significantly outperform existing feature sets for human detection. We study the influence of each stage of the computation on performance, concluding that fine-scale gradients, fine orientation binning, relatively coarse spatial binning, and high-quality local contrast normalization in overlapping descriptor blocks are all important for good results. The new approach gives near-perfect separation on the original MIT pedestrian database, so we introduce a more challenging dataset containing over 1800 annotated human images with a large range of pose variations and backgrounds.

/pdf/histograms-of-oriented-gradients-for-human-detection-3nteinziuq.pdf

Histograms of oriented gradients for human detection

The ImageNet Large Scale Visual Recognition Challenge is a benchmark in object category classification and detection on hundreds of object categories and millions of images. The challenge has been run annually from 2010 to present, attracting participation from more than fifty institutions. This paper describes the creation of this benchmark dataset and the advances in object recognition that have been possible as a result. We discuss the challenges of collecting large-scale ground truth annotation, highlight key breakthroughs in categorical object recognition, provide a detailed analysis of the current state of the field of large-scale image classification and object detection, and compare the state-of-the-art computer vision accuracy with human accuracy. We conclude with lessons learned in the 5 years of the challenge, and propose future directions and improvements.

/pdf/imagenet-large-scale-visual-recognition-challenge-caky5nmmix.pdf

ImageNet Large Scale Visual Recognition Challenge

Apparatuses and methods for capturing and generating composite images. In various aspects, the invention provides apparatuses and methods for correcting position information of captured images received by position sensors based on alignment of overlapping images. Corrected position information is then taken into account when displaying the locations of captured images on a display for providing guidance to users for generating composite images.

Methods and apparatus for generating composite images

Iterative alignment is one method for feature-based matching of an image and a model for the purpose of object recognition. The method alternately hypothesizes feature pairings and estimates a viewpoint transformation from those pairings; at each stage a refined transformation estimate is used to suggest additional pairings. This paper extends iterative alignment in the domain of 2D similarity transformations so that it represents the uncertainty in the position of each model and image feature, and that of the transformation estimate. A model describes probabilistically the significance, position, and intrinsic attributes of each feature, plus topological relations among features. A measure of the match between a model and an image integrates all four of these, and leads to an efficient matching procedure called probabilistic alignment. That procedure supports both recognition and a learning procedure for acquiring models from training images. By explicitly representing uncertainty, one model can satisfactorily describe appearance over a wider range of viewing conditions. Thus, when models represent 2D characteristic views of a 3D object, fewer models are needed. Experiments demonstrating the effectiveness of this approach are reported.

Modeling Positional Uncertainty in Object Recognition

This paper introduces the idea of temporally extending the results of a stereo algorithm in order to improve the algorithm's performance This approach anticipates the changes between two consecutive depth maps resulting from the motion of the cameras Uncertainties in motion are accounted for by computation of an ambiguity area and a resulting disparity range for each pixel The computation is used to verify and refine the anticipated values, rather than calculate them without prior knowledge The paper compares the performance of the algorithm under different constraints on motion Speedups of up to 400% are achieved without significant errors

Temporally coherent stereo: improving performance through knowledge of motion

To recognize an object in an image one must have some internal model of how that object may appear. We show how to learn such a model from a series of training images depicting a class of objects. The model represents a 3D object by a set of characteristic views, each defining a probability distribution over variation in object appearance. Features identified in an image through perceptual organization are represented by a graph whose nodes include feature labels and numeric measurements. Image graphs are partitioned into characteristic views by an incremental conceptual clustering algorithm. A learning procedure generalizes multiple image graphs to form a characteristic view graph in which the numeric measurements are described by probability distributions. A matching procedure, using a similarity metric based on a non‐ parametric probability density estimator, compares image and characteristic view graphs to identify an instance of a modeled object in an image. We present experimental results from a system constructed to test this approach. The system is demonstrated learning to recognize partially occluded objects in images using shape cues.

/pdf/learning-3d-object-recognition-models-from-2d-images-5eniznc81f.pdf

Learning 3D Object Recognition Models from 2D Images

Thanks to large-scale image repositories, vast amounts of data for object recognition are now easily available. However, acquiring training labels for arbitrary objects still requires tedious and expensive human effort. This is particularly true for localization, where humans must not only provide labels, but also training windows in an image. We present an approach for reducing the number of labelled training instances required to train an object classifier and for assisting the user in specifying optimal object location windows. As part of this process, the algorithm performs localization to find bounding windows for training examples that are best aligned with the current classification function, which optimizes learning and reduces human effort. To test this approach, we introduce an active learning extension to a latent SVM learning algorithm. Our user interface for training object detectors employs real-time interaction with a human user. Our active learning system provides a mean performance improvement of 4.5% in the average precision over a state of the art detector on the PASCAL Visual Object Classes Challenge 2007 with an average of just 40 minutes of human labelling effort per class.

/pdf/an-adaptive-interface-for-active-localization-xqp4wh8r62.pdf

David G. Lowe

Papers

Methods and apparatus for generating composite images

Modeling Positional Uncertainty in Object Recognition

Temporally coherent stereo: improving performance through knowledge of motion

Learning 3D Object Recognition Models from 2D Images

An adaptive interface for active localization