We have witnessed great interest and a wealth of promise in content-based image retrieval as an emerging technology. While the last decade laid foundation to such promise, it also paved the way for a large number of new techniques and systems, got many new people involved, and triggered stronger association of weakly related fields. In this article, we survey almost 300 key theoretical and empirical contributions in the current decade related to image retrieval and automatic image annotation, and in the process discuss the spawning of related subfields. We also discuss significant challenges involved in the adaptation of existing image retrieval techniques to build systems that can be useful in the real world. In retrospect of what has been achieved so far, we also conjecture what the future may hold for image retrieval research.

/pdf/image-retrieval-ideas-influences-and-trends-of-the-new-age-51whw4w7su.pdf

Image retrieval: Ideas, influences, and trends of the new age

IEEE Transactions on Pattern Analysis and Machine Intelligence

Computer and Robot Vision Vol. 1, by R.M. Haralick and Linda G. Shapiro, Addison-Wesley, 1992, ISBN 0-201-10887-1.

Computer and Robot Vision

While research on articulated human motion and pose estimation has progressed rapidly in the last few years, there has been no systematic quantitative evaluation of competing methods to establish the current state of the art. We present data obtained using a hardware system that is able to capture synchronized video and ground-truth 3D motion. The resulting HumanEva datasets contain multiple subjects performing a set of predefined actions with a number of repetitions. On the order of 40,000 frames of synchronized motion capture and multi-view video (resulting in over one quarter million image frames in total) were collected at 60 Hz with an additional 37,000 time instants of pure motion capture data. A standard set of error measures is defined for evaluating both 2D and 3D pose estimation and tracking algorithms. We also describe a baseline algorithm for 3D articulated tracking that uses a relatively standard Bayesian framework with optimization in the form of Sequential Importance Resampling and Annealed Particle Filtering. In the context of this baseline algorithm we explore a variety of likelihood functions, prior models of human motion and the effects of algorithm parameters. Our experiments suggest that image observation models and motion priors play important roles in performance, and that in a multi-view laboratory environment, where initialization is available, Bayesian filtering tends to perform well. The datasets and the software are made available to the research community. This infrastructure will support the development of new articulated motion and pose estimation algorithms, will provide a baseline for the evaluation and comparison of new methods, and will help establish the current state of the art in human pose estimation and tracking.

/pdf/humaneva-synchronized-video-and-motion-capture-dataset-and-4dx3ly40om.pdf

HumanEva: Synchronized Video and Motion Capture Dataset and Baseline Algorithm for Evaluation of Articulated Human Motion

In this paper the main problems and the available solutions are addressed for the generation of 3D models from terrestrial images. Close range photogrammetry has dealt for many years with manual or automatic image measurements for precise 3D modelling. Nowadays 3D scanners are also becoming a standard source for input data in many application areas, but image-based modelling still remains the most complete, economical, portable, flexible and widely used approach. In this paper the full pipeline is presented for 3D modelling from terrestrial image data, considering the different approaches and analysing all the steps involved.

/pdf/image-based-3d-modelling-a-review-5msqumyd5o.pdf

Image-based 3d modelling: a review

Skin detection is used in applications ranging from face detection, tracking of body parts, hand gesture analysis, to retrieval and blocking objectionable content We present a systematic approach for robust skin segmentation using graph cuts The skin segmentation process starts by exploiting the local skin information of detected faces The detected faces are used as foreground seeds for calculating the foreground weights of the graph If local skin information is not available, we opt for the universal seed To increase the robustness, the decision tree based classifier is used to augment the universal seed weights when no local information is available in the image With this setup, we achieve robust skin segmentation, outperforming off-line trained classifiers The setup also provides a generic skin detection system, using positive training data only With face detection, we take advantage of the contextual information present in the scene With the weight augmentation, we provide a setup for merging spatial and non-spatial data Experiments on two datasets with annotated pixel-level ground truth show that the systematic skin segmentation approach outperforms other approaches and provides robust skin detection

Systematic skin segmentation: merging spatial and non-spatial data

In this paper, we present a Remote Sens. approach to localize parking cars in a city in order to enable the development of parking space availability models. We propose to use high-resolution stereo satellite images for this problem, as they provide enough details to make individual cars recognizable and the time interval between the stereo shots allows to reason about the moving or static condition of a car. Consequently, we describe a complete processing pipeline where raw satellite images are georeferenced, ortho-rectified, equipped with a digital surface model and an inclusion layer generated from Open Street Model vector data, and finally analyzed for parking cars by means of an adapted Faster R-CNN oriented bounding box detector. As a test site for the proposed approach, a new publicly available dataset of the city of Barcelona labeled with parking cars is presented. On this dataset, a Faster R-CNN model directly trained on the two ortho-rectified stereo images achieves an average precision of 0.65 for parking car detection. Finally, an extensive empirical and analytical evaluation shows the validity of our idea, as parking space occupancy can be broadly derived in fully visible areas, whereas moving cars are efficiently ruled out. Our evaluation also includes an in-depth analysis of the stereo occlusion problem in view of our application scenario as well as the suitability of using a reconstructed Digital Surface Model (DSM) as additional data modality for car detection. While an additional adoption of the DSM in our pipeline does not provide a beneficial cue for the detection task, the stereo images provide essentially two views of the dynamic scene at different timestamps. Therefore, for future studies, we recommend a satellite image acquisition geometry with smaller incidence angles, to decrease occlusions by buildings and thus improve the results with respect to completeness.

Detection of Parking Cars in Stereo Satellite Images

Multi-spectral imaging for the analysis and preservation of ancient documents has gained high attention in recent years. While readability enhancement is based on the multi-spectral image corpus, foreground-background separation still relies mainly on gray level or color images. In this paper we propose a foreground-background separation algorithm designed for multi-spectral images. The main contribution is the simultaneously utilization of spectral and spatial features. While spectral features incorporate the spectral components of the multi-spectral images, the spatial features are based on stroke properties. Higher order Markov Random Fields enables an efficient way to combine both features. To solve higher order energy functions, we introduce a new message update rule in the well known belief propagation algorithm based on a higher order potential function.

Higher order MRF for foreground-background separation in multi-spectral images of historical manuscripts

Recent developments in computer vision are providing powerful tools for the evaluation of data gathered by art historians and archaeologists. New camera hardware allows new insights into cultural heritage, especially if infrared cameras are concerned, since they allow the of study structures that are visually hidden. In this paper preliminary results of developing a system for automatic analysis of infrared reflectograms are presented. We concentrate on an algorithm for the automatic segmentation of strokes in underdrawings - the basic concept of the artist - in ancient panel paintings and the removal of cracks in infrared images. The purpose of the stroke analysis is the determination of the drawing tool used to draft the painting. This information allows significant support for a systematic stylistic approach in the analysis of paintings. Stroke segmentation in paintings is related to the extraction and recognition of handwriting, therefore similar techniques to segment the strokes from the background incorporating boundary information are used. Results of the algorithms developed are presented for both test panels and real reflectograms.

Robert Sablatnig

Papers

Systematic skin segmentation: merging spatial and non-spatial data

Detection of Parking Cars in Stereo Satellite Images

Higher order MRF for foreground-background separation in multi-spectral images of historical manuscripts

Computer aided analysis of underdrawings in infrared reflectograms

Computer aided classification of ceramics