Content-based image retrieval

About: Content-based image retrieval is a research topic. Over the lifetime, 6916 publications have been published within this topic receiving 150696 citations. The topic is also known as: CBIR.

Evaluating strategies and systems for content based indexing of person images on the Web

Abstract: Content based indexing of multimedia has always been a challenging task. The enormity and the diversity of the multimedia content on the web adds another dimension to this challenge. In this paper, we examine ways of combining visual and textual information for content based indexing of multimedia on the web. In particular, we examine different methods of combining evidences due to face detection, Text/HTML analysis and face recognition for identifying person images. We provide experimental evaluation of the following strategies: i) Face detection on the image followed by Text/HTML analysis of the containing page; ii) face detection followed by face recognition; iii) face detection followed by a linear combination of evidences due to text/HTML analysis and face recognition; and iv) face detection followed by a Dempster-Shafer combination of evidences due to text/HTML analysis and face recognition. These strategies were implemented in an automatic web search agent named Diogenes1 and compared against some well known web image search engines. The latter includes commercial systems such as Alta Vista, Lycos and Ditto, and a research prototype, WebSEEk. We report the results of our experimental retrievals where Diogenes outperformed these search engines for celebrity image queries in terms of average precision.

Content-based image retrieval using a mobile device as a novel interface

Abstract: Given the large amount of research into content-based image retrieval currently taking place, new interfaces to systems that perform queries based on image content need to be considered. A new paradigm for content-based image retrieval is introduced, in which a mobile device is used to capture the query image and display the results. The system consists of a client-server architecture in which query images are captured on a mobile device and then transferred to a server for further processing. The server then returns the results of the query to the mobile device. The use of a mobile device as an interface to a content-based image retrieval or object recognition system presents a number of challenges because the query image from the device will have been degraded by noise and subjected to transformations through the imaging system. A methodology is presented that uses techniques inspired from the information retrieval community in order to aid efficient indexing and retrieval. In particular, a vector-space model is used in the efficient indexing of each image, and a two-stage pruning/ranking procedure is used to determine the correct matching image. The retrieval algorithm is shown to outperform existing algorithms when used with query images from the device.

Modelling image complexity by independent component analysis, with application to content-based image retrieval

Abstract: Estimating the degree of similarity between images is a challenging task as the similarity always depends on the context. Because of this context dependency, it seems quite impossible to create a universal metric for the task. The number of low-level features on which the judgement of similarity is based may be rather low, however. One approach to quantifying the similarity of images is to estimate the (joint) complexity of images based on these features. We present a novel method to estimate the complexity of images, based on ICA. We further use this to model joint complexity of images, which gives distances that can be used in content-based retrieval. We compare this new method to two other methods, namely estimating mutual information of images using marginal Kullback-Leibler divergence and approximating the Kolmogorov complexity of images using Normalized Compression Distance.

Mapping visual features to semantic profiles for retrieval in medical imaging

Abstract: Content based image retrieval is highly relevant in medical imaging, since it makes vast amounts of imaging data accessible for comparison during diagnosis. Finding image similarity measures that reflect diagnostically relevant relationships is challenging, since the overall appearance variability is high compared to often subtle signatures of diseases. To learn models that capture the relationship between semantic clinical information and image elements at scale, we have to rely on data generated during clinical routine (images and radiology reports), since expert annotation is prohibitively costly. Here we show that re-mapping visual features extracted from medical imaging data based on weak labels that can be found in corresponding radiology reports creates descriptions of local image content capturing clinically relevant information. We show that these semantic profiles enable higher recall and precision during retrieval compared to visual features, and that we can even map semantic terms describing clinical findings from radiology reports to localized image volume areas.

Deep Learning for Image Retrieval: What Works and What Doesn't

Abstract: To build an industrial content-based image retrieval system (CBIRs), it is highly recommended that feature extraction, feature processing and feature indexing need to be fully considered. Although research that bloomed in the past years suggest that the convolutional neural network (CNN) be in a leading position on feature extraction & representation for CBIRs, there are less instructions on the deep analysis of feature related topics, for example the kind of feature representation that has the best performance among the candidates provided by CNN, the extracted features generalization ability, the relationship between the dimensional reduction and the accuracy loss in CBIRs, the best distance measure technique in CBIRs and the benefit of the coding techniques in improving the efficiency of CBIRs, etc. Therefore, several practicing studies were conducted and a thorough analysis was made in this research attempting to answer the above questions. The results in the study on both ImageNet-2012 and an industrial dataset provided by Sogou demonstrate that fc4096a and fc4096b perform the best on the datasets from unseen categories. Several interesting and practicing conclusions are drawn, for instance, fc4096a and fc4096b are found to have a better generalization ability than other features of CNN and could be considered as the first choice for industrial CBIRs. Furthermore, a novel feature binarization approach is presented in this paper for better efficiency of CBIRs. More specifically, the binarization is capable of reducing 31/32 space usage of original data. To sum up, the conclusions seem to provide practical instructions on real industrial CBIRs.