Semantic content-based image retrieval: A comprehensive study ☆

As an emerging technology to support scalable content-based image retrieval (CBIR), hashing has recently received great attention and became a very active research domain. In this study, we propose a novel unsupervised visual hashing approach called semantic-assisted visual hashing (SAVH). Distinguished from semi-supervised and supervised visual hashing, its core idea is to effectively extract the rich semantics latently embedded in auxiliary texts of images to boost the effectiveness of visual hashing without any explicit semantic labels. To achieve the target, a unified unsupervised framework is developed to learn hash codes by simultaneously preserving visual similarities of images, integrating the semantic assistance from auxiliary texts on modeling high-order relationships of inter-images, and characterizing the correlations between images and shared topics. Our performance study on three publicly available image collections: Wiki , MIR Flickr , and NUS-WIDE indicates that SAVH can achieve superior performance over several state-of-the-art techniques.

/pdf/unsupervised-visual-hashing-with-semantic-assistant-for-2lodyz606p.pdf

Unsupervised Visual Hashing with Semantic Assistant for Content-Based Image Retrieval

With the prevalence of multimedia content on the Web, developing recommender solutions that can effectively leverage the rich signal in multimedia data is in urgent need. Owing to the success of deep neural networks in representation learning, recent advances on multimedia recommendation has largely focused on exploring deep learning methods to improve the recommendation accuracy. To date, however, there has been little effort to investigate the robustness of multimedia representation and its impact on the performance of multimedia recommendation. In this paper, we shed light on the robustness of multimedia recommender system. Using the state-of-the-art recommendation framework and deep image features, we demonstrate that the overall system is not robust, such that a small (but purposeful) perturbation on the input image will severely decrease the recommendation accuracy. This implies the possible weakness of multimedia recommender system in predicting user preference, and more importantly, the potential of improvement by enhancing its robustness. To this end, we propose a novel solution named Adversarial Multimedia Recommendation (AMR), which can lead to a more robust multimedia recommender model by using adversarial learning. The idea is to train the model to defend an adversary, which adds perturbations to the target image with the purpose of decreasing the model's accuracy. We conduct experiments on two representative multimedia recommendation tasks, namely, image recommendation and visually-aware product recommendation. Extensive results verify the positive effect of adversarial learning and demonstrate the effectiveness of our AMR method. Source codes are available in https://github.com/duxy-me/AMR .

Adversarial Training Towards Robust Multimedia Recommender System

Content based image retrieval (CBIR) systems provide potential solution of retrieving semantically similar images from large image repositories against any query image. The research community are competing for more effective ways of content based image retrieval, so they can be used in serving time critical applications in scientific and industrial domains. In this paper a Neural Network based architecture for content based image retrieval is presented. To enhance the capabilities of proposed work, an efficient feature extraction method is presented which is based on the concept of in-depth texture analysis. For this wavelet packets and Eigen values of Gabor filters are used for image representation purposes. To ensure semantically correct image retrieval, a partial supervised learning scheme is introduced which is based on K-nearest neighbors of a query image, and ensures the retrieval of images in a robust way. To elaborate the effectiveness of the presented work, the proposed method is compared with several existing CBIR systems, and it is proved that the proposed method has performed better then all of the comparative systems.

Embedding neural networks for semantic association in content based image retrieval

An integrated approach to region based image retrieval using firefly algorithm and support vector machine

Nowadays, content-based image retrieval (CBIR) is the mainstay of image retrieval systems. To be more profitable, relevance feedback techniques were incorporated into CBIR such that more precise results can be obtained by taking user's feedbacks into account. However, existing relevance feedback-based CBIR methods usually request a number of iterative feedbacks to produce refined search results, especially in a large-scale image database. This is impractical and inefficient in real applications. In this paper, we propose a novel method, Navigation-Pattern-based Relevance Feedback (NPRF), to achieve the high efficiency and effectiveness of CBIR in coping with the large-scale image data. In terms of efficiency, the iterations of feedback are reduced substantially by using the navigation patterns discovered from the user query log. In terms of effectiveness, our proposed search algorithm NPRFSearch makes use of the discovered navigation patterns and three kinds of query refinement strategies, Query Point Movement (QPM), Query Reweighting (QR), and Query Expansion (QEX), to converge the search space toward the user's intention effectively. By using NPRF method, high quality of image retrieval on RF can be achieved in a small number of feedbacks. The experimental results reveal that NPRF outperforms other existing methods significantly in terms of precision, coverage, and number of feedbacks.

Efficient Relevance Feedback for Content-Based Image Retrieval by Mining User Navigation Patterns

A content-based image retrieval method by integrating navigation patterns mining and relevance feedback is retrieved This method comprises an offline stage and an online stage The offline stage includes a data transformation step, a navigation pattern mining step and a pattern indexing step The online stage includes an initial query processing phase and a Q3 search phase The initial query processing phase includes a feature extraction step, an initial feedback step and a user interaction step The Q3 search phase includes a query-point generation step, a query expansion step, a query re-weighting step and a result filtering step

Content-based image retrieval method by integrating navigation patterns mining and relevance feedback

Semantic image retrieval basically can be viewed as a pattern recognition problem. For human, pattern recognition is inherent in herself/himself by the inference rules through a long time experience. However, for computer, on the one hand, the simulated human identification of objects is impressive at its experience (training) like a baby learns to identify objects; on the other hand, the precise identification is unreasonable because the similar features are usually shared by different objects, e.g., “an white animal like cat and dog”, “a structural transportation like car and truck”. In traditional approaches, disambiguate the images by eliminating irrelevant semantics does not fit in with human behavior. Accordingly, the ambiguous concepts of each image estimated throughout the collaboration of similarity function and membership function is sensible. To this end, in this paper, we propose a novel fuzzy matching technique named Fuzzy Content-Based Image Retrieval (FCBIR) that primarily contains three characteristics: 1) conceptualize image automatically, 2) identify image roughly, and 3) retrieve image efficiently. Out of human perspective, experiments reveal that our proposed approach can bring out good results effectively and efficiently in terms of image retrieval.

Wei-Jyun Huang

Papers

Efficient Relevance Feedback for Content-Based Image Retrieval by Mining User Navigation Patterns

Content-based image retrieval method by integrating navigation patterns mining and relevance feedback

FCBIR: A Fuzzy Matching Technique for Content-Based Image Retrieval