Recently, the wide deployment of practical face recognition systems gives rise to the emergence of the inter-modality face recognition problem. In this problem, the face images in the database and the query images captured on spot are acquired under quite different conditions or even using different equipments. Conventional approaches either treat the samples in a uniform model or introduce an intermediate conversion stage, both of which would lead to severe performance degradation due to the great discrepancies between different modalities. In this paper, we propose a novel algorithm called Common Discriminant Feature Extraction specially tailored to the inter-modality problem. In the algorithm, two transforms are simultaneously learned to transform the samples in both modalities respectively to the common feature space. We formulate the learning objective by incorporating both the empirical discriminative power and tlie local smoothness of the feature transformation. By explicitly controlling the model complexity through the smoothness constraint, we can effectively reduce the risk of overfitting and enhance the generalization capability. Furthermore, to cope with the nongaussian distribution and diverse variations in the sample space, we develop two non-linear extensions of the algorithm: one is based on kernelization, while the other is a multi-mode framework. These extensions substantially improve the recognition performance in complex situation. Extensive experiments are conducted to test our algorithms in two application scenarios: optical image-infrared image recognition and photo-sketch recognition. Our algorithms show excellent performance in the experiments.

Inter-modality Face Recognition

The emergence of Artificial Intelligence of Things (AIoT) has provided novel insights for many social computing applications such as group recommender systems. As the distances between people have been greatly shortened, there has been more general demand for the provision of personalized services aimed at groups instead of individuals. The existing methods for capturing group-level preference features from individuals have mostly been established via aggregation and face two challenges: secure data management workflows are absent, and implicit preference feedback is ignored. To tackle these current difficulties, this paper proposes secure AIoT for implicit group recommendations (SAIoT-GR). For the hardware module, a secure IoT structure is developed as the bottom support platform. For the software module, a collaborative Bayesian network model and noncooperative game are introduced as algorithms. This secure AIoT architecture is able to maximize the advantages of the two modules. In addition, a large number of experiments are carried out to evaluate the performance of SAIoT-GR in terms of efficiency and robustness.

Secure Artificial Intelligence of Things for Implicit Group Recommendations

The content based image retrieval aims to find the similar images from a large scale dataset against a query image. Generally, the similarity between the representative features of the query image and dataset images is used to rank the images for retrieval. In early days, various hand designed feature descriptors have been investigated based on the visual cues such as color, texture, shape, etc. that represent the images. However, the deep learning has emerged as a dominating alternative of hand-designed feature engineering from a decade. It learns the features automatically from the data. This paper presents a comprehensive survey of deep learning based developments in the past decade for content based image retrieval. The categorization of existing state-of-the-art methods from different perspectives is also performed for greater understanding of the progress. The taxonomy used in this survey covers different supervision, different networks, different descriptor type and different retrieval type. A performance analysis is also performed using the state-of-the-art methods. The insights are also presented for the benefit of the researchers to observe the progress and to make the best choices. The survey presented in this paper will help in further research progress in image retrieval using deep learning.

A Decade Survey of Content Based Image Retrieval using Deep Learning

Text classification is the most fundamental and essential task in natural language processing. The last decade has seen a surge of research in this area due to the unprecedented success of deep learning. Numerous methods, datasets, and evaluation metrics have been proposed in the literature, raising the need for a comprehensive and updated survey. This paper fills the gap by reviewing the state of the art approaches from 1961 to 2020, focusing on models from shallow to deep learning. We create a taxonomy for text classification according to the text involved and the models used for feature extraction and classification. We then discuss each of these categories in detail, dealing with both the technical developments and benchmark datasets that support tests of predictions. A comprehensive comparison between different techniques, as well as identifying the pros and cons of various evaluation metrics are also provided in this survey. Finally, we conclude by summarizing key implications, future research directions, and the challenges facing the research area.

A Survey on Text Classification: From Shallow to Deep Learning

Supervised 3D reconstruction has witnessed a significant progress through the use of deep neural networks. However, this increase in performance requires large scale annotations of 2D/3D data. In this paper, we explore inexpensive 2D supervision as an alternative for expensive 3D CAD annotation. Specifically, we use foreground masks as weak supervision through a raytrace pooling layer that enables perspective projection and backpropagation. Additionally, since the 3D reconstruction from masks is an ill posed problem, we propose to constrain the 3D reconstruction to the manifold of unlabeled realistic 3D shapes that match mask observations. We demonstrate that learning a log-barrier solution to this constrained optimization problem resembles the GAN objective, enabling the use of existing tools for training GANs. We evaluate and analyze the manifold constrained reconstruction on various datasets for single and multi-view reconstruction of both synthetic and real images.

Weakly supervised 3D Reconstruction with Adversarial Constraint

Recently, there is a new approach for bike usage that has emerged by bike-sharing system. While traveling on the road, more and more people will choose to ride shared bicycle at home and abroad. When using shared bikes, we also face to problems and challenges. As a result of the shared-bikes renting/returning at different stations in different periods are imbalanced, the bike-sharing system needs to be updated frequently. This is the motivation for our study of bike-sharing traffic prediction. In this paper, we propose a hierarchical prediction model that predicts the number of rents/returns to each cluster in a future period to achieve redistribution. Firstly, we propose a two-level Gaussian Mixture Model clustering algorithm to divide bike stations into groups where migration trends of bikes among stations as well as geographical locations information are considered. Secondly, we employ a gradient boosting regression tree to predict the entire traffic rents. Thirdly, we use a multi-similarity-based inference model to forecast the check-out proportion and inter-cluster transition. Based on the above, finally the rents/returns of bikes to each cluster are deduced. In order to verify the effectiveness of our hierarchical prediction model, we validate it on the bike-sharing system of New York City (NYC) and Washington D.C. (D.C.) respectively, and compare the results with those of other popular methods obtained. We prove that our method is robust. Experimental results demonstrate the superiority over other methods. Compared with the state-of-the-art model, our model reduces the error rate roughly by 8%and 22% respectively in the check-out and check-in prediction for NYC, and reduces roughly by 3%and 2% respectively in the check-out and check-in prediction for D.C. Compared with other baseline models, our model can reduce the error rate roughly by 20%and 30% respectively in the check-out and check-in prediction for the two cities.

Hierarchical prediction based on two-level Gaussian mixture model clustering for bike-sharing system

The increment of new words and text categories requires more accurate and robust classification methods. In this paper, we propose a novel multi-label text classification method that combines dynamic semantic representation model and deep neural network (DSRM-DNN). DSRM-DNN first utilizes word embedding model and clustering algorithm to select semantic words. Then the selected words are designated as the elements of DSRM-DNN and quantified by the weighted combination of word attributes. Finally, we construct a text classifier by combining deep belief network and back-propagation neural network. During the classification process, the low-frequency words and new words are re-expressed by the existing semantic words under sparse constraint. We evaluate the performance of DSRM-DNN on RCV1-v2, Reuters-21578, EUR-Lex, and Bookmarks. Experimental results show that our method outperforms the state-of-the-art methods.

A multi-label text classification method via dynamic semantic representation model and deep neural network

Nowadays, most recommendation approaches used to suggest appreciate items for individual users. However, due to the social nature of human beings, group activities have become an integral part of our daily life, thus motivating the study on group recommendation. Unfortunately, most existing approaches used in group recommender systems make recommendations through aggregating individual preferences or individual predictive results rather than comprehensively investigating the social features that govern user choices made within a group. As a result, such approaches often fail to detect many latent factors that could potentially improve the performance of the group recommender systems. Therefore, we propose a new approach, random walks based on a topic model (RTM), for group recommendations through combining an integrated probabilistic topic model − a  User Topic Model (UTM) with the Random Walk with Restart (RWR) method. The goal of the work in this paper is better identify group preferences by comprehensively detecting the latent associations among group members, in order to alleviate the data sparsity problem and improve the performance of group recommender systems. The UTM provides a latent framework of users, groups, and items by exploiting both the users’ preference profiles and the items’ content information, which together can describe group interests and item features in a more complete manner. This latent framework is then combined with RWR to predict the preference degrees of groups to unrated items. In particular, we develop two different recommendation strategies based on the proposed approach, and design a special random walk path for each developed recommendation strategy to comprehensively detect various latent associations. Finally, we conduct experiments to evaluate our approach and compare it with other state-of-the-art approaches using the real-world CAMRa2011 dataset. The results demonstrate the advantage of our approach over comparative ones.

Detecting the latent associations hidden in multi-source information for better group recommendation

Multi-modal hashing focuses on fusing different modalities and exploring the complementarity of heterogeneous multi-modal data for compact hash learning. However, existing multi-modal hashing methods still suffer from several problems, including: 1) Almost all existing methods generate unexplainable hash codes. They roughly assume that the contribution of each hash code bit to the retrieval results is the same, ignoring the discriminative information embedded in hash learning and semantic similarity in hash retrieval. Moreover, the length of hash code is empirically set, which will cause bit redundancy and affect retrieval accuracy. 2) Most existing methods exploit shallow models which fail to fully capture higher-level correlation of multi-modal data. 3) Most existing methods adopt online hashing strategy based on immutable direct projection, which generates query codes for new samples without considering the differences of semantic categories. In this paper, we propose a Semantic-driven Interpretable Deep Multi-modal Hashing (SIDMH) method to generate interpretable hash codes driven by semantic categories within a deep hashing architecture, which can solve all these three problems in an integrated model. The main contributions are: 1) A novel deep multi-modal hashing network is developed to progressively extract hidden representations of heterogeneous modality features and deeply exploit the complementarity of multi-modal data. 2) Learning interpretable hash codes, with discriminant information of different categories distinctively embedded into hash codes and their different impacts on hash retrieval intuitively explained. Besides, the code length depends on the number of categories in the dataset, which can reduce the bit redundancy and improve the retrieval accuracy. 3) The semantic-driven online hashing strategy encodes the significant branches and discards the negligible branches of each query sample according to the semantics contained in it, therefore it could capture different semantics in dynamic queries. Finally, we consider both the nearest neighbor similarity and semantic similarity of hash codes. Experiments on several public multimedia retrieval datasets validate the superiority of the proposed method.

Semantic-driven Interpretable Deep Multi-modal Hashing for large-scale multimedia retrieval

Visible infrared person re-identification (VI-reid) has gradually increased in popularity as an crucial branch of person re-identification (reid). It not only has intra-class variations caused by different viewpoint, pedestrian posture changes, complex backgrounds, resolution, occlusions existing in traditional visible visible person re-identification (VV-reid), but also has been subjected to enormous cross-modality discrepancies resulted from the difference in the reflection spectrum of visible light and infrared cameras. In this paper, we put forward a novel loss function, called dual-modality hard mining triplet-center loss (DTCL), to optimize intra-class and inter-class distance and to supervise the network to learn discriminative feature representations. The DTCL learns a visible modality center and an infrared modality center separately for each class and selects online novel cross-modality triplets and intra-modality triplets for each sample respectively. In particular, it makes the samples from the same class closer to the class center and pushes them away from the centers of the different classes. Besides, we also propose a dual-path part-based feature learning network (DPFLN) framework, to learn the local features and solve the problem of cross-modality discrepancies. We conduct experiments on two cross-modality reid datasets, and obtain promising results.

Li Liu

Papers

Hierarchical prediction based on two-level Gaussian mixture model clustering for bike-sharing system

A multi-label text classification method via dynamic semantic representation model and deep neural network

Detecting the latent associations hidden in multi-source information for better group recommendation

Semantic-driven Interpretable Deep Multi-modal Hashing for large-scale multimedia retrieval

Dual-modality hard mining triplet-center loss for visible infrared person re-identification