Label smoothing is an effective regularization tool for deep neural networks (DNNs), which generates soft labels by applying a weighted average between the uniform distribution and the hard label. It is often used to reduce the overfitting problem of training DNNs and further improve classification performance. In this paper, we aim to investigate how to generate more reliable soft labels. We present an Online Label Smoothing (OLS) strategy, which generates soft labels based on the statistics of the model prediction for the target category. The proposed OLS constructs a more reasonable probability distribution between the target categories and non-target categories to supervise DNNs. Experiments demonstrate that based on the same classification models, the proposed approach can effectively improve the classification performance on CIFAR-100, ImageNet, and fine-grained datasets. Additionally, the proposed method can significantly improve the robustness of DNN models to noisy labels compared to current label smoothing approaches.

Delving Deep into Label Smoothing

Label smoothing is an effective regularization tool for deep neural networks (DNNs), which generates soft labels by applying a weighted average between the uniform distribution and the hard label. It is often used to reduce the overfitting problem of training DNNs and further improve classification performance. In this paper, we aim to investigate how to generate more reliable soft labels. We present an Online Label Smoothing (OLS) strategy, which generates soft labels based on the statistics of the model prediction for the target category. The proposed OLS constructs a more reasonable probability distribution between the target categories and non-target categories to supervise DNNs. Experiments demonstrate that based on the same classification models, the proposed approach can effectively improve the classification performance on CIFAR-100, ImageNet, and fine-grained datasets. Additionally, the proposed method can significantly improve the robustness of DNN models to noisy labels compared to current label smoothing approaches. The source code is available at our project page: https://mmcheng.net/ols/

Delving Deep Into Label Smoothing

ABSTRACT Cellular automata (CA) model is a useful tool for simulating spatiotemporal changes of land-use evolution. However, previous CA models have usually ignored the inhomogeneous and mixed land-use situations and simplified the land grid cell as one single land-use type. This study developed a multi-label (ML) convolutional neural network CA (ML-CNN-CA) model to simulate the complex evolution of mixed land uses. The multi-label learning strategy in the proposed model can formulize mixed land uses by labeling multiple land-use types to each grid cell. The transition rules of CA were modified to accommodate the multi-label learning strategy and mined by coupling a CNN network and three interactive mechanisms, including the neighborhood effect, adaptive inertia coefficient, and random factor. The proposed ML-CNN-CA model was applied and examined in a fast-developing city (Huizhou) in the China’s Pearl River Delta region for the mixed land-use simulation during 2009–2013 and 2013–2020. The ML-CNN-CA model with different CNN architectures and a previous artificial neural network (ANN)-based multi-label CA model were also implemented for comparisons. Results show the capability of the ML-CNN-CA model to simulate fine-scale mixed land-use changes with satisfactory performances. Specifically, the simulated mixed land-use patterns agree well with the actual land uses from the morphological perspective. Furthermore, the quantitative assessments demonstrate the performance of the proposed model, showing an accuracy value of 0.912 for 2009–2013 and 0.896 for 2013–2020, and a hamming loss value of 0.048 for 2009–2013 and 0.055 for 2013–2020. The comparisons also show the best performance of the ML-CNN-CA model with the VGG-based architecture and significant outperformance of the proposed model against the previous ANN-based CA model. Multiple sensitivity analyses were also conducted to investigate the uncertainty of the proposed model. The ML-CNN-CA model proposed in this study can provide a new tool for better simulation of fine-scale mixed land-use changes and is expected to help formulate urban planning guidelines and achieve sustainable urban development.

https://www.tandfonline.com/doi/pdf/10.1080/15481603.2022.2049493?needAccess=true

Simulating mixed land-use change under multi-label concept by integrating a convolutional neural network and cellular automata: a case study of Huizhou, China

Most existing unsupervised active learning methods aim at minimizing the data reconstruction loss by using the linear models to choose representative samples for manually labeling in an unsupervised setting. Thus these methods often fail in modelling data with complex non-linear structure. To address this issue, we propose a new deep unsupervised Active Learning method for classification tasks, inspired by the idea of Matrix Sketching, called ALMS. Specifically, ALMS leverages a deep auto-encoder to embed data into a latent space, and then describes all the embedded data with a small size sketch to summarize the major characteristics of the data. In contrast to previous approaches that reconstruct the whole data matrix for selecting the representative samples, ALMS aims to select a representative subset of samples to well approximate the sketch, which can preserve the major information of data meanwhile significantly reducing the number of network parameters. This makes our algorithm alleviate the issue of model overfitting and readily cope with large datasets. Actually, the sketch provides a type of self-supervised signal to guide the learning of the model. Moreover, we propose to construct an auxiliary self-supervised task by classifying real/fake samples, in order to further improve the representation ability of the encoder. We thoroughly evaluate the performance of ALMS on both single-label and multi-label classification tasks, and the results demonstrate its superior performance against the state-of-the-art methods. The code can be found at https://github.com/lrq99/ALMS .

Deep Unsupervised Active Learning via Matrix Sketching

Code smells in a source code shows the weakness of design or implementation. To detect code smells, several detection tools have been developed. However, these tools generally produce different results, since code smells are subjectively interpreted, informally defined and configured by the developers, domain-dependent and based on opinions and experiences. To cope with these issues, in this paper, we have used machine learning techniques, especially multi-label classification methods, to classify whether the given source code is affected with more than one code smells or not. We have conducted experiments on four code smell datasets and transformed them into two multi-label datasets (one for method level and the other one for class level). Two multi-label classification methods (Classifier Chains and Label Combination) and their ensemble models performed on the converted datasets using five different base classifiers. The results show that, as a base classifier, Random Forest algorithm performs better than Decision Tree, Naive Bayes, Support Vector Machine and Neural Network algorithms.

Comparison of Multi-Label Classification Algorithms for Code Smell Detection

In this paper, we present a novel deep metric learning method to tackle the multi-label image classification problem. In order to better learn the correlations among images features, as well as labels, we attempt to explore a latent space, where images and labels are embedded via two unique deep neural networks, respectively. To capture the relationships between image features and labels, we aim to learn a two-way deep distance metric over the embedding space from two different views, i.e., the distance between one image and its labels is not only smaller than those distances between the image and its labels’ nearest neighbors but also smaller than the distances between the labels and other images corresponding to the labels’ nearest neighbors. Moreover, a reconstruction module for recovering correct labels is incorporated into the whole framework as a regularization term, such that the label embedding space is more representative. Our model can be trained in an end-to-end manner. Experimental results on publicly available image data sets corroborate the efficacy of our method compared with the state of the arts.

Reconstruction Regularized Deep Metric Learning for Multi-Label Image Classification

In this paper, we present a novel deep metric learning method to tackle the multi-label image classification problem. In order to better learn the correlations among images features, as well as labels, we attempt to explore a latent space, where images and labels are embedded via two unique deep neural networks, respectively. To capture the relationships between image features and labels, we aim to learn a \emph{two-way} deep distance metric over the embedding space from two different views, i.e., the distance between one image and its labels is not only smaller than those distances between the image and its labels' nearest neighbors, but also smaller than the distances between the labels and other images corresponding to the labels' nearest neighbors. Moreover, a reconstruction module for recovering correct labels is incorporated into the whole framework as a regularization term, such that the label embedding space is more representative. Our model can be trained in an end-to-end manner. Experimental results on publicly available image datasets corroborate the efficacy of our method compared with the state-of-the-arts.

Peng Gao

Papers

Reconstruction Regularized Deep Metric Learning for Multi-Label Image Classification

Reconstruction Regularized Deep Metric Learning for Multi-label Image Classification