This paper presents a new approach to identifying and eliminating mislabeled training instances for supervised learning. The goal of this approach is to improve classification accuracies produced by learning algorithms by improving the quality of the training data. Our approach uses a set of learning algorithms to create classifiers that serve as noise filters for the training data. We evaluate single algorithm, majority vote and consensus filters on five datasets that are prone to labeling errors. Our experiments illustrate that filtering significantly improves classification accuracy for noise levels up to 30%. An analytical and empirical evaluation of the precision of our approach shows that consensus filters are conservative at throwing away good data at the expense of retaining bad data and that majority filters are better at detecting bad data at the expense of throwing away good data. This suggests that for situations in which there is a paucity of data, consensus filters are preferable, whereas majority vote filters are preferable for situations with an abundance of data.

/pdf/identifying-mislabeled-training-data-ly6q8sogeg.pdf

Identifying mislabeled training data

Real-world data is never perfect and can often suffer from corruptions (noise) that may impact interpretations of the data, models created from the data and decisions made based on the data. Noise can reduce system performance in terms of classification accuracy, time in building a classifier and the size of the classifier. Accordingly, most existing learning algorithms have integrated various approaches to enhance their learning abilities from noisy environments, but the existence of noise can still introduce serious negative impacts. A more reasonable solution might be to employ some preprocessing mechanisms to handle noisy instances before a learner is formed. Unfortunately, rare research has been conducted to systematically explore the impact of noise, especially from the noise handling point of view. This has made various noise processing techniques less significant, specifically when dealing with noise that is introduced in attributes. In this paper, we present a systematic evaluation on the effect of noise in machine learning. Instead of taking any unified theory of noise to evaluate the noise impacts, we differentiate noise into two categories: class noise and attribute noise, and analyze their impacts on the system performance separately. Because class noise has been widely addressed in existing research efforts, we concentrate on attribute noise. We investigate the relationship between attribute noise and classification accuracy, the impact of noise at different attributes, and possible solutions in handling attribute noise. Our conclusions can be used to guide interested readers to enhance data quality by designing various noise handling mechanisms.

https://cs.nju.edu.cn/zhouzh/zhouzh.files/course/dm/reading/reading03/zhu_airev04.pdf

Class noise vs. attribute noise: a quantitative study of their impacts

To cleanse mislabeled examples from a training dataset for efficient and effective induction, most existing approaches adopt a major set oriented scheme: the training dataset is separated into two parts (a major set and a minor set). The classifiers learned from the major set are used to identify noise in the minor set. The obvious drawbacks of such a scheme are twofold: (1) when the underlying data volume keeps growing, it would be either physically impossible or time consuming to load the major set into the memory for inductive learning; and (2) for multiple or distributed datasets, it can be either technically infeasible or factitiously forbidden to download data from other sites (for security or privacy reasons). Therefore, these approaches have severe limitations in conducting effective global data cleansing from large, distributed datasets.

In this paper, we propose a solution to bridge the local and global analysis for noise cleansing. More specifically, the proposed effort tries to identify and eliminate mislabeled data items from large or distributed datasets through local analysis and global incorporation. For this purpose, we make use of distributed datasets or partition a large dataset into subsets, each of which is regarded as a local subset and is small enough to be processed by an induction algorithm at one time to construct a local model for noise identification. We construct good rules from each subset, and use the good rules to evaluate the whole dataset. For a given instance I k , two error count variables are used to count the number of times it has been identified as noise by all data subsets. The instance with higher error values will have a higher probability of being a mislabeled example. Two threshold schemes, majority and non-objection, are used to identify and eliminate the noisy examples. Experimental results and comparative studies on both real-world and synthetic datasets are reported to evaluate the effectiveness and efficiency of the proposed approach.

Bridging Local and Global Data Cleansing: Identifying Class Noise in Large, Distributed Data Datasets

SIMS—A precursor and partner to contemporary mass spectrometry

Real-world large-scale datasets are heteroskedastic and imbalanced -- labels have varying levels of uncertainty and label distributions are long-tailed Heteroskedasticity and imbalance challenge deep learning algorithms due to the difficulty of distinguishing among mislabeled, ambiguous, and rare examples Addressing heteroskedasticity and imbalance simultaneously is under-explored We propose a data-dependent regularization technique for heteroskedastic datasets that regularizes different regions of the input space differently Inspired by the theoretical derivation of the optimal regularization strength in a one-dimensional nonparametric classification setting, our approach adaptively regularizes the data points in higher-uncertainty, lower-density regions more heavily We test our method on several benchmark tasks, including a real-world heteroskedastic and imbalanced dataset, WebVision Our experiments corroborate our theory and demonstrate a significant improvement over other methods in noise-robust deep learning

Heteroskedastic and Imbalanced Deep Learning with Adaptive Regularization

Identifying inaccurate data has long been regarded as a significant and difficult problem in AI. In this paper, we present a new method for identifying inaccurate data on the basis of qualitative correlations among related data. First, we introduce the definitions of related data and qualitative correlations among related data. Then we put forward a new concept called support coefficient function (SCF). SCF can be used to extract, represent, and calculate qualitative correlations among related data within a dataset. We propose an approach to determining dynamic shift intervals of inaccurate data, and an approach to calculating possibility of identifying inaccurate data, respectively. Both of the approaches are based on SCF. Finally we present an algorithm for identifying inaccurate data by using qualitative correlations among related data as confirmatory or disconfirmatory evidence. We have developed a practical system for interpreting infrared spectra by applying the method, and have fully tested the system against several hundred real spectra. The experimental results show that the method is significantly better than the conventional methods used in many similar systems.

/pdf/using-qualitative-hypotheses-to-identify-inaccurate-data-591dz47zb3.pdf

Using qualitative hypotheses to identify inaccurate data

Infra-red spectrum processing (IRSP) is a very important problem both in academic research and industrial application which includes IR spectrum coding, interpreting and recognizing. The primary aim of IRSP is to identify what atomic groups an unknown compound contains and what the unknown compound is by interpreting its IR spectrum. This paper presents a knowledge model for IRSP and its application in an expert system. First, three new concepts of IRSP are put forward. Then the knowledge model is introduced. Finally, an expert system based on the knowledge model is presented.

A Knowledge Model for Infrared Spectrum Processing

Using Qualitative Hypotheses to Identify Inaccurate Data

A novel method for extracting, representing and propagating qualitative correlations among hypotheses as confirmatory or disconfirmatory evidence of uncertain reasoning is presented. First, two new concepts, qualitative correlations among hypotheses and qualitative correlation propagation, are introduced. Then, an algorithm for extracting and representing qualitative correlations among hypotheses and an algorithm for propagating qualitative correlations and updating possibilities of hypotheses are proposed. The advantages of the method include: (1) it can applied to the problems where evidence is not explicitly or completely given; (2) few numbers and assumptions need to be provided by domain experts in advance; and consequently, (3) the knowledge acquisition is simple, and the inconsistency in knowledge bases can be avoided. The method has been applied to a practical system for infrared spectrum interpretation. The experimental results show that it is significantly better than other methods used in similar systems.

/pdf/using-qualitative-correlations-as-evidence-of-uncertain-56d8a2vqq4.pdf

Qi Zhao

Papers

Using qualitative hypotheses to identify inaccurate data

A Knowledge Model for Infrared Spectrum Processing

Using Qualitative Hypotheses to Identify Inaccurate Data

Using Qualitative Correlations as Evidence of Uncertain Reasoning

Qualitative Decomposition and Recognition of Infrared Spectra