527 articles related to imbalanced data and rare events are reviewed.Viewing reviewed papers from both technical and practical perspectives.Summarizing existing methods and corresponding statistics by a new taxonomy idea.Categorizing 162 application papers into 13 domains and giving introduction.Some opening questions are discussed at the end of this manuscript. Rare events, especially those that could potentially negatively impact society, often require humans decision-making responses. Detecting rare events can be viewed as a prediction task in data mining and machine learning communities. As these events are rarely observed in daily life, the prediction task suffers from a lack of balanced data. In this paper, we provide an in depth review of rare event detection from an imbalanced learning perspective. Five hundred and seventeen related papers that have been published in the past decade were collected for the study. The initial statistics suggested that rare events detection and imbalanced learning are concerned across a wide range of research areas from management science to engineering. We reviewed all collected papers from both a technical and a practical point of view. Modeling methods discussed include techniques such as data preprocessing, classification algorithms and model evaluation. For applications, we first provide a comprehensive taxonomy of the existing application domains of imbalanced learning, and then we detail the applications for each category. Finally, some suggestions from the reviewed papers are incorporated with our experiences and judgments to offer further research directions for the imbalanced learning and rare event detection fields.

Learning from class-imbalanced data

The Synthetic Minority Oversampling Technique (SMOTE) preprocessing algorithm is considered "de facto" standard in the framework of learning from imbalanced data. This is due to its simplicity in the design of the procedure, as well as its robustness when applied to different type of problems. Since its publication in 2002, SMOTE has proven successful in a variety of applications from several different domains. SMOTE has also inspired several approaches to counter the issue of class imbalance, and has also significantly contributed to new supervised learning paradigms, including multilabel classification, incremental learning, semi-supervised learning, multi-instance learning, among others. It is standard benchmark for learning from imbalanced data. It is also featured in a number of different software packages -- from open source to commercial. In this paper, marking the fifteen year anniversary of SMOTE, we reect on the SMOTE journey, discuss the current state of affairs with SMOTE, its applications, and also identify the next set of challenges to extend SMOTE for Big Data problems.

/pdf/smote-for-learning-from-imbalanced-data-progress-and-3slcu62yo6.pdf

SMOTE for learning from imbalanced data: progress and challenges, marking the 15-year anniversary

Improving imbalanced learning through a heuristic oversampling method based on k-means and SMOTE

Application of conditional Generative Adversarial Networks as oversampling method.Generates minority class samples by recovering the training data distribution.Outperforms various standard oversampling algorithms.Performance advantage of the proposed method remains stable with higher imbalance ratios. Learning from imbalanced datasets is a frequent but challenging task for standard classification algorithms. Although there are different strategies to address this problem, methods that generate artificial data for the minority class constitute a more general approach compared to algorithmic modifications. Standard oversampling methods are variations of the SMOTE algorithm, which generates synthetic samples along the line segment that joins minority class samples. Therefore, these approaches are based on local information, rather on the overall minority class distribution. Contrary to these algorithms, in this paper the conditional version of Generative Adversarial Networks (cGAN) is used to approximate the true data distribution and generate data for the minority class of various imbalanced datasets. The performance of cGAN is compared against multiple standard oversampling algorithms. We present empirical results that show a significant improvement in the quality of the generated data when cGAN is used as an oversampling algorithm.

Effective data generation for imbalanced learning using conditional generative adversarial networks

Synthetic minority oversampling technique for multiclass imbalance problems

A new oversampling method for imbalanced dataset classification is presented.It clusters the minority class and identifies borderline minority instances.Considering majority class during minority class clustering improves oversampling.Cluster size after oversampling should be dependent on its misclassification error.Generated synthetic instances improved subsequent classification. In many applications, the dataset for classification may be highly imbalanced where most of the instances in the training set may belong to one of the classes (majority class), while only a few instances are from the other class (minority class). Conventional classifiers will strongly favor the majority class and ignore the minority instances. In this paper, we present a new oversampling method called Adaptive Semi-Unsupervised Weighted Oversampling (A-SUWO) for imbalanced binary dataset classification. The proposed method clusters the minority instances using a semi-unsupervised hierarchical clustering approach and adaptively determines the size to oversample each sub-cluster using its classification complexity and cross validation. Then, the minority instances are oversampled depending on their Euclidean distance to the majority class. A-SUWO aims to identify hard-to-learn instances by considering minority instances from each sub-cluster that are closer to the borderline. It also avoids generating synthetic minority instances that overlap with the majority class by considering the majority class in the clustering and oversampling stages. Results demonstrate that the proposed method achieves significantly better results in most datasets compared with other sampling methods.

Adaptive semi-unsupervised weighted oversampling (A-SUWO) for imbalanced datasets

Cluster-based Weighted Oversampling for Ordinal Regression (CWOS-Ord)

A method is presented to automatically track and segment pelvic organs on dynamic magnetic resonance imaging (MRI) followed by multiple-object trajectory classification to improve understanding of pelvic organ prolapse (POP). POP is a major health problem in women where pelvic floor organs fall from their normal position and bulge into the vagina. Dynamic MRI is presently used to analyze the organs' movements, providing complementary support for clinical examination. However, there is currently no automated or quantitative approach to measure the movement of the pelvic organs and their correlation with the severity of prolapse. In the proposed method, organs are first tracked and segmented using particle filters and [Formula: see text]-means clustering with prior information. Then, the trajectories of the pelvic organs are modeled using a coupled switched hidden Markov model to classify the severity of POP. Results demonstrate that the presented method can automatically track and segment pelvic organs with a Dice similarity index above 78% and Hausdorff distance of [Formula: see text] for 94 tested cases while demonstrating correlation between organ movement and POP. This work aims to enable automatic tracking and analysis of multiple deformable structures from images to improve understanding of medical disorders.

/pdf/automated-contour-tracking-and-trajectory-classification-of-4e44hxrx1b.pdf

Automated contour tracking and trajectory classification of pelvic organs on dynamic MRI.

Pelvic organ prolapse is a major health problem in women where pelvic floor organs (bladder, uterus, small bowel, and rectum) fall from their normal position and bulge into the vagina. Dynamic Magnetic Resonance Imaging (DMRI) is presently used to analyze the organs' movements from rest to maximum strain providing complementary support for diagnosis. However, there is currently no automated or quantitative approach to measure the movement of the pelvic organs and their correlation with the severity of prolapse. In this paper, a two-stage method is presented to automatically track and segment pelvic organs on DMRI followed by a multiple-object trajectory classification method to improve the diagnosis of pelvic organ prolapse. Organs are first tracked using particle filters and K-means clustering with prior information. Then, they are segmented using the convex hull of the cluster of particles. Finally, the trajectories of the pelvic organs are modeled using a new Coupled Switched Hidden Markov Model (CSHMM) to classify the severity of pelvic organ prolapse. The tracking and segmentation results are validated using Dice Similarity Index (DSI) whereas the classification results are compared with two manual clinical measurements. Results demonstrate that the presented method is able to automatically track and segment pelvic organs with a DSI above 82% for 26 out of 46 cases and DSI above 75% for all 46 tested cases. The accuracy of the trajectory classification model is also better than current manual measurements.

Automated tracking, segmentation and trajectory classification of pelvic organs on dynamic MRI

/pdf/oversampling-methods-for-imbalanced-dataset-classification-4om0phnptu.pdf

Iman Nekooeimehr

Papers

Adaptive semi-unsupervised weighted oversampling (A-SUWO) for imbalanced datasets

Cluster-based Weighted Oversampling for Ordinal Regression (CWOS-Ord)

Automated contour tracking and trajectory classification of pelvic organs on dynamic MRI.

Automated tracking, segmentation and trajectory classification of pelvic organs on dynamic MRI

Oversampling Methods for Imbalanced Dataset Classification and their Application to Gynecological Disorder Diagnosis