527 articles related to imbalanced data and rare events are reviewed.Viewing reviewed papers from both technical and practical perspectives.Summarizing existing methods and corresponding statistics by a new taxonomy idea.Categorizing 162 application papers into 13 domains and giving introduction.Some opening questions are discussed at the end of this manuscript. Rare events, especially those that could potentially negatively impact society, often require humans decision-making responses. Detecting rare events can be viewed as a prediction task in data mining and machine learning communities. As these events are rarely observed in daily life, the prediction task suffers from a lack of balanced data. In this paper, we provide an in depth review of rare event detection from an imbalanced learning perspective. Five hundred and seventeen related papers that have been published in the past decade were collected for the study. The initial statistics suggested that rare events detection and imbalanced learning are concerned across a wide range of research areas from management science to engineering. We reviewed all collected papers from both a technical and a practical point of view. Modeling methods discussed include techniques such as data preprocessing, classification algorithms and model evaluation. For applications, we first provide a comprehensive taxonomy of the existing application domains of imbalanced learning, and then we detail the applications for each category. Finally, some suggestions from the reviewed papers are incorporated with our experiences and judgments to offer further research directions for the imbalanced learning and rare event detection fields.

Learning from class-imbalanced data

Data imbalance is frequently encountered in biomedical applications Resampling techniques can be used in binary classification to tackle this issue However such solutions are not desired when the number of samples in the small class is limited Moreover the use of inadequate performance metrics, such as accuracy, lead to poor generalization results because the classifiers tend to predict the largest size class One of the good approaches to deal with this issue is to optimize performance metrics that are designed to handle data imbalance Matthews Correlation Coefficient (MCC) is widely used in Bioinformatics as a performance metric We are interested in developing a new classifier based on the MCC metric to handle imbalanced data We derive an optimal Bayes classifier for the MCC metric using an approach based on Frechet derivative We show that the proposed algorithm has the nice theoretical property of consistency Using simulated data, we verify the correctness of our optimality result by searching in the space of all possible binary classifiers The proposed classifier is evaluated on 64 datasets from a wide range data imbalance We compare both classification performance and CPU efficiency for three classifiers: 1) the proposed algorithm (MCC-classifier), the Bayes classifier with a default threshold (MCC-base) and imbalanced SVM (SVM-imba) The experimental evaluation shows that MCC-classifier has a close performance to SVM-imba while being simpler and more efficient

/pdf/optimal-classifier-for-imbalanced-data-using-matthews-326t5v9ce2.pdf

Optimal classifier for imbalanced data using Matthews Correlation Coefficient metric.

LB-194 Lung cancer is the most prevalent form of cancer worldwide and accounts for the most cancer deaths. MicroRNAs (miRNAs) are small, non-protein coding RNAs that have recently emerged as important regulators of gene expression and direct proper cellular growth, differentiation and cell death - all mechanisms that go awry in cancer. The let-7 miRNA is postulated to function as a tumor suppressor gene in a variety of human tissues, particularly in the lung, by negatively regulating the post-transcriptional expression of multiple oncogenes including RAS, MYC, and HMGA2, as well as other cell cycle progression genes. Here we have used both in vitro and in vivo approaches to show that let-7 directly represses cancer growth in  the lung. We show that let-7 inhibits the growth of multiple human lung cancer cell lines in culture, as well as the growth of lung cancer cell xenografts in immunodeficient mice. Using the established Lox-Stop-Lox K-ras mouse lung cancer model, we find that intranasal let-7 administration can reduce tumor formation in vivo in the lungs of animals expressing a G12D activating mutation for the K-ras oncogene. These findings support the notion that let-7 functions as a tumor suppressor in the lung and indicates that this miRNA could be used as a therapeutic agent to treat lung cancer.

The let-7 microrna reduces tumor growth in mouse models of lung cancer

As new fabrication and integration technologies reduce the cost and size of micro-sensors and wireless interfaces, it becomes feasible to deploy densely distributed wireless networks of sensors and actuators. These systems promise to revolutionize biological, earth, and environmental monitoring applications, providing data at granularities unrealizable by other means. In addition to the challenges of miniaturization, new system architectures and new network algorithms must be developed to transform the vast quantity of raw sensor data into a manageable stream of high-level data. To address this, we propose a tiered system architecture in which data collected at numerous, inexpensive sensor nodes is filtered by local processing on its way through to larger, more capable and more expensive nodes.We briefly describe Habitat monitoring as our motivating application and introduce initial system building blocks designed to support this application. The remainder of the paper presents details of our experimental platform.

Habitat monitoring: Application driver for wireless communications technology

Industry 4.0 and Health: Internet of Things, Big Data, and Cloud Computing for Healthcare 4.0

Bioinformatics is challenged by the fact that traditional analysis tools have difficulty in processing large-scale data from high-throughput sequencing. The open source Apache Hadoop project, which adopts the MapReduce framework and a distributed file system, has recently given bioinformatics researchers an opportunity to achieve scalable, efficient and reliable computing performance on Linux clusters and on cloud computing services. In this article, we present MapReduce frame-based applications that can be employed in the next-generation sequencing and other biological domains. In addition, we discuss the challenges faced by this field as well as the future works on parallel computing in bioinformatics.

/pdf/survey-of-mapreduce-frame-operation-in-bioinformatics-39zcyaboum.pdf

Survey of MapReduce frame operation in bioinformatics

MicroRNAs constitute an important class of noncoding, single-stranded, ~22 nucleotide long RNA molecules encoded by endogenous genes. They play an important role in regulating gene transcription and the regulation of normal development. MicroRNAs can be associated with disease; however, only a few microRNA-disease associations have been confirmed by traditional experimental approaches. We introduce two methods to predict microRNA-disease association. The first method, KATZ, focuses on integrating the social network analysis method with machine learning and is based on networks derived from known microRNA-disease associations, disease-disease associations, and microRNA-microRNA associations. The other method, CATAPULT, is a supervised machine learning method. We applied the two methods to 242 known microRNA-disease associations and evaluated their performance using leave-one-out cross-validation and 3-fold cross-validation. Experiments proved that our methods outperformed the state-of-the-art methods.

/pdf/prediction-of-microrna-disease-associations-based-on-social-3ver5wri6m.pdf

Prediction of MicroRNA-Disease Associations Based on Social Network Analysis Methods.

Finding the Best Classification Threshold in Imbalanced Classification

Biology is meaningful and important to identify cytokines and investigate their various functions and biochemical mechanisms. However, several issues remain, including the large scale of benchmark datasets, serious imbalance of data, and discovery of new gene families. In this paper, we employ the machine learning approach based on a novel ensemble classifier to predict cytokines. We directly selected amino acids sequences as research objects. First, we pretreated the benchmark data accurately. Next, we analyzed the physicochemical properties and distribution of whole amino acids and then extracted a group of 120-dimensional (120D) valid features to represent sequences. Third, in the view of the serious imbalance in benchmark datasets, we utilized a sampling approach based on the synthetic minority oversampling technique algorithm and K-means clustering undersampling algorithm to rebuild the training set. Finally, we built a library for dynamic selection and circulating combination based on clustering (LibD3C) and employed the new training set to realize cytokine classification. Experiments showed that the geometric mean of sensitivity and specificity obtained through our approach is as high as 93.3%, which proves that our approach is effective for identifying cytokines.

https://downloads.hindawi.com/journals/bmri/2013/686090.pdf

An approach for identifying cytokines based on a novel ensemble classifier.

Recent years there has been increasing concern about the rider demand responsive systems and the vehicular ad hoc networks. On one hand, centralised taxi platforms such as Uber and Didi Taxi are popular and changing our daily life; on the other hand, vehicles are equipped with more and more sensors and are capable to calculate, store, and communicate with other vehicles or road side units, forming vehicle-to-vehicle or vehicle-to-infrastructure communications. However, little effort has been devoted to integrating these two fields. In this paper, we propose a distributed public vehicle (PV) system that integrates the rider demand responsive system and ad hoc vehicular technologies, where the concept of fog computing and vehicular sensing are adopted for the system design. The challenges lie in that the PV scheduling problem itself is NP-hard, and careful design of scheduling and cooperation schemes among nodes are needed as they are ubiquitously connected at the edge of networks. The proposed PV system adopts a heuristic request insertion algorithm and a cooperative strategy among vehicle nodes, fog nodes, and the cloud to dispatch requests and to schedule routes for PVs. Experimental studies on real-world data sets demonstrate that the proposed scheme achieves higher service ratio of requests and better efficiency than other transit methods. Furthermore, the distributed vehicular sensing is demonstrated to be capable of collecting feasible metadata for scheduling applications. To the best of our knowledge, this paper is the first report on the integration of fog nodes and vehicular sensing for the rider request responsive scheduling systems.

Ziyu Lin

Papers

Survey of MapReduce frame operation in bioinformatics

Prediction of MicroRNA-Disease Associations Based on Social Network Analysis Methods.

Finding the Best Classification Threshold in Imbalanced Classification

An approach for identifying cytokines based on a novel ensemble classifier.

Distributed Public Vehicle System Based on Fog Nodes and Vehicular Sensing