One of the key challenges of machine learning-based anomaly detection relies on the difficulty of obtaining anomaly data for training, which is usually rare, diversely distributed, and difficult to collect. To address this challenge, we formulate anomaly detection as a Positive and Unlabeled (PU) learning problem where only labeled positive (normal) data and unlabeled (normal and anomaly) data are required for learning an anomaly detector. As a semi-supervised learning method, it does not require providing labeled anomaly data for the training, thus it is easily deployed to various applications. As the unlabeled data can be extremely unbalanced, we introduce a novel PU learning method, which can tackle the situation where an unlabeled data set is mostly composed of positive instances. We start by using a linear model to extract the most reliable negative instances followed by a self-learning process to add reliable negative and positive instances with different speeds based on the estimated positive class prior. Furthermore, when feedback is available, we adopt boosting in the self-learning process to advantageously exploit the instability characteristic of PU learning. The classifiers in the self-learning process are weighted combined based on the estimated error rate to build the final classifier. Extensive experiments on six real datasets and one synthetic dataset show that our methods have better results under different conditions compared to existing methods.

Boosting Positive and Unlabeled Learning for Anomaly Detection With Multi-Features

Microfinance institutions (MFIs) play a considerable role in providing capital to micro businesses (MBs) through microcredit services. However, the interplay between MFIs and MB owners has been hindered by several factors, such as challenges with information sharing. The current study aims at identifying the specific challenges of microcredit services in Dar es Salaam, Tanzania, and also at determining the enabling factors, preferred features, and general requirements for a mobile technology solution to support the interplay between MFIs and MBs. The participants in the study were 91 MB owners and 22 MFI officers, and the data for the study were collected via a questionnaire. The study confirmed that the interaction, communication, and relationship between MB owners and MFIs is affected by various challenges. It was confirmed that digital technology can be used to address the identified challenges, and that a tailor-made mobile technology solution would be appropriate for supporting the interaction, communication, and relationship between MFIs and MBs. Gomera et al. User requirement for mobile microcredit services The African Journal of Information Systems, Volume 12, Issue 1, Article 1 2

Enabling Factors and User Requirements for Microcredit Services through Mobile Devices - Dar es Salaam Tanzania Context.

A survey of multi-label classification based on supervised and semi-supervised learning

With the rapid expansion of big data in all fields of science and engineering, multi-label classification has become a more acute problem in real-world data sets, it is notably difficult to develop an efficient model by using traditional supervised learning method because of labeled samples being scarce. In this paper, we proposes a multi-label k-nearest neighbor classification method based on semi-supervised learning (SSML-kNN in short). SSML-kNN firstly proposed a semi-supervised self-training model,and multi-label k-nearest neighbor classification based on correlation degree is used to classify the unlabeled datasets. After that, the training intermediate results with high confidence level are selected and added to the training data set. Meanwhile the training model is continuously recycled to expand the labeled data set. Finally, the test set is classified by the trained model. Experiments on publicly emotions datasets and yeasts datasets demonstrate that SSML-kNN achieves more prominent results than other related approaches across various evaluation metrics.

Multi-label K-Nearest Neighbor Classification Method Based on Semi-supervised

Anomaly detection in temporal data from sensors under aviation scenarios is a practical but challenging task: 1) long temporal data is difficult to extract contextual information with temporal correlation; 2) the anomalous data are rare in time series, causing normal/abnormal imbalance in anomaly detection, making the detector classification degenerate or even fail. To remedy the aforementioned problems, we propose a Graphical Temporal Data Analysis (GTDA) framework. It consists three modules, named Series-to-Image (S2I), Cluster-based Resampling Approach using Euclidean Distance (CRD) and Variance-Based Loss (VBL). Specifically, for better extracts global information in temporal data from sensors, S2I converts the data to curve images to demonstrate abnormalities in data changes. CRD and VBL balance the classification to mitigate the unequal distribution of classes. CRD extracts minority samples with similar features to majority samples by clustering and over-samples them. And VBL fine-tunes the decision boundary by balancing the fitting degree of the network to each class. Ablation experiments on the Flights dataset indicate the effectiveness of CRD and VBL on precision and recall, respectively. Extensive experiments demonstrate the synergistic advantages of CRD and VBL on F1-score on Flights and three other temporal datasets.

Imbalanced Aircraft Data Anomaly Detection

With massive increase in the amount of data being generated each day, we need automated tools to oversee the evolution of the web and to quantify global effects like pagerank of webpages. Search engines crawl the web every now and then to build web graphs which store information about the structure of the web. This is an expensive and error prone process. Central to this problem is the notion of graph similarity (between two graphs spaced in time), which validates how well search engines secure content from web and the quality of the search results they produce. In this paper, we propose two different types of anomalies which occur during crawling and two novel similarity measures based on vertex neighbourhood, which overcomes the proposed anomalies. Extensive experimentation on real world datasets shows significant improvement over state of art signature similarity based methods.

Anomaly detection in web graphs using vertex neighbourhood based signature similarity methods

Many real world machine learning tasks suffer from the problem of scarce labeled data. In multi-label learning, each instance is associated with more than one label as in semantic scene understanding, text categorization and bio-informatics. Semi-supervised multi-label learning has attracted recent interest as gathering labeled data is both expensive and requires manual effort. Further, many of the labels have semantic correlation which manifests as co-occurrence and this information can be used to build effective classifiers in the multi-label scenario. In this paper, we propose two different graph based transductive methods, namely, the label correlation propagation and the k-nearest neighbors based label correlation propagation. Extensive experimentation on real-world datasets demonstrates the efficacy of the proposed methods and the importance of using the label correlation information in semi-supervised multi-label learning.

/pdf/label-correlation-propagation-for-semi-supervised-multi-3a8f8vxndq.pdf

Label Correlation Propagation for Semi-supervised Multi-label Learning

Microfinance institutions aim at offering financial services to people in low-income category, who typically lack access to traditional banking systems. Till date, greater than 15 billion U.S dollars has been infused into microfinancing, assisting more than 160 million people in developing countries. With the tremendous growth in the World Wide Web, a number of microfinance institutions have recently moved online. One such noble initiative is KIVA, a crowd sourced online microfinance platform which connects borrowers (small entrepreneurs and individuals) to lenders through the field partners. One particular interest to such microfinancing institutions, is the analysis of the network of borrowers which can help them improve the percentage of loan requests fulfilled. KIVA provides a rich dataset capturing the lending activities on the website. In this paper, we analyze the data to find and extract the structure in the KIVA framework. We formulate a novel tripartite extension of SimRank using the network of lenders, loans and borrowers to capture the inherent pattern in the system. We also propose a Multipartite extension of SimRank useful for real world settings. Extensive experiments validate the effectiveness of our modeling and the proposed disambiguation scheme for borrowers.

Aritra Ghosh

Papers

Anomaly detection in web graphs using vertex neighbourhood based signature similarity methods

Label Correlation Propagation for Semi-supervised Multi-label Learning

Inferring borrower network in a microfinancing framework (KIVA)