[신간의 별자리x] 우리/미술, 그리고 ‘슬픔의 박물관’

Data clustering is one of the most popular techniques in data mining. It is a process of partitioning an unlabeled dataset into groups, where each group contains objects which are similar to each other with respect to a certain similarity measure and different from those of other groups. Clustering high-dimensional data is the cluster analysis of data which have anywhere from a few dozen to many thousands of dimensions. Such high-dimensional data spaces are often encountered in areas such as medicine, bioinformatics, biology, recommendation systems and the clustering of text documents. Many algorithms for large data sets have been proposed in the literature using different techniques. However, conventional algorithms have some shortcomings such as the slowness of their convergence and their sensitivity to initialization values. Particle Swarm Optimization (PSO) is a population-based globalized search algorithm that uses the principles of the social behavior of swarms. PSO produces better results in complicated and multi-peak problems. This paper presents a literature survey on the PSO algorithm and its variants to clustering high-dimensional data. An attempt is made to provide a guide for the researchers who are working in the area of PSO and high-dimensional data clustering.

A review on particle swarm optimization algorithm and its variants to clustering high-dimensional data

Detecting outliers is a significant problem that has been studied in various research and application areas. Researchers continue to design robust schemes to provide solutions to detect outliers efficiently. In this survey, we present a comprehensive and organized review of the progress of outlier detection methods from 2000 to 2019. First, we offer the fundamental concepts of outlier detection and then categorize them into different techniques from diverse outlier detection techniques, such as distance-, clustering-, density-, ensemble-, and learning-based methods. In each category, we introduce some state-of-the-art outlier detection methods and further discuss them in detail in terms of their performance. Second, we delineate their pros, cons, and challenges to provide researchers with a concise overview of each technique and recommend solutions and possible research directions. This paper gives current progress of outlier detection techniques and provides a better understanding of the different outlier detection methods. The open research issues and challenges at the end will provide researchers with a clear path for the future of outlier detection methods.

Progress in Outlier Detection Techniques: A Survey

Due to the growing volume of electronic payments, the monetary strain of credit-card fraud is turning into a substantial challenge for financial institutions and service providers, thus forcing them to continuously improve their fraud detection systems However, modern data-driven and learning-based methods, despite their popularity in other domains, only slowly find their way into business applications In this paper, we phrase the fraud detection problem as a sequence classification task and employ Long Short-Term Memory (LSTM) networks to incorporate transaction sequences We also integrate state-of-the-art feature aggregation strategies and report our results by means of traditional retrieval metrics A comparison to a baseline random forest (RF) classifier showed that the LSTM improves detection accuracy on offline transactions where the card-holder is physically present at a merchant Both the sequential and non-sequential learning approaches benefit strongly from manual feature aggregation strategies A subsequent analysis of true positives revealed that both approaches tend to detect different frauds, which suggests a combination of the two We conclude our study with a discussion on both practical and scientific challenges that remain unsolved

Sequence Classification for Credit-Card Fraud Detection

The k-Nearest Neighbors classifier is a simple yet effective widely renowned method in data mining. The actual application of this model in the big data domain is not feasible due to time and memory restrictions. Several distributed alternatives based on MapReduce have been proposed to enable this method to handle large-scale data. However, their performance can be further improved with new designs that fit with newly arising technologies.In this work we provide a new solution to perform an exact k-nearest neighbor classification based on Spark. We take advantage of its in-memory operations to classify big amounts of unseen cases against a big training dataset. The map phase computes the k-nearest neighbors in different training data splits. Afterwards, multiple reducers process the definitive neighbors from the list obtained in the map phase. The key point of this proposal lies on the management of the test set, keeping it in memory when possible. Otherwise, it is split into a minimum number of pieces, applying a MapReduce per chunk, using the caching skills of Spark to reuse the previously partitioned training set. In our experiments we study the differences between Hadoop and Spark implementations with datasets up to 11 million instances, showing the scaling-up capabilities of the proposed approach. As a result of this work an open-source Spark package is available.

kNN-IS

The diagnosability of a system is defined as the maximum number of faulty processors that the system can guarantee to identify, which plays an important role in measuring of the reliability of multiprocessor systems. In the work of Peng et al. in 2012, they proposed a new measure for fault diagnosis of systems, namely, $g$ -good-neighbor conditional diagnosability. It is defined as the diagnosability of a multiprocessor system under the assumption that every fault-free node contains at least $g$ fault-free neighbors, which can measure the reliability of interconnection networks in heterogeneous environments more accurately than traditional diagnosability. The $k$ -ary $n$ -cube is a family of popular networks. In this study, we first investigate and determine the $R_g$ -connectivity of $k$ -ary $n$ -cube for $0\le g\le n.$ Based on this, we determine the $g$ -good-neighbor conditional diagnosability of $k$ -ary $n$ -cube under the PMC model and MM* model for $k\ge 4, n\ge 3$ and $0\le g\le n.$ Our study shows the $g$ -good-neighbor conditional diagnosability of $k$ -ary $n$ -cube is several times larger than the classical diagnosability of $k$ -ary $n$ -cube.

/pdf/the-schmi-g-good-neighbor-conditional-diagnosability-of-15um3hpd5g.pdf

The ${\schmi g}$ -Good-Neighbor Conditional Diagnosability of ${\schmi k}$ -Ary ${\schmi n}$ -Cubes under the PMC Modeland MM* Model

Existing parallel mining algorithms for frequent itemsets lack a mechanism that enables automatic parallelization, load balancing, data distribution, and fault tolerance on large clusters. As a solution to this problem, we design a parallel frequent itemsets mining algorithm called FiDoop using the MapReduce programming model. To achieve compressed storage and avoid building conditional pattern bases, FiDoop incorporates the frequent items ultrametric tree, rather than conventional FP trees. In FiDoop, three MapReduce jobs are implemented to complete the mining task. In the crucial third MapReduce job, the mappers independently decompose itemsets, the reducers perform combination operations by constructing small ultrametric trees, and the actual mining of these trees separately. We implement FiDoop on our in-house Hadoop cluster. We show that FiDoop on the cluster is sensitive to data distribution and dimensions, because itemsets with different lengths have different decomposition and construction costs. To improve FiDoop’s performance, we develop a workload balance metric to measure load balance across the cluster’s computing nodes. We develop FiDoop-HD, an extension of FiDoop, to speed up the mining performance for high-dimensional data analysis. Extensive experiments using real-world celestial spectral data demonstrate that our proposed solution is efficient and scalable.

FiDoop: Parallel Mining of Frequent Itemsets Using MapReduce

In this paper, a novel trajectory clustering algorithm - TAD - is proposed to extract trajectory Stays based on spatial-temporal density analysis of data. Two new metrics - NMAST (Neighbourhood Move Ability and Stay Time) density function and NT (Noise Tolerance) factor - are defined in this algorithm. Firstly, NMAST integrates the characteristics of Neighbourhood Move Ability (NMA, extended from the concept of Move Ability MA), Stay Time (ST), and evaluation factor Eμ to measure the spatial-temporal density of data. Secondly, NT utilizes the features of noise to dynamically evaluate and reduce the influence of noise. The experimental results on Geolife dataset shows that the distributions hidden in data are extracted more realistically, especially for various complex or special trajectories with long-duration gaps. Furthermore, our analytical method of trajectory data is particularly applied in the spectra of LAMOST survey to analyse the variation characteristics of sky-background. The results show a regular distribution on observational date which is relatively concentrated in the month of 1, 10, 11, 12 in each year. The laws discovered in this work would provide a reasonable support for the designation of observational plans, and the new trajectory analysis method would also provide the services for the astronomical data analysis and then for the further studies of formation and evolution of the universe.

TAD: A trajectory clustering algorithm based on spatial-temporal density analysis

Traditional parallel algorithms for mining frequent itemsets aim to balance load by equally partitioning data among a group of computing nodes. We start this study by discovering a serious performance problem of the existing parallel Frequent Itemset Mining algorithms. Given a large dataset, data partitioning strategies in the existing solutions suffer high communication and mining overhead induced by redundant transactions transmitted among computing nodes. We address this problem by developing a data partitioning approach called FiDoop-DP using the MapReduce programming model. The overarching goal of FiDoop-DP is to boost the performance of parallel Frequent Itemset Mining on Hadoop clusters. At the heart of FiDoop-DP is the Voronoi diagram-based data partitioning technique, which exploits correlations among transactions. Incorporating the similarity metric and the Locality-Sensitive Hashing technique, FiDoop-DP places highly similar transactions into a data partition to improve locality without creating an excessive number of redundant transactions. We implement FiDoop-DP on a 24-node Hadoop cluster, driven by a wide range of datasets created by IBM Quest Market-Basket Synthetic Data Generator. Experimental results reveal that FiDoop-DP is conducive to reducing network and computing loads by the virtue of eliminating redundant transactions on Hadoop nodes. FiDoop-DP significantly improves the performance of the existing parallel frequent-pattern scheme by up to 31 percent with an average of 18 percent.

Jifu Zhang

Papers

The ${\schmi g}$ -Good-Neighbor Conditional Diagnosability of ${\schmi k}$ -Ary ${\schmi n}$ -Cubes under the PMC Modeland MM* Model

FiDoop: Parallel Mining of Frequent Itemsets Using MapReduce

TAD: A trajectory clustering algorithm based on spatial-temporal density analysis

FiDoop-DP: Data Partitioning in Frequent Itemset Mining on Hadoop Clusters

g-Good-neighbor conditional diagnosability measures for 3-ary n-cube networks