Fast and stable clustering analysis based on Grid-mapping K-means algorithm and new clustering validity index

Document clustering is the partitioning of a given collection of documents into various K- groups based on some similarity/dissimilarity criterion. This task has applications in scope detection of journals/conferences, development of some automated peer-review support systems, topic-modeling, latest cognitive-inspired works on text summarization, and classification of documents based on semantics, etc. In the current paper, a cognitive-inspired multi-objective automatic document clustering technique is proposed which is a fusion of self-organizing map (SOM) and multi-objective differential evolution approach. The variable number of cluster centers are encoded in different solutions of the population to determine the number of clusters from a data set in an automated way. These solutions undergo various genetic operations during evolution. The concept of SOM is utilized in designing new genetic operators for the proposed clustering technique. In order to measure the goodness of a clustering solution, two cluster validity indices, Pakhira-Bandyopadhyay-Maulik index, and Silhouette index, are optimized simultaneously. The effectiveness of the proposed approach, namely self-organizing map based multi-objective document clustering technique (SMODoc_clust) is shown in automatic classification of some scientific articles and web-documents. Different representation schemas including tf, tf-idf and word-embedding are employed to convert articles in vector-forms. Comparative results with respect to internal cluster validity indices, namely, Dunn index and Davies-Bouldin index, are shown against several state-of-the-art clustering techniques including three multi-objective clustering techniques namely MOCK, VAMOSA, NSGA-II-Clust, single objective genetic algorithm (SOGA) based clustering technique, K-means, and single-linkage clustering. Results obtained clearly show that our approach is better than existing approaches. The validation of the obtained results is also shown using statistical significant t tests.

Automatic Scientific Document Clustering Using Self-organized Multi-objective Differential Evolution

Test scenario generation for testing automated and autonomous driving systems requires knowledge about the recurring traffic cases, known as scenario types. The most common approach in industry is to have experts create lists of scenario types. This poses the risk both that certain types are overlooked; and that the mental model that underlies the manual process is inadequate. We propose to extract scenario types from real driving data by clustering recorded scenario instances, which are composed of timeseries. Existing works in the domain of traffic data either cannot cope with multivariate timeseries; are limited to one or two vehicles per scenario instance; or they use handcrafted features that are based on the mental model of the data scientist. The latter suffers from similar shortcomings as manual scenario type derivation. Our approach clusters scenario instances relying as little as possible on a mental model. As such, we consider the approach an important complement to manual scenario type derivation. It may yield scenario types overlooked by the experts, and it may provide a different segmentation of a whole set of scenarios instances into scenario types, thus overall increasing confidence in the handcrafted scenario types. We present the application of the approach to a real driving dataset.

Clustering Traffic Scenarios Using Mental Models as Little as Possible

A cluster validity index is used to select which clustering algorithm to apply for a given problem. It works by evaluating the quality of a partition, as output by a candidate clustering algorithm, getting around the common case of the lack of an expert in the given domain of discourse. Most existing validity indexes make assumptions, such as each cluster of the partition having an underlying structure, for example, a hypersphere, yielding incorrect evaluations when they do not hold. Here, we propose a new cluster validity index, which attempts to avoid this bias using an ensemble of distinct supervised classifiers; this way the bias is not attributable to a specific classifier, but to a collection thereof, hence alleviating the problem. The rationale behind our index is that a good partition should induce the construction of also a good classifier; the better the classification performance, the better the quality of the partition under evaluation. Notice how we use the partition to be assessed as a sort of labeled dataset, where each object is labeled with the cluster label it belongs to. We have tested our index on 50 numerical datasets, grouped using six different clustering algorithms. In our experiments, our index outperforms five validity indexes, including the most popular ones.

Cluster validation using an ensemble of supervised classifiers

In this paper we present a modified Bag-of-Words algorithm used in image classification The classic Bag-of-Words algorithm is used in natural language processing A text (such as a sentence or a document) is represented as a bag of words In image retrieval or image classification this algorithm also works on one characteristic image feature and most often it is a descriptor defining the surrounding of a keypoint obtained by using eg the SURF algorithm The modification which we have introduced involves using two different types of image features – the descriptor of a keypoint and also the colour histogram, which can be obtained from the surrounding of a keypoint This additional feature will make it possible to obtain more information as the commonly used SURF algorithm works only on images with greyscale intensity The experiments which we have conducted show that using this additional image feature significantly improves image classification results by using the BoW algorithm

The Image Classification with Different Types of Image Features

In this paper, a new cluster validity index which can be considered as a measure of the accuracy of the partitioning of data sets is proposed. The new index, called the STR index, is defined as the product of two components which determine changes of compactness and separability of clusters during a clustering process. The maximum value of this index identifies the best clustering scheme. Three popular algorithms have been applied as underlying clustering techniques, namely complete-linkage, expectation maximization and K-means algorithms. The performance of the new index is demonstrated for several artificial and real-life data sets. Moreover, this new index has been compared with other well-known indices, i.e., Dunn, Davies-Bouldin, PBM and Silhouette indices, taking into account the number of clusters in a data set as the comparison criterion. The results prove superiority of the new index as compared to the above-mentioned indices.

/pdf/a-new-validity-index-for-crisp-clusters-1a3th36krk.pdf

A new validity index for crisp clusters

This article provides the performance evaluation of the Silhouette index, which is based on the so called silhouette width. However, the index can be calculated in two ways, and so, the first approach uses the mean of the mean silhouettes through all the clusters. On the other hand, the second one is realized by averaging the silhouettes over the whole data set. These various approaches of the index have significant influence on indicating the proper number of clusters in a data set. To study the performance of the index, as the underlying clustering algorithms, two popular hierarchical methods were applied, that is, the complete-linkage and the single-linkage algorithm. These methods have been used for artificial and real-life data sets, and the results confirm very good performances of the index and they also allow to choose the best approach.

Performance Evaluation of the Silhouette Index

Abstract Clustering is an attractive technique used in many fields in order to deal with large scale data. Many clustering algorithms have been proposed so far. The most popular algorithms include density-based approaches. These kinds of algorithms can identify clusters of arbitrary shapes in datasets. The most common of them is the Density-Based Spatial Clustering of Applications with Noise (DBSCAN). The original DBSCAN algorithm has been widely applied in various applications and has many different modifications. However, there is a fundamental issue of the right choice of its two input parameters, i.e the eps radius and the MinPts density threshold. The choice of these parameters is especially difficult when the density variation within clusters is significant. In this paper, a new method that determines the right values of the parameters for different kinds of clusters is proposed. This method uses detection of sharp distance increases generated by a function which computes a distance between each element of a dataset and its k-th nearest neighbor. Experimental results have been obtained for several different datasets and they confirm a very good performance of the newly proposed method.

/pdf/a-new-method-for-automatic-determining-of-the-dbscan-4h7e62olnr.pdf

A New Method for Automatic Determining of the DBSCAN Parameters

Clustering is an attractive technique used in many fields and lots of clustering algorithms have been proposed so far. The Density-Based Spatial Clustering of Applications with Noise (DBSCAN) is one of the most popular algorithms, which has been widely applied in many different applications. This algorithm can discover clusters of arbitrary shapes in large datasets. However, the fundamental issue is the right choice of two input parameters, i.e. radius eps and density threshold MinPts. In this paper, a new method is proposed to determine the value of eps. The suggested approach is based on an analysis of the sorted values of the distance function. The performance of the new approach has been demonstrated for several different datasets.

Determining the Eps Parameter of the DBSCAN Algorithm

In this paper a modification of the well-known Silhouette validity index is proposed. This index, which can be considered a measure of the data set partitioning accuracy, enjoys significant popularity and is often used by researchers. The proposed modification involves using an additional component in the original index. This approach improves performance of the index and provides better results during a clustering process, especially when changes of cluster separability are big. The new version of the index is called the SILA index and its maximum value identifies the best clustering scheme. The performance of the new index is demonstrated for several data sets, where the popular algorithm has been applied as underlying clustering techniques, namely the Complete–linkage algorithm. The results prove superiority of the new approach as compared to the original Silhouette validity index.

Artur Starczewski

Papers

A new validity index for crisp clusters

Performance Evaluation of the Silhouette Index

A New Method for Automatic Determining of the DBSCAN Parameters

Determining the Eps Parameter of the DBSCAN Algorithm

A Modification of the Silhouette Index for the Improvement of Cluster Validity Assessment