We explore the effect of dimensionality on the nearest neighbor problem. We show that under a broad set of conditions (much broader than independent and identically distributed dimensions), as dimensionality increases, the distance to the nearest data point approaches the distance to the farthest data point. To provide a practical perspective, we present empirical results on both real and synthetic data sets that demonstrate that this effect can occur for as few as 10-15 dimensions. These results should not be interpreted to mean that high-dimensional indexing is never meaningful; we illustrate this point by identifying some high-dimensional workloads for which this effect does not occur. However, our results do emphasize that the methodology used almost universally in the database literature to evaluate high-dimensional indexing techniques is flawed, and should be modified. In particular, most such techniques proposed in the literature are not evaluated versus simple linear scan, and are evaluated over workloads for which nearest neighbor is not meaningful. Often, even the reported experiments, when analyzed carefully, show that linear scan would outperform the techniques being proposed on the workloads studied in high (10-15) dimensionality!.

When is nearest neighbor meaningful

The growth of cancerous cells in lungs is called lung cancer. The mortality rate of both men and women has expanded due to the increasing rate of incidence of cancer. Lung cancer is a disease where cells in the lungs multiply uncontrollably. Lung cancer cannot be prevented but its risk can be reduced. So detection of lung cancer at the earliest is crucial for the survival rate of patients. The number of chain- smokers is directly proportional to the number of people affected with lung cancer. The lung cancer prediction was analysed using classification algorithms such as Naive Bayes, SVM, Decision tree and Logistic Regression. The keyobjective of this paper is the early diagnosis of lung cancer by examining the performance of classification algorithms.

A Comparative Study of Lung Cancer Detection using Machine Learning Algorithms

A simple and fast k-medoids algorithm that updates medoids by minimizing the total distance within clusters has been developed. Although it is simple and fast, as its name suggests, it nonetheless has neglected local optima and empty clusters that may arise. With the distance as an input to the algorithm, a generalized distance function is developed to increase the variation of the distances, especially for a mixed variable dataset. The variation of the distances is a crucial part of a partitioning algorithm due to different distances producing different outcomes. The experimental results of the simple k-medoids algorithm produce consistently good performances in various settings of mixed variable data. It also has a high cluster accuracy compared to other distance-based partitioning algorithms for mixed variable data.

https://www.mdpi.com/1999-4893/12/9/177/pdf

Simple K-Medoids Partitioning Algorithm for Mixed Variable Data

Chatbot is software that communicates using natural language. chatbots such as machine conversation systems, Chatterbot, virtual agents, and dialogue systems. This software enables to simulate human conversations. In this research, the chatbot system that will be created must be able to understand the natural language of what is entered by the user, and the chatbot will answer according to what the user is expecting. The researcher proposes a classification method to identify intent rather than user input or called intent classification on the chatbot system; the researcher also wants to know the level of accuracy, precision, and recall on the evaluation results of both methods. The classification method applied in this research is the Naive Bayes method and compared with the Logistic Regression method to determine the class intention. The evaluation results show the level of accuracy precision and recall in the Logistic Regression model is higher than the Naive Bayes model.

Comparison Of Multinomial Naive Bayes Algorithm And Logistic Regression For Intent Classification In Chatbot

Appropriate training data is a requirement for building good machine-learned models. In this paper, we study the notion of coverage for ordinal and continuous-valued attributes, by formalizing the intuition that the learned model can accurately predict only at data points for which there are "enough" similar data points in the training data set. We develop an efficient algorithm to identify uncovered regions in low-dimensional attribute feature space, by making a connection to Voronoi diagrams. We also develop a randomized approximation algorithm for use in high-dimensional attribute space. We evaluate our algorithms through extensive experiments on real datasets.

Identifying Insufficient Data Coverage for Ordinal Continuous-Valued Attributes

Recent years have explored various clustering strategies to partition datasets comprising of heterogeneous domains or types such as categorical, numerical and binary. Clustering algorithms seek to identify homogeneous groups of objects based on the values of their attributes. These algorithms either assume the attributes to be of homogeneous types or are converted into homogeneous types. However, datasets with heterogeneous data types are common in real life applications, which if converted, can lead to loss of information. This paper proposes a new similarity measure in the form of triplet to find the distance between two data objects with heterogeneous attribute types. A new k-medoid type of clustering algorithm is proposed by leveraging the similarity measure in the form of a vector. The proposed k-medoid type of clustering algorithm is compared with traditional clustering algorithms, based on cluster validation using Purity Index and Davies Bouldin index. Results show that the new clustering algorithm with new similarity measure outperforms the k-means clustering for mixed datasets.

K-Medoid Clustering for Heterogeneous DataSets☆

Apriori is one of the best algorithms for learning association rules. Due to the explosion of data, the storage and retrieval mechanisms in various database paradigms have revolutionized the technologies and methodologies used in the architecture. As a result, the database is not only utilized for mere information retrieval but also to infer the analytical aspect of data. Therefore it is essential to find association rules from high dimensional data because the correlation amongst the attributes can help in gaining deeper insight into the data and help in decision making, recommendations as well as reorganizing the data for effective retrieval. The traditional Apriori algorithm is computationally expensive and infeasible with high dimensional datasets. Hence we propose a variant of Apriori algorithm using the concept of QR decomposition for reducing the dimensions thereby reducing the complexity of the traditional Apriori algorithm.

Apriori algorithm for association rule mining in high dimensional data

The context of this paper is to come up with an analytical query model for data categorization within DBMS. DBMS being the asset for most of the organizations, classification can help in getting better insight and control over the data. Conventionally, classification algorithms like logistic regression, KNN, etc. are applied after exporting the data out of DBMS, using non DBMS tools like R, matrix packages, generic data mining programs or large scale systems like Hadoop and Spark. However, this leads to I/O overhead since the data within DBMS is updated quite frequently and usually cannot be accommodated in the main memory. This paper proposes an alternative strategy, based on SQL and UDFs, to integrate the logistic regression for data categorization as well as prediction query processing within DBMS. A comparison of SQL with user defined functions (UDFs) as well as with statistical packages like R is presented, by experimentation on real datasets. The empirical results show the viability and validity of this approach for predicting the class of a given query.

Logistic regression within DBMS

Due to the ever growing needs of managing huge volume of data, together with the desire for consistent, scalable, reliable and efficient retrieval of information, an intelligent mechanism to design the storage structure for distributing the databases has become inevitable. The two critical facets of distributed databases are data fragmentation and allocation. Existing fragmentation techniques are based on the frequency and type of the queries as well as the statistics of the empirical data. However, very limited work is done to fragment the data based on the pattern of the tuples and the attributes responsible for such patterns. This paper presents a unique approach towards hybridized fragmentation, by applying subspace clustering algorithm, to come up with a set of fragments which partitions the data with respect to tuples as well as attributes. Projected clustering is the one that determines the clusters in the subspaces of high dimensional data. This concept leads to find the closely correlated attributes for different sets of instances thereby giving good hybridized fragments for distributed databases. Experimental results show that fragmenting the database based on clustering, results in reduced database access time as compared to the fragments chosen at design time using certain statistics.

Hybridized fragmentation of very large databases using clustering

Epilepsy is a common neurological disease that affects more than 2 percent of the population globally. An imbalance in brain electrical activities causes unpredictable seizures, which eventually leads to epilepsy. Neurostimulators have the power to intervene in advance and avoid the occurrence of seizures. Its efficiency can be increased with the help of heuristics like advanced seizure prediction. Early identification of preictal state will help easy activation of neurostimulator on time. This research concentrates on the performance analysis of various machine learning algorithms on recorded EEG data. Through this study, we aim to find the best model, which can be used to create an ensemble model for better learning. This involves modeling and simulation of classical machine learning technique like Logistic regression, Naive Bayes model, K nearest neighbors Random Forest, and deep learning techniques like an Artificial neural network, Convolutional neural networks, Long short term memory, and Autoencoders. In this analysis, Random Forest and Long Short-Term Memory performed well among all models in terms of sensitivity and specificity.

https://iopscience.iop.org/article/10.1088/1742-6596/2161/1/012055/pdf

Sandhya Harikumar

Papers

K-Medoid Clustering for Heterogeneous DataSets☆

Apriori algorithm for association rule mining in high dimensional data

Logistic regression within DBMS

Hybridized fragmentation of very large databases using clustering

Comparative analysis of multiple machine learning algorithms for epileptic seizure prediction