Data analysis plays an indispensable role for understanding various phenomena. Cluster analysis, primitive exploration with little or no prior knowledge, consists of research developed across a wide variety of communities. The diversity, on one hand, equips us with many tools. On the other hand, the profusion of options causes confusion. We survey clustering algorithms for data sets appearing in statistics, computer science, and machine learning, and illustrate their applications in some benchmark data sets, the traveling salesman problem, and bioinformatics, a new field attracting intensive efforts. Several tightly related topics, proximity measure, and cluster validation, are also discussed.

/pdf/survey-of-clustering-algorithms-6zezqhbjyo.pdf

Survey of clustering algorithms

From the Publisher:
Pattern recognition has long been studied in relation to many different (and mainly unrelated) applications, such as remote sensing, computer vision, space research, and medical imaging. In this book Professor Ripley brings together two crucial ideas in pattern recognition; statistical methods and machine learning via neural networks. Unifying principles are brought to the fore, and the author gives an overview of the state of the subject. Many examples are included to illustrate real problems in pattern recognition and how to overcome them.This is a self-contained account, ideal both as an introduction for non-specialists readers, and also as a handbook for the more expert reader.

Pattern recognition and neural networks

Clustering is the division of data into groups of similar objects. In clustering, some details are disregarded in exchange for data simplification. Clustering can be viewed as a data modeling technique that provides for concise summaries of the data. Clustering is therefore related to many disciplines and plays an important role in a broad range of applications. The applications of clustering usually deal with large datasets and data with many attributes. Exploration of such data is a subject of data mining. This survey concentrates on clustering algorithms from a data mining perspective.

A Survey of Clustering Data Mining Techniques

Methods for voting classification algorithms, such as Bagging and AdaBoost, have been shown to be very successful in improving the accuracy of certain classifiers for artificial and real-world datasets. We review these algorithms and describe a large empirical study comparing several variants in conjunction with a decision tree inducer (three variants) and a Naive-Bayes inducer. The purpose of the study is to improve our understanding of why and when these algorithms, which use perturbation, reweighting, and combination techniques, affect classification error. We provide a bias and variance decomposition of the error to show how different methods and variants influence these two terms. This allowed us to determine that Bagging reduced variance of unstable methods, while boosting methods (AdaBoost and Arc-x4) reduced both the bias and variance of unstable methods but increased the variance for Naive-Bayes, which was very stable. We observed that Arc-x4 behaves differently than AdaBoost if reweighting is used instead of resampling, indicating a fundamental difference. Voting variants, some of which are introduced in this paper, include: pruning versus no pruning, use of probabilistic estimates, weight perturbations (Wagging), and backfitting of data. We found that Bagging improves when probabilistic estimates in conjunction with no-pruning are used, as well as when the data was backfit. We measure tree sizes and show an interesting positive correlation between the increase in the average tree size in AdaBoost trials and its success in reducing the error. We compare the mean-squared error of voting methods to non-voting methods and show that the voting methods lead to large and significant reductions in the mean-squared errors. Practical problems that arise in implementing boosting algorithms are explored, including numerical instabilities and underflows. We use scatterplots that graphically show how AdaBoost reweights instances, emphasizing not only “hard” areas but also outliers and noise.

/pdf/an-empirical-comparison-of-voting-classification-algorithms-10wg9wglay.pdf

An Empirical Comparison of Voting Classification Algorithms: Bagging, Boosting, and Variants

This paper proposes an unsupervised algorithm for learning a finite mixture model from multivariate data. The adjective "unsupervised" is justified by two properties of the algorithm: 1) it is capable of selecting the number of components and 2) unlike the standard expectation-maximization (EM) algorithm, it does not require careful initialization. The proposed method also avoids another drawback of EM for mixture fitting: the possibility of convergence toward a singular estimate at the boundary of the parameter space. The novelty of our approach is that we do not use a model selection criterion to choose one among a set of preestimated candidate models; instead, we seamlessly integrate estimation and model selection in a single algorithm. Our technique can be applied to any type of parametric mixture model for which it is possible to write an EM algorithm; in this paper, we illustrate it with experiments involving Gaussian mixtures. These experiments testify for the good performance of our approach.

Unsupervised learning of finite mixture models

The invention relates generally to methods, systems, and devices for measuring the concentration of target analytes present in a biological system using a series of measurements obtained from a monitoring system and a Mixtures of Experts (MOE) algorithm. In one embodiment, the present invention describes a method for measuring blood glucose in a subject.

Method and device for predicting physiological values

A pure adversarial optical character recognition (OCR) approach in identifying text content in images. An image and a search term are input to a pure adversarial OCR module, which searches the image for presence of the search term. The image may be extracted from an email by an email processing engine. The OCR module may split the image into several character-blocks that each has a reasonable probability of containing a character (e.g., an ASCII character). The OCR module may form a sequence of blocks that represent a candidate match to the search term and calculate the similarity of the candidate sequence to the search term. The OCR module may be configured to output whether or not the search term is found in the image and, if applicable, the location of the search term in the image.

Pure adversarial approach for identifying text content in images

A technique for classifying a message is disclosed. The technique includes determining the domain from which the message is purported to be sent, determining an IP address from which the message was relayed at some point in its transmission, associating the domain with the IP address, and classifying the message based on the associated domain and IP address.

Message classification based on likelihood of spoofing

Unsup ervisedLearningUsingMMLJonathan J. OliverComputer Science Dept.Monash University, Clayton,Victoria, 3168, Australiajono@cs.monash.edu.auRohan A. Baxterrohan@cs.monash.edu.auChris S. Wallacecsw@cs.monash.edu.auAbstractThis pap er discusses the unsup ervised learn-ing problem. An imp ortant part of the unsu-p ervised learning problem is determining thenumb er of constituent groups (comp onents orclasses) which b est describ essome data.Weapply the Minimum Message Length (MML)criteriontotheunsup ervisedlearning prob-lem,mo difying anearliersuchMMLappli-cation.We give an empirical comparison ofcriteriaprominentintheliteratureforesti-mating the numb er of comp onents in a dataset.WeconcludethattheMinimum Mes-sageLengthcriterionp erformsb etterthanthe alternatives on thedata consideredherefor unsup ervised learning tasks.1INTRODUCTIONWe discuss the unsup ervised learning problem.Therearemanyapproachestounsup ervisedlearning.Within AI there have b een systems such as (a) CLUS-TER{MichalskiandStepp[16],(b)COBWEBFisher[13], (c) AQ17 { Wnek and Michalski29, (d)AUTOCLASS { Cheeseman et al.[7]and (e) Snob {Wallaceetal.[25;26]whichusesMML.Spath23,Page 7]de nes unsup ervised learning (or cluster anal-ysis) in the following way:\The ob jective of cluster analysis is to sepa-ratea setof ob jectsinto constituentgroups(classes,clumps, clusters)so thatthemem-b ers of one group di er from one another aslittle as p ossible, according to a chosen crite-rion".An imp ortant part of the unsup ervised learning prob-lem is determining thenumb er of constituentgroups(which we shall call comp onents for the remainder ofthis pap er) which b est describ es some data.SnobandAUTOCLASSareinthecategoryofmix-ture model lers[12; 24; 15].They assume that the dataunder consideration was generated from a distributionwhich is the sum of simpler distributions.That is, oursample,x1;2: : : ;nisassumedtoarisefromadistribution taking the formf(x)=kXj=1pj;wherefj(x) is a distribution (with a simple form), andpjis the relative weight off(x).We b egin with the univariate case and concentrate onmo dels wherethecomp onent distributions are Gaus-sian,i.e.,fj(x)N;2j).Whenwegeneraliseto the multidimensional case, we replace the variance,2j,byacovariancematrix.Themathematics fordiagonal covariance matrix is discussed in Section 5.Toformsuchamo del,weestimatek,thenumb erof comp onents, and the following parameters for eachcomp onent from 1 tok:j| the mean of comp onentj,j| the standard deviation of comp onentj, andpj| the relative weight of comp onentj.Avarietyofapproacheshasb eenusedtoestimatetheseparameters.Wolfe[30]and Day11give max-imum likeliho o d estimates forj;andpgivenk.Thestandardmaximum likeliho o d approachhasdif- cultyifwealloeachcomp onenttohavdistinct

Unsupervised Learning Using MML.

Cryptographic hashes such as MD5 and SHA-1 are used for many data mining and security applications -- they are used as an identifier for files and documents. However, if a single byte of a file is changed, then cryptographic hashes result in a completely different hash value. It would be very useful to work with hashes which identify that files were similar based on their hash values. The security field has proposed similarity digests, and the data mining community has proposed locality sensitive hashes. Some proposals include the Nilsimsa hash (a locality sensitive hash), Ssdeep and Sdhash (both Ssdeep and Sdhash are similarity digests). Here, we describe a new locality sensitive hashing scheme the TLSH. We provide algorithms for evaluating and comparing hash values and provide a reference to its open source code. We do an empirical evaluation of publically available similarity digest schemes. The empirical evaluation highlights significant problems with previously proposed schemes; the TLSH scheme does not suffer from the flaws identified.

Jonathan J. Oliver

Papers

Method and device for predicting physiological values

Pure adversarial approach for identifying text content in images

Message classification based on likelihood of spoofing

Unsupervised Learning Using MML.

TLSH -- A Locality Sensitive Hash