Journal ArticleDOI
Minimum Cross-Entropy Pattern Classification and Cluster Analysis
John E. Shore,Robert M. Gray +1 more
Reads0
Chats0
TLDR
The approach is a generalization of a recently developed speech coding technique called speech coding by vector quantization based on the minimization of cross-entropy, and can be viewed as a refinement of a general classification method due to Kullback.Abstract:
This paper considers the problem of classifying an input vector of measurements by a nearest neighbor rule applied to a fixed set of vectors. The fixed vectors are sometimes called characteristic feature vectors, codewords, cluster centers, models, reproductions, etc. The nearest neighbor rule considered uses a non-Euclidean information-theoretic distortion measure that is not a metric, but that nevertheless leads to a classification method that is optimal in a well-defined sense and is also computationally attractive. Furthermore, the distortion measure results in a simple method of computing cluster centroids. Our approach is based on the minimization of cross-entropy (also called discrimination information, directed divergence, K-L number), and can be viewed as a refinement of a general classification method due to Kullback. The refinement exploits special properties of cross-entropy that hold when the probability densities involved happen to be minimum cross-entropy densities. The approach is a generalization of a recently developed speech coding technique called speech coding by vector quantization.read more
Citations
More filters
Journal Article
Vector quantization
TL;DR: During the past few years several design algorithms have been developed for a variety of vector quantizers and the performance of these codes has been studied for speech waveforms, speech linear predictive parameter vectors, images, and several simulated random processes.
Journal ArticleDOI
Distance measures for signal processing and pattern recognition
TL;DR: Some classical results about error bounds in classification and feature selection for pattern recognition are recalled, which are obtained with the aid of properties of distance measures.
Journal ArticleDOI
On measuring the distance between histograms
Sung-Hyuk Cha,Sargur N. Srihari +1 more
TL;DR: The proposed distance measure has the advantage over the traditional distance measures regarding the overlap between two distributions; it takes the similarity of the non-overlapping parts into account as well as that of overlapping parts.
Journal ArticleDOI
Properties of cross-entropy minimization
John E. Shore,R. Johnson +1 more
TL;DR: The principle of minimum cross-entropy (minimum directed divergence, minimum discrimination information) is a general method of inference about an unknown probability density when there exists a prior estimate of the density and new information in the form of constraints on expected values.
Journal ArticleDOI
Optimal partitioning for classification and regression trees
TL;DR: An iterative algorithm that finds a locally optimal partition for an arbitrary loss function, in time linear in N for each iteration, is presented and it is proven that the globally optimal partition must satisfy a nearest neighbour condition using divergence as the distance measure.
References
More filters
Some methods for classification and analysis of multivariate observations
TL;DR: The k-means algorithm as mentioned in this paper partitions an N-dimensional population into k sets on the basis of a sample, which is a generalization of the ordinary sample mean, and it is shown to give partitions which are reasonably efficient in the sense of within-class variance.
Journal ArticleDOI
Information Theory and Statistical Mechanics. II
TL;DR: In this article, the authors consider statistical mechanics as a form of statistical inference rather than as a physical theory, and show that the usual computational rules, starting with the determination of the partition function, are an immediate consequence of the maximum-entropy principle.
Journal ArticleDOI
An Algorithm for Vector Quantizer Design
Y. Linde,A. Buzo,Robert M. Gray +2 more
TL;DR: An efficient and intuitive algorithm is presented for the design of vector quantizers based either on a known probabilistic model or on a long training sequence of data.
Journal ArticleDOI
Axiomatic derivation of the principle of maximum entropy and the principle of minimum cross-entropy
John E. Shore,R. Johnson +1 more
TL;DR: Jaynes's principle of maximum entropy and Kullbacks principle of minimum cross-entropy (minimum directed divergence) are shown to be uniquely correct methods for inductive inference when new information is given in the form of expected values.