The primary goal of pattern recognition is supervised or unsupervised classification. Among the various frameworks in which pattern recognition has been traditionally formulated, the statistical approach has been most intensively studied and used in practice. More recently, neural network techniques and methods imported from statistical learning theory have been receiving increasing attention. The design of a recognition system requires careful attention to the following issues: definition of pattern classes, sensing environment, pattern representation, feature extraction and selection, cluster analysis, classifier design and learning, selection of training and test samples, and performance evaluation. In spite of almost 50 years of research and development in this field, the general problem of recognizing complex patterns with arbitrary orientation, location, and scale remains unsolved. New and emerging applications, such as data mining, web searching, retrieval of multimedia data, face recognition, and cursive handwriting recognition, require robust and efficient pattern recognition techniques. The objective of this review paper is to summarize and compare some of the well-known methods used in various stages of a pattern recognition system and identify research topics and applications which are at the forefront of this exciting and challenging field.

/pdf/statistical-pattern-recognition-a-review-1kz1b0tte4.pdf

Statistical pattern recognition: a review

Planning algorithms are impacting technical disciplines and industries around the world, including robotics, computer-aided design, manufacturing, computer graphics, aerospace applications, drug design, and protein folding. This coherent and comprehensive book unifies material from several sources, including robotics, control theory, artificial intelligence, and algorithms. The treatment is centered on robot motion planning but integrates material on planning in discrete spaces. A major part of the book is devoted to planning under uncertainty, including decision theory, Markov decision processes, and information spaces, which are the “configuration spaces” of all sensor-based planning problems. The last part of the book delves into planning under differential constraints that arise when automating the motions of virtually any mechanical system. Developed from courses taught by the author, the book is intended for students, engineers, and researchers in robotics, artificial intelligence, and control theory as well as computer graphics, algorithms, and computational biology.

Planning Algorithms: Introductory Material

Gradient Boosting Decision Tree (GBDT) is a popular machine learning algorithm, and has quite a few effective implementations such as XGBoost and pGBRT. Although many engineering optimizations have been adopted in these implementations, the efficiency and scalability are still unsatisfactory when the feature dimension is high and data size is large. A major reason is that for each feature, they need to scan all the data instances to estimate the information gain of all possible split points, which is very time consuming. To tackle this problem, we propose two novel techniques: Gradient-based One-Side Sampling (GOSS) and Exclusive Feature Bundling (EFB). With GOSS, we exclude a significant proportion of data instances with small gradients, and only use the rest to estimate the information gain. We prove that, since the data instances with larger gradients play a more important role in the computation of information gain, GOSS can obtain quite accurate estimation of the information gain with a much smaller data size. With EFB, we bundle mutually exclusive features (i.e., they rarely take nonzero values simultaneously), to reduce the number of features. We prove that finding the optimal bundling of exclusive features is NP-hard, but a greedy algorithm can achieve quite good approximation ratio (and thus can effectively reduce the number of features without hurting the accuracy of split point determination by much). We call our new GBDT implementation with GOSS and EFB LightGBM. Our experiments on multiple public datasets show that, LightGBM speeds up the training process of conventional GBDT by up to over 20 times while achieving almost the same accuracy.

/pdf/lightgbm-a-highly-efficient-gradient-boosting-decision-tree-f835dtmi7b.pdf

LightGBM: a highly efficient gradient boosting decision tree

may 7th, 1986, Professor A. F. M. Smith in the Chair] SUMMARY A continuous two-dimensional region is partitioned into a fine rectangular array of sites or "pixels", each pixel having a particular "colour" belonging to a prescribed finite set. The true colouring of the region is unknown but, associated with each pixel, there is a possibly multivariate record which conveys imperfect information about its colour according to a known statistical model. The aim is to reconstruct the true scene, with the additional knowledge that pixels close together tend to have the same or similar colours. In this paper, it is assumed that the local characteristics of the true scene can be represented by a nondegenerate Markov random field. Such information can be combined with the records by Bayes' theorem and the true scene can be estimated according to standard criteria. However, the computational burden is enormous and the reconstruction may reflect undesirable largescale properties of the random field. Thus, a simple, iterative method of reconstruction is proposed, which does not depend on these large-scale characteristics. The method is illustrated by computer simulations in which the original scene is not directly related to the assumed random field. Some complications, including parameter estimation, are discussed. Potential applications are mentioned briefly.

https://stat.duke.edu/~scs/Courses/Stat376/Papers/GibbsFieldEst/BesagDirtyPicsJRSSB1986.pdf

On the statistical analysis of dirty pictures

/pdf/semi-supervised-learning-literature-survey-4ccxy3rmmj.pdf

Semi-Supervised Learning Literature Survey

To help better understand the problems of specifying data acquisition parameters and extracting desired information from the voluminous data, research has been focused on understanding the remote sensing process as a system and investigating the interrelated effects of various parameters. A system model is used to explore system parameter trade-offs for a model sensor based on the High-Resolution Imaging Spectrometer (HIRIS). Radiometric performance was studied, along with the effect on classification accuracy of several system parameters. The atmosphere and sensor have significant effects on the mean received signal and noise performance. The effect of random uncorrelated errors in the radiometric calibration of the detector array is discussed. Accurate pixel-to-pixel relative radiometric calibration and the use of the image motion compensation (IMC) option are seen to improve classification accuracy. The selection of feature sets based on combining spectral bands was studied under a variety of observational conditions. >

Parameter trade-offs for imaging spectroscopy systems (remote sensing)

In Gaussian maximum likelihood classification, the mean vector and covariance matrix are usually estimated from training samples. When the training sample size is small compared to dimensionality, the sample estimates, especially the covariance matrix becomes highly variable and consequently, the classifier performs poorly. In particular, if the number of training samples is less than dimensionality, the sample covariance estimate becomes singular so the quadratic classifier cannot be applied. Unfortunately, the problem of limited training samples is prevalent in remote sensing applications. While the recent progress in sensor technology has increased the number of spectral features making possible more classes to be identified, the training data remain expensive and difficult to acquire. In this work, the problem of small training set size on the classification performance is addressed by introducing a covariance estimation method for limited training samples. The proposed approach can be viewed as an intermediate method between linear and quadratic classifiers by selecting an appropriate mixture of covariance matrices. The mixture of covariance matrices is formulated under an empirical Bayesian setting which is advantageous when the training sample size reflects the prior of each class. The experimental results show that the proposed method improves the classification performance when training sample sizes are limited.

https://engineering.purdue.edu/~landgreb/saldju_bLOOC_comm.pdf

Covariance estimation for limited training samples

Research in recent years into methods for hyperspectral image data analysis has shown that there is a strong relationship between the precision with which classes are defined and the classification accuracy that results. There is also a relationship between these two factors and the complexity of the classifier algorithm used in the analysis. The authors illustrate this relationship empirically using a moderate dimensional, moderately difficult classification task. This example is also used to explore the effect of two recently introduced algorithms that are intended to mitigate the effect of use of a limited number of training samples on classifier performance. The results tend to confirm the theory with regard to training sample size vs. classifier complexity. They also show the two algorithms to be moderately useful in improving classifier performance when training data is limited.

On the relationship between class definition precision and classification accuracy in hyperspectral analysis

Advances in sensor technology have increased the spectral resolution of remote sensing data significantly. Higher spectral resolution for each pixel should make possible the discrimination of a larger number of classes in more detail. However, due to the scarcity of training samples in remote sensing applications, the increase in spectral dimensionality only complicates the design of classifiers which, if not properly done, may cause the deterioration of classification accuracy. In this work, we propose a new design procedure for a hybrid decision tree classifier which improves the classification efficiency and accuracy for classifying high-dimensional data with a small training sample size. We further propose to use a feature extraction technique based on maximizing the statistical distance between two subgroups. Experimental results show that the proposed tree classifier is more effective in classifying high-dimensional data with limited training samples than a single-layer classifier and a previously proposed hybrid tree classifier.

A decision tree classifier design for high-dimensional data with limited training samples

A spatio-temporal contextual classifier that can utilize both spatial and temporal information is investigated. Experiments carried out with Landsat TM data are reported. They show that spatial correlation contexts are more useful than the other contexts. The use of the homogeneity test followed by a selective application of the contextual rule is more effective than the totally recursive case in the sense of both classification accuracy and computation. Classification performance is compared with that of the maximum-likelihood classifier and the ECHO (extraction and classification of homogeneous objects) classifier. >

David A. Landgrebe

Papers

Parameter trade-offs for imaging spectroscopy systems (remote sensing)

Covariance estimation for limited training samples

On the relationship between class definition precision and classification accuracy in hyperspectral analysis

A decision tree classifier design for high-dimensional data with limited training samples

Spatio-temporal contextual classification of remotely sensed multispectral data