The primary goal of pattern recognition is supervised or unsupervised classification. Among the various frameworks in which pattern recognition has been traditionally formulated, the statistical approach has been most intensively studied and used in practice. More recently, neural network techniques and methods imported from statistical learning theory have been receiving increasing attention. The design of a recognition system requires careful attention to the following issues: definition of pattern classes, sensing environment, pattern representation, feature extraction and selection, cluster analysis, classifier design and learning, selection of training and test samples, and performance evaluation. In spite of almost 50 years of research and development in this field, the general problem of recognizing complex patterns with arbitrary orientation, location, and scale remains unsolved. New and emerging applications, such as data mining, web searching, retrieval of multimedia data, face recognition, and cursive handwriting recognition, require robust and efficient pattern recognition techniques. The objective of this review paper is to summarize and compare some of the well-known methods used in various stages of a pattern recognition system and identify research topics and applications which are at the forefront of this exciting and challenging field.

/pdf/statistical-pattern-recognition-a-review-1kz1b0tte4.pdf

Statistical pattern recognition: a review

Planning algorithms are impacting technical disciplines and industries around the world, including robotics, computer-aided design, manufacturing, computer graphics, aerospace applications, drug design, and protein folding. This coherent and comprehensive book unifies material from several sources, including robotics, control theory, artificial intelligence, and algorithms. The treatment is centered on robot motion planning but integrates material on planning in discrete spaces. A major part of the book is devoted to planning under uncertainty, including decision theory, Markov decision processes, and information spaces, which are the “configuration spaces” of all sensor-based planning problems. The last part of the book delves into planning under differential constraints that arise when automating the motions of virtually any mechanical system. Developed from courses taught by the author, the book is intended for students, engineers, and researchers in robotics, artificial intelligence, and control theory as well as computer graphics, algorithms, and computational biology.

Planning Algorithms: Introductory Material

Gradient Boosting Decision Tree (GBDT) is a popular machine learning algorithm, and has quite a few effective implementations such as XGBoost and pGBRT. Although many engineering optimizations have been adopted in these implementations, the efficiency and scalability are still unsatisfactory when the feature dimension is high and data size is large. A major reason is that for each feature, they need to scan all the data instances to estimate the information gain of all possible split points, which is very time consuming. To tackle this problem, we propose two novel techniques: Gradient-based One-Side Sampling (GOSS) and Exclusive Feature Bundling (EFB). With GOSS, we exclude a significant proportion of data instances with small gradients, and only use the rest to estimate the information gain. We prove that, since the data instances with larger gradients play a more important role in the computation of information gain, GOSS can obtain quite accurate estimation of the information gain with a much smaller data size. With EFB, we bundle mutually exclusive features (i.e., they rarely take nonzero values simultaneously), to reduce the number of features. We prove that finding the optimal bundling of exclusive features is NP-hard, but a greedy algorithm can achieve quite good approximation ratio (and thus can effectively reduce the number of features without hurting the accuracy of split point determination by much). We call our new GBDT implementation with GOSS and EFB LightGBM. Our experiments on multiple public datasets show that, LightGBM speeds up the training process of conventional GBDT by up to over 20 times while achieving almost the same accuracy.

/pdf/lightgbm-a-highly-efficient-gradient-boosting-decision-tree-f835dtmi7b.pdf

LightGBM: a highly efficient gradient boosting decision tree

may 7th, 1986, Professor A. F. M. Smith in the Chair] SUMMARY A continuous two-dimensional region is partitioned into a fine rectangular array of sites or "pixels", each pixel having a particular "colour" belonging to a prescribed finite set. The true colouring of the region is unknown but, associated with each pixel, there is a possibly multivariate record which conveys imperfect information about its colour according to a known statistical model. The aim is to reconstruct the true scene, with the additional knowledge that pixels close together tend to have the same or similar colours. In this paper, it is assumed that the local characteristics of the true scene can be represented by a nondegenerate Markov random field. Such information can be combined with the records by Bayes' theorem and the true scene can be estimated according to standard criteria. However, the computational burden is enormous and the reconstruction may reflect undesirable largescale properties of the random field. Thus, a simple, iterative method of reconstruction is proposed, which does not depend on these large-scale characteristics. The method is illustrated by computer simulations in which the original scene is not directly related to the assumed random field. Some complications, including parameter estimation, are discussed. Potential applications are mentioned briefly.

https://stat.duke.edu/~scs/Courses/Stat376/Papers/GibbsFieldEst/BesagDirtyPicsJRSSB1986.pdf

On the statistical analysis of dirty pictures

/pdf/semi-supervised-learning-literature-survey-4ccxy3rmmj.pdf

Semi-Supervised Learning Literature Survey

The main purpose of this work is to find an improved regularized covariance estimator of each class with the advantages of LOOC, and BLOOC, which are useful for high dimensional pattern recognition problems. The searching ranges of LOOC and BLOOC are between the linear combinations of three pair covariance estimators. The first proposed covariance estimator (mixed-LOOC1) extended the searching range and is a general case of LOOC and BLOOC. By observing that the optimal value of leave-one-out likelihood function of LOOC usually occurs at near the end point of the parameter domain, the second covariance estimator (mixed-LOOC2), which needs less computation, was proposed. Using the proposed covariance estimator to improve the linear feature extraction methods when the multivariate data are singular or nearly so is demonstrated.

https://engineering.purdue.edu/~landgreb/Kuo2.IGARSS02.pdf

Regularized covariance estimators for hyperspectral data classification and its application to feature extraction

A very important technological development in sensor technology is the ability to construct multispectral imaging sensors with very large numbers (/spl ap/200) of spectral bands. A significant number of such hyperspectral sensor systems are now in use or under various stages of development in various countries. The existence of such sensors raises the question of how best to analyze the data with so many spectral bands. Though existing conventional multispectral analysis methods will still be useful in the hyperspectral era ahead, it becomes clear that they fall short of the ideal in several senses. For example, conventional methods often prove computationally unreasonable for large numbers of bands, they may not enable the extraction of all the information from the data that might otherwise be possible, and they may prove excessively cumbersome for researchers in their work. In the paper, after a brief introduction, some of the basic characteristics fundamental to the analysis process are outlined. This is followed by an example showing how they may be applied to hyperspectral data analysis. >

A perspective on the analysis of hyperspectral data

A focused research program has been under way for several years to discover optimally effective means for analysis of multispectral and hyperspectral data. The methods pursued are based upon fundamental principles of signal theory and signal processing. The basic approach revolves around viewing N spectral bands of data from a pixel as a single point in N dimensional space, thus, an important aspect of the work has been to discover unique aspects of higher dimensional spaces which can be exploited for their information-bearing aspects. Substantial progress on this problem has been made in the last several years, with several key algorithms having been defined. Among these are algorithms for transforms which define optimal case-specific features, and which improve the ability of the classifier to generalize. A more fundamental finding has been to understand the characteristics of high dimensional space and the significance of design samples and their use in defining the classifier. These results have been published in separate papers over the last several years. The purpose of this paper is to survey these results and to show how they relate to one another in achieving an effective overall analysis procedure for analyzing a hyperspectral image data set.

https://engineering.purdue.edu/~landgreb/SPIE97.pdf

Progress toward information-extraction methods for hyperspectral data

A stochastic model for the data acquisition system in a multispectral scanner system, like the one utilized by the LANDSAT satellites, is presented. A list of noise sources which are known or presumed to have a significant effect in the information extraction process was constructed. Since the shot noise introduced by the photodetectors in the sensor system is signal level dependent, an atmospheric model was adopted which could adequately describe the amount of radiation that gets into the sensors based on the atmospheric transmittance. An analysis was carried out to find the output spectral statistics in terms of the input signal statistics and the system parameters. This was integrated into a set of FORTRAN programs that when supplied with, the class statistics, the noise levels introduced by the sensor system, the atmospheric transmittance, and the atmospheric path radiance, can be used to estimate the classification performance. In order to show the beneficts of this model a series of runs were performed in which the Thematic Mapper multispectral scanner was the system under consideration.

The relationship of sensor parameters to applications data analysis

https://docs.lib.purdue.edu/cgi/viewcontent.cgi?article=1677&context=ecetr

David A. Landgrebe

Papers

Regularized covariance estimators for hyperspectral data classification and its application to feature extraction

A perspective on the analysis of hyperspectral data

Progress toward information-extraction methods for hyperspectral data

The relationship of sensor parameters to applications data analysis

RSSIM: A Simulation Program for Optical Remote Sensing Systems