The primary goal of pattern recognition is supervised or unsupervised classification. Among the various frameworks in which pattern recognition has been traditionally formulated, the statistical approach has been most intensively studied and used in practice. More recently, neural network techniques and methods imported from statistical learning theory have been receiving increasing attention. The design of a recognition system requires careful attention to the following issues: definition of pattern classes, sensing environment, pattern representation, feature extraction and selection, cluster analysis, classifier design and learning, selection of training and test samples, and performance evaluation. In spite of almost 50 years of research and development in this field, the general problem of recognizing complex patterns with arbitrary orientation, location, and scale remains unsolved. New and emerging applications, such as data mining, web searching, retrieval of multimedia data, face recognition, and cursive handwriting recognition, require robust and efficient pattern recognition techniques. The objective of this review paper is to summarize and compare some of the well-known methods used in various stages of a pattern recognition system and identify research topics and applications which are at the forefront of this exciting and challenging field.

/pdf/statistical-pattern-recognition-a-review-1kz1b0tte4.pdf

Statistical pattern recognition: a review

Planning algorithms are impacting technical disciplines and industries around the world, including robotics, computer-aided design, manufacturing, computer graphics, aerospace applications, drug design, and protein folding. This coherent and comprehensive book unifies material from several sources, including robotics, control theory, artificial intelligence, and algorithms. The treatment is centered on robot motion planning but integrates material on planning in discrete spaces. A major part of the book is devoted to planning under uncertainty, including decision theory, Markov decision processes, and information spaces, which are the “configuration spaces” of all sensor-based planning problems. The last part of the book delves into planning under differential constraints that arise when automating the motions of virtually any mechanical system. Developed from courses taught by the author, the book is intended for students, engineers, and researchers in robotics, artificial intelligence, and control theory as well as computer graphics, algorithms, and computational biology.

Planning Algorithms: Introductory Material

Gradient Boosting Decision Tree (GBDT) is a popular machine learning algorithm, and has quite a few effective implementations such as XGBoost and pGBRT. Although many engineering optimizations have been adopted in these implementations, the efficiency and scalability are still unsatisfactory when the feature dimension is high and data size is large. A major reason is that for each feature, they need to scan all the data instances to estimate the information gain of all possible split points, which is very time consuming. To tackle this problem, we propose two novel techniques: Gradient-based One-Side Sampling (GOSS) and Exclusive Feature Bundling (EFB). With GOSS, we exclude a significant proportion of data instances with small gradients, and only use the rest to estimate the information gain. We prove that, since the data instances with larger gradients play a more important role in the computation of information gain, GOSS can obtain quite accurate estimation of the information gain with a much smaller data size. With EFB, we bundle mutually exclusive features (i.e., they rarely take nonzero values simultaneously), to reduce the number of features. We prove that finding the optimal bundling of exclusive features is NP-hard, but a greedy algorithm can achieve quite good approximation ratio (and thus can effectively reduce the number of features without hurting the accuracy of split point determination by much). We call our new GBDT implementation with GOSS and EFB LightGBM. Our experiments on multiple public datasets show that, LightGBM speeds up the training process of conventional GBDT by up to over 20 times while achieving almost the same accuracy.

/pdf/lightgbm-a-highly-efficient-gradient-boosting-decision-tree-f835dtmi7b.pdf

LightGBM: a highly efficient gradient boosting decision tree

may 7th, 1986, Professor A. F. M. Smith in the Chair] SUMMARY A continuous two-dimensional region is partitioned into a fine rectangular array of sites or "pixels", each pixel having a particular "colour" belonging to a prescribed finite set. The true colouring of the region is unknown but, associated with each pixel, there is a possibly multivariate record which conveys imperfect information about its colour according to a known statistical model. The aim is to reconstruct the true scene, with the additional knowledge that pixels close together tend to have the same or similar colours. In this paper, it is assumed that the local characteristics of the true scene can be represented by a nondegenerate Markov random field. Such information can be combined with the records by Bayes' theorem and the true scene can be estimated according to standard criteria. However, the computational burden is enormous and the reconstruction may reflect undesirable largescale properties of the random field. Thus, a simple, iterative method of reconstruction is proposed, which does not depend on these large-scale characteristics. The method is illustrated by computer simulations in which the original scene is not directly related to the assumed random field. Some complications, including parameter estimation, are discussed. Potential applications are mentioned briefly.

https://stat.duke.edu/~scs/Courses/Stat376/Papers/GibbsFieldEst/BesagDirtyPicsJRSSB1986.pdf

On the statistical analysis of dirty pictures

/pdf/semi-supervised-learning-literature-survey-4ccxy3rmmj.pdf

Semi-Supervised Learning Literature Survey

In this paper, a new regularization method for both parametric and nonparametric feature extraction algorithms is proposed for mitigating the singularity of within-class scatter matrix and improving hyperspectral data feature extraction. Keywords-regularization; feature extraction; hyperspectral data classification;

Regularized feature extractions for hyperspectral data classification

> In this paper, a novel approach to feature selection for classification is proposed based directly on the decision boundaries. It is shown how discriminantly redundant features and discriminantly informative features are related to decision boundaries. A novel characteristic of the proposed method arises by noting that only a portion of the decision boundary is effective in discriminating between classes. Next a procedure to extract discriminantly informative features based on a decision boundary is proposed. The proposed feature selection algorithm has several desirable properties: (1) it predicts the minimum number of features necessary to achieve the same classification accuracy as in the original space (2) it finds the necessary feature vectors. Experiments show that the performance of the proposed algorithm compares favorable with that of previous algorithms. Key word: Decision boundary, decision boundary feature matrix, discriminantly redundant, discriminantly informative, intrinsic discriminant dimension. I. FEATURE SELECTION AND' SUBSPACE An observation consisting of N features in pattern recognition can be viewed as a point in the Ndimensional Euclidean space EN. Since an Ndimensional vector space can be spanned by a basis consisting of N independent vectors {a, ,a2,..,aN}, an observation can be expressed by a linear combination of these ai. Then feature selection is equivalent to finding an effective subspace, W , and the new features can be found by projecting an observation into the subspace. It is desired that dim(wj < N. Now consider briefly Bayes' decision rule, which will be used later in the proposed feature selection algorithm. Let X be an observation in the Ndimensional Euclidean space EN under hypothesis HI: X E w, i=l,2. Decisions will be made according to the following rule. h(X) < t X E 01 h(X) > t X E o2

Feature Selection Based on Decision Boundaries

Supervised classification techniques use labeled samples in order to train the classifier. Usually the number of such samples is limited, and as the number of bands available increases, this limitation becomes more severe, and can become dominate over the projected added value of having the additional bands available. This suggests the need for reducing the dimensionality via a preprocessing method. Such reduction should enable the estimation of feature extraction parameters to be more accurate. Using a technique referred to as projection pursuit, two parametric projection pursuit algorithms have been developed: parallel parametric projection pursuit and sequential parametric projection pursuit. In the present paper both methods are presented, and an iterative procedure of the sequential approach that mitigates the computation time problem is shown.

Projection pursuit for high dimensional feature reduction: parallel and sequential approaches

A method for classifying high dimensional remote sensing data is described. The technique uses a radiometric adjustment to allow a human operator to identify and label training pixels by visually comparing the remotely sensed spectra to laboratory reflectance spectra. Training pixels for material without obvious spectral features are identified by traditional means. Features which are effective for discriminating between the classes are then derived from the original radiance data and used to classify the scene. This technique is applied to Airborne Visible/Infrared Imaging Spectrometer (AVIRIS) data taken over Cuprite, Nevada in 1992, and the results are compared to an existing geologic map. This technique performed well even with noisy data and the fact that some of the materials in the scene lack absorption features. No adjustment for the atmosphere or other scene variables was made to the data classified. While the experimental results compare favorably with an existing geologic map, the primary purpose of this research was to demonstrate the classification method, as compared to the geology of the Cuprite scene.

https://ntrs.nasa.gov/api/citations/19950017448/downloads/19950017448.pdf

Classification of high dimensional multispectral image data

.......................................................................................................................................................... V CHAPTER 1: INTRODUCTION ..................................................................................................................... 1 CHAPTER 2: EFFECT OF SEMI-LABELED SAMPLES IN REDUCING THE SMALL SAMPLE SIZE PROBLEM AND MITIGATING THE HUGHES PHENOMENON ................................................. 9 2.

/pdf/design-of-an-adaptive-classification-procedure-for-the-2hk1z9onqc.pdf

David A. Landgrebe

Papers

Regularized feature extractions for hyperspectral data classification

Feature Selection Based on Decision Boundaries

Projection pursuit for high dimensional feature reduction: parallel and sequential approaches

Classification of high dimensional multispectral image data

Design of an Adaptive Classification Procedure for the Analysis of High-Dimensional Data with Limited Training Samples