Showing papers on "Linear discriminant analysis published in 1997"

Eigenfaces vs. Fisherfaces: recognition using class specific linear projection

Abstract: We develop a face recognition algorithm which is insensitive to large variation in lighting direction and facial expression. Taking a pattern classification approach, we consider each pixel in an image as a coordinate in a high-dimensional space. We take advantage of the observation that the images of a particular face, under varying illumination but fixed pose, lie in a 3D linear subspace of the high dimensional image space-if the face is a Lambertian surface without shadowing. However, since faces are not truly Lambertian surfaces and do indeed produce self-shadowing, images will deviate from this linear subspace. Rather than explicitly modeling this deviation, we linearly project the image into a subspace in a manner which discounts those regions of the face with large deviation. Our projection method is based on Fisher's linear discriminant and produces well separated classes in a low-dimensional subspace, even under severe variation in lighting and facial expressions. The eigenface technique, another method based on linearly projecting the image space to a low dimensional subspace, has similar computational requirements. Yet, extensive experimental results demonstrate that the proposed "Fisherface" method has error rates that are lower than those of the eigenface technique for tests on the Harvard and Yale face databases.

ADE-4: a multivariate analysis and graphical display software

Abstract: We present ADE-4, a multivariate analysis and graphical display software. Multivariate analysis methods available in ADE-4 include usual one-table methods like principal component analysis and correspondence analysis, spatial data analysis methods (using a total variance decomposition into local and global components, analogous to Moran and Geary indices), discriminant analysis and within/between groups analyses, many linear regression methods including lowess and polynomial regression, multiple and PLS (partial least squares) regression and orthogonal regression (principal component regression), projection methods like principal component analysis on instrumental variables, canonical correspondence analysis and many other variants, coinertia analysis and the RLQ method, and several three-way table (k-table) analysis methods. Graphical display techniques include an automatic collection of elementary graphics corresponding to groups of rows or to columns in the data table, thus providing a very efficient way for automatic k-table graphics and geographical mapping options. A dynamic graphic module allows interactive operations like searching, zooming, selection of points, and display of data values on factor maps. The user interface is simple and homogeneous among all the programs; this contributes to making the use of ADE-4 very easy for non- specialists in statistics, data analysis or computer science.

Improvements on Cross-Validation: The 632+ Bootstrap Method

Abstract: A training set of data has been used to construct a rule for predicting future responses. What is the error rate of this rule? This is an important question both for comparing models and for assessing a final selected model. The traditional answer to this question is given by cross-validation. The cross-validation estimate of prediction error is nearly unbiased but can be highly variable. Here we discuss bootstrap estimates of prediction error, which can be thought of as smoothed versions of cross-validation. We show that a particular bootstrap method, the .632+ rule, substantially outperforms cross-validation in a catalog of 24 simulation experiments. Besides providing point estimates, we also consider estimating the variability of an error rate estimate. All of the results here are nonparametric and apply to any possible prediction rule; however, we study only classification problems with 0–1 loss in detail. Our simulations include “smooth” prediction rules like Fisher's linear discriminant fun...

Discriminant analysis for recognition of human face images

Abstract: In this paper the discriminatory power of various human facial features is studied and a new scheme for Automatic Face Recognition (AFR) is proposed. Using Linear Discriminant Analysis (LDA) of different aspects of human faces in spatial domain, we first evaluate the significance of visual information in different parts/features of the face for identifying the human subject. The LDA of faces also provides us with a small set of features that carry the most relevant information for classification purposes. The features are obtained through eigenvector analysis of scatter matrices with the objective of maximizing between-class and minimizing within-class variations. The result is an efficient projection-based feature extraction and classification scheme for AFR. Soft decisions made based on each of the projections are combined, using probabilistic or evidential approaches to multisource data analysis. For medium-sized databases of human faces, good classification accuracy is achieved using very low-dimensional feature vectors.

Identification of protein coding regions in the human genome by quadratic discriminant analysis

Abstract: A new method for predicting internal coding exons in genomic DNA sequences has been developed. This method is based on a prediction algorithm that uses the quadratic discriminant function for multivariate statistical pattern recognition. Substantial improvements have been made (with only 9 discriminant variables) when compared with existing methods: hexon [Solovyev, V. V., Salamov, A. A. & Lawrence, C. B. (1994) Nucleic Acids Res. 22, 5156–5163] (based on linear discriminant analysis) and grail2 [Uberbacher, E. C. & Mural, R. J. (1991) Proc. Natl. Acad. Sci. USA 88, 11261–11265] (based on neural networks). A computer program called mzef is freely available to the genome community and allows users to adjust prior probability and to output alternative overlapping exons.

Bankruptcy prediction using case-based reasoning, neural networks, and discriminant analysis

Abstract: Bankruptcy prediction is one of the major business classification problems. 1n this paper, we use three different techniques: (1) Multivariate discriminant analysis, (2) case-based forecasting, and (3) neural network to predict Korean bankrupt and nonbankrupt firms. The average hit ratios of three methods range from 81.5 to 83.8%. Neural network performs better than discriminant analysis and the case-based forecasting system.

Investor Risk Tolerance: Testing The Efficacy Of Demographics As Differentiating and Classifying Factors

Abstract: This study was designed to determine what variables would differentiate between levels of investor risk tolerance and classify individuals into risk tolerance categories. A model was developed and empirically tested using data from the 1992 Survey of Consumer Finances. Multiple discriminant analysis indicated that the educational level of respondents was the most significant differentiating and classifying factor. Gender, self-employment status, and income also were found to be effective in discriminating among levels of risk tolerance. Demographic characteristic provide only a starting point in accessing investor risk tolerance. More research is needed to explain variations in risk tolerance.

A Comparative Analysis of Neural Networks and Statistical Methods for Predicting Consumer Choice

Abstract: This paper presents a definitive description of neural network methodology and provides an evaluation of its advantages and disadvantages relative to statistical procedures. The development of this rich class of models was inspired by the neural architecture of the human brain. These models mathematically emulate the neurophysical structure and decision making of the human brain, and, from a statistical perspective, are closely related to generalized linear models. Artificial neural networks are, however, nonlinear and use a different estimation procedure feed forward and back propagation than is used in traditional statistical models least squares or maximum likelihood. Additionally, neural network models do not require the same restrictive assumptions about the relationship between the independent variables and dependent variables. Consequently, these models have already been very successfully applied in many diverse disciplines, including biology, psychology, statistics, mathematics, business, insurance, and computer science. We propose that neural networks will prove to be a valuable tool for marketers concerned with predicting consumer choice. We will demonstrate that neural networks provide superior predictions regarding consumer decision processes. In the context of modeling consumer judgment and decision making, for example, neural network models can offer significant improvement over traditional statistical methods because of their ability to capture nonlinear relationships associated with the use of noncompensatory decision rules. Our analysis reveals that neural networks have great potential for improving model predictions in nonlinear decision contexts without sacrificing performance in linear decision contexts. This paper provides a detailed introduction to neural networks that is understandable to both the academic researcher and the practitioner. This exposition is intended to provide both the intuition and the rigorous mathematical models needed for successful applications. In particular, a step-by-step outline of how to use the models is provided along with a discussion of the strengths and weaknesses of the model. We also address the robustness of the neural network models and discuss how far wrong you might go using neural network models versus traditional statistical methods. Herein we report the results of two studies. The first is a numerical simulation comparing the ability of neural networks with discriminant analysis and logistic regression at predicting choices made by decision rules that vary in complexity. This includes simulations involving two noncompensatory decision rules and one compensatory decision rule that involves attribute thresholds. In particular, we test a variant of the satisficing rule used by Johnson et al. Johnson, Eric J., Robert J. Meyer, Sanjoy Ghose. 1989. When choice models fail: Compensatory models in negatively correlated environments. J. Marketing Res.26August 255--270. that sets a lower bound threshold on all attribute values and a “latitude of acceptance” model that sets both a lower threshold and an upper threshold on attribute values, mimicking an “ideal point” model Coombs and Avrunin [Coombs, Clyde H., George S. Avrunin. 1977. Single peaked functions and the theory of preference. Psych. Rev.84 216--230.]. We also test a compensatory rule that equally weights attributes and judges the acceptability of an alternative based on the sum of its attribute values. Thus, the simulations include both a linear environment, in which traditional statistical models might be deemed appropriate, as well as a nonlinear environment where statistical models might not be appropriate. The complexity of the decision rules was varied to test for any potential degradation in model performance. For these simulated data it is shown that, in general, the neural network model outperforms the commonly used statistical procedures in terms of explained variance and out-of-sample predictive accuracy. An empirical study bridging the behavioral and statistical lines of research was also conducted. Here we examine the predictive relationship between retail store image variables and consumer patronage behavior. A direct comparison between a neural network model and the more commonly encountered techniques of discriminant analysis and factor analysis followed by logistic regression is presented. Again the results reveal that the neural network model outperformed the statistical procedures in terms of explained variance and out-of-sample predictive accuracy. We conclude that neural network models offer superior predictive capabilities over traditional statistical methods in predicting consumer choice in nonlinear and linear settings.

Investigation of silicon auditory models and generalization of linear discriminant analysis for improved speech recognition

Abstract: Biologically motivated feature extraction algorithms have been found to provide significantly robust performance in speech recognition systems, in the presence of channel and noise degradation, when compared to the standard features such as mel-cepstrum coefficients. However, auditory feature extraction is computationally expensive, and makes these features useless for real-time speech recognition systems. In this thesis, I investigate the use of low power techniques and custom analog VLSI for auditory feature extraction. I first investigated the basilar-membrane model and the hair-cell model chips that were designed by Liu (Liu, 1992). I performed speech recognition experiments to evaluate how well these chips would perform as a front-end to a speech recognizer. Based on the experience gained by these experiments, I propose an alternate architecture that goes beyond the basilar-membrane model, and, using which, auditory features can be computed in real time. These chips have been designed and tested, and consume only a few milliwatts of power as compared to general purpose digital machines that consume several Watts. I have also investigated Linear Discriminant Analysis (LDA) for dimension reduction of auditory features. Researchers have used Fisher-Rao linear discriminant analysis (LDA) to reduce the feature dimension. They model the low-dimensional features obtained from LDA as the outputs of a Markov process with hidden states (HMM). I present a unified framework for the problem of dimension reduction and HMM parameter estimation by modeling the original features with reduced-rank HMM. This re-formulation also leads to a generalization of LDA that is consistent with the heteroscedastic state models used in HMM, and give better performance when tested on a digit recognition task.

The use and misuse of chemometrics for treating classification problems

Abstract: In this article, we examine the increasing use by analytical chemists of chemometric methods for treating classification problems. The methods considered are principal component analysis (PCA), canonical variates analysis (CVA), discriminant analysis (DA), and discriminant partial least squares (PLS). Overfitting, a potential hazard of multivariate modelling, is illustrated using examples of real and simulated data, and the importance of model validation is discussed.

Multivariate analysis techniques in social science research : from problem to analysis

Abstract: PART ONE: FROM THEORY TO METHODOLOGY Types of Research Problems and Research Situations A Classification of Classical Techniques of Multivariate Analysis The Analysis Technique as the Mirror of the Research Problem PART TWO: FROM METHODOLOGY TO ANALYSIS The Experimental Design The Effect of Humour on Financial Concession Multiple Regression Analysis The Causes of Childlessness Partial Correlation and Path Analysis The Causal Influence of Christian Beliefs on Anti-Semitism Analysis of Variance and Covariance The Effect of Interaction of Monetary Rewards and Task Interest on Motivation Two-Group Discriminant Analysis Poor and Rich Neighbourhoods Factor Analysis The Investigation of Marital Adjustment Canonical Correlation Analysis The Study of Economic Inequality and Political Instability Techniques with Multiple Dependent Variables

Feature extraction for multisource data classification with artificial neural networks

Abstract: Classification of multisource remote sensing and geographic data by neural networks is discussed with respect to feature extraction. Several feature extraction methods are reviewed, including principal component analysis, discriminant analysis, and the recently proposed decision boundary feature extraction method. The feature extraction methods are then applied in experiments in conjunction with classification by multilayer neural networks. The decision boundary feature extraction method shows excellent performance in the experiments.

Adaptive autoregressive modeling used for single-trial EEG classification.

Abstract: An adaptive autoregressive (AAR) model is used for analyzing event-related EEG changes. Such an AAR model is applied to single EEG trials of three subjects, recorded over both sensorimotor areas during imagination of left and right hand movements. It is found that discrimination between both types of motor-imagery is possible using linear discriminant analysis, but the time point for optimal classification is different in each subject. For the estimation of the AAR parameters, the Least-mean-squares and the Recursive-least-squares algorithms are compared. In both methods, the update coefficient plays a key role: it determines the adaptation ratio as well as the estimation accuracy. A new method, based on minimizing the prediction error, is introduced for determining the update coefficient.

High-Breakdown Linear Discriminant Analysis

Abstract: The classification rules of linear discriminant analysis are defined by the true mean vectors and the common covariance matrix of the populations from which the data come. Because these true parameters are generally unknown, they are commonly estimated by the sample mean vector and covariance matrix of the data in a training sample randomly drawn from each population. However, these sample statistics are notoriously susceptible to contamination by outliers, a problem compounded by the fact that the outliers may be invisible to conventional diagnostics. High-breakdown estimation is a procedure designed to remove this cause for concern by producing estimates that are immune to serious distortion by a minority of outliers, regardless of their severity. In this article we motivate and develop a high-breakdown criterion for linear discriminant analysis and give an algorithm for its implementation. The procedure is intended to supplement rather than replace the usual sample-moment methodology of discri...

Texture fusion and feature selection applied to SAR imagery

Abstract: The discrimination ability of four different methods for texture computation in ERS SAR imagery is examined and compared. Feature selection methodology and discriminant analysis are applied to find the optimal combination of texture features. By combining features derived from different texture models, the classification accuracy increased significantly.

Multifeature analysis of Gd-enhanced MR images of breast lesions.

Abstract: The objective of this study was to determine whether linear discriminant analysis of different independent features of MR images of breast lesions can increase the sensitivity and specificity of this technique. For MR images of 23 benign and 20 malignant breast lesions, three independent classes of features, including characteristics of Gd-DTPA-uptake curve, boundary, and texture were evaluated. The three classes included five, four and eight features each, respectively. Discriminant analysis was applied both within and across the three classes, to find the best combination of features yielding the highest classification accuracy. The highest specificity and sensitivity of the different classes considered independently were as follows: Gd-up-take curves, 83% and 70%; boundary features, 86% and 70%; and texture, 70% and 75%, respectively. A combination of one feature each from the first two classes and age yielded a specificity of 79% and sensitivity of 90%, whereas highest figures of 93% and 95%, respectively, were obtained when a total of 10 features were combined across different classes. Statistical analysis of different independent classes of features in MR images of breast lesions can improve the classification accuracy of this technique significantly.

Multivariate statistical analysis for geographers

Abstract: I. INTRODUCTION AND REVIEW. 1. Elementary Statistics Background. 2. Information Content in Geo-Referenced Data. 3. Introduction to Matrix Algebra. 4. Multiple Linear Regression Analysis and Correlation Analysis. II. INSTANCES OF THE GENERAL LINEAR MODEL. 5. Multivariate Analysis of Variance. 6. Principal Components and Factor Analysis. 7. Discriminant Function Analysis. 8. Cluster Analysis. 9. Canonical Correlation Analysis. III. NONLINEAR AND CATEGORICAL DATA MODELING. 10. Nonlinear Regression Analysis. 11. Spatial Autoregressive Analysis. 12. Special Nonlinear Regression Applications in Spatial Analysis. Epilogue. Appendices. Index.

Analyzing Credit Risk Data: A Comparison of Logistic Discrimination, Classification Tree Analysis, and Feedforward Networks

Abstract: Three different discriminant techniques are applied and compared to analyze a complex data set of credit risks. A large sample is split into a training, a validation, and a test sample. The dependent variable is whether a loan is paid back without problems or not. Predictor variables are sex, job duration, age, car ownership, telephone ownership, and marital status. The statistical techniques are logistic discriminant analysis with a simple mean effects model, classification tree analysis, and a feedforward network with one hidden layer consisting of three units. It turns out, that in the given test sample, the predictive power is about equal for all techniques with the logistic discrimination as the best technique. However, the feedforward network produces different classification rules from the logistic discrimination and the classification tree analysis. Therefore, an additional coupling procedure for forecasts is applied to produce a combined forecast. However, this forecast turns out to be slightly worse than the logit model.

Image segmentation and discriminant analysis for the identification of land cover units in ecology

Abstract: The textured nature of most natural land cover units as represented in remotely sensed imagery causes limited results of per-pixel classifications The segmentation algorithm, iterative mutually optimum region merging (IMORM), is presented and used to partition images into elements that are thereafter classified by linear canonical discriminant analysis and a maximum likelihood allocation rule This per-segment approach results in much higher accuracy than the conventional per-pixel approach Furthermore, separability matrices indicate that many land cover categories cannot be correctly defined by per-pixel statistics

False-positive reduction technique for detection of masses on digital mammograms: Global and local multiresolution texture analysis

Abstract: We investigated the application of multiresolution global and local texture features to reduce false-positive detection in a computerized mass detection program. One hundred and sixty-eight digitized mammograms were randomly and equally divided into training and test groups. From these mammograms, two datasets were formed. The first dataset (manual) contained four regions of interest (ROIs) selected manually from each of the mammograms. One of the four ROIs contained a biopsy-proven mass and the other three contained normal parenchyma, including dense, mixed dense/fatty, and fatty tissues. The second dataset (hybrid) contained the manually extracted mass ROIs, along with normal tissue ROIs extracted by an automated Density-Weighted Contrast Enhancement (DWCE) algorithm as false-positive detections. A wavelet transform was used to decompose an ROI into several scales. Global texture features were derived from the low-pass coefficients in the wavelet transformed images. Local texture features were calculated from the suspicious object and the peripheral subregions. Linear discriminant models using effective features selected from the global, local, or combined feature spaces were established to maximize the separation between masses and normal tissue. Receiver Operating Characteristic (ROC) analysis was conducted to evaluate the classifier performance. The classification accuracy using global features were comparable to that using local features. With both global and local features, the average area, A z , under the test ROC curve, reached 0.92 for the manual dataset and 0.96 for the hybrid dataset, demonstrating statistically significant improvement over those obtained with global or local features alone. The results indicated the effectiveness of the combined global and local features in the classification of masses and normal tissue for false-positive reduction.

A simulation study of microevolutionary inferences by spatial autocorrelation analysis

Abstract: To explore the extent to which microevolutionary inference can be made using spatial autocorrelation analysis of gene frequency surfaces, we simulated sets of surfaces for nine evolutionary scenarios, and subjected spatially-based summary statistics of these to linear discriminant analysis. Scenarios varied the amounts of dispersion, selection, migration, and deme sizes, and included: panmixia, drift, intrusion, and stepping-stone models with 0-2 migrations, 0-2 selection gradients, and migration plus selection. To discover how weak evolutionary forces could be and still allow discrimination, each scenario had both a strong and a weak configuration. Discriminant rules were calculated using one collection of data (the training set) consisting of 250 sets of 15 surfaces for each of the nine scenarios. Misclassification rates were verified against a second, entirely new set of data (the test set) equal in size. Test set misclassification rates for the 20 best discriminating variables ranged from 39·3% (weak) to 3·6% (strong), far lower than the expected rate of 88·9% absent any discriminating ability. Misclassification was highest when discriminating the number of migrational events or the presence or number of selection events. Discrimination of drift and panmixia from the other scenarios was perfect. A subsequent subjective analysis of a subset of the data by one of us yielded comparable, although somewhat higher, misclassification rates. Judging by these results, spatial autocorrelation variables describing sets of gene frequency surfaces permit some microevolutionary inferences.

Self-organizing algorithms for generalized eigen-decomposition

Abstract: We discuss a new approach to self-organization that leads to novel adaptive algorithms for generalized eigen-decomposition and its variance for a single-layer linear feedforward neural network. First, we derive two novel iterative algorithms for linear discriminant analysis (LDA) and generalized eigen-decomposition by utilizing a constrained least-mean-squared classification error cost function, and the framework of a two-layer linear heteroassociative network performing a one-of-m classification. By using the concept of deflation, we are able to find sequential versions of these algorithms which extract the LDA components and generalized eigenvectors in a decreasing order of significance. Next, two new adaptive algorithms are described to compute the principal generalized eigenvectors of two matrices (as well as LDA) from two sequences of random matrices. We give a rigorous convergence analysis of our adaptive algorithms by using stochastic approximation theory, and prove that our algorithms converge with probability one.

Multigroup Discriminant Analysis Using Linear Programming

Abstract: In this paper we introduce a nonparametric linear programming formulation for the general multigroup classification problem. Previous research using linear programming formulations has either been limited to the two-group case, or required complicated constraints and many zero-one variables. We develop general properties of our multigroup formulation and illustrate its use with several small example problems and previously published real data sets. A comparative analysis on the real data sets shows that our formulation may offer an interesting robust alternative to parametric statistical formulations for the multigroup discriminant problem.

The proximity of an individual to a population with applications in discriminant analysis

Abstract: We develop a proximity function between an individual and a population from a distance between multivariate observations. We study some properties of this construction and apply it to a distance{based discrimination rule, which contains the classic linear discriminant function as a particular case. Additionally, this rule can be used advantageously for categorical or mixed variables, or in problems where a probabilistic model is not well determined. This approach is illustrated and compared with other classic procedures using four real data sets. The authors thank M.Abrahamowicz, J.C. Gower and M. Greenacre for their helpful comments, and W.J. Krzanowski for providing us with a data set and his quadratic location model program.

Differentiation of wine vinegars based on phenolic composition

Abstract: Phenolic composition of 92 wine vinegars produced from different wines from the south of Spain (Jerez, Montilla, El Condado) is determined by HPLC with diode array detection. Pattern recognition techniques were applied to distinguish between different methods of elaboration (slow traditional methods with surface culture or quick methods carried out in bioreactors with submerged culture) or wines employed as substrate. Multivariate analysis of data includes principal component analysis, cluster analysis, and linear discriminant analysis (LDA) as well as artificial neural networks trained by back-propagation (BPANN). The classification depending on the acetification process leads to good recalling rates in both LDA (mean = 92.5) and BPANN (mean = 99.6). With respect to the classification on the basis of the geographical origin, the obtained recalling rates were 88.8 for LDA and of 96.5 for BPANN (mean values). Keywords: Vinegar; phenols; discriminant analysis; artificial neural network; HPLC