The tutorial starts with an overview of the concepts of VC dimension and structural risk minimization. We then describe linear Support Vector Machines (SVMs) for separable and non-separable data, working through a non-trivial example in detail. We describe a mechanical analogy, and discuss when SVM solutions are unique and when they are global. We describe how support vector training can be practically implemented, and discuss in detail the kernel mapping technique which is used to construct SVM solutions which are nonlinear in the data. We show how Support Vector machines can have very large (even infinite) VC dimension by computing the VC dimension for homogeneous polynomial and Gaussian radial basis function kernels. While very high VC dimension would normally bode ill for generalization performance, and while at present there exists no theory which shows that good generalization performance is guaranteed for SVMs, there are several arguments which support the observed high accuracy of SVMs, which we review. Results of some experiments which were inspired by these arguments are also presented. We give numerous examples and proofs of most of the key theorems. There is new material, and I hope that the reader will find that even old material is cast in a fresh light.

/pdf/a-tutorial-on-support-vector-machines-for-pattern-5cn6x20psd.pdf

A Tutorial on Support Vector Machines for Pattern Recognition

Summary 1. While teaching statistics to ecologists, the lead authors of this paper have noticed common statistical problems. If a random sample of their work (including scientific papers) produced before doing these courses were selected, half would probably contain violations of the underlying assumptions of the statistical techniques employed. 2. Some violations have little impact on the results or ecological conclusions; yet others increase type I or type II errors, potentially resulting in wrong ecological conclusions. Most of these violations can be avoided by applying better data exploration. These problems are especially troublesome in applied ecology, where management and policy decisions are often at stake. 3. Here, we provide a protocol for data exploration; discuss current tools to detect outliers, heterogeneity of variance, collinearity, dependence of observations, problems with interactions, double zeros in multivariate analysis, zero inflation in generalized linear modelling, and the correct type of relationships between dependent and independent variables; and provide advice on how to address these problems when they arise. We also address misconceptions about normality, and provide advice on data transformations. 4. Data exploration avoids type I and type II errors, among other problems, thereby reducing the chance of making wrong ecological conclusions and poor recommendations. It is therefore essential for good quality management and policy based on statistical analyses.

https://besjournals.onlinelibrary.wiley.com/doi/epdf/10.1111/j.2041-210X.2009.00001.x

A protocol for data exploration to avoid common statistical problems

(2002). Introduction to Linear Regression Analysis. Technometrics: Vol. 44, No. 2, pp. 191-192.

Introduction to Linear Regression Analysis

A wide range of methods for analysis of airborne- and satellite-derived imagery continues to be proposed and assessed. In this paper, we review remote sensing implementations of support vector machines (SVMs), a promising machine learning methodology. This review is timely due to the exponentially increasing number of works published in recent years. SVMs are particularly appealing in the remote sensing field due to their ability to generalize well even with limited training samples, a common limitation for remote sensing applications. However, they also suffer from parameter assignment issues that can significantly affect obtained results. A summary of empirical results is provided for various applications of over one hundred published works (as of April, 2010). It is our hope that this survey will provide guidelines for future applications of SVMs and possible areas of algorithm enhancement.

https://www.aboutgis.com/Publications/Mountrakis_SVM_review_in_remote_sensing_ISPRS2010.pdf

Support vector machines in remote sensing: A review

Publisher Summary This chapter concerns data analysis techniques that assist the interpretation of community composition in terms of species' responses to environmental gradients in the broadest sense. All species occur in a characteristic, limited range of habitats; and within their range, they tend to be most abundant around their particular environmental optimum. The composition of biotic communities thus changes along environmental gradients. Direct gradient analysis is a regression problem—fitting curves or surfaces to the relation between each species' abundance, probability of occurrence, and one or more environmental variables. Ecologists have independently developed a variety of alternative techniques. Many of these techniques are essentially heuristic, and have a less secure theoretical basis. This chapter presents a theory of gradient analysis, in which the heuristic techniques are integrated with regression, calibration, ordination and constrained ordination as distinct, well-defined statistical problems. The various techniques used for each type of problem are classified in families according to their implicit response model and the method used to estimate parameters of the model. Three such families are considered. The treatment shown here unites such apparently disparate data analysis techniques as linear regression, principal components analysis, redundancy analysis, Gaussian ordination, weighted averaging, reciprocal averaging, detrended correspondence analysis, and canonical correspondence analysis in a single theoretical framework.

/pdf/a-theory-of-gradient-analysis-351pq0a6mk.pdf

A Theory of Gradient Analysis

Preface. Introduction. Simple Linear Regression. Multiple Linear Regression. Model Adequacy Checking. Transformations and Weighting to Correct Model Inadequacies. Diagnostics for Leverage and Influence. Polynomial Regression Models. Indicator Variables. Variable Selection and Model Building. Multicollinearity. Robust Regression. Introduction to Nonlinear Regression. Generalized Linear Models. Other Topics in the Use of Regression Analysis. Validation of Regression Models. Appendix A. Statistical Tables. Appendix B. Data Sets for Exercises. Appendix C. Supplemental Technical Material. References. Index.

Introduction to Linear Regression Analysis, Solutions Manual (Wiley Series in Probability and Statistics)

Praise for the Fourth Edition "As with previous editions, the authors have produced a leading textbook on regression." -- Journal of the American Statistical Association A comprehensive and up-to-date introduction to the fundamentals of regression analysis Introduction to Linear Regression Analysis, Fifth Edition continues to present both the conventional and less common uses of linear regression in today's cutting-edge scientific research. The authors blend both theory and application to equip readers with an understanding of the basic principles needed to apply regression model-building techniques in various fields of study, including engineering, management, and the health sciences. Following a general introduction to regression modeling, including typical applications, a host of technical tools are outlined such as basic inference procedures, introductory aspects of model adequacy checking, and polynomial regression models and their variations. The book then discusses how transformations and weighted least squares can be used to resolve problems of model inadequacy and also how to deal with influential observations. The Fifth Edition features numerous newly added topics, including: A chapter on regression analysis of time series data that presents the Durbin-Watson test and other techniques for detecting autocorrelation as well as parameter estimation in time series regression models Regression models with random effects in addition to a discussion on subsampling and the importance of the mixed model Tests on individual regression coefficients and subsets of coefficients Examples of current uses of simple linear regression models and the use of multiple regression models for understanding patient satisfaction data. In addition to Minitab, SAS, and S-PLUS, the authors have incorporated JMP and the freely available R software to illustrate the discussed techniques and procedures in this new edition. Numerous exercises have been added throughout, allowing readers to test their understanding of the material, and a related FTP site features the presented data sets, extensive problem solutions, software hints, and PowerPoint slides to facilitate instructional use of the book. Introduction to Linear Regression Analysis, Fifth Edition is an excellent book for statistics and engineering courses on regression at the upper-undergraduate and graduate levels. The book also serves as a valuable, robust resource for professionals in the fields of engineering, life and biological sciences, and the social sciences.

Introduction to linear regression analysis : solutions manual to accompany

A computer program for performing two diagnostic procedures suggested by Daniel and Wood for a multiple regression model is presented. The first procedure is designed to detect points that are remote in x-space and that may be overly influential on the model summary statistics. The second procedure allows the analyst to identify pairs of points that are near neighbors in x-space and to compute an estimate of pure error using the residuals at these points. Two examples are given.

Elizabeth A. Peck

Papers

Introduction to Linear Regression Analysis

Introduction to Linear Regression Analysis, Solutions Manual (Wiley Series in Probability and Statistics)

Introduction to linear regression analysis : solutions manual to accompany

Interior Analysis of the Observations in Multiple Linear Regression

Introduction to Linear Regression Analysis.@@@Applied Linear Regression Models.