Generalized Inverses, Ridge Regression, Biased Linear Estimation, and Nonlinear Estimation
TL;DR: In this article, the authors discuss a class of biased linear estimators employing generalized inverses and establish a unifying perspective on nonlinear estimation from nonorthogonal data.
Abstract: A principal objective of this paper is to discuss a class of biased linear estimators employing generalized inverses. A second objective is to establish a unifying perspective. The paper exhibits theoretical properties shared by generalized inverse estimators, ridge estimators, and corresponding nonlinear estimation procedures. From this perspective it becomes clear why all these methods work so well in practical estimation from nonorthogonal data.
Citations
More filters
TL;DR: In this article, the authors examined the effect of the variance inflation factor (VIF) on the results of regression analyses, and found that threshold values of the VIF need to be evaluated in the context of several other factors that influence the variance of regression coefficients.
Abstract: The Variance Inflation Factor (VIF) and tolerance are both widely used measures of the degree of multi-collinearity of the ith independent variable with the other independent variables in a regression model. Unfortunately, several rules of thumb – most commonly the rule of 10 – associated with VIF are regarded by many practitioners as a sign of severe or serious multi-collinearity (this rule appears in both scholarly articles and advanced statistical textbooks). When VIF reaches these threshold values researchers often attempt to reduce the collinearity by eliminating one or more variables from their analysis; using Ridge Regression to analyze their data; or combining two or more independent variables into a single index. These techniques for curing problems associated with multi-collinearity can create problems more serious than those they solve. Because of this, we examine these rules of thumb and find that threshold values of the VIF (and tolerance) need to be evaluated in the context of several other factors that influence the variance of regression coefficients. Values of the VIF of 10, 20, 40, or even higher do not, by themselves, discount the results of regression analyses, call for the elimination of one or more independent variables from the analysis, suggest the use of ridge regression, or require combining of independent variable into a single index.
7,165 citations
Cites methods from "Generalized Inverses, Ridge Regress..."
...Marquardt (1970) uses a VIF greater than 10 as a guideline for serious multi-collinearity....
[...]
TL;DR: The generalized cross-validation (GCV) method as discussed by the authors is a generalized version of Allen's PRESS, which can be used in subset selection and singular value truncation, and even to choose from among mixtures of these methods.
Abstract: Consider the ridge estimate (λ) for β in the model unknown, (λ) = (X T X + nλI)−1 X T y. We study the method of generalized cross-validation (GCV) for choosing a good value for λ from the data. The estimate is the minimizer of V(λ) given by where A(λ) = X(X T X + nλI)−1 X T . This estimate is a rotation-invariant version of Allen's PRESS, or ordinary cross-validation. This estimate behaves like a risk improvement estimator, but does not require an estimate of σ2, so can be used when n − p is small, or even if p ≥ 2 n in certain cases. The GCV method can also be used in subset selection and singular value truncation methods for regression, and even to choose from among mixtures of these methods.
3,697 citations
TL;DR: In this article, the authors examined partial least squares and principal components regression from a statistical perspective and compared them with other statistical methods intended for those situations, such as variable subset selection and ridge regression.
Abstract: Chemometrics is a field of chemistry that studies the application of statistical methods to chemical data analysis. In addition to borrowing many techniques from the statistics and engineering literatures, chemometrics itself has given rise to several new data-analytical methods. This article examines two methods commonly used in chemometrics for predictive modeling—partial least squares and principal components regression—from a statistical perspective. The goal is to try to understand their apparent successes and in what situations they can be expected to work well and to compare them with other statistical methods intended for those situations. These methods include ordinary least squares, variable subset selection, and ridge regression.
2,309 citations
TL;DR: In this article, the use of Partial Least Squares (PLS) for handling collinearities among the independent variables X in multiple regression is discussed, and successive estimates are obtained using the residuals from previous rank as a new dependent variable y.
Abstract: The use of partial least squares (PLS) for handling collinearities among the independent variables X in multiple regression is discussed. Consecutive estimates $({\text{rank }}1,2,\cdots )$ are obtained using the residuals from previous rank as a new dependent variable y. The PLS method is equivalent to the conjugate gradient method used in Numerical Analysis for related problems.To estimate the “optimal” rank, cross validation is used. Jackknife estimates of the standard errors are thereby obtained with no extra computation.The PLS method is compared with ridge regression and principal components regression on a chemical example of modelling the relation between the measured biological activity and variables describing the chemical structure of a set of substituted phenethylamines.
2,290 citations
Cites methods from "Generalized Inverses, Ridge Regress..."
...In a forthcoming report we will also investigate statistical and numerical aspects on the PLS method, and show how the 8 theorems given by Marquardt (1970) for ridge and principal component regression translate into this situation....
[...]
TL;DR: The authors empirically tested the ability of stakeholder theory to explain one specific corporate social responsibility activity (social responsibility disclosure) and found that measures of investor power, strategic posture, and economic performance are significantly related to levels of corporate social disclosure.
Abstract: A lack of sufficient theoretical support for models designed to explain corporate social responsibility activity led Ullmann (Academy of Management Review, 1985, pp. 540–577) to develop a framework for predicting corporate social activity based on a stakeholder theory of strategic management. This study empirically tests the ability of stakeholder theory to explain one specific corporate social responsibility activity — social responsibility disclosure. Results support this application, finding that measures of stakeholder power, strategic posture, and economic performance are significantly related to levels of corporate social disclosure.
2,147 citations
References
More filters
28,888 citations
"Generalized Inverses, Ridge Regress..." refers methods in this paper
...See [13, 17]....
[...]
...The antecedents of this work are: (1) my earlier work on algorithms for nonlinear estimation [17], where it was shown that a small positive quantity should be added to the diagonal of the linearized normal equations at each iteration, in order that the parameter correction vector can be confined to a region where the linearization yields meaningful approximations to the nonlinear function; (2) the work of A....
[...]
...The preceding theorems suggest that an algorithm (hereafter, the "alternate algorithm") for nonlinear estimation can be developed based on generalized inverses, and that this algorithm would have properties similar to those previously described [17] for the method (hereafter, the "original algorithm") based on ordinary inverses of (A + kI)....
[...]
Book•
01 Jan 1965TL;DR: Algebra of Vectors and Matrices, Probability Theory, Tools and Techniques, and Continuous Probability Models.
Abstract: Algebra of Vectors and Matrices. Probability Theory, Tools and Techniques. Continuous Probability Models. The Theory of Least Squares and Analysis of Variance. Criteria and Methods of Estimation. Large Sample Theory and Methods. Theory of Statistical Inference. Multivariate Analysis. Publications of the Author. Author Index. Subject Index.
8,300 citations
"Generalized Inverses, Ridge Regress..." refers background in this paper
..., by Rao ([22], [23], pp 183, 4), Golub [5, 6], and Goldman and Zelen [4]....
[...]
...Faced with the apparent singularity of A, it is tempting to invert A by means of a generalized inverse [7, 8, 22, 23]....
[...]
TL;DR: In this paper, an estimation procedure based on adding small positive quantities to the diagonal of X′X was proposed, which is a method for showing in two dimensions the effects of nonorthogonality.
Abstract: In multiple regression it is shown that parameter estimates based on minimum residual sum of squares have a high probability of being unsatisfactory, if not incorrect, if the prediction vectors are not orthogonal. Proposed is an estimation procedure based on adding small positive quantities to the diagonal of X′X. Introduced is the ridge trace, a method for showing in two dimensions the effects of nonorthogonality. It is then shown how to augment X′X to obtain biased estimates with smaller mean square error.
8,091 citations
TL;DR: The theory of least squares and analysis of variance has been studied in the literature for a long time, see as mentioned in this paper for a review of some of the most relevant works. But the main focus of this paper is on the analysis of variance.
Abstract: Algebra of Vectors and Matrices. Probability Theory, Tools and Techniques. Continuous Probability Models. The Theory of Least Squares and Analysis of Variance. Criteria and Methods of Estimation. Large Sample Theory and Methods. Theory of Statistical Inference. Multivariate Analysis. Publications of the Author. Author Index. Subject Index.
5,182 citations
TL;DR: In this paper, the use of ridge regression methods is discussed and recommendations are made for obtaining a better regression equation than that given by ordinary least squares estimation. But the authors focus on the RIDGE TRACE which is a two-dimensional graphical procedure for portraying the complex relationships in multifactor data.
Abstract: This paper is an exposition of the use of ridge regression methods. Two examples from the literature are used as a base. Attention is focused on the RIDGE TRACE which is a two-dimensional graphical procedure for portraying the complex relationships in multifactor data. Recommendations are made for obtaining a better regression equation than that given by ordinary least squares estimation.
2,345 citations