Showing papers on "Proper linear model published in 2012"

Interpreting Multiple Linear Regression: A Guidebook of Variable Importance

Abstract: Copyright is retained by the first or sole author, who grants right of first publication to the Practical Assessment, Research & Evaluation. Permission is granted to distribute this article for nonprofit, educational purposes if it is copied in its entirety and the journal is credited. PARE has the right to authorize third party reproduction of this article in print, electronic and database forms.

Scaled sparse linear regression

Abstract: SUMMARY Scaled sparse linear regression jointly estimates the regression coefficients and noise level in a linear model. It chooses an equilibrium with a sparse regression method by iteratively estimating the noise level via the mean residual square and scaling the penalty in proportion to the estimated noise level. The iterative algorithm costs little beyond the computation of a path or grid of the sparse regression estimator for penalty levels above a proper threshold. For the scaled lasso, the algorithm is a gradient descent in a convex minimization of a penalized joint loss function for the regression coefficients and noise level. Under mild regularity conditions, we prove that the scaled lasso simultaneously yields an estimator for the noise level and an estimated coefficient vector satisfying certain oracle inequalities for prediction, the estimation of the noise level and the regression coefficients. These inequalities provide sufficient conditions for the consistency and asymptotic normality of the noise-level estimator, including certain cases where the number of variables is of greater order than the sample size. Parallel results are provided for least-squares estimation after model selection by the scaled lasso. Numerical results demonstrate the superior performance of the proposed methods over an earlier proposal of joint convex minimization. Somekeywords:Convexminimization;Estimationaftermodelselection;Iterativealgorithm;Linearregression;Oracle inequality; Penalized least squares; Scale invariance; Variance estimation.

IBM SPSS statistics 19 advanced statistical procedures companion

Abstract: 1. Model Selection Loglinear Analysis 2. Logit Loglinear Analysis 3. Multinomial Logistic Regression 4. Ordinal Regression 5. Probit Regression 6. Kaplan-Meier Survival Analysis 7. Life Tables 8. Cox Regression 9. Variance Components 10. Linear Mixed Models 11. Generalized Linear Models 12. Generalized Estimating Equations 13. Generalized Linear Mixed Models 14. Nonlinear Regression 15. Two-Stage Least-Squares Regression 16. Weighted Least-Squares Regression 17. Multidimensional Scaling

Sparse Reduced-Rank Regression for Simultaneous Dimension Reduction and Variable Selection

Abstract: The reduced-rank regression is an effective method in predicting multiple response variables from the same set of predictor variables. It reduces the number of model parameters and takes advantage of interrelations between the response variables and hence improves predictive accuracy. We propose to select relevant variables for reduced-rank regression by using a sparsity-inducing penalty. We apply a group-lasso type penalty that treats each row of the matrix of the regression coefficients as a group and show that this penalty satisfies certain desirable invariance properties. We develop two numerical algorithms to solve the penalized regression problem and establish the asymptotic consistency of the proposed method. In particular, the manifold structure of the reduced-rank regression coefficient matrix is considered and studied in our theoretical analysis. In our simulation study and real data analysis, the new method is compared with several existing variable selection methods for multivariate regression...

Estimating the evidence – a review

Abstract: The model evidence is a vital quantity in the comparison of statistical models under the Bayesian paradigm. This study presents a review of commonly used methods. We outline some guidelines and offer some practical advice. The reviewed methods are compared for two examples; non-nested Gaussian linear regression and covariate subset selection in logistic regression.

Linear regression with compositional explanatory variables

Abstract: Compositional explanatory variables should not be directly used in a linear regression model because any inference statistic can become misleading. While various approaches for this problem were proposed, here an approach based on the isometric logratio (ilr) transformation is used. It turns out that the resulting model is easy to handle, and that parameter estimation can be done in like in usual linear regression. Moreover, it is possible to use the ilr variables for inference statistics in order to obtain an appropriate interpretation of the model.

Regression Analysis by Example. 5th Edition.

Abstract: Praise for the Fourth Edition: "This book is . . . an excellent source of examples for regression analysis. It has been and still is readily readable and understandable." —Journal of the American Statistical Association Regression analysis is a conceptually simple method for investigating relationships among variables. Carrying out a successful application of regression analysis, however, requires a balance of theoretical results, empirical rules, and subjective judgment. Regression Analysis by Example, Fifth Edition has been expanded and thoroughly updated to reflect recent advances in the field. The emphasis continues to be on exploratory data analysis rather than statistical theory. The book offers in-depth treatment of regression diagnostics, transformation, multicollinearity, logistic regression, and robust regression.

A Covariance Regression Model

Abstract: Classical regression analysis relates the expectation of a response vari- able to a linear combination of explanatory variables. In this article, we propose a covariance regression model that parameterizes the covariance matrix of a mul- tivariate response vector as a parsimonious quadratic function of explanatory vari- ables. The approach is analogous to the mean regression model, and is similar to a factor analysis model in which the factor loadings depend on the explanatory vari- ables. Using a random-effects representation, parameter estimation for the model is straightforward using either an EM-algorithm or an MCMC approximation via Gibbs sampling. The proposed methodology provides a simple but flexible repre- sentation of heteroscedasticity across the levels of an explanatory variable, improves estimation of the mean function and gives better calibrated prediction regions when compared to a homoscedastic model.

A New Robust Regression Model for Proportions

Abstract: A new regression model for proportions is presented by considering the Beta rectangular distribution proposed by Hahn (2008). This new model includes the Beta regression model introduced by Ferrari and Cribari-Neto (2004) and the variable dispersion Beta regression model introduced by Smithson and Verkuilen (2006) as particular cases. Like Branscum, Johnson, and Thurmond (2007), a Bayesian inference approach is adopted using Markov Chain Monte Carlo (MCMC) algorithms. Simulation studies on the influence of outliers by considering contaminated data under four perturbation patterns to generate outliers were carried out and confirm that the Beta rectangular regression model seems to be a new robust alternative for modeling proportion data and that the Beta regression model shows sensitivity to the estimation of regression coefficients, to the posterior distribution of all parameters and to the model comparison criteria considered. Furthermore, two applications are presented to illustrate the robustness of the Beta rectangular model.

Multi-target regression with rule ensembles

Abstract: Methods for learning decision rules are being successfully applied to many problem domains, in particular when understanding and interpretation of the learned model is necessary. In many real life problems, we would like to predict multiple related (nominal or numeric) target attributes simultaneously. While several methods for learning rules that predict multiple targets at once exist, they are all based on the covering algorithm, which does not work well for regression problems. A better solution for regression is the rule ensemble approach that transcribes an ensemble of decision trees into a large collection of rules. An optimization procedure is then used to select the best (and much smaller) subset of these rules and to determine their respective weights. We introduce the FIRE algorithm for solving multi-target regression problems, which employs the rule ensembles approach. We improve the accuracy of the algorithm by adding simple linear functions to the ensemble. We also extensively evaluate the algorithm with and without linear functions. The results show that the accuracy of multi-target regression rule ensembles is high. They are more accurate than, for instance, multi-target regression trees, but not quite as accurate as multi-target random forests. The rule ensembles are significantly more concise than random forests, and it is also possible to create compact rule sets that are smaller than a single regression tree but still comparable in accuracy.

Local Distance-Based Generalized Linear Models Using the Dbstats Package for R

Abstract: This paper introduces local distance-based generalized linear models. These models extend (weighted) distance-based linear models firstly with the generalized linear model concept, then by localizing. Distances between individuals are the only predictor information needed to fit these models. Therefore they are applicable to mixed (qualitative and quantitative) explanatory variables or when the regressor is of functional type. Models can be fitted and analysed with the R package dbstats, which implements several distance-based prediction methods.

ℓ1-regularized linear regression: persistence and oracle inequalities

Abstract: We study the predictive performance of l 1-regularized linear regression in a model-free setting, including the case where the number of covariates is substantially larger than the sample size. We introduce a new analysis method that avoids the boundedness problems that typically arise in model-free empirical minimization. Our technique provides an answer to a conjecture of Greenshtein and Ritov (Bernoulli 10(6):971–988, 2004) regarding the “persistence” rate for linear regression and allows us to prove an oracle inequality for the error of the regularized minimizer. It also demonstrates that empirical risk minimization gives optimal rates (up to log factors) of convex aggregation of a set of estimators of a regression function.

Mixture of Regression Models With Varying Mixing Proportions: A Semiparametric Approach

Abstract: In this article, we study a class of semiparametric mixtures of regression models, in which the regression functions are linear functions of the predictors, but the mixing proportions are smoothing functions of a covariate. We propose a one-step backfitting estimation procedure to achieve the optimal convergence rates for both regression parameters and the nonparametric functions of mixing proportions. We derive the asymptotic bias and variance of the one-step estimate, and further establish its asymptotic normality. A modified expectation-maximization-type (EM-type) estimation procedure is investigated. We show that the modified EM algorithms preserve the asymptotic ascent property. Numerical simulations are conducted to examine the finite sample performance of the estimation procedures. The proposed methodology is further illustrated via an analysis of a real dataset.

Semiparametric Regression Pursuit.

Abstract: The semiparametric partially linear model allows flexible modeling of covariate effects on the response variable in regression. It combines the flexibility of nonparametric regression and parsimony of linear regression. The most important assumption in the existing methods for the estimation in this model is to assume a priori that it is known which covariates have a linear effect and which do not. However, in applied work, this is rarely known in advance. We consider the problem of estimation in the partially linear models without assuming a priori which covariates have linear effects. We propose a semiparametric regression pursuit method for identifying the covariates with a linear effect. Our proposed method is a penalized regression approach using a group minimax concave penalty. Under suitable conditions we show that the proposed approach is model-pursuit consistent, meaning that it can correctly determine which covariates have a linear effect and which do not with high probability. The performance of the proposed method is evaluated using simulation studies, which support our theoretical results. A real data example is used to illustrated the application of the proposed method.

Some Liu and ridge-type estimators and their properties under the ill-conditioned Gaussian linear regression model

Abstract: The estimation of the regression parameters for the ill-conditioned Gaussian linear regression model are considered in this paper. Accordingly, we consider some improved Liu [A new class of biased estimate in linear regression, Commun. Stat. Theory Methods 22 (1993), pp. 393–402] type estimators, namely the unrestricted Liu estimator, restricted Liu estimator and the preliminary test Liu estimator (PTLE) for estimating the regression parameters. The performances of the proposed estimators are compared based on the quadratic bias and risk functions under both null and alternative hypotheses. The conditions of superiority of the proposed estimators for departure parameter, Δ, and biasing parameter, d, are given. We also numerically compared the performance of PTLE with the preliminary test ridge regression estimator (PTRRE) and concluded that for small values of d and ridge parameter k, PTLE performed better than the PTRRE; otherwise the PTRRE performed better than PTLE in the sense of smaller MSE.

Simultaneously Leveraging Output and Task Structures for Multiple-Output Regression

Abstract: Multiple-output regression models require estimating multiple parameters, one for each output. Structural regularization is usually employed to improve parameter estimation in such models. In this paper, we present a multiple-output regression model that leverages the covariance structure of the latent model parameters as well as the conditional covariance structure of the observed outputs. This is in contrast with existing methods that usually take into account only one of these structures. More importantly, unlike some of the other existing methods, none of these structures need be known a priori in our model, and are learned from the data. Several previously proposed structural regularization based multiple-output regression models turn out to be special cases of our model. Moreover, in addition to being a rich model for multiple-output regression, our model can also be used in estimating the graphical model structure of a set of variables (multivariate outputs) conditioned on another set of variables (inputs). Experimental results on both synthetic and real datasets demonstrate the effectiveness of our method.

Multiple Linear Regression

Abstract: This chapter provides a comprehensive introduction to multiple regression analysis (MR), a highly flexible system for examining the relationship of a collection of independent variables (predictors) to a single dependent variable (criterion). The independent variables may be quantitative (e.g., personality traits, family income) or categorical (e.g., ethnic group, treatment conditions in an experiment). The present chapter explores ordinary least squares (OLS) regression, which requires a continuous dependent variable. The chapter emphasizes: (a) testing theoretical predictions through multiple regression, and (b) identifying problems with implementation of regression analysis. First the structure of MR is described, including the overall regression equation, estimation of partial regression coefficients for individual predictors, effect size measures for both overall model fit and for the contribution of individual predictors and sets of predictors to prediction accuracy. Treatment of categorical predictors through effects, dummy, and contrast coding is explained. Polynomial regression for capturing curvilinear relationships is explored. The specification, testing, and posthoc probing of interactions between continuous variables and between a continuous and a categorical variable are explicated. Second, detection of violations of MR assumptions is addressed, as this helps to identify problems with regression models. Regression diagnostics (case statistics) that are used to identify problematic cases which bias results are explained. Graphic displays for regression analysis that characterize the overall nature of the regression model, curvilinear, and interactive relationships among variables, and checking of assumptions are provided. An empirical example illustrates the interplay between theory and empirical findings in the specification, testing, and revision of regression models. Keywords: multiple regression analysis; ordinary least squares (OLS) regression; interactions in regression; model checking in multiple regression; graphics for multiple regression

Factor models and variable selection in high-dimensional regression analysis

Abstract: The paper considers linear regression problems where the number of predictor variables is possibly larger than the sample size. The basic motivation of the study is to combine the points of view of model selection and functional regression by using a factor approach: it is assumed that the predictor vector can be decomposed into a sum of two uncorrelated random components reflecting common factors and specific variabilities of the explanatory variables. It is shown that the traditional assumption of a sparse vector of parameters is restrictive in this context. Common factors may possess a significant influence on the response variable which cannot be captured by the specific effects of a small number of individual variables. We therefore propose to include principal components as additional explanatory variables in an augmented regression model. We give finite sample inequalities for estimates of these components. It is then shown that model selection procedures can be used to estimate the parameters of the augmented model, and we derive theoretical properties of the estimators. Finite sample performance is illustrated by a simulation study.

A parameters reduction method for monitoring multiple linear regression profiles

Abstract: In certain applications of statistical process control, it is possible to model quality of a product or process using a multiple linear regression profile Some methods exist in the literature which could be used for monitoring multiple linear regression profiles However, the performance of most of these methods deteriorates as the number of regression parameters increases In this paper, we specifically concentrate on phase II monitoring of multiple linear regression profiles and propose a new dimension reduction method to overcome the dimensionality problem of some of the existing methods The robustness, effectiveness, and limitations of the proposed method are also discussed Simulation results show that in term of average run length criterion, the proposed method outperforms the traditional methods and has comparable performance with another dimension reduction method in the literature

Different slopes for different folks: mining for exceptional regression models with cook's distance

Abstract: Exceptional Model Mining (EMM) is an exploratory data analysis technique that can be regarded as a generalization of subgroup discovery. In EMM we look for subgroups of the data for which a model fitted to the subgroup differs substantially from the same model fitted to the entire dataset. In this paper we develop methods to mine for exceptional regression models. We propose a measure for the exceptionality of regression models (Cook's distance), and explore the possibilities to avoid having to fit the regression model to each candidate subgroup. The algorithm is evaluated on a number of real life datasets. These datasets are also used to illustrate the results of the algorithm. We find interesting subgroups with deviating models on datasets from several different domains. We also show that under certain circumstances one can forego fitting regression models on up to 40% of the subgroups, and these 40% are the relatively expensive regression models to compute.

D-MORPH regression for modeling with fewer unknown parameters than observation data

Abstract: D-MORPH regression is a procedure for the treatment of a model prescribed as a linear superposition of basis functions with less observation data than the number of expansion parameters. In this case, there is an infinite number of solutions exactly fitting the data. D-MORPH regression provides a practical systematic means to search over the solutions seeking one with desired ancillary properties while preserving fitting accuracy. This paper extends D-MORPH regression to consider the common case where there is more observation data than unknown parameters. This situation is treated by utilizing a proper subset of the normal equation of least-squares regression to judiciously reduce the number of linear algebraic equations to be less than the number of unknown parameters, thereby permitting application of D-MORPH regression. As a result, no restrictions are placed on model complexity, and the model with the best prediction accuracy can be automatically and efficiently identified. Ignition data for a H2/air combustion model as well as laboratory data for quantum-control-mechanism identification are used to illustrate the method.

Using regression equations built from summary data in the psychological assessment of the individual case: extension to multiple regression.

Abstract: Regression equations have many useful roles in psychological assessment. Moreover, there is a large reservoir of published data that could be used to build regression equations; these equations could then be employed to test a wide variety of hypotheses concerning the functioning of individual cases. This resource is currently underused because (a) not all psychologists are aware that regression equations can be built not only from raw data but also using only basic summary data for a sample, and (b) the computations involved are tedious and prone to error. In an attempt to overcome these barriers, Crawford and Garthwaite (2007) provided methods to build and apply simple linear regression models using summary statistics as data. In the present study, we extend this work to set out the steps required to build multiple regression models from sample summary statistics and the further steps required to compute the associated statistics for drawing inferences concerning an individual case. We also develop, describe, and make available a computer program that implements these methods. Although there are caveats associated with the use of the methods, these need to be balanced against pragmatic considerations and against the alternative of either entirely ignoring a pertinent data set or using it informally to provide a clinical “guesstimate.” Upgraded versions of earlier programs for regression in the single case are also provided; these add the point and interval estimates of effect size developed in the present article.

Applications of Linear and Nonlinear Models: Fixed Effects, Random Effects, and Total Least Squares

Abstract: The first problem of algebraic regression.- The first problem of algebraic regression: the bias problem Special Gauss-Markov model with datum defects, LUMBE.- The second problem of algebraic regression Inconsistent system of linear observational equations.- The second problem of probabilistic regression Special Gauss-Markov model without datum defect.- The third problem of algebraic regression.- The third problem of probabilistic regression Special Gauss-Markov model without datum defect.- Overdetermined system of nonlinear equations on curved manifolds inconsistent system of directional observational equations.- The fourth problem of probabilistic regression Special Gauss-Markov model with random effects.- Appendix A-D.- References.- Index.

Linear Regression with Limited Observation

Abstract: We consider the most common variants of linear regression, including Ridge, Lasso and Support-vector regression, in a setting where the learner is allowed to observe only a fixed number of attributes of each example at training time. We present simple and efficient algorithms for these problems: for Lasso and Ridge regression they need the same total number of attributes (up to constants) as do full-information algorithms, for reaching a certain accuracy. For Support-vector regression, we require exponentially less attributes compared to the state of the art. By that, we resolve an open problem recently posed by Cesa-Bianchi et al. (2010). Experiments show the theoretical bounds to be justified by superior performance compared to the state of the art.