Showing papers on "Linear model published in 2021"

performance: An R Package for Assessment, Comparison and Testing of Statistical Models

Abstract: A crucial part of statistical analysis is evaluating a model’s quality and fit, or performance. During analysis, especially with regression models, investigating the fit of models to data also often involves selecting the best fitting model amongst many competing models. Upon investigation, fit indices should also be reported both visually and numerically to bring readers in on the investigative effort.

Logistic or linear? Estimating causal effects of experimental treatments on binary outcomes using regression analysis.

Abstract: When the outcome is binary, psychologists often use nonlinear modeling strategies such as logit or probit. These strategies are often neither optimal nor justified when the objective is to estimate causal effects of experimental treatments. Researchers need to take extra steps to convert logit and probit coefficients into interpretable quantities, and when they do, these quantities often remain difficult to understand. Odds ratios, for instance, are described as obscure in many textbooks (e.g., Gelman & Hill, 2006, p. 83). I draw on econometric theory and established statistical findings to demonstrate that linear regression is generally the best strategy to estimate causal effects of treatments on binary outcomes. Linear regression coefficients are directly interpretable in terms of probabilities and, when interaction terms or fixed effects are included, linear regression is safer. I review the Neyman-Rubin causal model, which I use to prove analytically that linear regression yields unbiased estimates of treatment effects on binary outcomes. Then, I run simulations and analyze existing data on 24,191 students from 56 middle schools (Paluck, Shepherd, & Aronow, 2013) to illustrate the effectiveness of linear regression. Based on these grounds, I recommend that psychologists use linear regression to estimate treatment effects on binary outcomes. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

Deep learning: a statistical viewpoint

Abstract: The remarkable practical success of deep learning has revealed some major surprises from a theoretical perspective. In particular, simple gradient methods easily find near-optimal solutions to non-convex optimization problems, and despite giving a near-perfect fit to training data without any explicit effort to control model complexity, these methods exhibit excellent predictive accuracy. We conjecture that specific principles underlie these phenomena: that overparametrization allows gradient methods to find interpolating solutions, that these methods implicitly impose regularization, and that overparametrization leads to benign overfitting, that is, accurate predictions despite overfitting training data. In this article, we survey recent progress in statistical learning theory that provides examples illustrating these principles in simpler settings. We first review classical uniform convergence results and why they fall short of explaining aspects of the behaviour of deep learning methods. We give examples of implicit regularization in simple settings, where gradient methods lead to minimal norm functions that perfectly fit the training data. Then we review prediction methods that exhibit benign overfitting, focusing on regression problems with quadratic loss. For these methods, we can decompose the prediction rule into a simple component that is useful for prediction and a spiky component that is useful for overfitting but, in a favourable setting, does not harm prediction accuracy. We focus specifically on the linear regime for neural networks, where the network can be approximated by a linear model. In this regime, we demonstrate the success of gradient flow, and we consider benign overfitting with two-layer networks, giving an exact asymptotic analysis that precisely demonstrates the impact of overparametrization. We conclude by highlighting the key challenges that arise in extending these insights to realistic deep learning settings.

partR2: partitioning R2 in generalized linear mixed models.

Abstract: The coefficient of determination R 2 quantifies the amount of variance explained by regression coefficients in a linear model. It can be seen as the fixed-effects complement to the repeatability R (intra-class correlation) for the variance explained by random effects and thus as a tool for variance decomposition. The R 2 of a model can be further partitioned into the variance explained by a particular predictor or a combination of predictors using semi-partial (part) R 2 and structure coefficients, but this is rarely done due to a lack of software implementing these statistics. Here, we introduce partR2, an R package that quantifies part R 2 for fixed effect predictors based on (generalized) linear mixed-effect model fits. The package iteratively removes predictors of interest from the model and monitors the change in the variance of the linear predictor. The difference to the full model gives a measure of the amount of variance explained uniquely by a particular predictor or a set of predictors. partR2 also estimates structure coefficients as the correlation between a predictor and fitted values, which provide an estimate of the total contribution of a fixed effect to the overall prediction, independent of other predictors. Structure coefficients can be converted to the total variance explained by a predictor, here called 'inclusive' R 2, as the square of the structure coefficients times total R 2. Furthermore, the package reports beta weights (standardized regression coefficients). Finally, partR2 implements parametric bootstrapping to quantify confidence intervals for each estimate. We illustrate the use of partR2 with real example datasets for Gaussian and binomial GLMMs and discuss interactions, which pose a specific challenge for partitioning the explained variance among predictors.

Derivative-Based Koopman Operators for Real-Time Control of Robotic Systems

Abstract: This article presents a generalizable methodology for data-driven identification of nonlinear dynamics that bounds the model error in terms of the prediction horizon and the magnitude of the derivatives of the system states Using higher order derivatives of general nonlinear dynamics that need not be known, we construct a Koopman-operator-based linear representation and utilize Taylor series accuracy analysis to derive an error bound The resulting error formula is used to choose the order of derivatives in the basis functions and obtain a data-driven Koopman model using a closed-form expression that can be computed in real time Using the inverted pendulum system, we illustrate the robustness of the error bounds given noisy measurements of unknown dynamics, where the derivatives are estimated numerically When combined with control, the Koopman representation of the nonlinear system has marginally better performance than competing nonlinear modeling methods, such as SINDy and NARX In addition, as a linear model, the Koopman approach lends itself readily to efficient control design tools, such as linear–quadratic regulator, whereas the other modeling approaches require nonlinear control methods The efficacy of the approach is further demonstrated with simulation and experimental results on the control of a tail-actuated robotic fish Experimental results show that the proposed data-driven control approach outperforms a tuned proportional–integral–derivative controller and that updating the data-driven model online significantly improves performance in the presence of unmodeled fluid disturbance This article is complemented with a video available at https://youtube/9_wx0tdDta0

Standard SOGI-FLL and Its Close Variants: Precise Modeling in LTP Framework and Determining Stability Region/Robustness Metrics

Abstract: In recent years, single-phase frequency-locked loops (FLLs) are gaining more popularity as a signal processing and synchronization tool in a wide variety of engineering applications. In the power and energy area, a basic structure in designing the majority of available single-phase FLLs is the second-order generalized integrator-based FLL (SOGI-FLL), which is a nonlinear feedback control system. This nonlinearity makes the SOGI-FLL analysis complicated. To deal with this problem, some attempts to derive linear models for the SOGI-FLL have been made in very recent years. The available linear models, however, are not able to accurately predict the dynamic behavior, stability region, and robustness metrics of the SOGI-FLL. The situation is even worse for close variants of the SOGI-FLL because some of them have no linear model at all. Filling these gaps in research is the main goal of this article. To this end, the structural relationship among the SOGI-FLL and its variants is identified first. Based on this information and deriving the linear time-periodic (LTP) model of a recently proposed extended SOGI-FLL, the LTP models of the standard SOGI-FLL and its close variants are obtained. The accuracy assessment of these LTP models, discussion about their limitations, and performing the stability analysis using them are other contributions of this article.

The box-cox transformation: review and extensions

Abstract: The Box-Cox power transformation family for non-negative responses in linear models has a long and interesting history in both statistical practice and theory, which we summarize. The relationship between generalized linear models and log transformed data is illustrated. Extensions investigated include the transform both sides model and the Yeo-Johnson transformation for observations that can be positive or negative. The paper also describes an extended Yeo-Johnson transformation that allows positive and negative responses to have different power transformations. Analyses of data show this to be necessary. Robustness enters in the fan plot for which the forward search provides an ordering of the data. Plausible transformations are checked with an extended fan plot. These procedures are used to compare parametric power transformations with nonparametric transformations produced by smoothing.

Prediction of geometry deviations in additive manufactured parts: comparison of linear regression with machine learning algorithms

Abstract: Dimensional accuracy in additive manufacturing (AM) is still an issue compared with the tolerances for injection molding. In order to make AM suitable for the medical, aerospace, and automotive industries, geometry variations should be controlled and managed with a tight tolerance range. In the previously published article, the authors used statistical analysis to develop linear models for the prediction of dimensional features of laser-sintered specimens. Two identical builds with the same material, process, and build parameters were produced, resulting in 434 samples for mechanical testing (ISO 527-2 1BA). The developed linear models had low accuracy, and therefore needed an application of more advanced data analysis techniques. In this work, machine learning techniques are applied for the same data, and results are compared with the previously reported linear models. The linear regression model is the best for width. Multilayer perceptron and gradient boost regressor models have outperformed other for thickness and length. The recommendations on how the developed models can be used in the future are proposed.

Theoretical Analysis of Self-Training with Deep Networks on Unlabeled Data

Abstract: Self-training algorithms, which train a model to fit pseudolabels predicted by another previously-learned model, have been very successful for learning with unlabeled data using neural networks. However, the current theoretical understanding of self-training only applies to linear models. This work provides a unified theoretical analysis of self-training with deep networks for semi-supervised learning, unsupervised domain adaptation, and unsupervised learning. At the core of our analysis is a simple but realistic ``"expansion" assumption, which states that a low-probability subset of the data must expand to a neighborhood with large probability relative to the subset. We also assume that neighborhoods of examples in different classes have minimal overlap. We prove that under these assumptions, the minimizers of population objectives based on self-training and input-consistency regularization will achieve high accuracy with respect to ground-truth labels. By using off-the-shelf generalization bounds, we immediately convert this result to sample complexity guarantees for neural nets that are polynomial in the margin and Lipschitzness. Our results help explain the empirical successes of recently proposed self-training algorithms which use input consistency regularization.

The Application of Multiple Linear Regression and Artificial Neural Network Models for Yield Prediction of Very Early Potato Cultivars before Harvest

Abstract: Yield forecasting is a rational and scientific way of predicting future occurrences in agriculture—the level of production effects. Its main purpose is reducing the risk in the decision-making process affecting the yield in terms of quantity and quality. The aim of the following study was to generate a linear and non-linear model to forecast the tuber yield of three very early potato cultivars: Arielle, Riviera, and Viviana. In order to achieve the set goal of the study, data from the period 2010–2017 were collected, coming from official varietal experiments carried out in northern and northwestern Poland. The linear model has been created based on multiple linear regression analysis (MLR), while the non-linear model has been built using artificial neural networks (ANN). The created models can predict the yield of very early potato varieties on 20th June. Agronomic, phytophenological, and meteorological data were used to prepare the models, and the correctness of their operation was verified on the basis of separate sets of data not participating in the construction of the models. For the proper validation of the model, six forecast error metrics were used: i.e., global relative approximation error (RAE), root mean square error (RMS), mean absolute error (MAE), and mean absolute percentage error (MAPE). As a result of the conducted analyses, the forecast error results for most models did not exceed 15% of MAPE. The predictive neural model NY1 was characterized by better values of quality measures and ex post forecast errors than the regression model RY1.

Confounding and regression adjustment in difference-in-differences studies.

Abstract: Objective To define confounding bias in difference-in-difference studies and compare regression- and matching-based estimators designed to correct bias due to observed confounders. Data sources We simulated data from linear models that incorporated different confounding relationships: time-invariant covariates with a time-varying effect on the outcome, time-varying covariates with a constant effect on the outcome, and time-varying covariates with a time-varying effect on the outcome. We considered a simple setting that is common in the applied literature: treatment is introduced at a single time point and there is no unobserved treatment effect heterogeneity. Study design We compared the bias and root mean squared error of treatment effect estimates from six model specifications, including simple linear regression models and matching techniques. Data collection Simulation code is provided for replication. Principal findings Confounders in difference-in-differences are covariates that change differently over time in the treated and comparison group or have a time-varying effect on the outcome. When such a confounding variable is measured, appropriately adjusting for this confounder (ie, including the confounder in a regression model that is consistent with the causal model) can provide unbiased estimates with optimal SE. However, when a time-varying confounder is affected by treatment, recovering an unbiased causal effect using difference-in-differences is difficult. Conclusions Confounding in difference-in-differences is more complicated than in cross-sectional settings, from which techniques and intuition to address observed confounding cannot be imported wholesale. Instead, analysts should begin by postulating a causal model that relates covariates, both time-varying and those with time-varying effects on the outcome, to treatment. This causal model will then guide the specification of an appropriate analytical model (eg, using regression or matching) that can produce unbiased treatment effect estimates. We emphasize the importance of thoughtful incorporation of covariates to address confounding bias in difference-in-difference studies.

Distributed linear regression by averaging

Abstract: Distributed statistical learning problems arise commonly when dealing with large datasets. In this setup, datasets are partitioned over machines, which compute locally, and communicate short messages. Communication is often the bottleneck. In this paper, we study one-step and iterative weighted parameter averaging in statistical linear models under data parallelism. We do linear regression on each machine, send the results to a central server and take a weighted average of the parameters. Optionally, we iterate, sending back the weighted average and doing local ridge regressions centered at it. How does this work compared to doing linear regression on the full data? Here, we study the performance loss in estimation and test error, and confidence interval length in high dimensions, where the number of parameters is comparable to the training data size. We find the performance loss in one-step weighted averaging, and also give results for iterative averaging. We also find that different problems are affected differently by the distributed framework. Estimation error and confidence interval length increases a lot, while prediction error increases much less. We rely on recent results from random matrix theory, where we develop a new calculus of deterministic equivalents as a tool of broader interest.

A machine learning approximation of the 2015 Portuguese high school student grades: A hybrid approach

Abstract: This article uses an anonymous 2014–15 school year dataset from the Directorate-General for Statistics of Education and Science (DGEEC) of the Portuguese Ministry of Education as a means to carry out a predictive power comparison between the classic multilinear regression model and a chosen set of machine learning algorithms. A multilinear regression model is used in parallel with random forest, support vector machine, artificial neural network and extreme gradient boosting machine stacking ensemble implementations. Designing a hybrid analysis is intended where classical statistical analysis and artificial intelligence algorithms are blended to augment the ability to retain valuable conclusions and well-supported results. The machine learning algorithms attain a higher level of predictive ability. In addition, the stacking appropriateness increases as the base learner output correlation matrix determinant increases and the random forest feature importance empirical distributions are correlated with the structure of p-values and the statistical significance test ascertains of the multiple linear model. An information system that supports the nationwide education system should be designed and further structured to collect meaningful and precise data about the full range of academic achievement antecedents. The article concludes that no evidence is found in favour of smaller classes.

Big Data as a Tool for Building a Predictive Model of Mill Roll Wear

Abstract: Big data analysis is becoming a daily task for companies all over the world as well as for Russian companies. With advances in technology and reduced storage costs, companies today can collect and store large amounts of heterogeneous data. The important step of extracting knowledge and value from such data is a challenge that will ultimately be faced by all companies seeking to maintain their competitiveness and place in the market. An approach to the study of metallurgical processes using the analysis of a large array of operational control data is considered. Using the example of steel rolling production, the development of a predictive model based on processing a large array of operational control data is considered. The aim of the work is to develop a predictive model of rolling mill roll wear based on a large array of operational control data containing information about the time of filling and unloading of rolls, rolled assortment, roll material, and time during which the roll is in operation. Preliminary preparation of data for modeling was carried out, which includes the removal of outliers, uncharacteristic and random measurement results (misses), as well as data gaps. Correlation analysis of the data showed that the dimensions and grades of rolled steel sheets, as well as the material from which the rolls are made, have the greatest influence on the wear of rolling mill rolls. Based on the processing of a large array of operational control data, various predictive models of the technological process were designed. The adequacy of the models was assessed by the value of the mean square error (MSE), the coefficient of determination (R2), and the value of the Pearson correlation coefficient (R) between the calculated and experimental values of the mill roll wear. In addition, the adequacy of the models was assessed by the symmetry of the values predicted by the model relative to the straight line Ypredicted = Yactual. Linear models constructed using the least squares method and cross-validation turned out to be inadequate (the coefficient of determination R2 does not exceed 0.3) to the research object. The following regressions were built on the basis of the same operational control database: Linear Regression multivariate, Lasso multivariate, Ridge multivariate, and ElasticNet multivariate. However, these models also turned out to be inadequate to the object of the research. Testing these models for symmetry showed that, in all cases, there is an underestimation of the predicted values. Models using algorithm composition have also been built. The methods of random forest and gradient boosting are considered. Both methods were found to be adequate for the object of the research (for the random forest model, the coefficient of determination is R2 = 0.798; for the gradient boosting model, the coefficient of determination is R2 = 0.847). However, the gradient boosting algorithm is recognized as preferable thanks to its high accuracy compared with the random forest algorithm. Control data for symmetry in reference to the straight line Ypredicted = Yactual showed that, in the case of developing the random forest model, there is a tendency to underestimate the predicted values (the calculated values are located below the straight line). In the case of developing a gradient boosting model, the predicted values are located symmetrically regarding the straight line Ypredicted = Yactual. Therefore, the gradient boosting model is preferred. The predictive model of mill roll wear will allow rational use of rolls in terms of minimizing overall roll wear. Thus, the proposed model will make it possible to redistribute the existing work rolls between the stands in order to reduce the total wear of the rolls.

GIMLi: Global Ionospheric total electron content model based on machine learning

Abstract: EXtreme Gradient Boosting over Decision Trees (XGBoost or XGBDT) is a powerful tool to model a wide range of processes. We propose a new approach to create a global total electron content model, using machine-learning-based techniques, in particular, gradient boosting. The model is based on the Global Ionospheric Maps computed by Universitat Politecnica de Catalunya with a tomographic-kriging combined technique (UQRG). To reduce the problem complexity, we used empirical orthogonal functions (EOFs). The created model involves the first 16 spatial EOFs. For training and validation we used the 1998–2016 data sets, and the 2017 data as a test data set. To drive the model, we used the following features: (1) geomagnetic activity indexes (Kp, Ap, AE, AU, AL) and solar activity indexes (R, F10.7); (2) derivative values from these indexes such as the mean value and standard deviations within the last 12 h, last 11 days, and last 40 days; (3) day of the year (DOY); (4) averaged EOFs for given Kp and UT, and those for a given DOY and UT. The validation data set revealed the following hyperparameters for XGBoost learning: number of trees is 100, tree depth is 6, and learning rate is 0.1. Comparisons with the NeQuick2, Klobuchar, and GEMTEC models show that machine learning achieves higher accuracy for the 2017 test data set. The global averaged root-mean-square errors and mean absolute percentage errors were about 2.5 TECU and 19% for the nonlinear GIMLi-XGBDT model, about 4 TECU and 30–40% for NeQuick2, GEMTEC, and the linear model GIMLi-LM, and about 5.2 TECU and 73% for the Klobuchar model. A 4-fully-connected-layer artificial neural network provided a higher error (3.28 TECU and 27.7%) as compared to GIMLi-XGBDT. For all models mentioned, the error peaked in the equatorial anomaly region. The solar activity increase does not affect the error of the nonlinear GIMLi-XGBDT model. However, an increase in geomagnetic activity strongly affects that model.

Vehicle dispatching in modular transit networks: A mixed-integer nonlinear programming model

Abstract: Modular vehicle (MV) technology offers the possibility of flexibly adjusting the vehicle capacity by docking/undocking modular pods into vehicles of different sizes en route to satisfy passenger demand Based on the MV technology, a modular transit network system (MTNS) concept is proposed to overcome the mismatch between fixed vehicle capacity and spatially varying travel demand in traditional public transportation systems To achieve the optimal MTNS design, a mixed-integer nonlinear programming model is developed to balance the tradeoff between the vehicle operation cost and the passenger trip time cost The nonlinear model is reformulated into a computationally tractable linear model The linear model solves the lower and upper bounds of the original nonlinear model to produce a near-optimal solution to the MTNS design This reformulated linear model can be solved with off-the-shelf commercial solvers (eg, Gurobi) Two numerical examples are used to demonstrate the applicability of the proposed model and its effectiveness in reducing system costs

Autoregressive neural networks with exogenous variables for indoor temperature prediction in buildings

Abstract: Thermal models of buildings are helpful to forecast their energy use and to enhance the control of their mechanical systems. However, these models are building-specific and require a tedious, error-prone and time-consuming development effort relying on skilled building energy modelers. Compared to white-box and gray-box models, data-driven (black-box) models require less development time and a minimal amount of information about the building characteristics. In this paper, autoregressive neural network models are compared to gray-box and black-box linear models to simulate indoor temperatures. These models are trained, validated and compared to actual experimental data obtained for an existing commercial building in Montreal (QC, Canada) equipped with roof top units for air conditioning. Results show that neural networks mimic more accurately the thermal behavior of the building when limited information is available, compared to gray-box and black-box linear models. The gray-box model does not perform adequately due to its under-parameterized nature, while the linear models cannot capture non-linear phenomena such as radiative heat transfer and occupancy. Therefore, the neural network models outperform the alternative models in the presented application, reaching a coefficient of determination R2 up to 0.824 and a root mean square error down to 1.11 °C, including the error propagation over time for a 1-week period with a 5-minute time-step. When considering a 50-hour time horizon, the best neural networks reach a much lower root mean square error of around 0.6 °C, which is suitable for applications such as model predictive control.

Nonlinear Double-Capacitor Model for Rechargeable Batteries: Modeling, Identification, and Validation

Abstract: This article proposes a new equivalent circuit model for rechargeable batteries by modifying a double-capacitor model in the literature. It is known that the original model can address the rate capacity effect and energy recovery effect inherent to batteries better than other models. However, it is a purely linear model and includes no representation of a battery’s nonlinear phenomena. Hence, this article transforms the original model by introducing a nonlinear-mapping-based voltage source and a serial RC circuit. The modification is justified by an analogy with the single-particle model. Two off-line parameter estimation approaches, termed 1.0 and 2.0, are designed for the new model to deal with the scenarios of constant-current and variable-current charging/discharging, respectively. In particular, the 2.0 approach proposes the notion of Wiener system identification based on the maximum a posteriori estimation, which allows all the parameters to be estimated in one shot while overcoming the nonconvexity or local minima issue to obtain physically reasonable estimates. Extensive experimental evaluation shows that the proposed model offers excellent accuracy and predictive capability. A comparison against the Rint and Thevenin models further points to its superiority. With high fidelity and low mathematical complexity, this model is beneficial for various real-time battery management applications.

3-D Modeling and Testing of a Stator-Magnet Transverse-Flux Linear Oscillatory Machine for Direct Compressor Drive

Abstract: In this article, a stator-magnet transverse-flux linear oscillatory machine is proposed for direct compressor drive. The robust transverse-flux structure with permanent magnets (PMs) embedded in the stator yoke and a moving-iron translator can yield high reliability and is relatively simple to fabricate. For electromagnetic performance analysis, a linear model under the no-load condition and a nonlinear model under the loaded condition are developed by taking into account the axial leakage flux and saturation effects of iron core, respectively. The effectiveness and accuracy of the proposed analytical models are verified by comparing the results with those of the finite-element analysis and the static experimental tests. Based on the measured static characteristics and damping coefficient, a system kinetic model is developed in the form of coupled equivalent electromechanical circuit, and validated by the results of dynamic test on a prototype. The key indices of the new machine are compared with those of an existing moving-magnet linear oscillatory machine, including the amount of PM usage, efficiency, and thrust density, etc. The case study results show that the proposed linear oscillatory machine is suitable for linear compressor drives.