scispace - formally typeset
Search or ask a question
Posted Content

Efficient Estimation in Single Index Models through Smoothing splines

TL;DR: In this paper, the authors developed a method to compute the penalized least squares estimators (PLSEs) of the parametric and nonparametric components given independent and identically distributed (i.i.d.) data.
Abstract: We consider estimation and inference in a single index regression model with an unknown but smooth link function. In contrast to the standard approach of using kernels or regression splines, we use smoothing splines to estimate the smooth link function. We develop a method to compute the penalized least squares estimators (PLSEs) of the parametric and the nonparametric components given independent and identically distributed (i.i.d.)~data. We prove the consistency and find the rates of convergence of the estimators. We establish asymptotic normality under under mild assumption and prove asymptotic efficiency of the parametric component under homoscedastic errors. A finite sample simulation corroborates our asymptotic theory. We also analyze a car mileage data set and a Ozone concentration data set. The identifiability and existence of the PLSEs are also investigated.
Citations
More filters
Journal ArticleDOI
TL;DR: This work considers estimation in the single‐index model where the link function is monotone and shows how one can solve this score equation without any reparametrization, which makes it easy to solve the score equations in high dimensions.
Abstract: We consider estimation in the single index model where the link function is monotone. For this model a profile least squares estimator has been proposed to estimate the unknown link function and index. Although it is natural to propose this procedure, it is still unknown whether it produces index estimates which converge at the parametric rate. We show that this holds if we solve a score equation corresponding to this least squares problem. Using a Lagrangian formulation, we show how one can solve this score equation without any reparametrization. This makes it easy to solve the score equations in high dimensions. We also compare our method with the Effective Dimension Reduction (EDR) and the Penalized Least Squares Estimator (PLSE) methods, both available on CRAN as R packages, and compare with link-free methods, where the covariates are ellipticallly symmetric.

36 citations

Book ChapterDOI
TL;DR: In this paper, a profile least squares estimator was proposed to estimate a fixed regression parameter in a monotone single index regression model, which is shown to be convergence and asymptotic normal.
Abstract: We consider least squares estimators of the finite regression parameter \(\boldsymbol{\alpha }\) in the single index regression model \(Y=\psi (\boldsymbol{\alpha }^T\boldsymbol{X})+\varepsilon \), where \(\boldsymbol{X}\) is a d-dimensional random vector, \({\mathbb E}(Y|\boldsymbol{X})=\psi (\boldsymbol{\alpha }^T\boldsymbol{X})\), and \(\psi \) is a monotone. It has been suggested to estimate \(\boldsymbol{\alpha }\) by a profile least squares estimator, minimizing \(\sum _{i=1}^n(Y_i-\psi (\boldsymbol{\alpha }^T\boldsymbol{X}_i))^2\) over monotone \(\psi \) and \(\boldsymbol{\alpha }\) on the boundary \(\mathcal {S}_{d-1}\) of the unit ball. Although this suggestion has been around for a long time, it is still unknown whether the estimate is \(\sqrt{n}\)-convergent. We show that a profile least squares estimator, using the same pointwise least squares estimator for fixed \(\boldsymbol{\alpha }\), but using a different global sum of squares, is \(\sqrt{n}\)-convergent and asymptotically normal. The difference between the corresponding loss functions is studied and also a comparison with other methods is given.

8 citations

Journal ArticleDOI
TL;DR: The results show that the estimation accuracy depends almost exclusively on the components of the distribution that correspond to desired subspaces or directions, relevant and important for problems where the behavior of data along a lower-dimensional space is of specific interest, such as dimension reduction or structured regression problems.
Abstract: We study the accuracy of estimating the covariance and the precision matrix of a $D$-variate sub-Gaussian distribution along a prescribed subspace or direction using the finite sample covariance. Our results show that the estimation accuracy depends almost exclusively on the components of the distribution that correspond to desired subspaces or directions. This is relevant and important for problems where the behavior of data along a lower-dimensional space is of specific interest, such as dimension reduction or structured regression problems. We also show that estimation of precision matrices is almost independent of the condition number of the covariance matrix. The presented applications include direction-sensitive eigenspace perturbation bounds, relative bounds for the smallest eigenvalue, and the estimation of the single-index model. For the latter, a new estimator, derived from the analysis, with strong theoretical guarantees and superior numerical performance is proposed.

6 citations

Posted Content
TL;DR: In this paper, a Lipschitz constrained least squares estimator (LLSE) for both the parametric and the nonparametric components given independent and identically distributed observations is proposed.
Abstract: We consider estimation and inference in a single index regression model with an unknown convex link function We propose a Lipschitz constrained least squares estimator (LLSE) for both the parametric and the nonparametric components given independent and identically distributed observations We prove the consistency and find the rates of convergence of the LLSE when the errors are assumed to have only $q \ge 2$ moments and are allowed to depend on the covariates In fact, we prove a general theorem which can be used to find the rates of convergence of LSEs in a variety of nonparametric/semiparametric regression problems under the same assumptions on the errors Moreover when $q\ge 5$, we establish $n^{-1/2}$-rate of convergence and asymptotic normality of the estimator of the parametric component Moreover the LLSE is proved to be semiparametrically efficient if the errors happen to be homoscedastic Furthermore, we develop the R package \texttt{simest} to compute the proposed estimator

6 citations

References
More filters
Book
01 Mar 1990
TL;DR: In this paper, a theory and practice for the estimation of functions from noisy data on functionals is developed, where convergence properties, data based smoothing parameter selection, confidence intervals, and numerical methods are established which are appropriate to a number of problems within this framework.
Abstract: This book serves well as an introduction into the more theoretical aspects of the use of spline models. It develops a theory and practice for the estimation of functions from noisy data on functionals. The simplest example is the estimation of a smooth curve, given noisy observations on a finite number of its values. Convergence properties, data based smoothing parameter selection, confidence intervals, and numerical methods are established which are appropriate to a number of problems within this framework. Methods for including side conditions and other prior information in solving ill posed inverse problems are provided. Data which involves samples of random variables with Gaussian, Poisson, binomial, and other distributions are treated in a unified optimization context. Experimental design questions, i.e., which functionals should be observed, are studied in a general context. Extensions to distributed parameter system identification problems are made by considering implicitly defined functionals.

6,120 citations

Book
24 May 2010
TL;DR: The author presents Perron-Frobenius theory of nonnegative matrices Index, a theory of matrices that combines linear equations, vector spaces, and matrix algebra with insights into eigenvalues and Eigenvectors.
Abstract: Preface 1. Linear equations 2. Rectangular systems and echelon forms 3. Matrix algebra 4. Vector spaces 5. Norms, inner products, and orthogonality 6. Determinants 7. Eigenvalues and Eigenvectors 8. Perron-Frobenius theory of nonnegative matrices Index.

4,979 citations

Journal ArticleDOI
TL;DR: In this article, sliced inverse regression (SIR) is proposed to reduce the dimension of the input variable without going through any parametric or nonparametric model-fitting process.
Abstract: Modern advances in computing power have greatly widened scientists' scope in gathering and investigating information from many variables, information which might have been ignored in the past. Yet to effectively scan a large pool of variables is not an easy task, although our ability to interact with data has been much enhanced by recent innovations in dynamic graphics. In this article, we propose a novel data-analytic tool, sliced inverse regression (SIR), for reducing the dimension of the input variable x without going through any parametric or nonparametric model-fitting process. This method explores the simplicity of the inverse view of regression; that is, instead of regressing the univariate output variable y against the multivariate x, we regress x against y. Forward regression and inverse regression are connected by a theorem that motivates this method. The theoretical properties of SIR are investigated under a model of the form, y = f(β 1 x, …, β K x, e), where the β k 's are the unknown...

2,158 citations

Book
16 Apr 2013
TL;DR: How to Construct Nonparametric Regression Estimates * Lower Bounds * Partitioning Estimates * Kernel Estimates * k-NN Estimates * Splitting the Sample * Cross Validation * Uniform Laws of Large Numbers
Abstract: Why is Nonparametric Regression Important? * How to Construct Nonparametric Regression Estimates * Lower Bounds * Partitioning Estimates * Kernel Estimates * k-NN Estimates * Splitting the Sample * Cross Validation * Uniform Laws of Large Numbers * Least Squares Estimates I: Consistency * Least Squares Estimates II: Rate of Convergence * Least Squares Estimates III: Complexity Regularization * Consistency of Data-Dependent Partitioning Estimates * Univariate Least Squares Spline Estimates * Multivariate Least Squares Spline Estimates * Neural Networks Estimates * Radial Basis Function Networks * Orthogonal Series Estimates * Advanced Techniques from Empirical Process Theory * Penalized Least Squares Estimates I: Consistency * Penalized Least Squares Estimates II: Rate of Convergence * Dimension Reduction Techniques * Strong Consistency of Local Averaging Estimates * Semi-Recursive Estimates * Recursive Estimates * Censored Observations * Dependent Observations

1,931 citations