Variable selection is fundamental to high-dimensional statistical modeling, including nonparametric regression. Many approaches in use are stepwise selection procedures, which can be computationally expensive and ignore stochastic errors in the variable selection process. In this article, penalized likelihood approaches are proposed to handle these kinds of problems. The proposed methods select variables and estimate coefficients simultaneously. Hence they enable us to construct confidence intervals for estimated parameters. The proposed approaches are distinguished from others in that the penalty functions are symmetric, nonconcave on (0, ∞), and have singularities at the origin to produce sparse solutions. Furthermore, the penalty functions should be bounded by a constant to reduce bias and satisfy certain conditions to yield continuous solutions. A new algorithm is proposed for optimizing penalized likelihood functions. The proposed ideas are widely applicable. They are readily applied to a variety of ...

/pdf/variable-selection-via-nonconcave-penalized-likelihood-and-ay6pzm02g6.pdf

Variable Selection via Nonconcave Penalized Likelihood and its Oracle Properties

Convergence of Probability Measures. By P. Billingsley. Chichester, Sussex, Wiley, 1968. xii, 253 p. 9 1/4“. 117s.

Convergence of Probability Measures

From the Publisher:
Pattern recognition has long been studied in relation to many different (and mainly unrelated) applications, such as remote sensing, computer vision, space research, and medical imaging. In this book Professor Ripley brings together two crucial ideas in pattern recognition; statistical methods and machine learning via neural networks. Unifying principles are brought to the fore, and the author gives an overview of the state of the subject. Many examples are included to illustrate real problems in pattern recognition and how to overcome them.This is a self-contained account, ideal both as an introduction for non-specialists readers, and also as a handbook for the more expert reader.

Pattern recognition and neural networks

CA : A Cancer Journal for Clinicians

B-splines are attractive for nonparametric modelling, but choosing the optimal number and positions of knots is a complex task. Equidistant knots can be used, but their small and discrete number allows only limited control over smoothness and fit. We propose to use a relatively large number of knots and a difference penalty on coefficients of adjacent B-splines. We show connections to the familiar spline penalty on the integral of the squared second derivative. A short overview of B-splines, of their construction and of penalized likelihood is presented. We discuss properties of penalized B-splines and propose various criteria for the choice of an optimal penalty parameter. Nonparametric logistic regression, density estimation and scatterplot smoothing are used as examples. Some details of the computations are presented.

/pdf/flexible-smoothing-with-b-splines-and-penalties-aqyuxvm7bh.pdf

Flexible smoothing with B-splines and penalties

Analysis of variance type models are considered for a regression function or for the logarithm of a probability function, conditional probability function, density function, conditional density function, hazard function, conditional hazard function or spectral density function. Polynomial splines are used to model the main effects, and their tensor products are used to model any interaction components that are included. In the special context of survival analysis, the baseline hazard function is modeled and nonproportionality is allowed. In general, the theory involves the $L_2$ rate of convergence for the fitted model and its components. The methodology involves least squares and maximum likelihood estimation, stepwise addition of basis functions using Rao statistics, stepwise deletion using Wald statistics and model selection using the Bayesian information criterion, cross-validation or an independent test set. Publicly available software, written in C and interfaced to S/S-PLUS, is used to apply this methodology to real data.

https://projecteuclid.org/download/pdf_1/euclid.aos/1031594728

Polynomial splines and their tensor products in extended linear modeling: 1994 Wald memorial lecture

The effect of errors in variables in nonparametric regression estimation is examined. To account for errors in covariates, deconvolution is involved in the construction of a new class of kernel estimators. It is shown that optimal local and global rates of convergence of these kernel estimators can be characterized by the tail behavior of the characteristic function of the error distribution. In fact, there are two types of rates of convergence according to whether the error is ordinary smooth or super smooth. It is also shown that these results hold uniformly over a class of joint distributions of the response and the covariate, which is rich enough for many practical applications. Furthermore, to achieve optimality, we show that the convergence rates of all possible estimators have a lower bound possessed by the kernel estimators.

/pdf/nonparametric-regression-with-errors-in-variables-6hi2mi2r8s.pdf

Nonparametric regression with errors in variables

The bias of kernel methods based on local constant fits can have an adverse effect when the derivative of the marginal density or that of the regression function is large. The drawback can be repaired by considering a class of kernel estimators based on local linear fits. These estimators have the desired asymptotic properties and can be used to estimate conditional quantiles and to robustify the usual mean regression. The conditional asymptotic normality of these estimators at both boundary and interior points is established. An important consequence of the study is that the proposed method has the desired sampling properties at both boundary and interior points of the support of the design density. Therefore, our procedure does not require boundary modifications. Applications of such a local linear approximation method are discussed.

Robust Non-parametric Function Estimation

Consider a stationary time series $(\mathbf{X}_t, Y_t), t = 0, \pm 1,\ldots,$ with $\mathbf{X}_t$ being $\mathbb{R}^d$-valued and $Y_t$ real-valued The conditional mean function is given by $\theta(\mathbf{X}_0) = E(Y_0\mid\mathbf{X}_0)$ Under appropriate regularity conditions, a local average estimator of this function based on a finite realization $(\mathbf{X}_1, Y_1),\ldots,(\mathbf{X}_n, Y_n)$ can be chosen to achieve the optimal rate of convergence $n^{-1/(2 + d)}$ both pointwise and in $L_2$ norms restricted to a compact; and it can also be chosen to achieve the optimal rate of convergence $(n^{-1} \log(n))^{1/(2 + d)}$ in $L_\infty$ norm restricted to a compact Similar results hold for local median estimators of the conditional median function, which is given by $\theta(\mathbf{X}_0) = \operatorname{med}(Y_0\mid\mathbf{X}_0)$

/pdf/nonparametric-function-estimation-involving-time-series-542xm5wx9e.pdf

Nonparametric function estimation involving time series

(Headache 2010;50:1273-1277)

Objective.— To determine the prevalence of neck pain at the time of migraine treatment relative to the prevalence of nausea, a defining associated symptom of migraine.

Methods.— This is a prospective, observational cross-sectional study of 113 migraineurs, ranging in attack frequency from episodic to chronic migraine. Subjects were examined by headache medicine specialists to confirm the diagnosis of migraine and exclude both cervicogenic headache and fibromyalgia. Details of all migraines were recorded over the course of at least 1 month and until 6 qualifying migraines had been treated. For each attack, subjects recorded the presence or absence of nausea as well as the intensity of headache and neck pain (graded as none, mild, moderate, or severe).

Results.— Subjects recorded 2411 headache days, 786 of which were migraines. The majority of migraines were treated in the moderate pain stage. Regardless of the intensity of headache pain at time of treatment, neck pain was a more frequent accompaniment of migraine than was nausea (P < .0001). Prevalence of neck pain correlated with chronicity of headache as attacks moved from episodic to chronic daily headache.

Conclusions.— In this representative cross-section of migraineurs, neck pain was more commonly associated with migraine than was nausea, a defining characteristic of the disorder. Awareness of neck pain as a common associated feature of migraine may improve diagnostic accuracy and have a beneficial impact on time to treatment.

Young K. Truong

Papers

Polynomial splines and their tensor products in extended linear modeling: 1994 Wald memorial lecture

Nonparametric regression with errors in variables

Robust Non-parametric Function Estimation

Nonparametric function estimation involving time series

The prevalence of neck pain in migraine.