These investigations and several others that have beenpublishedwithin recentyears compel us us to hold the view that the ratio of the vital capacity to the body length, trunk length, chest circumference,surfacearea or weight or any combination of thesemeasurements, is too variable to admit of any workable standardor normal value. On the other hand the vital capacity of each individual, after he had becomeaccustomedto the use of the spirometer,will be found to be subjectto but small variations as long as good health is maintained. Thereseems to beevidenceto show that a reductionin the vital capacityis ofen the first sign of a progressivedamageto the respiratorytissue.

The cancer cell

High dimension, low sample size data are emerging in various areas of science. We find a common structure underlying many such data sets by using a non-standard type of asymptotics: the dimension tends to ∞ while the sample size is fixed. Our analysis shows a tendency for the data to lie deterministically at the vertices of a regular simplex. Essentially all the randomness in the data appears only as a random rotation of this simplex. This geometric representation is used to obtain several new statistical insights. Copyright 2005 Royal Statistical Society.

Geometric representation of high dimension, low sample size data

We propose a method for nonparametric density estimation that exhibits robustness to contamination of the training sample. This method achieves robustness by combining a traditional kernel density estimator (KDE) with ideas from classical M-estimation. We interpret the KDE based on a positive semi-definite kernel as a sample mean in the associated reproducing kernel Hilbert space. Since the sample mean is sensitive to outliers, we estimate it robustly via M-estimation, yielding a robust kernel density estimator (RKDE).

An RKDE can be computed efficiently via a kernelized iteratively re-weighted least squares (IRWLS) algorithm. Necessary and sufficient conditions are given for kernelized IRWLS to converge to the global minimizer of the M-estimator objective function. The robustness of the RKDE is demonstrated with a representer theorem, the influence function, and experimental results for density estimation and anomaly detection.

Robust kernel density estimation

Data based identification and prediction of nonlinear and complex dynamical systems

Applied nonparametric statistical methods (2nd edition ), by P. Sprent. Pp 342. £19-50. 1993. ISBN 0-412-44980-3 (Chapman and Hall)

Statistical depth functions provide from the deepest point a center-outward ordering of multidimensional data. In this sense, depth functions can measure the extremeness or outlyingness of a data point with respect to a given data set. Hence, they can detect outliers observations that appear extreme relative to the rest of the observations. Of the various statistical depths, the spatial depth is especially appealing because of its computational efficiency and mathematical tractability. In this article, we propose a novel statistical depth, the kernelized spatial depth (KSD), which generalizes the spatial depth via positive definite kernels. By choosing a proper kernel, the KSD can capture the local structure of a data set while the spatial depth fails. We demonstrate this by the half-moon data and the ring-shaped data. Based on the KSD, we propose a novel outlier detection algorithm, by which an observation with a depth value less than a threshold is declared as an outlier. The proposed algorithm is simple in structure: the threshold is the only one parameter for a given kernel. It applies to a one-class learning setting, in which normal observations are given as the training data, as well as to a missing label scenario, where the training set consists of a mixture of normal observations and outliers with unknown labels. We give upper bounds on the false alarm probability of a depth-based detector. These upper bounds can be used to determine the threshold. We perform extensive experiments on synthetic data and data sets from real applications. The proposed outlier detector is compared with existing methods. The KSD outlier detector demonstrates a competitive performance.

Outlier Detection with the Kernelized Spatial Depth Function

Consistency and asymptotic distribution of the Theil–Sen estimator

Motivated by applications to goodness of fit testing, the empirical likelihood approach is generalized to allow for the number of constraints to grow with the sample size and for the constraints to use estimated criteria functions. The latter is needed to deal with nuisance parameters. The proposed empirical likelihood based goodness of fit tests are asymptotically distribution free. For univariate observations, tests for a specified distribution, for a distribution of parametric form, and for a symmetric distribution are presented. For bivariate observations, tests for independence are developed.

Empirical likelihood approach to goodness of fit testing

Summary: This paper deals with the construction of efficient estimators in semiparametric models without the sample splitting technique. Schick (1987) gave sufficient conditions using the leaveone-out technique for a construction without sample splitting. His conditions are stronger and more cumbersome to verify than the necessary and sufficient conditions for the existence of efficient estimators which suffice for the construction based on sample splitting. In this paper we use a conditioning argument to weaken Schick’s conditions. We shall then show that in a large class of semiparametric models and for properly chosen estimators of the score function the resulting weaker conditions reduce to the minimal conditions for the construction with sample splitting. In other words, in these models efficient estimators can be constructed without sample splitting under the same conditions as those used for the construction with sample splitting. We demonstrate our results by constructing an efficient estimator using these ideas in a semiparametric additive regression model.

On the construction of efficient estimators in semiparametric models

Background
Mean-based clustering algorithms such as bisecting k-means generally lack robustness. Although componentwise median is a more robust alternative, it can be a poor center representative for high dimensional data. We need a new algorithm that is robust and works well in high dimensional data sets e.g. gene expression data.

/pdf/robust-clustering-in-high-dimensional-data-using-statistical-2q35lrr7pd.pdf

Hanxiang Peng

Papers

Outlier Detection with the Kernelized Spatial Depth Function

Consistency and asymptotic distribution of the Theil–Sen estimator

Empirical likelihood approach to goodness of fit testing

On the construction of efficient estimators in semiparametric models

Robust clustering in high dimensional data using statistical depths