Showing papers on "Resampling published in 2009"

Permutation Tests for Studying Classifier Performance

Abstract: We explore the framework of permutation-based p-values for assessing the behavior of the classification error. In this paper we study two simple permutation tests. The first test estimates the null distribution by permuting the labels in the data; this has been used extensively in classification problems in computational biology. The second test produces permutations of the features within classes, inspired by restricted randomization techniques traditionally used in statistics. We study the properties of these tests and present an extensive empirical evaluation on real and synthetic data. Our analysis shows that studying the classification error via permutation tests is effective; in particular, the restricted permutation test clearly reveals whether the classifier exploits the interdependency between the features in the data.

Basic validation procedures for regression models in QSAR and QSPR studies: theory and application

Abstract: Four quantitative structure-activity relationships (QSAR) and quantitative structure-property relationship (QSPR) data sets were selected from the literature and used to build regression models with 75, 56, 50 and 15 training samples. The models were validated by leave-one-out crossvalidation, leave-N-out crossvalidation (LNO), external validation, y-randomization and bootstrapping. Validations have shown that the size of the training sets is the crucial factor in determining model performance, which deteriorates as the data set becomes smaller. Models from very small data sets suffer from the impossibility of being thoroughly validated, failure and atypical behavior in certain validations (chance correlation, lack of robustness to resampling and LNO), regardless of their good performance in leave-one-out crossvalidation, fitting and even in external validation. A simple determination of the critical Nin LNO has been introduced by using the limit of 0.1 for oscillations in Q2, quantified as the variation range in single LNO and two standard deviations in multiple LNO. It has been demonstrated that it is sufficient to perform 10 -25 y-randomization and bootstrap runs for a typical model validation. The bootstrap schemes based on hierarchical cluster analysis give more reliable and reasonable results than bootstraps relying only on randomization of the complete data set. Data quality in terms of statistical significance of descriptor -yrelationships is the second important factor for model performance.Variable selection that does not eliminate insignificant descriptor - yrelationships may lead to situations in which they are not detected during model validation, especially when dealing with large data sets.

Rapid and Accurate Multiple Testing Correction and Power Estimation for Millions of Correlated Markers

Abstract: With the development of high-throughput sequencing and genotyping technologies, the number of markers collected in genetic association studies is growing rapidly, increasing the importance of methods for correcting for multiple hypothesis testing. The permutation test is widely considered the gold standard for accurate multiple testing correction, but it is often computationally impractical for these large datasets. Recently, several studies proposed efficient alternative approaches to the permutation test based on the multivariate normal distribution (MVN). However, they cannot accurately correct for multiple testing in genome-wide association studies for two reasons. First, these methods require partitioning of the genome into many disjoint blocks and ignore all correlations between markers from different blocks. Second, the true null distribution of the test statistic often fails to follow the asymptotic distribution at the tails of the distribution. We propose an accurate and efficient method for multiple testing correction in genome-wide association studies—SLIDE. Our method accounts for all correlation within a sliding window and corrects for the departure of the true null distribution of the statistic from the asymptotic distribution. In simulations using the Wellcome Trust Case Control Consortium data, the error rate of SLIDE's corrected p-values is more than 20 times smaller than the error rate of the previous MVN-based methods' corrected p-values, while SLIDE is orders of magnitude faster than the permutation test and other competing methods. We also extend the MVN framework to the problem of estimating the statistical power of an association study with correlated markers and propose an efficient and accurate power estimation method SLIP. SLIP and SLIDE are available at http://slide.cs.ucla.edu.

Implicit sampling for particle filters

Abstract: We present a particle-based nonlinear filtering scheme, related to recent work on chainless Monte Carlo, designed to focus particle paths sharply so that fewer particles are required. The main features of the scheme are a representation of each new probability density function by means of a set of functions of Gaussian variables (a distinct function for each particle and step) and a resampling based on normalization factors and Jacobians. The construction is demonstrated on a standard, ill-conditioned test problem.

Intrinsic Regression Models for Positive-Definite Matrices With Applications to Diffusion Tensor Imaging

Abstract: The aim of this paper is to develop an intrinsic regression model for the analysis of positive-definite matrices as responses in a Riemannian manifold and their association with a set of covariates, such as age and gender, in a Euclidean space. The primary motivation and application of the proposed methodology is in medical imaging. Because the set of positive-definite matrices do not form a vector space, directly applying classical multivariate regression may be inadequate in establishing the relationship between positive-definite matrices and covariates of interest, such as age and gender, in real applications. Our intrinsic regression model, which is a semiparametric model, uses a link function to map from the Euclidean space of covariates to the Riemannian manifold of positive-definite matrices. We develop an estimation procedure to calculate parameter estimates and establish their limiting distributions. We develop score statistics to test linear hypotheses on unknown parameters and develop a test procedure based on a resampling method to simultaneously assess the statistical significance of linear hypotheses across a large region of interest. Simulation studies are used to demonstrate the methodology and examine the finite sample performance of the test procedure for controlling the family-wise error rate. We apply our methods to the detection of statistical significance of diagnostic effects on the integrity of white matter in a diffusion tensor study of human immunodeficiency virus. Supplemental materials for this article are available online.

On resampling detection in re-compressed images

Abstract: Resampling detection has become a standard tool in digital image forensics. This paper investigates the important case of resampling detection in re-compressed JPEG images. We show how blocking artifacts of the previous compression step can help to increase the otherwise drastically reduced detection performance in JPEG compressed images. We give a formulation on how affine transformations of JPEG compressed images affect state-of-the-art resampling detectors and derive a new efficient detection variant, which better suits this relevant detection scenario. The principal appropriateness of using JPEG pre-compression artifacts for the detection of resampling in re-compressed images is backed with experimental evidence on a large image set and for a variety of different JPEG qualities.

A statistical framework to evaluate virtual screening

Abstract: Receiver operating characteristic (ROC) curve is widely used to evaluate virtual screening (VS) studies. However, the method fails to address the "early recognition" problem specific to VS. Although many other metrics, such as RIE, BEDROC, and pROC that emphasize "early recognition" have been proposed, there are no rigorous statistical guidelines for determining the thresholds and performing significance tests. Also no comparisons have been made between these metrics under a statistical framework to better understand their performances. We have proposed a statistical framework to evaluate VS studies by which the threshold to determine whether a ranking method is better than random ranking can be derived by bootstrap simulations and 2 ranking methods can be compared by permutation test. We found that different metrics emphasize "early recognition" differently. BEDROC and RIE are 2 statistically equivalent metrics. Our newly proposed metric SLR is superior to pROC. Through extensive simulations, we observed a "seesaw effect" – overemphasizing early recognition reduces the statistical power of a metric to detect true early recognitions. The statistical framework developed and tested by us is applicable to any other metric as well, even if their exact distribution is unknown. Under this framework, a threshold can be easily selected according to a pre-specified type I error rate and statistical comparisons between 2 ranking methods becomes possible. The theoretical null distribution of SLR metric is available so that the threshold of SLR can be exactly determined without resorting to bootstrap simulations, which makes it easy to use in practical virtual screening studies.

A rate and history-preserving resampling algorithm for neural spike trains

Abstract: Resampling methods are popular tools for exploring the statistical structure of neural spike trains. In many applications, it is desirable to have resamples that preserve certain non-Poisson properties, like refractory periods and bursting, and that are also robust to trial-to-trial variability. Pattern jitter is a resampling technique that accomplishes this by preserving the recent spiking history of all spikes and constraining resampled spikes to remain close to their original positions. The resampled spike times are maximally random up to these constraints. Dynamic programming is used to create an efficient resampling algorithm.

Online data processing: Comparison of Bayesian regularized particle filters

Abstract: The aim of this paper is to compare three regularized particle filters in an online data processing context. We carry out the comparison in terms of hidden states filtering and parameters estimation, considering a Bayesian paradigm and a univariate stochastic volatility model. We discuss the use of an improper prior distribution in the initialization of the filtering procedure and show that the Regularized Auxiliary Particle Filter (R-APF) outperforms the Regularized Sequential Importance Sampling (R-SIS) and the Regularized Sampling Importance Resampling (R-SIR).

Inference based on kernel estimates of the relative risk function in geographical epidemiology.

Abstract: Kernel smoothing is a popular approach to estimating relative risk surfaces from data on the locations of cases and controls in geographical epidemiology. The interpretation of such surfaces is facilitated by plotting of tolerance contours which highlight areas where the risk is sufficiently high to reject the null hypothesis of unit relative risk. Previously it has been recommended that these tolerance intervals be calculated using Monte Carlo randomization tests. We examine a computationally cheap alternative whereby the tolerance intervals are derived from asymptotic theory. We also examine the performance of global tests of hetereogeneous risk employing statistics based on kernel risk surfaces, paying particular attention to the choice of smoothing parameters on test power.

Introduction to Permutation and Resampling-Based Hypothesis Tests

Abstract: A resampling-based method of inference -- permutation tests -- is often used when distributional assumptions are questionable or unmet. Not only are these methods useful for obvious departures from parametric assumptions (e.g., normality) and small sample sizes, but they are also more robust than their parametric counterparts in the presences of outliers and missing data, problems that are often found in clinical child and adolescent psychology research. These methods are increasingly found in statistical software programs, making their use more feasible. In this article, we use an application-based approach to provide a brief tutorial on permutation testing. We present some historical perspectives, describe how the tests are formulated, and provide examples of common and specific situations under which the methods are most useful. Finally, we demonstrate the utility of these methods to clinical and adolescent psychology by examining four recent articles employing these methods.

Sequential Implementation of Monte Carlo Tests With Uniformly Bounded Resampling Risk

Abstract: This paper introduces an open-ended sequential algorithm for computing the p-value of a test using Monte Carlo simulation. It guarantees that the resampling risk, the probability of a different decision than the one based on the theoretical p-value, is uniformly bounded by an arbitrarily small constant. Previously suggested sequential or nonsequential algorithms, using a bounded sample size, do not have this property. Although the algorithm is open-ended, the expected number of steps is finite, except when the p-value is on the threshold between rejecting and not rejecting. The algorithm is suitable as standard for implementing tests that require (re)sampling. It can also be used in other situations: to check whether a test is conservative, iteratively to implement double bootstrap tests, and to determine the sample size required for a certain power. An R-package implementing the sequential algorithm is available online.

Selecting the number of change-points in segmented line regression.

Abstract: Segmented line regression has been used in many applications, and the problem of estimating the number of change-points in segmented line regression has been discussed in Kim et al. (2000). This paper studies asymptotic properties of the number of change-points selected by the permutation procedure of Kim et al. (2000). This procedure is based on a sequential application of likelihood ratio type tests, and controls the over-fitting probability by its design. In this paper we show that, under some conditions, the number of change-points selected by the permutation procedure is consistent. Via simulations, the permutation procedure is compared with such information-based criterior as the Bayesian Information Criterion (BIC), the Akaike Information Criterion (AIC), and Generalized Cross Validation (GCV).

Model selection by resampling penalization

Abstract: We present a new family of model selection algorithms based on the resampling heuristics. It can be used in several frameworks, do not require any knowledge about the unknown law of the data, and may be seen as a generalization of local Rademacher complexities and $V$-fold cross-validation. In the case example of least-square regression on histograms, we prove oracle inequalities, and that these algorithms are naturally adaptive to both the smoothness of the regression function and the variability of the noise level. Then, interpretating $V$-fold cross-validation in terms of penalization, we enlighten the question of choosing $V$. Finally, a simulation study illustrates the strength of resampling penalization algorithms against some classical ones, in particular with heteroscedastic data.

On the unified method with nuisance parameters

Abstract: In this paper we consider the problem of constructing confidence intervals in the presence of nuisance parameters. We discuss a generalization of the unified method of Feldman and Cousins (1998) with nuisance parameters. We demonstrate our method with several examples that arise frequently in High Energy Physics and Astronomy. We also discuss the hybrid resampling method of Chuang and Lai (1998, 2000), and implement it in some of the problems.

Resampling-Based Multiple Hypothesis Testing with Applications to Genomics: New Developments in the R/Bioconductor Package multtest

Abstract: The multtest package is a standard Bioconductor package containing a suite of functions useful for executing, summarizing, and displaying the results from a wide variety of multiple testing procedures (MTPs). In addition to many popular MTPs, the central methodological focus of the multtest package is the implementation of powerful joint multiple testing procedures. Joint MTPs are able to account for the dependencies between test statistics by effectively making use of (estimates of) the test statistics joint null distribution. To this end, two additional bootstrap-based estimates of the test statistics joint null distribution have been developed for use in the package. For asymptotically linear estimators involving single-parameter hypotheses (such as tests of means, regression parameters, and correlation parameters using t-statistics), a computationally efficient joint null distribution estimate based on influence curves is now also available. New MTPs implemented in multtest include marginal adaptive procedures for control of the false discovery rate (FDR) as well as empirical Bayes joint MTPs which can control any Type I error rate defined as a function of the numbers of false positives and true positives. Examples of such error rates include, among others, the familywise error rate and the FDR. S4 methods are available for objects of the new class EBMTP, and particular attention has been given to reducing the need for repeated resampling between function calls.

Model selection by resampling penalization

Abstract: In this paper, a new family of resampling-based penalization procedures for model selection is defined in a general framework. It generalizes several methods, including Efron's bootstrap penalization and the leave-one-out penalization recently proposed by Arlot (2008), to any exchangeable weighted bootstrap resampling scheme. In the heteroscedastic regression framework, assuming the models to have a particular structure, these resampling penalties are proved to satisfy a non-asymptotic oracle inequality with leading constant close to 1. In particular, they are asympotically optimal. Resampling penalties are used for defining an estimator adapting simultaneously to the smoothness of the regression function and to the heteroscedasticity of the noise. This is remarkable because resampling penalties are general-purpose devices, which have not been built specifically to handle heteroscedastic data. Hence, resampling penalties naturally adapt to heteroscedasticity. A simulation study shows that resampling penalties improve on V-fold cross-validation in terms of final prediction error, in particular when the signal-to-noise ratio is not large.

Using the Model-Based Residual Bootstrap to Quantify Uncertainty in Fiber Orientations From $Q$ -Ball Analysis

Abstract: Bootstrapping of repeated diffusion-weighted image datasets enables nonparametric quantification of the uncertainty in the inferred fiber orientation. The wild bootstrap and the residual bootstrap are model-based residual resampling methods which use a single dataset. Previously, the wild bootstrap method has been presented as an alternative to conventional bootstrapping for diffusion tensor imaging. Here we present a study of an implementation of model-based residual bootstrapping using q -ball analysis and compare the outputs with conventional bootstrapping. We show that model-based residual bootstrap q-ball generates results that closely match the output of the conventional bootstrap. Both the residual and conventional bootstrap of multifiber methods can be used to estimate the probability of different numbers of fiber populations existing in different brain tissues. Also, we have shown that these methods can be used to provide input for probabilistic tractography, avoiding existing limitations associated with data calibration and model selection.

Agreement among statistical significance tests for information retrieval evaluation at varying sample sizes

Abstract: Research has shown that little practical difference exists between the randomization, Student's paired t, and bootstrap tests of statistical significance for TREC ad-hoc retrieval experiments with 50 topics We compared these three tests on runs with topic sizes down to 10 topics We found that these tests show increasing disagreement as the number of topics decreases At smaller numbers of topics, the randomization test tended to produce smaller p-values than the t-test for p-values less than 01 The bootstrap exhibited a systematic bias towards p-values strictly less than the t-test with this bias increasing as the number of topics decreased We recommend the use of the randomization test although the t-test appears to be suitable even when the number of topics is small

Estimating the regional mean status and change of soil properties: two distinct objectives for soil survey

Abstract: There is a general requirement for inventories to establish the status of soils with respect to key indicator properties, and for monitoring to detect changes in these indicators. The design of sampling protocols for soil survey must meet these dual requirements. This paper emphasizes that the status and change of an indicator are different variables and so their variability may differ. We therefore may not assume that a sampling scheme that is suitable for inventory is also suitable for monitoring; this poses a practical problem since information on the variability of the change in soil is not in general available at present. In this paper some plausible statistical models of change in the soil are examined, and their implications for sampling to estimate mean change in large regions are considered. Paired sampling, with baseline and resampling at common sites, is generally preferable to sampling at independent sites on the two dates, unless the error from relocation of the sample sites is large by comparison to other sources of variation. A simple process model of soil organic carbon in soils of lowland tropical forest is used to examine the problem of resampling to estimate change. This shows the expected advantages of paired sampling for change, and the very different sampling requirements that may pertain for inventory and monitoring. One implication of these results is that, while sampling schemes for inventory may be designed on the basis of available information (from samples taken at a single time), the precise requirements for monitoring might not be apparent until a reconnaissance resampling is undertaken subsequently.

Bootstrapping for Penalized Spline Regression

Abstract: We describe and contrast several different bootstrap procedures for penalized spline smoothers. The bootstrap methods considered are variations on existing methods, developed under two different probabilistic frameworks. Under the first framework, penalized spline regression is considered as an estimation technique to find an unknown smooth function. The smooth function is represented in a high-dimensional spline basis, with spline coefficients estimated in a penalized form. Under the second framework, the unknown function is treated as a realization of a set of random spline coefficients, which are then predicted in a linear mixed model. We describe how bootstrap methods can be implemented under both frameworks, and we show theoretically and through simulations and examples that bootstrapping provides valid inference in both cases. We compare the inference obtained under both frameworks, and conclude that the latter generally produces better results than the former. The bootstrap ideas are extended to hy...

Direct estimation of the area under the receiver operating characteristic curve in the presence of verification bias.

Abstract: The area under a receiver operating characteristic (ROC) curve (AUC) is a commonly used index for summarizing the ability of a continuous diagnostic test to discriminate between healthy and diseased subjects. If all subjects have their true disease status verified, one can directly estimate the AUC nonparametrically using the Wilcoxon statistic. In some studies, verification of the true disease status is performed only for a subset of subjects, possibly depending on the result of the diagnostic test and other characteristics of the subjects. Because estimators of the AUC based only on verified subjects are typically biased, it is common to estimate the AUC from a bias-corrected ROC curve. The variance of the estimator, however, does not have a closed-form expression and thus resampling techniques are used to obtain an estimate. In this paper, we develop a new method for directly estimating the AUC in the setting of verification bias based on U-statistics and inverse probability weighting (IPW). Closed-form expressions for the estimator and its variance are derived. We also show that the new estimator is equivalent to the empirical AUC derived from the bias-corrected ROC curve arising from the IPW approach.

Method for processing sampling value of digitalization protection test control apparatus

Abstract: The present invention relates to a sampling value processing method of integrative device based on a sampling value interface, which is used for protecting and monitoring a digital substation. The method is characterized in that the sampling value interface module of the integrative device is used for real-time judgment on the effectiveness and rationality of sampling values which are transmitted from an incorporating unit; the secondary Lagrange interpolation are calculated for a small quantity of broken or lost sampling values; a window function is used for designing a high-order FIR digital filter which is used for filtering the sampling values; the on-line parameters of the threshold and the filter are regulated in a self-adapting way according to the parameters of conventional data sets transmitted by the sampling values; the protection function is resampled by adopting a secondary Lagrange algorithm of fixed frequency according to the requirements of protecting and monitoring functions; the monitoring function is first processed by iterative computation of frequency; the real-time resampling frequency s regulated according to the signal frequency; the sampling values are resampled by adopting the secondary Lagrange algorithm according to the novel sampling frequency, thus the processing method not only satisfies the requirements of protection and application, but also improves the measurement accuracy.

Kolmogorov–Smirnov test for spatially correlated data

Abstract: The Kolmogorov–Smirnov test is a convenient method for investigating whether two underlying univariate probability distributions can be regarded as undistinguishable from each other or whether an underlying probability distribution differs from a hypothesized distribution. Application of the test requires that the sample be unbiased and the outcomes be independent and identically distributed, conditions that are violated in several degrees by spatially continuous attributes, such as topographical elevation. A generalized form of the bootstrap method is used here for the purpose of modeling the distribution of the statistic D of the Kolmogorov–Smirnov test. The innovation is in the resampling, which in the traditional formulation of bootstrap is done by drawing from the empirical sample with replacement presuming independence. The generalization consists of preparing resamplings with the same spatial correlation as the empirical sample. This is accomplished by reading the value of unconditional stochastic realizations at the sampling locations, realizations that are generated by simulated annealing. The new approach was tested by two empirical samples taken from an exhaustive sample closely following a lognormal distribution. One sample was a regular, unbiased sample while the other one was a clustered, preferential sample that had to be preprocessed. Our results show that the p-value for the spatially correlated case is always larger that the p-value of the statistic in the absence of spatial correlation, which is in agreement with the fact that the information content of an uncorrelated sample is larger than the one for a spatially correlated sample of the same size.

Invalidity of the bootstrap and the m out of n bootstrap for confidence interval endpoints defined by moment inequalities

Abstract: Summary This paper analyses the finite-sample and asymptotic properties of several bootstrap and m out of n bootstrap methods for constructing confidence interval (CI) endpoints in models defined by moment inequalities. In particular, we consider using these methods directly to construct CI endpoints. By considering two very simple models, the paper shows that neither the bootstrap nor the m out of n bootstrap is valid in finite samples or in a uniform asymptotic sense in general when applied directly to construct CI endpoints. In contrast, other results in the literature show that other ways of applying the bootstrap, m out of n bootstrap, and subsampling do lead to uniformly asymptotically valid confidence sets in moment inequality models. Thus, the uniform asymptotic validity of resampling methods in moment inequality models depends on the way in which the resampling methods are employed.