Showing papers on "Resampling published in 2012"

Permutation Tests: A Practical Guide to Resampling Methods for Testing Hypotheses

Abstract: This book provides a step-by-step manual on the application of permutation tests in biology, medicine, science, and engineering. Its intuitive and informal style will ideally suit it as a text for students and researchers coming to these methods for the first time. In particular, it shows how the problems of missing and censored data, nonresponders, after-the-fact covariates, and outliers may be handled.

Double-check: validation of diagnostic statistics for PLS-DA models in metabolomics studies.

Abstract: Partial Least Squares-Discriminant Analysis (PLS-DA) is a PLS regression method with a special binary ‘dummy’ y-variable and it is commonly used for classification purposes and biomarker selection in metabolomics studies. Several statistical approaches are currently in use to validate outcomes of PLS-DA analyses e.g. double cross validation procedures or permutation testing. However, there is a great inconsistency in the optimization and the assessment of performance of PLS-DA models due to many different diagnostic statistics currently employed in metabolomics data analyses. In this paper, properties of four diagnostic statistics of PLS-DA, namely the number of misclassifications (NMC), the Area Under the Receiver Operating Characteristic (AUROC), Q 2 and Discriminant Q 2 (DQ 2) are discussed. All four diagnostic statistics are used in the optimization and the performance assessment of PLS-DA models of three different-size metabolomics data sets obtained with two different types of analytical platforms and with different levels of known differences between two groups: control and case groups. Statistical significance of obtained PLS-DA models was evaluated with permutation testing. PLS-DA models obtained with NMC and AUROC are more powerful in detecting very small differences between groups than models obtained with Q 2 and Discriminant Q 2 (DQ 2). Reproducibility of obtained PLS-DA models outcomes, models complexity and permutation test distributions are also investigated to explain this phenomenon. DQ 2 and Q 2 (in contrary to NMC and AUROC) prefer PLS-DA models with lower complexity and require higher number of permutation tests and submodels to accurately estimate statistical significance of the model performance. NMC and AUROC seem more efficient and more reliable diagnostic statistics and should be recommended in two group discrimination metabolomic studies.

Testing the Significance of a Correlation With Nonnormal Data: Comparison of Pearson, Spearman, Transformation, and Resampling Approaches

Abstract: It is well known that when data are nonnormally distributed, a test of the significance of Pearson's r may inflate Type I error rates and reduce power. Statistics textbooks and the simulation literature provide several alternatives to Pearson's correlation. However, the relative performance of these alternatives has been unclear. Two simulation studies were conducted to compare 12 methods, including Pearson, Spearman's rank-order, transformation, and resampling approaches. With most sample sizes (n ≥ 20), Type I and Type II error rates were minimized by transforming the data to a normal shape prior to assessing the Pearson correlation. Among transformation approaches, a general purpose rank-based inverse normal transformation (i.e., transformation to rankit scores) was most beneficial. However, when samples were both small (n ≤ 10) and extremely nonnormal, the permutation test often outperformed other alternatives, including various bootstrap tests.

RESAMPLING FEWER THAN n OBSERVATIONS: GAINS, LOSSES, AND REMEDIES FOR LOSSES

Abstract: We discuss a number of resampling schemes in which m = o(n) observations are resampled. We review nonparametric bootstrap failure and give results old and new on how the m out of n with replacement and without replacement bootstraps work. We extend work of Bickel and Yahav (1988) to show that m out of n bootstraps can be made second order correct, if the usual nonparametric bootstrap is correct and study how these extrapolation techniques work when the nonparametric bootstrap does not.

Explanation of Two Anomalous Results in Statistical Mediation Analysis

Abstract: Previous studies of different methods of testing mediation models have consistently found two anomalous results. The first result is elevated Type I error rates for the bias-corrected and accelerated bias-corrected bootstrap tests not found in nonresampling tests or in resampling tests that did not include a bias correction. This is of special concern as the bias-corrected bootstrap is often recommended and used due to its higher statistical power compared with other tests. The second result is statistical power reaching an asymptote far below 1.0 and in some conditions even declining slightly as the size of the relationship between X and M, a, increased. Two computer simulations were conducted to examine these findings in greater detail. Results from the first simulation found that the increased Type I error rates for the bias-corrected and accelerated bias-corrected bootstrap are a function of an interaction between the size of the individual paths making up the mediated effect and the sample size, such...

On the effectiveness of preprocessing methods when dealing with different levels of class imbalance

Abstract: The present paper investigates the influence of both the imbalance ratio and the classifier on the performance of several resampling strategies to deal with imbalanced data sets. The study focuses on evaluating how learning is affected when different resampling algorithms transform the originally imbalanced data into artificially balanced class distributions. Experiments over 17 real data sets using eight different classifiers, four resampling algorithms and four performance evaluation measures show that over-sampling the minority class consistently outperforms under-sampling the majority class when data sets are strongly imbalanced, whereas there are not significant differences for databases with a low imbalance. Results also indicate that the classifier has a very poor influence on the effectiveness of the resampling strategies.

Spatial cross-validation and bootstrap for the assessment of prediction rules in remote sensing: The R package sperrorest

Abstract: Novel computational and statistical prediction methods such as the support vector machine are becoming increasingly popular in remote-sensing applications and need to be compared to more traditional approaches like maximum-likelihood classification. However, the accuracy assessment of such predictive models in a spatial context needs to account for the presence of spatial autocorrelation in geospatial data by using spatial cross-validation and bootstrap strategies instead of their now more widely used non-spatial equivalent. These spatial resampling-based estimation procedures were therefore implemented in a new package ‘sperrorest’ for the open-source statistical data analysis software R. This package is introduced using the example of the detection of rock-glacier flow structures from IKONOS-derived Gabor texture features and terrain attribute data.

Bootstrap LR tests of stationarity, common trends and cointegration

Abstract: This paper considers the likelihood ratio (LR) tests of stationarity, common trends and cointegration for multivariate time series. As the distribution of these tests is not known, a bootstrap version is proposed via a state- space representation. The bootstrap samples are obtained from the Kalman filter innovations under the null hypothesis. Monte Carlo simulations for the Gaussian univariate random walk plus noise model show that the bootstrap LR test achieves higher power for medium-sized deviations from the null hypothesis than a locally optimal and one-sided Lagrange Multiplier (LM) test that has a known asymptotic distribution. The power gains of the bootstrap LR test are significantly larger for testing the hypothesis of common trends and cointegration in multivariate time series, as the alternative asymptotic procedure – obtained as an extension of the LM test of stationarity – does not possess properties of optimality. Finally, it is shown that the (pseudo-)LR tests maintain good size and power p...

On adaptive resampling strategies for sequential Monte Carlo methods

Abstract: Sequential Monte Carlo (SMC) methods are a class of techniques to sample approximately from any sequence of probability distributions using a combination of importance sampling and resampling steps. This paper is concerned with the convergence analysis of a class of SMC methods where the times at which resampling occurs are computed online using criteria such as the effective sample size. This is a popular approach amongst practitioners but there are very few convergence results available for these methods. By combining semigroup techniques with an original coupling argument, we obtain functional central limit theorems and uniform exponential concentration estimates for these algorithms.

On adaptive resampling strategies for sequential Monte Carlo methods

Abstract: Sequential Monte Carlo (SMC) methods are a class of techniques to sample approximately from any sequence of probability distributions using a combination of importance sampling and resampling steps. This paper is concerned with the convergence analysis of a class of SMC methods where the times at which resampling occurs are computed online using criteria such as the effective sample size. This is a popular approach amongst practitioners but there are very few convergence results available for these methods. By combining semigroup techniques with an original coupling argument, we obtain functional central limit theorems and uniform exponential concentration estimates for these algorithms.

Resampling methods for meta-model validation with recommendations for evolutionary computation

Abstract: Meta-modeling has become a crucial tool in solving expensive optimization problems. Much of the work in the past has focused on finding a good regression method to model the fitness function. Examples include classical linear regression, splines, neural networks, Kriging and support vector regression. This paper specifically draws attention to the fact that assessing model accuracy is a crucial aspect in the meta-modeling framework. Resampling strategies such as cross-validation, subsampling, bootstrapping, and nested resampling are prominent methods for model validation and are systematically discussed with respect to possible pitfalls, shortcomings, and specific features. A survey of meta-modeling techniques within evolutionary optimization is provided. In addition, practical examples illustrating some of the pitfalls associated with model selection and performance assessment are presented. Finally, recommendations are given for choosing a model validation technique for a particular setting.

Conditional modeling and the jitter method of spike resampling.

Abstract: The existence and role of fine-temporal structure in the spiking activity of central neurons is the subject of an enduring debate among physiologists. To a large extent, the problem is a statistical one: what inferences can be drawn from neurons monitored in the absence of full control over their presynaptic environments? In principle, properly crafted resampling methods can still produce statistically correct hypothesis tests. We focus on the approach to resampling known as jitter. We review a wide range of jitter techniques, illustrated by both simulation experiments and selected analyses of spike data from motor cortical neurons. We rely on an intuitive and rigorous statistical framework known as conditional modeling to reveal otherwise hidden assumptions and to support precise conclusions. Among other applications, we review statistical tests for exploring any proposed limit on the rate of change of spiking probabilities, exact tests for the significance of repeated fine-temporal patterns of spikes, and the construction of acceptance bands for testing any purported relationship between sensory or motor variables and synchrony or other fine-temporal events.

Semi-Supervised Learning of Class Balance under Class-Prior Change by Distribution Matching

Abstract: In real-world classification problems, the class balance in the training dataset does not necessarily reflect that of the test dataset, which can cause significant estimation bias. If the class ratio of the test dataset is known, instance re-weighting or resampling allows systematical bias correction. However, learning the class ratio of the test dataset is challenging when no labeled data is available from the test domain. In this paper, we propose to estimate the class ratio in the test dataset by matching probability distributions of training and test input data. We demonstrate the utility of the proposed approach through experiments.

Closed-form density-based framework for automatic detection of cellular morphology changes

Abstract: A primary method for studying cellular function is to examine cell morphology after a given manipulation. Fluorescent markers attached to proteins/intracellular structures of interest in conjunction with 3D fluorescent microscopy are frequently exploited for functional analysis. Despite the central role of morphology comparisons in cell biological approaches, few statistical tools are available that allow biological scientists without a high level of statistical training to quantify the similarity or difference of fluorescent images containing multifactorial information. We transform intracellular structures into kernels and develop a multivariate two-sample test that is nonparametric and asymptotically normal to directly and quantitatively compare cellular morphologies. The asymptotic normality bypasses the computationally intensive calculations used by the usual resampling techniques to compute the P-value. Because all parameters required for the statistical test are estimated directly from the data, it does not require any subjective decisions. Thus, we provide a black-box method for unbiased, automated comparison of cell morphology. We validate the performance of our test statistic for finite synthetic samples and experimental data. Employing our test for the comparison of the morphology of intracellular multivesicular bodies, we detect changes in their distribution after disruption of the cellular microtubule cytoskeleton with high statistical significance in fixed samples and live cell analysis. These results demonstrate that density-based comparison of multivariate image information is a powerful tool for automated detection of cell morphology changes. Moreover, the underlying mathematics of our test statistic is a general technique, which can be applied in situations where two data samples are compared.

Position weight matrix, gibbs sampler, and the associated significance tests in motif characterization and prediction.

Abstract: Position weight matrix (PWM) is not only one of the most widely used bioinformatic methods, but also a key component in more advanced computational algorithms (e.g., Gibbs sampler) for characterizing and discovering motifs in nucleotide or amino acid sequences. However, few generally applicable statistical tests are available for evaluating the significance of site patterns, PWM, and PWM scores (PWMS) of putative motifs. Statistical significance tests of the PWM output, that is, site-specific frequencies, PWM itself, and PWMS, are in disparate sources and have never been collected in a single paper, with the consequence that many implementations of PWM do not include any significance test. Here I review PWM-based methods used in motif characterization and prediction (including a detailed illustration of the Gibbs sampler for de novo motif discovery), present statistical and probabilistic rationales behind statistical significance tests relevant to PWM, and illustrate their application with real data. The multiple comparison problem associated with the test of site-specific frequencies is best handled by false discovery rate methods. The test of PWM, due to the use of pseudocounts, is best done by resampling methods. The test of individual PWMS for each sequence segment should be based on the extreme value distribution.

The importance of parameter resampling for soil moisture data assimilation into hydrologic models using the particle filter

Abstract: The Sequential Importance Sampling with Resampling (SISR) particle filter and the SISR with parameter resampling particle filter (SISR-PR) are evaluated for their performance in soil moisture assimilation and the consequent effect on baseflow generation. With respect to the resulting soil moisture time series, both filters perform appropriately. However, the SISR filter has a negative effect on the baseflow due to inconsistency between the parameter values and the states after the assimilation. In order to overcome this inconsistency, parameter resampling is applied along with the SISR filter, to obtain consistent parameter values with the analyzed soil moisture state. Extreme parameter replication, which could lead to a particle collapse, is avoided by the perturbation of the parameters with white noise. Both the modeled soil moisture and baseflow are improved if the complementary parameter resampling is applied. The SISR filter with parameter resampling offers an efficient way to deal with biased observations. The robustness of the methodology is evaluated for 3 model parameter sets and 3 assimilation frequencies. Overall, the results in this paper indicate that the particle filter is a promising tool for hydrologic modeling purposes, but that an additional parameter resampling may be necessary to consistently update all state variables and fluxes within the model.

Blind Sampling Rate Offset Estimation and Compensation in Wireless Acoustic Sensor Networks with Application to Beamforming

Abstract: Beamforming methods for speech enhancement in wireless acoustic sensor networks (WASNs) have recently attracted the attention of the research community. One of the major obstacles in implementing speech processing algorithms in WASN is the sampling rate offsets between the nodes. As nodes utilize individual clock sources, sampling rate offsets are inevitable and may cause severe performance degradation. In this paper, a blind procedure for estimating the sampling rate offsets is derived. The procedure is applicable to speech-absent time segments with slow time-varying interference statistics. The proposed procedure is based on the phase drift of the coherence between two signals sampled at different sampling rates. Resampling the signals with Lagrange polynomials interpolation method compensates for the sampling rate offsets. An extensive experimental study, utilizing the transfer function generalized sidelobe canceller (TFGSC), exemplifies the problem and its solution.

Applying adaptive over-sampling technique based on data density and cost-sensitive SVM to imbalanced learning

Abstract: Resampling method is a popular and effective technique to imbalanced learning. However, most resampling methods ignore data density information and may lead to overfitting. A novel adaptive over-sampling technique based on data density (ASMOBD) is proposed in this paper. Compared with existing resampling algorithms, ASMOBD can adaptively synthesize different number of new samples around each minority sample according to its level of learning difficulty. Therefore, this method makes the decision region more specific and can eliminate noise. What's more, to avoid over generalization, two smoothing methods are proposed. Cost- Sensitive learning is also an effective technique to imbalanced learning. In this paper, ASMOBD and Cost-Sensitive SVM are combined. Experiments show that our methods perform better than various state-of-art approaches on 9 UCI datasets by using metrics of G-mean and area under the receiver operation curve (AUC).

A Studentized Permutation Test for the Comparison of Spatial Point Patterns

Abstract: In this study, a new test is proposed for the hypothesis that two (or more) observed point patterns are realizations of the same spatial point process model To this end, the point patterns are divided into disjoint quadrats, on each of which an estimate of Ripley’s K-function is calculated The two groups of empirical K-functions are compared by a permutation test using a Studentized test statistic The proposed test performs convincingly in terms of empirical level and power in a simulation study, even for point patterns where the K-function estimates on neighboring subsamples are not strictly exchangeable It also shows improved behavior compared with a test suggested by Diggle et al for the comparison of groups of independently replicated point patterns In an application to two point patterns from pathology that represent capillary positions in sections of healthy and cancerous tissue, our Studentized permutation test indicates statistical significance, although the patterns cannot be clearly distin

Goodness-of-fit testing based on a weighted bootstrap: A fast large-sample alternative to the parametric bootstrap

Abstract: The process comparing the empirical cumulative distribution function of the sample with a parametric estimate of the cumulative distribution function is known as the empirical process with estimated parameters and has been extensively employed in the literature for goodness-of-fit testing. The simplest way to carry out such goodness-of-fit tests, especially in a multivariate setting, is to use a parametric bootstrap. Although very easy to implement, the parametric bootstrap can become very computationally expensive as the sample size, the number of parameters, or the dimension of the data increase. An alternative resampling technique based on a fast weighted bootstrap is proposed in this paper, and is studied both theoretically and empirically. The outcome of this work is a generic and computationally efficient multiplier goodness-of-fit procedure that can be used as a large-sample alternative to the parametric bootstrap. In order to approximately determine how large the sample size needs to be for the parametric and weighted bootstraps to have roughly equivalent powers, extensive Monte Carlo experiments are carried out in dimension one, two and three, and for models containing up to nine parameters. The computational gains resulting from the use of the proposed multiplier goodness-of-fit procedure are illustrated on trivariate financial data. A by-product of this work is a fast large-sample goodness-of-fit procedure for the bivariate and trivariate t distribution whose degrees of freedom are fixed. The Canadian Journal of Statistics 40: 480–500; 2012 © 2012 Statistical Society of Canada

Adaptive estimation of heavy right tails: resampling-based methods in action

Abstract: In this paper, we discuss an algorithm for the adaptive estimation of a positive extreme value index, γ, the primary parameter in Statistics of Extremes. Apart from the classical extreme value index estimators, we suggest the consideration of associated second-order corrected-bias estimators, and propose the use of resampling-based computer-intensive methods for an asymptotically consistent choice of the thresholds to use in the adaptive estimation of γ. The algorithm is described for a classical γ-estimator and associated corrected-bias estimator, but it can work similarly for the estimation of other parameters of extreme events, like a high quantile, the probability of exceedance or the return period of a high level.

Information Ratio Test for Model Misspecification in Quasi-Likelihood Inference

Abstract: In this article, we focus on the circumstances in quasi-likelihood inference that the estimation accuracy of mean structure parameters is guaranteed by correct specification of the first moment, but the estimation efficiency could be diminished due to misspecification of the second moment. We propose an information ratio (IR) statistic to test for model misspecification of the variance/covariance structure through a comparison between two forms of information matrix: the negative sensitivity matrix and the variability matrix. We establish asymptotic distributions of the proposed IR test statistics. We also suggest an approximation to the asymptotic distribution of the IR statistic via a perturbation resampling method. Moreover, we propose a selection criterion based on the IR test to select the best fitting variance/covariance structure from a class of candidates. Through simulation studies, it is shown that the IR statistic provides a powerful statistical tool to detect different scenarios of misspecific...

Optimal model selection in density estimation

Abstract: We build penalized least-squares estimators using the slope heuristic and resampling penalties. We prove oracle inequalities for the selected estimator with leading constant asymptotically equal to $1$. We compare the practical performances of these methods in a short simulation study.