Resampling

About: Resampling is a research topic. Over the lifetime, 5428 publications have been published within this topic receiving 242291 citations.

Bootstrap methods for adaptive designs.

Abstract: Adaptive designs generate dependent sequences of random variables that are not exchangeable. Therefore, it is not obvious how to employ a resampling scheme for confidence interval estimation. We propose a simple procedure where observed response rates from an adaptive experiment are input to a simulation program. The program then generates sequences from the adaptive sampling scheme. We compare, via simulation, three bootstrap confidence intervals with the asymptotic confidence interval for two adaptive designs useful for clinical trials. A simple ranking of simulated response rates yields a confidence interval approximation with coverage close to 1 - α in most cases. The method allows us to incorporate such complexities as staggered entry and delayed response. We give an example of its utility on a clinical trial of fluoxetine in depression.

An evaluation of resampling methods for assessment of survival risk prediction in high-dimensional settings.

Abstract: Resampling techniques are often used to provide an initial assessment of accuracy for prognostic prediction models developed using high-dimensional genomic data with binary outcomes. Risk prediction is most important, however, in medical applications and frequently the outcome measure is a right-censored time-to-event variable such as survival. Although several methods have been developed for survival risk prediction with high-dimensional genomic data, there has been little evaluation of the use of resampling techniques for the assessment of such models. Using real and simulated datasets, we compared several resampling techniques for their ability to estimate the accuracy of risk prediction models. Our study showed that accuracy estimates for popular resampling methods, such as sample splitting and leave-one-out cross validation (Loo CV), have a higher mean square error than for other methods. Moreover, the large variability of the split-sample and Loo CV may make the point estimates of accuracy obtained using these methods unreliable and hence should be interpreted carefully. A k-fold cross-validation with k = 5 or 10 was seen to provide a good balance between bias and variability for a wide range of data settings and should be more widely adopted in practice.

Detecting trends in raptor counts: power and Type I error rates of various statistical tests

Abstract: We conducted simulations that estimated power and Type I error rates of statistical tests for detecting trends in raptor population count data collected from a single monitoring site. Results of the simulations were used to help analyze count data of bald eagles (HaIi- aeetus leucocephalus) from 7 national forests in Michigan, Minnesota, and Wisconsin during 1 980-1 989. Seven statistical tests were evaluated, including simple linear regres- sion on the log scale and linear regression with a permutation test. Using 1,000 replica- tions each, we simulated n = 1 0 and n = 50 years of count data and trends ranging from -5 to 5O/% change/year. We evaluated the tests at 3 critical levels (a = 0.01, 0.05, and 0.10) for both upper- and lower-tailed tests. Exponential count data were simulated by adding sampling error with a coefficient of variation of 400/o from either a log-normal or autocorrelated log-normal distribution. Not surprisingly, tests performed with 50 years of data were much more powerful than tests with 1 O years of data. Positive autocorrelation inflated a-levels upward from their nominal levels, making the tests less conservative and more likely to reject the null hypothesis of no trend. Of the tests studied, Cox and Stuart's test and Pollard's test clearly had lower power than the others. Surprisingly, the linear re- gression t-test, Collins' linear regression permutation test, and the nonparametric Lehmann's and Mann's tests all had similar power in our simulations. Analyses of the count data suggested that bald eagles had increasing trends on at least 2 of the 7 national forests during 1 980-1 989.

Antithetic resampling for the bootstrap

Abstract: SUMMARY An antithetic variates method for the bootstrap is proposed and discussed. It is applicable quite generally, to bias estimation, distribution function estimation and quantile estimation, for example. It is based on an 'antithetic permutation' of the sample, which amounts to ranking values of a certain function of the data. Once this has been done, B uniform resampling operations may be immediately converted into 2B 'effective' resampling operations, yielding greater statistical efficiency than 2B totally independent resampling operations. We show that antithetic resampling leads to positive nonnegligible gains in performance, for the same level of labour, when compared with ordinary uniform resampling.

Residual uncertainty estimation using instance-based learning with applications to hydrologic forecasting

Abstract: . A non-parametric method is applied to quantify residual uncertainty in hydrologic streamflow forecasting. This method acts as a post-processor on deterministic model forecasts and generates a residual uncertainty distribution. Based on instance-based learning, it uses a k nearest-neighbour search for similar historical hydrometeorological conditions to determine uncertainty intervals from a set of historical errors, i.e. discrepancies between past forecast and observation. The performance of this method is assessed using test cases of hydrologic forecasting in two UK rivers: the Severn and Brue. Forecasts in retrospect were made and their uncertainties were estimated using kNN resampling and two alternative uncertainty estimators: quantile regression (QR) and uncertainty estimation based on local errors and clustering (UNEEC). Results show that kNN uncertainty estimation produces accurate and narrow uncertainty intervals with good probability coverage. Analysis also shows that the performance of this technique depends on the choice of search space. Nevertheless, the accuracy and reliability of uncertainty intervals generated using kNN resampling are at least comparable to those produced by QR and UNEEC. It is concluded that kNN uncertainty estimation is an interesting alternative to other post-processors, like QR and UNEEC, for estimating forecast uncertainty. Apart from its concept being simple and well understood, an advantage of this method is that it is relatively easy to implement.