Showing papers on "Resampling published in 2005"
TL;DR: This work compares several methods for estimating the 'true' prediction error of a prediction model in the presence of feature selection, and finds that LOOCV and 10-fold CV have the smallest bias for linear discriminant analysis and the .632+ bootstrap has the lowest mean square error.
Abstract: Motivation: In genomic studies, thousands of features are collected on relatively few samples. One of the goals of these studies is to build classifiers to predict the outcome of future observations. There are three inherent steps to this process: feature selection, model selection and prediction assessment. With a focus on prediction assessment, we compare several methods for estimating the 'true' prediction error of a prediction model in the presence of feature selection.
Results: For small studies where features are selected from thousands of candidates, the resubstitution and simple split-sample estimates are seriously biased. In these small samples, leave-one-out cross-validation (LOOCV), 10-fold cross-validation (CV) and the .632+ bootstrap have the smallest bias for diagonal discriminant analysis, nearest neighbor and classification trees. LOOCV and 10-fold CV have the smallest bias for linear discriminant analysis. Additionally, LOOCV, 5- and 10-fold CV, and the .632+ bootstrap have the lowest mean square error. The .632+ bootstrap is quite biased in small sample sizes with strong signal-to-noise ratios. Differences in performance among resampling methods are reduced as the number of specimens available increase.
Contact: annette.molinaro@yale.edu
Supplementary Information: A complete compilation of results and R code for simulations and analyses are available in Molinaro et al. (2005) (http://linus.nci.nih.gov/brb/TechReport.htm).
1,128 citations
Posted Content•
TL;DR: In this paper, a comparison of various resampling approaches that have been proposed in the literature on particle filtering is made, and it is shown using simple arguments that the so-called residual and stratified methods do yield an improvement over the basic multinomial re-sampling approach.
Abstract: This contribution is devoted to the comparison of various resampling approaches that have been proposed in the literature on particle filtering. It is first shown using simple arguments that the so-called residual and stratified methods do yield an improvement over the basic multinomial resampling approach. A simple counter-example showing that this property does not hold true for systematic resampling is given. Finally, some results on the large-sample behavior of the simple bootstrap filter algorithm are given. In particular, a central limit theorem is established for the case where resampling is performed using the residual approach.
832 citations
TL;DR: In a recent review as mentioned in this paper, the authors clarified the concepts of bias, precision and accuracy as they are commonly defined in the biostatistical literature, with a focus on the use of these concepts in quantitatively testing the performance of point estimators.
Abstract: The purpose of this review is to clarify the concepts of bias, precision and accuracy as they are commonly defined in the biostatistical literature, with our focus on the use of these concepts in quantitatively testing the performance of point estimators (specifically species richness estimators). We first describe the general concepts underlying bias, precision and accuracy, and then describe a number of commonly used unscaled and scaled performance measures of bias, precision and accuracy (e.g. mean error, variance, standard deviation, mean square error, root mean square error, mean absolute error, and all their scaled counterparts) which may be used to evaluate estimator performance. We also provide mathematical formulas and a worked example for most performance measures. Since every measure of estimator performance should be viewed as suggestive, not prescriptive, we also mention several other performance measures that have been used by biostatisticians or ecologists. We then outline several guidelines of how to test the performance of species richness estimators: the detailed description of data simulation models and resampling schemes, the use of real and simulated data sets on as many different estimators as possible, mathematical expressions for all estimators and performance measures, and the presentation of results for each scaled performance measure in numerical tables with increasing levels of sampling effort. We finish with a literature review of promising new research related to species richness estimation, and summarize the results of 14 studies that compared estimator performance, which confirm that with most data sets, non-parametric estimators (mostly the Chao and jackknife estimators) perform better than other estimators, e.g. curve models or fitting species-abundance distributions.
757 citations
24 Oct 2005
TL;DR: It is first shown using simple arguments that the so-called residual and stratified methods do yield an improvement over the basic multinomial resampling approach, and a central limit theorem is established for the case where resamplings is performed using the residual approach.
Abstract: This contribution is devoted to the comparison of various resampling approaches that have been proposed in the literature on particle filtering. It is first shown using simple arguments that the so-called residual and stratified methods do yield an improvement over the basic multinomial resampling approach. A simple counter-example showing that this property does not hold true for systematic resampling is given. Finally, some results on the large-sample behavior of the simple bootstrap filter algorithm are given. In particular, a central limit theorem is established for the case where resampling is performed using the residual approach.
692 citations
TL;DR: This paper presents a simple computational method for measuring the difference of independent empirical distributions estimated by bootstrapping or other resampling approaches, using data from a field test of external scope in contingent valuation.
Abstract: This paper presents a simple computational method for measuring the difference of independent empirical distributions estimated by bootstrapping or other resampling approaches. Using data from a field test of external scope in contingent valuation, this complete combinatorial method is compared with other methods (empirical convolutions, repeated sampling, normality, nonoverlapping confidence intervals) that have been suggested in the literature. Tradeoffs between methods are discussed in terms of programming complexity, time and computer resources required, bias, and the precision of the estimate.
552 citations
TL;DR: In a case study from the Ecuadorian Andes, logistic regression with stepwise backward variable selection yields lowest error rates and demonstrates the best generalization capabilities.
Abstract: . The predictive power of logistic regression, support vector machines and bootstrap-aggregated classification trees (bagging, double-bagging) is compared using misclassification error rates on independent test data sets. Based on a resampling approach that takes into account spatial autocorrelation, error rates for predicting "present" and "future" landslides are estimated within and outside the training area. In a case study from the Ecuadorian Andes, logistic regression with stepwise backward variable selection yields lowest error rates and demonstrates the best generalization capabilities. The evaluation outside the training area reveals that tree-based methods tend to overfit the data.
531 citations
TL;DR: This article proposes to use Kendall’s coefficient of concordance (W) to identify groups of significantly associated species in field survey data and shows that when the number of judges is small, the classical χ2 test is overly conservative, whereas the permutations test has correct Type 1 error; power of the permutation test is thus also higher.
Abstract: The search for species associations is one of the classical problems of community ecology. This article proposes to use Kendall’s coefficient of concordance (W) to identify groups of significantly associated species in field survey data. An overall test of independence of all species is first carried out. If the null hypothesis is rejected, one looks for groups of correlated species and, within each group, tests the contribution of each species to the overall statistic, using a permutation test. A field survey of oribatid mites in the peat blanket surrounding a bog lake is presented as an example. In the permutation framework, an a posteriori test of the contribution of each “judge” (species) to the overall W concordance statistic is possible; this is not the case in the classical testing framework. A simulation study showed that when the number of judges is small, which is the case in most real-life applications of Kendall’s test of concordance, the classical χ2 test is overly conservative, whereas the permutation test has correct Type 1 error; power of the permutation test is thus also higher. The interpretation and usefulness of the a posteriori tests are discussed in the framework of environmental studies. They can help identify groups of concordant species that can be used as indices of the quality of the environment, in particular in cases of pollution or contamination of the environment.
460 citations
TL;DR: The proposed algorithms improve the scalability of the filter architectures affected by the resampling process and reduce communication through the interconnection network is reduced and made deterministic, which results in simpler network structure and increased sampling frequency.
Abstract: In this paper, we propose novel resampling algorithms with architectures for efficient distributed implementation of particle filters. The proposed algorithms improve the scalability of the filter architectures affected by the resampling process. Problems in the particle filter implementation due to resampling are described, and appropriate modifications of the resampling algorithms are proposed so that distributed implementations are developed and studied. Distributed resampling algorithms with proportional allocation (RPA) and nonproportional allocation (RNA) of particles are considered. The components of the filter architectures are the processing elements (PEs), a central unit (CU), and an interconnection network. One of the main advantages of the new resampling algorithms is that communication through the interconnection network is reduced and made deterministic, which results in simpler network structure and increased sampling frequency. Particle filter performances are estimated for the bearings-only tracking applications. In the architectural part of the analysis, the area and speed of the particle filter implementation are estimated for a different number of particles and a different level of parallelism with field programmable gate array (FPGA) implementation. In this paper, only sampling importance resampling (SIR) particle filters are considered, but the analysis can be extended to any particle filters with resampling.
360 citations
TL;DR: The concept of bootstrap is described, to demonstrate how to estimate confidence intervals for the median and the Spearman rank correlation coefficient for non-normally-distributed data from a recent clinical study using two commonly used statistical software packages (SAS and Stata), and to discuss specific limitations of the bootstrap.
Abstract: The use of confidence intervals in reporting results of research has increased dramatically and is now required or highly recommended by editors of many scientific journals. Many resources describe methods for computing confidence intervals for statistics with mathematically simple distributions. Computing confidence intervals for descriptive statistics with distributions that are difficult to represent mathematically is more challenging. The bootstrap is a computationally intensive statistical technique that allows the researcher to make inferences from data without making strong distributional assumptions about the data or the statistic being calculated. This allows the researcher to estimate confidence intervals for statistics that do not have simple sampling distributions (e.g., the median). The purposes of this article are to describe the concept of bootstrapping, to demonstrate how to estimate confidence intervals for the median and the Spearman rank correlation coefficient for non-normally-distributed data from a recent clinical study using two commonly used statistical software packages (SAS and Stata), and to discuss specific limitations of the bootstrap.
326 citations
TL;DR: These techniques can supplement current nonparametric statistical methods and should be included, where appropriate, in the armamentarium of data processing methodologies.
267 citations
TL;DR: By using a realistic artificial RR interval generator, interpolation and resampling is shown to result in consistent over-estimations of the power spectral density (PSD) compared with the theoretical solution.
Abstract: Spectral estimates of heart rate variability (HRV) often involve the use of techniques such as the fast Fourier transform (FFT), which require an evenly sampled time series. HRV is calculated from the variations in the beat-to-beat (RR) interval timing of the cardiac cycle which are inherently irregularly spaced in time. In order to produce an evenly sampled time series prior to FFT-based spectral estimation, linear or cubic spline resampling is usually employed. In this paper, by using a realistic artificial RR interval generator, interpolation and resampling is shown to result in consistent over-estimations of the power spectral density (PSD) compared with the theoretical solution. The Lomb-Scargle (LS) periodogram, a more appropriate spectral estimation technique for unevenly sampled time series that uses only the original data, is shown to provide a superior PSD estimate. Ectopy removal or replacement is shown to be essential regardless of the spectral estimation technique. Resampling and phantom beat replacement is shown to decrease the accuracy of PSD estimation, even at low levels of ectopy or artefact. A linear relationship between the frequency of ectopy/artefact and the error (mean and variance) of the PSD estimate is demonstrated. Comparisons of PSD estimation techniques performed on real RR interval data during minimally active segments (sleep) demonstrate that the LS periodogram provides a less noisy spectral estimate of HRV.
TL;DR: In this paper, a simple estimation procedure for the Cox model with time-varying coefficients based on a kernel-weighted partial likelihood approach was proposed, and the authors derived a prediction method for future patients' survival with any specific set of covariates.
Abstract: In the analysis of censored failure time observations, the standard Cox proportional hazards model assumes that the regression coefficients are time invariant. Often, these parameters vary over time, and the temporal covariate effects on the failure time are of great interest. In this article, following previous work of Cai and Sun, we propose a simple estimation procedure for the Cox model with time-varying coefficients based on a kernel-weighted partial likelihood approach. We construct pointwise and simultaneous confidence intervals for the regression parameters over a properly chosen time interval via a simple resampling technique. We derive a prediction method for future patients' survival with any specific set of covariates. Building on the estimates for the time-varying coefficients, we also consider the mixed case and present an estimation procedure for time-independent parameters in the model. Furthermore, we show how to use an integrated function of the estimate for a specific regression coeffic...
TL;DR: A theoretical framework for inference problems in benchmark experiments is introduced and it is shown that standard statistical test procedures can be used to test for differences in the performances.
Abstract: The assessment of the performance of learners by means of benchmark experiments is an established exercise. In practice, benchmark studies are a tool to compare the performance of several competing algorithms for a certain learning problem. Cross-validation or resampling techniques are commonly used to derive point estimates of the performances which are compared to identify algorithms with good properties. For several benchmarking problems, test procedures taking the variability of those point estimates into account have been suggested. Most of the recently proposed inference procedures are based on special variance estimators for the cross-validated performance. We introduce a theoretical framework for inference problems in benchmark experiments and show that standard statistical test procedures can be used to test for differences in the performances. The theory is based on well-defined distributions of performance measures which can be compared with established tests. To demonstrate the usefulness in p...
TL;DR: Applications to simulated and real data demonstrate that the proposed Monte Carlo approach provides accurate error control, and can be substantially more powerful than the Bonferroni and Holm methods, especially when the test statistics are highly correlated.
Abstract: Motivation: Multiple hypothesis testing is a common problem in genome research, particularly in microarray experiments and genomewide association studies. Failure to account for the effects of multiple comparisons would result in an abundance of false positive results. The Bonferroni correction and Holm's step-down procedure are overly conservative, whereas the permutation test is time-consuming and is restricted to simple problems.
Results: We developed an efficient Monte Carlo approach to approximating the joint distribution of the test statistics along the genome. We then used the Monte Carlo distribution to evaluate the commonly used criteria for error control, such as familywise error rates and positive false discovery rates. This approach is applicable to any data structures and test statistics. Applications to simulated and real data demonstrate that the proposed approach provides accurate error control, and can be substantially more powerful than the Bonferroni and Holm methods, especially when the test statistics are highly correlated.
Contact: [email protected]
TL;DR: In this article, the authors proposed a time-saving resampling method based on a simple modification of the Markov chain marginal bootstrap (MCMB) to construct confidence intervals in quantile regression.
Abstract: Routine applications of quantile regression analysis require reliable and practical algorithms for estimating standard errors, variance-covariance matrices, as well as confidence intervals. Because the asymptotic variance of a quantile estimator depends on error densities, some standard large-sample approximations have been found to be highly sensitive to minor deviations from the iid error assumption. In this article we propose a time-saving resampling method based on a simple but useful modification of the Markov chain marginal bootstrap (MCMB) to construct confidence intervals in quantile regression. This method is compared to several existing methods with favorable performance in speed, accuracy, and reliability. We also make practical recommendations based on the quantreg package contributed by Roger Koenker and a new package rqmcmb2 developed by the first two authors. These recommendations also apply to users of the new SAS procedure PROC QUANTREG, available from Version 9.2 of SAS.
TL;DR: In this article, a simple method for using bootstrap resampling to derive confidence intervals is described, which can be used for a wide variety of statistics, including the mean and median, the difference of two means or proportions, and correlation and regression coefficients.
Abstract: Confidence intervals are in many ways a more satisfactory basis for statistical inference than hypothesis tests. This article explains a simple method for using bootstrap resampling to derive confidence intervals. This method can be used for a wide variety of statistics—including the mean and median, the difference of two means or proportions, and correlation and regression coefficients. It can be implemented by an Excel spreadsheet, which is available to readers on the Web. The rationale behind the method is transparent, and it relies on almost no sophisticated statistical concepts.
TL;DR: In this paper, a generalized bootstrap technique for estimators obtained by solving estimating equations is introduced. But the use of the proposed technique is discussed in some examples, and distributional consistency of the method is established and an asymptotic representation of the resampling variance estimator is obtained.
Abstract: We introduce a generalized bootstrap technique for estimators obtained by solving estimating equations. Some special cases of this generalized bootstrap are the classical bootstrap of Efron, the delete-d jackknife and variations of the Bayesian bootstrap. The use of the proposed technique is discussed in some examples. Distributional consistency of the method is established and an asymptotic representation of the resampling variance estimator is obtained.
Proceedings Article•
26 Jul 2005TL;DR: In this paper, the marginal particle filter (MPF) algorithm was proposed to approximate the joint posterior distribution using sequential importance sampling, which is useful for state estimation in non-linear, non-Gaussian dynamic models.
Abstract: Sequential Monte Carlo techniques are useful for state estimation in non-linear, non-Gaussian dynamic models. These methods allow us to approximate the joint posterior distribution using sequential importance sampling. In this framework, the dimension of the target distribution grows with each time step, thus it is necessary to introduce some resampling steps to ensure that the estimates provided by the algorithm have a reasonable variance. In many applications, we are only interested in the marginal filtering distribution which is defined on a space of fixed dimension. We present a Sequential Monte Carlo algorithm called the Marginal Particle Filter which operates directly on the marginal distribution, hence avoiding having to perform importance sampling on a space of growing dimension. Using this idea, we also derive an improved version of the auxiliary particle filter. We show theoretic and empirical results which demonstrate a reduction in variance over conventional particle filtering, and present techniques for reducing the cost of the marginal particle filter with N particles from O(N2) to O(N log N).
TL;DR: Calculated bootstrap or jackknife proportions with narrow confidence intervals even for very large datasets can be achieved with less expense than often thought, and iterated bootstrap methods that aim at reducing statistical bias inherent to these proportions are more feasible when the individual bootstrap searches require less time.
Abstract: For parsimony analyses, the most common way to estimate confidence is by resampling plans (nonparametric bootstrap, jackknife), and Bremer support (Decay indices). The recent literature reveals that parameter settings that are quite commonly employed are not those that are recommended by theoretical considerations and by previous empirical studies. The optimal search strategy to be applied during resampling was previously addressed solely via standard search strategies available in PAUP*. The question of a compromise between search extensiveness and improved support accuracy for Bremer support received even less attention. A set of experiments was conducted on different datasets to find an empirical cut-off point at which increased search extensiveness does not significantly change Bremer support and jackknife or bootstrap proportions any more. For the number of replicates needed for accurate estimates of support in resampling plans, a diagram is provided that helps to address the question whether apparently different support values really differ significantly. It is shown that the use of random addition cycles and parsimony ratchet iterations during bootstrapping does not translate into higher support, nor does any extension of the search extensiveness beyond the rather moderate effort of TBR (tree bisection and reconnection branch swapping) plus saving one tree per replicate. Instead, in case of very large matrices, saving more than one shortest tree per iteration and using a strict consensus tree of these yields decreased support compared to saving only one tree. This can be interpreted as a small risk of overestimating support but should be more than compensated by other factors that counteract an enhanced type I error. With regard to Bremer support, a rule of thumb can be derived stating that not much is gained relative to the surplus computational effort when searches are extended beyond 20 ratchet iterations per constrained node, at least not for datasets that fall within the size range found in the current literature. In view of these results, calculating bootstrap or jackknife proportions with narrow confidence intervals even for very large datasets can be achieved with less expense than often thought. In particular, iterated bootstrap methods that aim at reducing statistical bias inherent to these proportions are more feasible when the individual bootstrap searches require less time.
TL;DR: It is reported that evidence from these analyses that this impact of the correlation between gene expression levels on the statistical inference based on the empirical Bayes methodology may be quite strong, leading to a high variance of the number of differentially expressed genes.
Abstract: Stochastic dependence between gene expression levels in microarray data is of critical importance for the methods of statistical inference that resort to pooling test statistics across genes. The empirical Bayes methodology in the nonparametric and parametric formulations, as well as closely related methods employing a two-component mixture model, represent typical examples. It is frequently assumed that dependence between gene expressions (or associated test statistics) is sufficiently weak to justify the application of such methods for selecting differentially expressed genes. By applying resampling techniques to simulated and real biological data sets, we have studied a potential impact of the correlation between gene expression levels on the statistical inference based on the empirical Bayes methodology. We report evidence from these analyses that this impact may be quite strong, leading to a high variance of the number of differentially expressed genes. This study also pinpoints specific components of the empirical Bayes method where the reported effect manifests itself.
29 Jun 2005
TL;DR: This paper develops importance resampling into a variance reduction technique for Monte Carlo integration and demonstrates the robustness of this technique on common global illumination problems and achieves 10%-70% variance reduction over standard importance sampling for direct lighting.
Abstract: This paper develops importance resampling into a variance reduction technique for Monte Carlo integration. Importance resampling is a sample generation technique that can be used to generate more equally weighted samples for importance sampling. This can lead to significant variance reduction over standard importance sampling for common rendering problems. We show how to select the importance resampling parameters for near optimal variance reduction. We demonstrate the robustness of this technique on common global illumination problems and achieve a 10%-70% variance reduction over standard importance sampling for direct lighting. We conclude that further variance reduction could be achieved with cheaper sampling methods.
TL;DR: In this article, a set of particle filtering methods for tracking and manoeuvering target in clutter from angle-only measurements is presented and evaluated, comparing PFs to a well-established tracking algorithm, the IMM-PDA-EKF (interacting multiple model, probabilistic data association, extended Kalman filter).
Abstract: A particle filter (PF) is a recursive numerical technique which uses random sampling to approximate the optimal solution to target tracking problems involving nonlinearities and/or non-Gaussianity. A set of particle filtering methods for tracking and manoeuvering target in clutter from angle-only measurements is presented and evaluated. The aim is to compare PFs to a well-established tracking algorithm, the IMM-PDA-EKF (interacting multiple model, probabilistic data association, extended Kalman filter), and to provide an insight into which aspects of PF design are of most importance under given conditions. Monte Carlo simulations show that the use of a resampling scheme which produces particles with distinct values offers significant improvements under almost all conditions. Interestingly, under all conditions considered here,using this resampling scheme with blind particle proposals is shown to be superior, in the sense of providing improved performance for a fixed computational expense, to measurement-directed particle proposals with the same resampling scheme. This occurs even under conditions favourable to the use of measurement-directed proposals. The IMM-PDA-EKF performs poorly compared with the PFs for large clutter densities but is more effective when the measurements are precise.
TL;DR: The resampling-based method relies on an efficient technique for estimating the Hessian matrix, introduced as part of the adaptive form of the simultaneous perturbation stochastic approximation (SPSA) optimization algorithm.
Abstract: The Fisher information matrix summarizes the amount of information in the data relative to the quantities of interest. There are many applications of the information matrix in modeling, systems analysis, and estimation, including confidence region calculation, input design, prediction bounds, and “noninformative” priors for Bayesian analysis. This article reviews some basic principles associated with the information matrix, presents a resampling-based method for computing the information matrix together with some new theory related to efficient implementation, and presents some numerical results. The resampling-based method relies on an efficient technique for estimating the Hessian matrix, introduced as part of the adaptive (“second-order”) form of the simultaneous perturbation stochastic approximation (SPSA) optimization algorithm.
TL;DR: A simple permutation test of the hypothesis that a regression parameter is zero can overcome limitations of logistic regression and can be used whether or not maximum likelihood estimates exist, but can be overly conservative.
Abstract: Inference based on large sample results can be highly inaccurate if applied to logistic regression with small data sets. Furthermore, maximum likelihood estimates for the regression parameters will on occasion not exist, and large sample results will be invalid. Exact conditional logistic regression is an alternative that can be used whether or not maximum likelihood estimates exist, but can be overly conservative. This approach also requires grouping the values of continuous variables corresponding to nuisance parameters, and inference can depend on how this is done. A simple permutation test of the hypothesis that a regression parameter is zero can overcome these limitations. The variable of interest is replaced by the residuals from a linear regression of it on all other independent variables. Logistic regressions are then done for permutations of these residuals, and a p-value is computed by comparing the resulting likelihood ratio statistics to the original observed value. Simulations of binary outcome data with two independent variables that have binary or lognormal distributions yield the following results: (a) in small data sets consisting of 20 observations, type I error is well-controlled by the permutation test, but poorly controlled by the asymptotic likelihood ratio test; (b) in large data sets consisting of 1000 observations, performance of the permutation test appears equivalent to that of the asymptotic test; and (c) in small data sets, the p-value for the permutation test is usually similar to the mid-p-value for exact conditional logistic regression.
TL;DR: This paper compares three multiple testing procedures in the microarray context: the original Bonferroni method, a Bonferronsi-type improved single-step method and a step-down method based on nonparametric resampling, by which the null distribution can be derived with the dependency structure among gene expressions preserved and the family-wise error rate accurately controlled at the desired level.
Abstract: Microarray technology is rapidly emerging for genome-wide screening of differentially expressed genes between clinical subtypes or different conditions of human diseases. Traditional statistical testing approaches, such as the two-sample t-test or Wilcoxon test, are frequently used for evaluating statistical significance of informative expressions but require adjustment for large-scale multiplicity. Due to its simplicity, Bonferroni adjustment has been widely used to circumvent this problem. It is well known, however, that the standard Bonferroni test is often very conservative. In the present paper, we compare three multiple testing procedures in the microarray context: the original Bonferroni method, a Bonferroni-type improved single-step method and a step-down method. The latter two methods are based on nonparametric resampling, by which the null distribution can be derived with the dependency structure among gene expressions preserved and the family-wise error rate accurately controlled at the desired level. We also present a sample size calculation method for designing microarray studies. Through simulations and data analyses, we find that the proposed methods for testing and sample size calculation are computationally fast and control error and power precisely.
TL;DR: In this article, a new statistical quantity, the energy, is introduced to test whether two samples originate from the same distribution, which is a simple logarithmic function of the distances of the observations in the variate space.
Abstract: We introduce a new statistical quantity, the energy, to test whether two samples originate from the same distributions. The energy is a simple logarithmic function of the distances of the observations in the variate space. The distribution of the test statistic is determined by a resampling method. The power of the energy test in one dimension was studied for a variety of different test samples and compared to several nonparametric tests. In two and four dimensions, a comparison was performed with the Friedman–Rafsky and nearest neighbor tests. The two-sample energy test was shown to be especially powerful in multidimensional applications.
TL;DR: This report examines statistical resampling techniques, including cross‐validation and the bootstrap, applied to the initial sample to improve the estimation of locus‐specific effects and suggests that cross‐ validation and bootstrap methods can substantially reduce the estimation bias.
Abstract: The accuracy of gene localization, the reliability of locus-specific effect estimates, and the ability to replicate initial claims of linkage and/or association have emerged as major methodological concerns in genomewide studies of complex diseases and quantitative traits. To address the issue of multiple comparisons inherent in genomewide studies, the use of stringent criteria for assessing statistical significance has been generally acknowledged as a strategy to control type I error. However, the application of genomewide significance criteria does not take account of the selection bias introduced into parameter estimates, e.g., estimates of locus-specific effect size of disease/trait loci. Some have argued that reliable locus-specific parameter estimates can only be obtained in an independent sample. In this report, we examine statistical resampling techniques, including cross-validation and the bootstrap, applied to the initial sample to improve the estimation of locus-specific effects. We compare them with the naive method in which all data are used for both hypothesis testing and parameter estimation, as well as with the split-sample approach in which part of the data are reserved for estimation. Upward bias of the naive estimator and inadequacy of the split-sample approach are derived analytically under a simple quantitative trait model. Simulation studies of the resampling methods are performed for both the simple model and a more realistic genomewide linkage analysis. Our results suggest that cross-validation and bootstrap methods can substantially reduce the estimation bias, especially when the effect size is small or there is no genetic effect.
01 Dec 2005
TL;DR: An experimental bias-variance analysis of bagged and random aggregated ensembles of SVMs is presented in order to verify their theoretical variance reduction properties and suggests new directions for research to improve on classical bagging.
Abstract: Recently, bias-variance decomposition of error has been used as a tool to study the behavior of learning algorithms and to develop new ensemble methods well suited to the bias-variance characteristics of base learners. We propose methods and procedures, based on Domingo's unified bias-variance theory, to evaluate and quantitatively measure the bias-variance decomposition of error in ensembles of learning machines. We apply these methods to study and compare the bias-variance characteristics of single support vector machines (SVMs) and ensembles of SVMs based on resampling techniques, and their relationships with the cardinality of the training samples. In particular, we present an experimental bias-variance analysis of bagged and random aggregated ensembles of SVMs in order to verify their theoretical variance reduction properties. The experimental bias-variance analysis quantitatively characterizes the relationships between bagging and random aggregating, and explains the reasons why ensembles built on small subsamples of the data work with large databases. Our analysis also suggests new directions for research to improve on classical bagging.
01 Jan 2005
TL;DR: This paper considers the computation from two aspects: estimation and inference of quantile regression from three algorithms: simplex, interior point, and smoothing, which include optimization, parallelization, and sparse computation.
Abstract: In this paper, we discuss some practical computational issues for quantile regression. We consider the computation from two aspects: estimation and inference. For estimation, we cover three algorithms: simplex, interior point, and smoothing. We describe and compare these algorithms, then discuss implementation of some computing techniques, which include optimization, parallelization, and sparse computation, with these algorithms in practice. For inference, we focus on confidence intervals. We discuss three methods: sparsity, rank-score, and resampling. Their performances are compared for data sets with a large number of covariates. AMS (2000) subject classification. Primary 62F35; secondary 62J99.
TL;DR: In this paper, the authors studied the performance of different AR bootstrap schemes of the unit root hypothesis, including a new proposal based on unrestricted residuals, and showed that bootstrap procedures based on differencing the observed series suffer from power problems.
Abstract: The theory developed for bootstrapping unit root tests in an autoregressive (AR) context has been concerned mainly with the large-sample behavior of the methods proposed under the assumption that the null hypothesis is true. No results exist for the relative performance and the power behavior of the bootstrap methods under the alternative. This article studies the properties of different AR bootstrap schemes of the unit root hypothesis, including a new proposal based on unrestricted residuals. It shows that bootstrap procedures based on differencing the observed series suffer from power problems as compared with bootstrap procedures based on unrestricted residuals. Whereas for finite-order AR processes differencing leads to just a loss of power, for infinite-order autoregressions such a differencing makes the application of sieve AR bootstrap schemes inappropriate if the alternative is true. The superiority of the new bootstrap proposal is shown, and some numerical examples illustrate our theoretical find...