Thresholding of statistical maps in functional neuroimaging using the false discovery rate.
TL;DR: This paper introduces to the neuroscience literature statistical procedures for controlling the false discovery rate (FDR) and demonstrates this approach using both simulations and functional magnetic resonance imaging data from two simple experiments.
About: This article is published in NeuroImage.The article was published on 2002-04-01 and is currently open access. It has received 4838 citations till now. The article focuses on the topics: Multiple comparisons problem & False discovery rate.
Figures (6)
TABLE 2 
FIG. 5. Results of applying the Bonferroni method to t-statistics to the same 20 data sets used in Fig. 4. 
TABLE 1 
FIG. 7. Axial slice of suprathreshold pixels overlayed on T1 structural image. Colored pixels are log10 of the P value. Top left, t 4 threshold. Top right, threshold-controlling FDR at 5% and c(V) 1; the equivalent t threshold is 3.8119. Bottom left, Bonferroni-corrected threshold; the equivalent t threshold is 5.2485. Bottom right, threshold-controlling FDR at 5% making no assumptions on P value distribution; the equivalent t threshold is 5.0747. 
FIG. 1. A graphical display of the FDR-controlling procedures. Sorted P values are plotted in order (with index i at i/V for i 1, . . ., V). The largest P value below the line through the origin with slope q is the corresponding threshold, 0 if all P values are above the line. One rejects the null hypotheses for tests whose P values are at or below this threshold. Using the false discovery rate, the threshold for significance is approximately 0.0049, with five rejections; using the Bonferroni correction, the threshold is approximately 0.00280, with four rejections. 
FIG. 3. Histogram of equivalent t thresholds generated by the FDR-controlling procedure across simulation runs with 128 128 image size and 10 10 block. The vertical line on the histogram is the corresponding Bonferroni threshold. Note, however, that all FDR thresholds above Bonferroni yield exactly the same rejections as would the Bonferroni threshold.
Citations
More filters
TL;DR: In right anterior insular/opercular cortex, neural activity predicted subjects' accuracy in the heartbeat detection task and local gray matter volume correlated with both interoceptive accuracy and subjective ratings of visceral awareness.
Abstract: Influential theories of human emotion argue that subjective feeling states involve representation of bodily responses elicited by emotional events. Within this framework, individual differences in intensity of emotional experience reflect variation in sensitivity to internal bodily responses. We measured regional brain activity by functional magnetic resonance imaging (fMRI) during an interoceptive task wherein subjects judged the timing of their own heartbeats. We observed enhanced activity in insula, somatomotor and cingulate cortices. In right anterior insular/opercular cortex, neural activity predicted subjects' accuracy in the heartbeat detection task. Furthermore, local gray matter volume in the same region correlated with both interoceptive accuracy and subjective ratings of visceral awareness. Indices of negative emotional experience correlated with interoceptive accuracy across subjects. These findings indicate that right anterior insula supports a representation of visceral responses accessible to awareness, providing a substrate for subjective feeling states.
2,972 citations
TL;DR: A meta-analysis of functional magnetic resonance imaging and positron emission tomography studies of posttraumatic stress disorder, social anxiety disorder, specific phobia, and fear conditioning in healthy individuals provided neuroimaging evidence for common brain mechanisms in anxiety disorders and normal fear.
Abstract: Objective: The study of human anxiety disorders has benefited greatly from functional neuroimaging approaches. Individual studies, however, vary greatly in their findings. The authors searched for common and disorder-specific functional neurobiological deficits in several anxiety disorders. The authors also compared these deficits to the neural systems engaged during anticipatory anxiety in healthy subjects. Method: Functional magnetic resonance imaging and positron emission tomography studies of posttraumatic stress disorder (PTSD), social anxiety disorder, specific phobia, and fear conditioning in healthy individuals were compared by quantitative meta-analysis. Included studies compared negative emotional processing to baseline, neutral, or positive emotion conditions. Results: Patients with any of the three disorders consistently showed greater activity than matched comparison subjects in the amygdala and insula, structures linked to negative emotional responses. A similar pattern was observed during f...
2,848 citations
Cites methods from "Thresholding of statistical maps in..."
...Probability values were corrected for multiple comparisons using false-discovery-rate control (64) at q<0....
[...]
TL;DR: This paper presents a generic framework for permutation inference for complex general linear models (glms) when the errors are exchangeable and/or have a symmetric distribution, and shows that, even in the presence of nuisance effects, these permutation inferences are powerful while providing excellent control of false positives in a wide range of common and relevant imaging research scenarios.
2,756 citations
Cites background from "Thresholding of statistical maps in..."
...the map of uncorrected p-values (Benjamini and Hochberg, 1995; Genovese et al., 2002) or an FDR-adjusted p-value can be calculated at each point (Yekutieli and Benjamini, 1999)....
[...]
...Either a single FDR threshold can be applied to the map of uncorrected p-values (Benjamini and Hochberg, 1995; Genovese et al., 2002) or an FDR-adjusted p-value can be calculated at each point (Yekutieli and Benjamini, 1999)....
[...]
TL;DR: It is argued that systems neuroscience needs to adjust some widespread practices to avoid the circularity that can arise from selection, and 'double dipping' the use of the same dataset for selection and selective analysis is suggested.
Abstract: A neuroscientific experiment typically generates a large amount of data, of which only a small fraction is analyzed in detail and presented in a publication. However, selection among noisy measurements can render circular an otherwise appropriate analysis and invalidate results. Here we argue that systems neuroscience needs to adjust some widespread practices to avoid the circularity that can arise from selection. In particular, ‘double dipping’, the use of the same dataset for selection and selective analysis, will give distorted descriptive statistics and invalid statistical inference whenever the results statistics are not inherently independent of the selection criteria under the null hypothesis. To demonstrate the problem, we apply widely used analyses to noise data known to not contain the experimental effects in question. Spurious effects can appear in the context of both univariate activation analysis and multivariate pattern-information analysis. We suggest a policy for avoiding circularity.
2,511 citations
TL;DR: The development of high-resolution neuroimaging and multielectrode electrophysiological recording provides neuroscientists with huge amounts of multivariate data, but the local averaging standardly applied to this end may obscure the effects of greatest neuroscientific interest.
Abstract: The development of high-resolution neuroimaging and multielectrode electrophysiological recording provides neuroscientists with huge amounts of multivariate data. The complexity of the data creates a need for statistical summary, but the local averaging standardly applied to this end may obscure the effects of greatest neuroscientific interest. In neuroimaging, for example, brain mapping analysis has focused on the discovery of activation, i.e., of extended brain regions whose average activity changes across experimental conditions. Here we propose to ask a more general question of the data: Where in the brain does the activity pattern contain information about the experimental condition? To address this question, we propose scanning the imaged volume with a "searchlight," whose contents are analyzed multivariately at each location in the brain.
2,082 citations
References
More filters
TL;DR: In this paper, a different approach to problems of multiple significance testing is presented, which calls for controlling the expected proportion of falsely rejected hypotheses -the false discovery rate, which is equivalent to the FWER when all hypotheses are true but is smaller otherwise.
Abstract: SUMMARY The common approach to the multiplicity problem calls for controlling the familywise error rate (FWER). This approach, though, has faults, and we point out a few. A different approach to problems of multiple significance testing is presented. It calls for controlling the expected proportion of falsely rejected hypotheses -the false discovery rate. This error rate is equivalent to the FWER when all hypotheses are true but is smaller otherwise. Therefore, in problems where the control of the false discovery rate rather than that of the FWER is desired, there is potential for a gain in power. A simple sequential Bonferronitype procedure is proved to control the false discovery rate for independent test statistics, and a simulation study shows that the gain in power is substantial. The use of the new procedure and the appropriateness of the criterion are illustrated with examples.
83,420 citations
"Thresholding of statistical maps in..." refers methods in this paper
...The FDR-controlling techniques introduced by Benjamini and Hochberg (1995) are easily implemented, even for very large data sets....
[...]
...In this paper, we describe a recent development in statistics that can be adapted to automatic and implicit threshold selection in neuroimaging: procedures that control the false discovery rate (FDR) (Benjamini and Hochberg, 1995; Benjamini and Liu, 1999; Benjamini and Yekutieli, 2001)....
[...]
TL;DR: In this paper, it was shown that a simple FDR controlling procedure for independent test statistics can also control the false discovery rate when test statistics have positive regression dependency on each of the test statistics corresponding to the true null hypotheses.
Abstract: Benjamini and Hochberg suggest that the false discovery rate may be the appropriate error rate to control in many applied multiple testing problems. A simple procedure was given there as an FDR controlling procedure for independent test statistics and was shown to be much more powerful than comparable procedures which control the traditional familywise error rate. We prove that this same procedure also controls the false discovery rate when the test statistics have positive regression dependency on each of the test statistics corresponding to the true null hypotheses. This condition for positive dependency is general enough to cover many problems of practical interest, including the comparisons of many treatments with a single control, multivariate normal test statistics with positive correlation matrix and multivariate $t$. Furthermore, the test statistics may be discrete, and the tested hypotheses composite without posing special difficulties. For all other forms of dependency, a simple conservative modification of the procedure controls the false discovery rate. Thus the range of problems for which a procedure with proven FDR control can be offered is greatly increased.
9,335 citations
Book•
01 Jan 1966
TL;DR: In this article, the authors presented a case of two means regression method for the family error rate, which was used to estimate the probability of a family having a nonzero family error.
Abstract: 1 Introduction.- 1 Case of two means.- 2 Error rates.- 2.1 Probability of a nonzero family error rate.- 2.2 Expected family error rate.- 2.3 Allocation of error.- 3 Basic techniques.- 3.1 Repeated normal statistics.- 3.2 Maximum modulus (Tukey).- 3.3 Bonferroni normal statistics.- 3.4 ?2 projections (Scheffe).- 3.5 Allocation.- 3.6 Multiple modulus tests (Duncan).- 3.7 Least significant difference test (Fisher).- 4 p-mean significance levels.- 5 Families.- 2 Normal Univariate Techniques.- 1 Studentized range (Tukey).- 1.1 Method.- 1.2 Applications.- 1.3 Comparison.- 1.4 Derivation.- 1.5 Distributions and tables.- 2 F projections (Scheffe)48.- 2.1 Method.- 2.2 Applications.- 2.3 Comparison.- 2.4 Derivation.- 2.5 Distributions and tables.- 3 Bonferroni t statistics.- 3.1 Method.- 3.2 Applications.- 3.3 Comparison.- 3.4 Derivation.- 3.5 Distributions and tables.- 4 Studentized maximum modulus.- 4.1 Method.- 4.2 Applications.- 4.3 Comparison.- 4.4 Derivation.- 4.5 Distributions and tables.- 5 Many-one t statistics76.- 5.1 Method.- 5.2 Applications.- 5.3 Comparison.- 5.4 Derivation.- 5.5 Distributions and tables.- 6 Multiple range tests (Duncan).- 6.1 Method.- 6.2 Applications.- 6.3 Comparison.- 6.4 Derivation.- 6.5 Distributions and tables.- 7 Least significant difference test (Fisher).- 7.1 Method.- 7.2 Applications.- 7.3 Comparison.- 7.4 Derivation.- 7.5 Distributions and tables.- 8 Other techniques.- 8.1 Tukey's gap-straggler-variance test.- 8.2 Shortcut methods.- 8.3 Multiple F tests.- 8.4 Two-sample confidence intervals of predetermined length.- 8.5 An improved Bonferroni inequality.- 9 Power.- 10 Robustness.- 3 Regression Techniques.- 1 Regression surface confidence bands.- 1.1 Method.- 1.2 Comparison.- 1.3 Derivation.- 2 Prediction.- 2.1 Method.- 2.2 Comparison.- 2.3 Derivation.- 3 Discrimination.- 3.1 Method.- 3.2 Comparison.- 3.3 Derivation.- 4 Other techniques.- 4.1 Linear confidence bands.- 4.2 Tolerance intervals.- 4.3 Unlimited discrimination intervals.- 4 Nonparametric Techniques.- 1 Many-one sign statistics (Steel).- 1.1 Method.- 1.2 Applications.- 1.3 Comparison.- 1.4 Derivation.- 1.5 Distributions and tables.- 2 k-sample sign statistics.- 2.1 Method.- 2.2 Applications.- 2.3 Comparison.- 2.4 Derivation.- 2.5 Distributions and tables.- 3 Many-one rank statistics (Steel).- 3.1 Method.- 3.2 Applications.- 3.3 Comparison.- 3.4 Derivation.- 3.5 Distributions and tables.- 4 k-sample rank statistics.- 4.1 Method.- 4.2 Applications.- 4.3 Comparison.- 4.4 Derivation.- 4.5 Distributions and tables.- 5 Signed-rank statistics.- 6 Kruskal-Wallis rank statistics (Nemenyi).- 6.1 Method.- 6.2 Applications.- 6.3 Comparison.- 6.4 Derivation.- 6.5 Distributions and tables.- 7 Friedman rank statistics (Nemenyi).- 7.1 Method.- 7.2 Applications.- 7.3 Comparison.- 7.4 Derivation.- 7.5 Distributions and tables.- 8 Other techniques.- 8.1 Permutation tests.- 8.2 Median tests (Nemenyi).- 8.3 Kolmogorov-Smirnov statistics.- 5 Multivariate Techniques.- 1 Single population covariance scalar unknown.- 1.1 Method.- 1.2 Applications.- 1.3 Comparison.- 1.4 Derivation.- 1.5 Distributions and tables.- 2 Single population covariance matrix unknown.- 2.1 Method.- 2.2 Applications.- 2.3 Comparison.- 2.4 Derivation.- 2.5 Distributions and tables.- 3 k populations covariance matrix unknown.- 3.1 Method.- 3.2 Applications.- 3.3 Comparison.- 3.4 Derivation.- 3.5 Distributions and tables.- 4 Other techniques.- 4.1 Variances known covariances unknown.- 4.2 Variance-covariance intervals.- 4.3 Two-sample confidence intervals of predetermined length.- 6 Miscellaneous Techniques.- 1 Outlier detection.- 2 Multinomial populations.- 2.1 Single population.- 2.2 Several populations.- 2.3 Cross-product ratios.- 2.4 Logistic response curves.- 3 Equality of variances.- 4 Periodogram analysis.- 5 Alternative approaches: selection, ranking, slippage.- A Strong Law For The Expected Error Rate.- B TABLES.- I Percentage points of the studentized range.- II Percentage points of the Bonferroni t statistic.- III Percentage points of the studentized maximum modulus.- IV Percentage points of the many-one t statistics.- V Percentage points of the Duncan multiple range test.- VI Percentage points of the many-one sign statistics.- VIII Percentage points of the many-one rank statistics.- IX Percentage points of the k-sample rank statistics.- Developments in Multiple Comparisons 1966-).- 3.5 Allocation.- 3.6 Multiple modulus tests (Duncan).- 3.7 Least significant difference test (Fisher).- 4 p-mean significance levels.- 5 Families.- 2 Normal Univariate Techniques.- 1 Studentized range (Tukey).- 1.1 Method.- 1.2 Applications.- 1.3 Comparison.- 1.4 Derivation.- 1.5 Distributions and tables.- 2 F projections (Scheffe)48.- 2.1 Method.- 2.2 Applications.- 2.3 Comparison.- 2.4 Derivation.- 2.5 Distributions and tables.- 3 Bonferroni t statistics.- 3.1 Method.- 3.2 Applications.- 3.3 Comparison.- 3.4 Derivation.- 3.5 Distributions and tables.- 4 Studentized maximum modulus.- 4.1 Method.- 4.2 Applications.- 4.3 Comparison.- 4.4 Derivation.- 4.5 Distributions and tables.- 5 Many-one t statistics76.- 5.1 Method.- 5.2 Applications.- 5.3 Comparison.- 5.4 Derivation.- 5.5 Distributions and tables.- 6 Multiple range tests (Duncan).- 6.1 Method.- 6.2 Applications.- 6.3 Comparison.- 6.4 Derivation.- 6.5 Distributions and tables.- 7 Least significant difference test (Fisher).- 7.1 Method.- 7.2 Applications.- 7.3 Comparison.- 7.4 Derivation.- 7.5 Distributions and tables.- 8 Other techniques.- 8.1 Tukey's gap-straggler-variance test.- 8.2 Shortcut methods.- 8.3 Multiple F tests.- 8.4 Two-sample confidence intervals of predetermined length.- 8.5 An improved Bonferroni inequality.- 9 Power.- 10 Robustness.- 3 Regression Techniques.- 1 Regression surface confidence bands.- 1.1 Method.- 1.2 Comparison.- 1.3 Derivation.- 2 Prediction.- 2.1 Method.- 2.2 Comparison.- 2.3 Derivation.- 3 Discrimination.- 3.1 Method.- 3.2 Comparison.- 3.3 Derivation.- 4 Other techniques.- 4.1 Linear confidence bands.- 4.2 Tolerance intervals.- 4.3 Unlimited discrimination intervals.- 4 Nonparametric Techniques.- 1 Many-one sign statistics (Steel).- 1.1 Method.- 1.2 Applications.- 1.3 Comparison.- 1.4 Derivation.- 1.5 Distributions and tables.- 2 k-sample sign statistics.- 2.1 Method.- 2.2 Applications.- 2.3 Comparison.- 2.4 Derivation.- 2.5 Distributions and tables.- 3 Many-one rank statistics (Steel).- 3.1 Method.- 3.2 Applications.- 3.3 Comparison.- 3.4 Derivation.- 3.5 Distributions and tables.- 4 k-sample rank statistics.- 4.1 Method.- 4.2 Applications.- 4.3 Comparison.- 4.4 Derivation.- 4.5 Distributions and tables.- 5 Signed-rank statistics.- 6 Kruskal-Wallis rank statistics (Nemenyi).- 6.1 Method.- 6.2 Applications.- 6.3 Comparison.- 6.4 Derivation.- 6.5 Distributions and tables.- 7 Friedman rank statistics (Nemenyi).- 7.1 Method.- 7.2 Applications.- 7.3 Comparison.- 7.4 Derivation.- 7.5 Distributions and tables.- 8 Other techniques.- 8.1 Permutation tests.- 8.2 Median tests (Nemenyi).- 8.3 Kolmogorov-Smirnov statistics.- 5 Multivariate Techniques.- 1 Single population covariance scalar unknown.- 1.1 Method.- 1.2 Applications.- 1.3 Comparison.- 1.4 Derivation.- 1.5 Distributions and tables.- 2 Single population covariance matrix unknown.- 2.1 Method.- 2.2 Applications.- 2.3 Comparison.- 2.4 Derivation.- 2.5 Distributions and tables.- 3 k populations covariance matrix unknown.- 3.1 Method.- 3.2 Applications.- 3.3 Comparison.- 3.4 Derivation.- 3.5 Distributions and tables.- 4 Other techniques.- 4.1 Variances known covariances unknown.- 4.2 Variance-covariance intervals.- 4.3 Two-sample confidence intervals of predetermined length.- 6 Miscellaneous Techniques.- 1 Outlier detection.- 2 Multinomial populations.- 2.1 Single population.- 2.2 Several populations.- 2.3 Cross-product ratios.- 2.4 Logistic response curves.- 3 Equality of variances.- 4 Periodogram analysis.- 5 Alternative approaches: selection, ranking, slippage.- A Strong Law For The Expected Error Rate.- B TABLES.- I Percentage points of the studentized range.- II Percentage points of the Bonferroni t statistic.- III Percentage points of the studentized maximum modulus.- IV Percentage points of the many-one t statistics.- V Percentage points of the Duncan multiple range test.- VI Percentage points of the many-one sign statistics.- VIII Percentage points of the many-one rank statistics.- IX Percentage points of the k-sample rank statistics.- Developments in Multiple Comparisons 1966-1976.- 1 Introduction.- 2 Papers of special interest.- 2.1 Probability inequalities.- 2.2 Methods for unbalanced ANOVA.- 2.3 Conditional confidence levels.- 2.4 Empirical Bayes approach.- 2.5 Confidence bands in regression.- 3 References.- 4 Bibliography 1966-1976.- 4.1 Survey articles.- 4.2 Probability inequalities.- 4.3 Tables.- 4.4 Normal multifactor methods.- 4.5 Regression.- 4.6 Categorical data.- 4.7 Nonparametric techniques.- 4.8 Multivariate methods.- 4.9 Miscellaneous.- 4.10 Pre-1966 articles missed in [6].- 4.11 Late additions.- 5 List of journals scanned.- Addendum New Table of the Studentized Maximum Modulus.- Table IIIA Percentage points of the studentized maximum modulus.- Author Index.
4,763 citations
"Thresholding of statistical maps in..." refers background in this paper
...© 2002 Elsevier Science (USA) Key Words: functional neuroimaging; false discovery rate; multiple testing; Bonferroni correction....
[...]
TL;DR: A unified statistical theory for assessing the significance of apparent signal observed in noisy difference images is presented and an estimate of the P‐value for local maxima of Gaussian, t, χ2 and F fields over search regions of any shape or size in any number of dimensions is estimated.
Abstract: We present a unified statistical theory for assessing the significance of apparent signal observed in noisy difference images. The results are usable in a wide range of applications, including fMRI, but are discussed with particular reference to PET images which represent changes in cerebral blood flow elicited by a specific cognitive or sensorimotor task. Our main result is an estimate of the P-value for local maxima of Gaussian, t, chi(2) and F fields over search regions of any shape or size in any number of dimensions. This unifies the P-values for large search areas in 2-D (Friston et al. [1991]: J Cereb Blood Flow Metab 11:690-699) large search regions in 3-D (Worsley et al. [1992]: J Cereb Blood Flow Metab 12:900-918) and the usual uncorrected P-value at a single pixel or voxel.
2,707 citations
"Thresholding of statistical maps in..." refers methods in this paper
...There have been a number of efforts to find an objective and effective method for threshold determination (Genovese et al., 1997; Worsley et al., 1996; Holmes et al., 1996)....
[...]
...More involved methods include random field approaches (such as Worsley et al., 1996) or permutation based methods (such as Holmes et al., 1996)....
[...]
TL;DR: A method for detecting significant and regionally specific correlations between sensory input and the brain's physiological response, as measured with functional magnetic resonance imaging (MRI), is presented in this paper.
Abstract: A method for detecting significant and regionally specific correlations between sensory input and the brain's physiological response, as measured with functional magnetic resonance imaging (MRI), is presented in this paper. The method involves testing for correlations between sensory input and the hemodynamic response after convolving the sensory input with an estimate of the hernodynamic response function. This estimate is obtained without reference to any assumed input. To lend the approach statistical validity, it is brought into the framework of statistical parametric mapping by using a measure of cross-correlations between sensory input and hemodynamic response that is valid in the presence of intrinsic autocorrelations. These autocorrelations are necessarily present, due to the hemodynamic response function or temporal point spread function.
1,748 citations
"Thresholding of statistical maps in..." refers methods in this paper
...Many recent methods for the analysis of fMRI data rely on fitting sophisticated statistical models to the data (see, for example, Friston et al., 1994; Genovese, 2000; Lange and Zeger, 1997)....
[...]
...Many recent methods for the analysis of fMRI data rely on fitting sophisticated statistical models to the data (see, for example, Friston et al., 1994; Genovese, 2000; Lange and Zeger, 1997)....
[...]