Innovated Higher Criticism for Detecting Sparse Signals in Correlated Noise
Peter A. Hall,Jiashun Jin +1 more
Reads0
Chats0
TLDR
In this paper, it was shown that correlation can be used to improve the performance of higher-criticism in the presence of correlated signals. But, it was also shown that the case of independent noise is the most difficult of all, from a statistical viewpoint, and that more accurate signal detection can be obtained when correlation is present.Abstract:
Higher criticism is a method for detecting signals that are both sparse and weak. Although first proposed in cases where the noise variables are independent, higher criticism also has reasonable performance in settings where those variables are correlated. In this paper we show that, by exploiting the nature of the correlation, performance can be improved by using a modified approach which exploits the potential advantages that correlation has to offer. Indeed, it turns out that the case of independent noise is the most difficult of all, from a statistical viewpoint, and that more accurate signal detection (for a given level of signal sparsity and strength) can be obtained when correlation is present. We characterize the advantages of correlation by showing how to incorporate them into the definition of an optimal detection boundary. The boundary has particularly attractive properties when correlation decays at a polynomial rate or the correlation matrix is Toeplitz.read more
Citations
More filters
Journal ArticleDOI
Two-sample test of high dimensional means under dependence
T. Tony Cai,Weidong Liu,Yin Xia +2 more
TL;DR: A new test statistic is introduced that is based on a linear transformation of the data by the precision matrix which incorporates the correlations between the variables and is shown to be particularly powerful against sparse alternatives and enjoys certain optimality.
Journal ArticleDOI
Global testing under sparse alternatives: ANOVA, multiple comparisons and the higher criticism
TL;DR: In this article, the authors show that under moderate sparsity levels, that is, 0 ≤ α ≤ 1/2, the analysis of variance (ANOVA) is essentially optimal under some conditions on the design.
Journal ArticleDOI
Cauchy combination test: a powerful test with analytic p-value calculation under arbitrary dependency structures.
Yaowu Liu,Jun Xie +1 more
TL;DR: Combining individual p-values to aggregate multiple small effects has a long-standing interest in statistics, dating back to the classic Fisher's combination test in modern large-scale data sets as discussed by the authors.
Journal ArticleDOI
Detection of an anomalous cluster in a network
TL;DR: In this article, the authors consider the problem of detecting whether or not in a given sensor network, there is a cluster of sensors which exhibit an "unusual behavior." Formally, suppose we are given a set of nodes and attach a random variable to each node.
Journal ArticleDOI
Detection of an anomalous cluster in a network
TL;DR: This work considers the problem of detecting whether or not in a given sensor network, there is a cluster of sensors which exhibit an "unusual behavior", and considers classes of clusters that are quite general, for which a lower bound is obtained on their respective minimax detection rate.
References
More filters
Journal ArticleDOI
Controlling the false discovery rate: a practical and powerful approach to multiple testing
Yoav Benjamini,Yosef Hochberg +1 more
TL;DR: In this paper, a different approach to problems of multiple significance testing is presented, which calls for controlling the expected proportion of falsely rejected hypotheses -the false discovery rate, which is equivalent to the FWER when all hypotheses are true but is smaller otherwise.
Book
Matrix Analysis
Roger A. Horn,Charles R. Johnson +1 more
TL;DR: In this article, the authors present results of both classic and recent matrix analyses using canonical forms as a unifying theme, and demonstrate their importance in a variety of applications, such as linear algebra and matrix theory.
Journal ArticleDOI
The control of the false discovery rate in multiple testing under dependency
Yoav Benjamini,Daniel Yekutieli +1 more
TL;DR: In this paper, it was shown that a simple FDR controlling procedure for independent test statistics can also control the false discovery rate when test statistics have positive regression dependency on each of the test statistics corresponding to the true null hypotheses.
Book
Empirical processes with applications to statistics
Galen R. Shorack,Jon A. Wellner +1 more
TL;DR: In this paper, a broad cross-section of the literature available on one-dimensional empirical processes is summarized, with emphasis on real random variable processes as well as a wide-ranging selection of applications in statistics.
Related Papers (5)
Controlling the false discovery rate: a practical and powerful approach to multiple testing
Yoav Benjamini,Yosef Hochberg +1 more