Showing papers on "False positive paradox published in 2007"

What's under the ROC? An Introduction to Receiver Operating Characteristics Curves

Abstract: It is often necessary to dichotomize a continuous scale to separate respondents into normal and abnormal groups. However, because the distributions of the scores in these 2 groups most often overlap, any cut point that is chosen will result in 2 types of errors: false negatives (that is, abnormal cases judged to be normal) and false positives (that is, normal cases placed in the abnormal group). Changing the cut point will alter the numbers of erroneous judgments but will not eliminate the problem. A technique called receiver operating characteristic (ROC) curves allows us to determine the ability of a test to discriminate between groups, to choose the optimal cut point, and to compare the performance of 2 or more tests. We discuss how to calculate and compare ROC curves and the factors that must be considered in choosing an optimal cut point.

How to reduce false positive results when undertaking in vitro genotoxicity testing and thus avoid unnecessary follow-up animal tests: Report of an ECVAM Workshop.

Abstract: Workshop participants agreed that genotoxicity tests in mammalian cells in vitro produce a remarkably high and unacceptable occurrence of irrelevant positive results (e.g. when compared with rodent carcinogenicity). As reported in several recent reviews, the rate of irrelevant positives (i.e. low specificity) for some studies using in vitro methods (when compared to this "gold standard") means that an increased number of test articles are subjected to additional in vivo genotoxicity testing, in many cases before, e.g. the efficacy (in the case of pharmaceuticals) of the compound has been evaluated. If in vitro tests were more predictive for in vivo genotoxicity and carcinogenicity (i.e. fewer false positives) then there would be a significant reduction in the number of animals used. Beyond animal (or human) carcinogenicity as the "gold standard", it is acknowledged that genotoxicity tests provide much information about cellular behaviour, cell division processes and cellular fate to a (geno)toxic insult. Since the disease impact of these effects is seldom known, and a verification of relevant toxicity is normally also the subject of (sub)chronic animal studies, the prediction of in vivo relevant results from in vitro genotoxicity tests is also important for aspects that may not have a direct impact on carcinogenesis as the ultimate endpoint of concern. In order to address the high rate of in vitro false positive results, a 2-day workshop was held at the European Centre for the Validation of Alternative Methods (ECVAM), Ispra, Italy in April 2006. More than 20 genotoxicity experts from academia, government and industry were invited to review data from the currently available cell systems, to discuss whether there exist cells and test systems that have a reduced tendency to false positive results, to review potential modifications to existing protocols and cell systems that might result in improved specificity, and to review the performance of some new test systems that show promise of improved specificity without sacrificing sensitivity. It was concluded that better guidance on the likely mechanisms resulting in positive results that are not biologically relevant for human health, and how to obtain evidence for those mechanisms, is needed both for practitioners and regulatory reviewers. Participants discussed the fact that cell lines commonly used for genotoxicity testing have a number of deficiencies that may contribute to the high false positive rate. These include, amongst others, lack of normal metabolism leading to reliance on exogenous metabolic activation systems (e.g. Aroclor-induced rat S9), impaired p53 function and altered DNA repair capability. The high concentrations of test chemicals (i.e. 10 mM or 5000 microg/ml, unless precluded by solubility or excessive toxicity) and the high levels of cytotoxicity currently required in mammalian cell genotoxicity tests were discussed as further potential sources of false positive results. Even if the goal is to detect carcinogens with short in vitro tests under more or less acute conditions, it does not seem logical to exceed the capabilities of cellular metabolic turnover, activation and defence processes. The concept of "promiscuous activation" was discussed. For numerous mutagens, the decisive in vivo enzymes are missing in vitro. However, if the substrate concentration is increased sufficiently, some other enzymes (that are unimportant in vivo) may take over the activation-leading to the same or a different active metabolite. Since we often do not use the right enzyme systems for positive controls in vitro, we have to rely on their promiscuous activation, i.e. to use excessive concentrations to get an empirical correlation between genotoxicity and carcinogenicity. A thorough review of published and industry data is urgently needed to determine whether the currently required limit concentration of 10mM or 5000 microg/ml, and high levels of cytotoxicity, are necessary for the detection of in vivo genotoxins and DNA-reactive, mutagenic carcinogens. In addition, various measures of cytotoxicity are currently allowable under OECD test guidelines, but there are few comparative data on whether different measures would result in different maximum concentrations for testing. A detailed comparison of cytotoxicity assessment strategies is needed. An assessment of whether test endpoints can be selected that are not intrinsically associated with cytotoxicity, and therefore are less susceptible to artefacts produced by cytotoxicity, should also be undertaken. There was agreement amongst the workshop participants that cell systems which are p53 and DNA-repair proficient, and have defined Phase 1 and Phase 2 metabolism, covering a broad set of enzyme forms, and used within the context of appropriately set limits of concentration and cytotoxicity, offer the best hope for reduced false positives. Whilst there is some evidence that human lymphocytes are less susceptible to false positives than the current rodent cell lines, other cell systems based on HepG2, TK6 and MCL-5 cells, as well as 3D skin models based on primary human keratinocytes also show some promise. Other human cell lines such as HepaRG, and human stem cells (the target for carcinogenicity) have not been used for genotoxicity investigations and should be considered for evaluation. Genetic engineering is also a valuable tool to incorporate missing enzyme systems into target cells. A collaborative research programme is needed to identify, further develop and evaluate new cell systems with appropriate sensitivity but improved specificity. In order to review current data for selection of appropriate top concentrations, measures and levels of cytotoxicity, metabolism, and to be able to improve existing or validate new assay systems, the participants called for the establishment of an expert group to identify the in vivo genotoxins and DNA-reactive, mutagenic carcinogens that we expect our in vitro genotoxicity assays to detect as well as the non-genotoxins and non-carcinogens we expect them not to detect.

Object detection combining recognition and segmentation

Abstract: We develop an object detection method combining top-down recognition with bottom-up image segmentation. There are two main steps in this method: a hypothesis generation step and a verification step. In the top-down hypothesis generation step, we design an improved Shape Context feature, which is more robust to object deformation and background clutter. The improved Shape Context is used to generate a set of hypotheses of object locations and figure-ground masks, which have high recall and low precision rate. In the verification step, we first compute a set of feasible segmentations that are consistent with top-down object hypotheses, then we propose a False Positive Pruning (FPP) procedure to prune out false positives. We exploit the fact that false positive regions typically do not align with any feasible image segmentation. Experiments show that this simple framework is capable of achieving both high recall and high precision with only a few positive training examples and that this method can be generalized to many object classes.

PathFinder: mining signal transduction pathway segments from protein-protein interaction networks

Abstract: A Signal transduction pathway is the chain of processes by which a cell converts an extracellular signal into a response. In most unicellular organisms, the number of signal transduction pathways influences the number of ways the cell can react and respond to the environment. Discovering signal transduction pathways is an arduous problem, even with the use of systematic genomic, proteomic and metabolomic technologies. These techniques lead to an enormous amount of data and how to interpret and process this data becomes a challenging computational problem. In this study we present a new framework for identifying signaling pathways in protein-protein interaction networks. Our goal is to find biologically significant pathway segments in a given interaction network. Currently, protein-protein interaction data has excessive amount of noise, e.g., false positive and false negative interactions. First, we eliminate false positives in the protein-protein interaction network by integrating the network with microarray expression profiles, protein subcellular localization and sequence information. In addition, protein families are used to repair false negative interactions. Then the characteristics of known signal transduction pathways and their functional annotations are extracted in the form of association rules. Given a pair of starting and ending proteins, our methodology returns candidate pathway segments between these two proteins with possible missing links (recovered false negatives). In our study, S. cerevisiae (yeast) data is used to demonstrate the effectiveness of our method.

On the operational characteristics of the Benjamini and Hochberg False Discovery Rate procedure.

Abstract: Multiple testing procedures are commonly used in gene expression studies for the detection of differential expression, where typically thousands of genes are measured over at least two experimental conditions. Given the need for powerful testing procedures, and the attendant danger of false positives in multiple testing, the False Discovery Rate (FDR) controlling procedure of Benjamini and Hochberg (1995) has become a popular tool. When simultaneously testing hypotheses, suppose that R rejections are made, of which Fp are false positives. The Benjamini and Hochberg procedure ensures that the expectation of Fp/R is bounded above by some pre-specified proportion. In practice, the procedure is applied to a single experiment. In this paper we investigate the across-experiment variability of the proportion Fp/R as a function of three experimental parameters. The operational characteristics of the procedure when applied to dependent hypotheses are also considered.

Building high accuracy bloom filters using partitioned hashing

Abstract: The growing importance of operations such as packet-content inspection, packet classification based on non-IP headers, maintaining flow-state, etc. has led to increased interest in the networking applications of Bloom filters. This is because Bloom filters provide a relatively easy method for hardware implementation of set-membership queries. However, the tradeoff is that Bloom filters only provide a probabilistic test and membership queries can result in false positives. Ideally, we would like this false positive probability to be very low. The main contribution of this paper is a method for significantly reducing this false positive probability in comparison to existing schemes. This is done by developing a partitioned hashing method which results in a choice of hash functions that set far fewer bits in the Bloom filter bit vector than would be the case otherwise. This lower fill factor of the bit vector translates to a much lower false positive probability. We show experimentally that this improved choice can result in as much as a ten-fold increase in accuracy over standard Bloom filters. We also show that the scheme performs much better than other proposed schemes for improving Bloom filters.

False positive reduction in protein-protein interaction predictions using gene ontology annotations

Abstract: Many crucial cellular operations such as metabolism, signalling, and regulations are based on protein-protein interactions However, the lack of robust protein-protein interaction information is a challenge One reason for the lack of solid protein-protein interaction information is poor agreement between experimental findings and computational sets that, in turn, comes from huge false positive predictions in computational approaches Reduction of false positive predictions and enhancing true positive fraction of computationally predicted protein-protein interaction datasets based on highly confident experimental results has not been adequately investigated Gene Ontology (GO) annotations were used to reduce false positive protein-protein interactions (PPI) pairs resulting from computational predictions Using experimentally obtained PPI pairs as a training dataset, eight top-ranking keywords were extracted from GO molecular function annotations The sensitivity of these keywords is 6421% in the yeast experimental dataset and 8083% in the worm experimental dataset The specificities, a measure of recovery power, of these keywords applied to four predicted PPI datasets for each studied organisms, are 4832% and 4649% (by average of four datasets) in yeast and worm, respectively Based on eight top-ranking keywords and co-localization of interacting proteins a set of two knowledge rules were deduced and applied to remove false positive protein pairs The 'strength', a measure of improvement provided by the rules was defined based on the signal-to-noise ratio and implemented to measure the applicability of knowledge rules applying to the predicted PPI datasets Depending on the employed PPI-predicting methods, the strength varies between two and ten-fold of randomly removing protein pairs from the datasets Gene Ontology annotations along with the deduced knowledge rules could be implemented to partially remove false predicted PPI pairs Removal of false positives from predicted datasets increases the true positive fractions of the datasets and improves the robustness of predicted pairs as compared to random protein pairing, and eventually results in better overlap with experimental results

Estimating the statistical significance of peptide identifications from shotgun proteomics experiments.

Abstract: We present a wrapper-based approach to estimate and control the false discovery rate for peptide identifications using the outputs from multiple commercially available MS/MS search engines. Features of the approach include the flexibility to combine output from multiple search engines with sequence and spectral derived features in a flexible classification model to produce a score associated with correct peptide identifications. This classification model score from a reversed database search is taken as the null distribution for estimating p-values and false discovery rates using a simple and established statistical procedure. Results from 10 analyses of rat sera on an LTQ-FT mass spectrometer indicate that the method is well calibrated for controlling the proportion of false positives in a set of reported peptide identifications while correctly identifying more peptides than rule-based methods using one search engine alone.

Understanding False Positives in Reporter Gene Assays: in Silico Chemogenomics Approaches To Prioritize Cell-Based HTS Data

Abstract: High throughput screening (HTS) data is often noisy, containing both false positives and negatives. Thus, careful triaging and prioritization of the primary hit list can save time and money by identifying potential false positives before incurring the expense of followup. Of particular concern are cell-based reporter gene assays (RGAs) where the number of hits may be prohibitively high to be scrutinized manually for weeding out erroneous data. Based on statistical models built from chemical structures of 650 000 compounds tested in RGAs, we created “frequent hitter” models that make it possible to prioritize potential false positives. Furthermore, we followed up the frequent hitter evaluation with chemical structure based in silico target predictions to hypothesize a mechanism for the observed “off target” response. It was observed that the predicted cellular targets for the frequent hitters were known to be associated with undesirable effects such as cytotoxicity. More specifically, the most frequently p...

A 3D Teacher for Car Detection in Aerial Images

Abstract: This paper demonstrates how to reduce the hand labeling effort considerably by 3D information in an object detection task. In particular, we demonstrate how an efficient car detector for aerial images with minimal hand labeling effort can be build. We use an on-line boosting algorithm to incrementally improve the detection results. Initially, we train the classifier with a single positive (car) example, randomly drawn from a fixed number of given samples. When applying this detector to an image we obtain many false positive detections. We use information from a stereo matcher to detect some of these false positives (e.g. detected cars on a facade) and feed back this information to the classifier as negative updates. This improves the detector considerably, thus reducing the number of false positives. We show that we obtain similar results to hand labeling by iteratively applying this strategy. The performance of our algorithm is demonstrated on digital aerial images of urban environments.

SNiPer-HD: improved genotype calling accuracy by an expectation-maximization algorithm for high-density SNP arrays

Abstract: Motivation: The technology to genotype single nucleotide polymorphisms (SNPs) at extremely high densities provides for hypothesis-free genome-wide scans for common polymorphisms associated with complex disease. However, we find that some errors introduced by commonly employed genotyping algorithms may lead to inflation of false associations between markers and phenotype. Results: We have developed a novel SNP genotype calling program, SNiPer-High Density (SNiPer-HD), for highly accurate genotype calling across hundreds of thousands of SNPs. The program employs an expectation-maximization (EM) algorithm with parameters based on a training sample set. The algorithm choice allows for highly accurate genotyping for most SNPs. Also, we introduce a quality control metric for each assayed SNP, such that poor-behaving SNPs can be filtered using a metric correlating to genotype class separation in the calling algorithm. SNiPer-HD is superior to the standard dynamic modeling algorithm and is complementary and non-redundant to other algorithms, such as BRLMM. Implementing multiple algorithms together may provide highly accurate genotyping calls, without inflation of false positives due to systematically miss-called SNPs. A reliable and accurate set of SNP genotypes for increasingly dense panels will eliminate some false association signals and false negative signals, allowing for rapid identification of disease susceptibility loci for complex traits. Availability: SNiPer-HD is available at TGen's website: http://www.tgen.org/neurogenomics/data. Contact: dstephan@tgen.org

Towards development of a 3-state self-paced brain-computer interface

Abstract: Most existing brain-computer interfaces (BCIs) detect specific mental activity in a so-called synchronous paradigm. Unlike synchronous systems which are operational at specific system-defined periods, self-paced (asynchronous) interfaces have the advantage of being operational at all times. The low-frequency asynchronous switch design (LF-ASD) is a 2-state self-paced BCI that detects the presence of a specific finger movement in the ongoing EEG. Recent evaluations of the 2-state LF-ASD show an average true positive rate of 41% at the fixed false positive rate of 1%. This paper proposes two designs for a 3-state self-paced BCI that is capable of handling idle brain state. The two proposed designs aim at detecting right- and left-hand extensions from the ongoing EEG. They are formed of two consecutive detectors. The first detects the presence of a right- or a left-hand movement and the second classifies the detected movement as a right or a left one. In an offline analysis of the EEG data collected from four able-bodied individuals, the 3-state brain-computer interface shows a comparable performance with a 2-state system and significant performance improvement if used as a 2-state BCI, that is, in detecting the presence of a right- or a left-hand movement (regardless of the type of movement). It has an average true positive rate of 37.5% and 42.8% (at false positives rate of 1%) in detecting right- and left-hand extensions, respectively, in the context of a 3-state self-paced BCI and average detection rate of 58.1% (at false positive rate of 1%) in the context of a 2-state self-paced BCI.

Cache-, hash- and space-efficient bloom filters

Abstract: A Bloom filter is a very compact data structure that supports approximate membership queries on a set, allowing false positives. We propose several new variants of Bloom filters and replacements with similar functionality. All of them have a better cache-efficiency and need less hash bits than regular Bloom filters. Some use SIMD functionality, while the others provide an even better space efficiency. As a consequence, we get a more flexible trade-off between false positive rate, space-efficiency, cache-efficiency, hash-efficiency, and computational effort. We analyze the efficiency of Bloom filters and the proposed replacements in detail, in terms of the false positive rate, the number of expected cache-misses, and the number of required hash bits. We also describe and experimentally evaluate the performance of highly-tuned implementations. For many settings, our alternatives perform better than the methods proposed so far.

Adaptively ranking alerts generated from automated static analysis

Abstract: Static analysis tools are useful for finding common programming mistakes that often lead to field failures. However, static analysis tools regularly generate a high number of false positive alerts, requiring manual inspection by the developer to determine if an alert is an indication of a fault. The adaptive ranking model presented in this paper utilizes feedback from developers about inspected alerts in order to rank the remaining alerts by the likelihood that an alert is an indication of a fault. Alerts are ranked based on the homogeneity of populations of generated alerts, historical developer feedback in the form of suppressing false positives and fixing true positive alerts, and historical, application-specific data about the alert ranking factors. The ordering of alerts generated by the adaptive ranking model is compared to a baseline of randomly-, optimally-, and static analysis tool-ordered alerts in a small role-based health care application. The adaptive ranking model provides developers with 81% of true positive alerts after investigating only 20% of the alerts whereas an average of 50 random orderings of the same alerts found only 22% of true positive alerts after investigating 20% of the generated alerts.

ATLANTIDES: an architecture for alert verification in network intrusion detection systems

Abstract: We present an architecture designed for alert verification (i.e., to reduce false positives) in network intrusion-detection systems. Our technique is based on a systematic (and automatic) anomaly-based analysis of the system output, which provides useful context information regarding the network services. The false positives raised by the NIDS analyzing the incoming traffic (which can be either signature- or anomaly-based) are reduced by correlating them with the output anomalies. We designed our architecture for TCP-based network services which have a client/server architecture (such as HTTP). Benchmarks show a substantial reduction of false positives between 50% and 100%.

A Cooperative Approach for Analyzing Intrusions in Mobile Ad hoc Networks

Abstract: In this paper, we consider the problem of reducing the number of false positives generated by cooperative intrusion detection systems (IDSs) in mobile ad hoc networks (MANETs). We define a flexible scheme using security classes, where an IDS is able to operate in different modes at each security class. This scheme helps in minimizing false alarms and informing the prevention system accurately about the severity of an intrusion. Shapley value is used to formally express the cooperation among all the nodes. To the best of our knowledge, there has not been any study for the case where the intrusions in MANETs are analyzed, in order to decrease false positives, using cooperative game theory. Our game theoretic model assists in analyzing the contribution of each mobile node on each security class in order to decrease false positives taking into consideration the reputation of nodes. Simulation results are given to validate the efficiency of our model in detecting intrusions and reducing false positives.

Accuracy requirements for a hypoglycemia detector: an analytical model to evaluate the effects of bias, precision, and rate of glucose change.

Abstract: BACKGROUND There has been considerable debate on what constitutes a good hypoglycemia (Hypo) detector and what is the accuracy required from the continuous monitoring sensor to meet the requirements of such a detector. The performance of most continuous monitoring sensors today is characterized by the mean absolute relative difference (MARD), whereas Hypo detectors are characterized by the number of false positive and false negative alarms, which are more relevant to the performance of a Hypo detector. This article shows that the overall accuracy of the system and not just the sensor plays a key role. METHODS A mathematical model has been developed to investigate the relationship between the accuracy of the continuous monitoring system as described by the MARD, and the number of false negatives and false positives as a function of blood glucose rate change is established. A simulation method with N = 10,000 patients is used in developing the model and generating the results. RESULTS Based on simulation for different scenarios for rate of change (0.5, 1.0, and 5.0 mg/dl per minute), sampling rate (from 1, 2.5, 5, and 10 minutes), and MARD (5, 7.5, 10, 12.5, and 15%), the false positive and false negative ratios are computed. The following key results are from these computations. 1. For a given glucose rate of change, there is an optimum sampling time. 2. The optimum sampling time as defined in the critical sampling rate section gives the best combination of low false positives and low false negatives. 3. There is a strong correlation between MARD and false positives and false negatives. 4. For false positives of <10% and false negatives of <5%, a MARD of <7.5% is needed. CONCLUSIONS Based on the model, assumptions in the model, and the simulation on N = 10,000 patients for different scenarios for rate of glucose change, sampling rate, and MARD, it is concluded that the false negative and false positive ratio will vary depending on the alarm Hypo threshold set by the patient and the MARD value. Also, to achieve a false negative ratio <5% and a false positive ratio <10% would require continuous glucose monitoring to have an MARD < or =7.5%.

Content-based Publish/Subscribe using Distributed R-trees

Abstract: Publish/subscribe systems provide a useful paradigm for selective data dissemination and most of the complexity related to addressing and routing is encapsulated within the network infrastructure. The challenge of such systems is to organize the peers so as to best match the interests of the consumers, minimizing false positives and avoiding false negatives. In this paper, we propose and evaluate the use of R-trees for organizing the peers of a content-based routing network. We adapt three well-known variants of R-trees to the content dissemination problem striving to minimize the occurrence of false positives while avoiding false negatives. The effectiveness and accuracy of each structure is analyzed by extensive simulations.

High-throughput screening by RNA interference : Control of two distinct types of variance

Abstract: The availability of genome-wide RNAi libraries has enabled researchers to rapidly assess the functions of thousands of genes; however the fact that these screens are run in living biological systems add complications above and beyond that normally seen in high-throughput screening (HTS). Specifically, error due to variance in both measurement and biology are large in such screens, leading to the conclusion that the majority of "hits" are expected to be false positives. Here, we outline basic guidelines for screen development that will help the researcher to control these forms of variance. By running a large number of positive and negative control genes, error of measurement can be accurately estimated and false negatives reduced. Likewise, by using a complex readout for the screen, which is not easily mimicked by other biological pathways and phenomena, false positives, can be minimized. By controlling variance in these ways, the researcher can maximize the utility of genome-wide RNAi screening.

A Class of Single-Class Minimax Probability Machines for Novelty Detection

Abstract: Single-class minimax probability machines (MPMs) offer robust novelty detection with distribution-free worst case bounds on the probability that a pattern will fall inside the normal region. However, in practice, they are too cautious in labeling patterns as outlying and so have a high false negative rate (FNR). In this paper, we propose a more aggressive version of the single-class MPM that bounds the best case probability that a pattern will fall inside the normal region. These two MPMs can then be used together to delimit the solution space. By using the hyperplane lying in the middle of this pair of MPMs, a better compromise between false positives (FPs) and false negatives (FNs), and between recall and precision can be obtained. Experiments on the real-world data sets show encouraging results

Alert Verification Based on Attack Classification in Collaborative Intrusion Detection

Abstract: The traditional intrusion detection system has the disadvantages of alert flooding and high false positive due to weak collaboration-awareness. The collaborative intrusion detection mechanism is advocated to overcome shortcomings of traditional IDS and alert verification and correlation are two important techniques to perform collaborative mechanisms. The goal of alert verification is to distinguish the false positives from true positives or confirm the confidence of the alert by integrating context information of protected network with alerts. In this paper, we present an alert verification scheme based on attack classification to achieve the objectives of low cost and high efficiency of verification process.

The local lymph node assay: current position in the regulatory classification of skin sensitizing chemicals.

Abstract: The local lymph node assay (LLNA) is being used increasingly in the identification of skin sensitizing chemicals for regulatory purposes. In the context of new chemicals legislation (REACH) in Europe, it is the preferred assay. The rationale for this is that the LLNA quantitative and objective approach to skin sensitization testing allied with the important animal welfare benefits that the method offers. However, as with certain guinea pig sensitization tests before it, this increasing use also brings experience with an increasingly wide range of industrial and other chemicals where the outcome of the assay does not always necessarily meet with the expectations of those conducting it. Sometimes, the result appears to be a false negative, but rather more commonly, the complaint is that the chemical represents a false positive. Against this background we have here reviewed a number of instances where false positive and false negative results have been described and have sought to reconcile science with expectation. Based on these analyses, it is our conclusion that false positives and false negatives do occur in the LLNA, as they do with any other skin sensitization assay (and indeed with all tests used for hazard identification), and that this occurs for a number of reasons. We further conclude, however, that false positive results in the LLNA, as with the guinea pig maximization test, arise most commonly via failure to distinguish what is scientifically correct from that which is unpalatable. The consequences of this confusion are discussed in the article, particularly in relation to the need to integrate both potency measurement and risk assessments into classification and labelling schemes that aim to manage potential risks to human health.

Re-sampling strategy to improve the estimation of number of null hypotheses in FDR control under strong correlation structures.

Abstract: When conducting multiple hypothesis tests, it is important to control the number of false positives, or the False Discovery Rate (FDR). However, there is a tradeoff between controlling FDR and maximizing power. Several methods have been proposed, such as the q-value method, to estimate the proportion of true null hypothesis among the tested hypotheses, and use this estimation in the control of FDR. These methods usually depend on the assumption that the test statistics are independent (or only weakly correlated). However, many types of data, for example microarray data, often contain large scale correlation structures. Our objective was to develop methods to control the FDR while maintaining a greater level of power in highly correlated datasets by improving the estimation of the proportion of null hypotheses. We showed that when strong correlation exists among the data, which is common in microarray datasets, the estimation of the proportion of null hypotheses could be highly variable resulting in a high level of variation in the FDR. Therefore, we developed a re-sampling strategy to reduce the variation by breaking the correlations between gene expression values, then using a conservative strategy of selecting the upper quartile of the re-sampling estimations to obtain a strong control of FDR. With simulation studies and perturbations on actual microarray datasets, our method, compared to competing methods such as q-value, generated slightly biased estimates on the proportion of null hypotheses but with lower mean square errors. When selecting genes with controlling the same FDR level, our methods have on average a significantly lower false discovery rate in exchange for a minor reduction in the power.