Showing papers on "False positive paradox published in 2001"

Assessing Gene Significance from cDNA Microarray Expression Data via Mixed Models

Abstract: The determination of a list of differentially expressed genes is a basic objective in many cDNA microarray experiments. We present a statistical approach that allows direct control over the percentage of false positives in such a list and, under certain reasonable assumptions, improves on existing methods with respect to the percentage of false negatives. The method accommodates a wide variety of experimental designs and can simultaneously assess significant differences between multiple types of biological samples. Two interconnected mixed linear models are central to the method and provide a flexible means to properly account for variability both across and within genes. The mixed model also provides a convenient framework for evaluating the statistical power of any particular experimental design and thus enables a researcher to a priori select an appropriate number of replicates. We also suggest some basic graphics for visualizing lists of significant genes. Analyses of published experiments studying hu...

Is There a Bias in Proteome Research

Abstract: Advances in technology have enabled us to take a fresh look at data acquired by traditional single experiments and to compare them with genomewide data. The differences can be tremendous, as we show here, in the field of proteomics. We have compared data sets of protein-protein interactions in Saccharomyces cerevisiae that were detected by an identical underlying technical method, the yeast two-hybrid system. We found that the individually identified protein-protein interactions are considerably different from those identified by two genomewide scans. Interacting proteins in the pooled database from single publications are much more closely related to each other with respect to transcription profiles when compared to genomewide data. This difference may have been introduced by two factors: by a selection process in individual publications and by false positives in the whole-genome scans. If we assume that the differences are a result of false positives in the whole-genome data, the scans would contain 47%, 44%, and 91% of false positives for the UETZ, ITO-core, and ITO-full data, respectively. If, however, the true fraction of false positives is considerably lower than estimated here, the data from hypothesis-driven experiments must have been subjected to a serious selection process.

Mining needle in a haystack: classifying rare classes via two-phase rule induction

Abstract: Learning models to classify rarely occurring target classes is an important problem with applications in network intrusion detection, fraud detection, or deviation detection in general. In this paper, we analyze our previously proposed two-phase rule induction method in the context of learning complete and precise signatures of rare classes. The key feature of our method is that it separately conquers the objectives of achieving high recall and high precision for the given target class. The first phase of the method aims for high recall by inducing rules with high support and a reasonable level of accuracy. The second phase then tries to improve the precision by learning rules to remove false positives in the collection of the records covered by the first phase rules. Existing sequential covering techniques try to achieve high precision for each individual disjunct learned. In this paper, we claim that such approach is inadequate for rare classes, because of two problems: splintered false positives and error-prone small disjuncts. Motivated by the strengths of our two-phase design, we design various synthetic data models to identify and analyze the situations in which two state-of-the-art methods, RIPPER and C4.5 rules, either fail to learn a model or learn a very poor model. In all these situations, our two-phase approach learns a model with significantly better recall and precision levels. We also present a comparison of the three methods on a challenging real-life network intrusion detection dataset. Our method is significantly better or comparable to the best competitor in terms of achieving better balance between recall and precision.

A new source detection algorithm using FDR

Abstract: The False Discovery Rate (FDR) method has recently been described by Miller et al (2001), along with several examples of astrophysical applications. FDR is a new statistical procedure due to Benjamini and Hochberg (1995) for controlling the fraction of false positives when performing multiple hypothesis testing. The importance of this method to source detection algorithms is immediately clear. To explore the possibilities offered we have developed a new task for performing source detection in radio-telescope images, Sfind 2.0, which implements FDR. We compare Sfind 2.0 with two other source detection and measurement tasks, Imsad and SExtractor, and comment on several issues arising from the nature of the correlation between nearby pixels and the necessary assumption of the null hypothesis. The strong suggestion is made that implementing FDR as a threshold defining method in other existing source-detection tasks is easy and worthwhile. We show that the constraint on the fraction of false detections as specified by FDR holds true even for highly correlated and realistic images. For the detection of true sources, which are complex combinations of source-pixels, this constraint appears to be somewhat less strict. It is still reliable enough, however, for a priori estimates of the fraction of false source detections to be robust and realistic.

Reliabilities of Parsimony-based and Likelihood-based Methods for Detecting Positive Selection at Single Amino Acid Sites

Abstract: The reliabilities of parsimony-based and likelihood-based methods for inferring positive selection at single amino acid sites were studied using the nucleotide sequences of human leukocyte antigen (HLA) genes, in which positive selection is known to be operating at the antigen recognition site. The results indicate that the inference by parsimony-based methods is robust to the use of different evolutionary models and generally more reliable than that by likelihood-based methods. In contrast, the results obtained by likelihood-based methods depend on the models and on the initial parameter values used. It is sometimes difficult to obtain the maximum likelihood estimates of parameters for a given model, and the results obtained may be false negatives or false positives depending on the initial parameter values. It is therefore preferable to use parsimony-based methods as long as the number of sequences is relatively large and the branch lengths of the phylogenetic tree are relatively small.

Elevations in cardiac troponin measurements: false false-positives: the real truth.

Abstract: The development of assays for troponin markers has improved cardiovascular diagnosis. However, because the assays are so sensitive and specific, elevations are often detected in patients without overt coronary artery disease. Some would like to attribute such elevations to biological “false positives.” That is rarely the case. There can be analytic false positives but most such elevations are indicative of subtle degrees of cardiac injury, many not related to ischemic heart disease. The spectrum of such increases and their potential etiologies are elaborated in this article.

An Achilles Heel in Signature-Based IDS : Squealing False Positives in SNORT

Abstract: We report a vulnerability to network signature-based IDS which we have tested using Snort and we call “Squealing” This vulnerability has significant implications since it can easily be generalized to any IDS The vulnerability of signature-based IDS to high false positive rates has been welldocumented but we go further to show (at a high level) how packets can be crafted to match attack signatures such that a alarms on a target IDS can be conditioned or disabled and then exploited This is the first academic treatment of this vulnerability that has already been reported to the CERT Coordination Center and the National Infrastructure Protection Center Independently, other tools based on “squealing” are poised to appear that, while validating our ideas, also gives cause for concern keywords: squealing, false positive, intrusion detection, IDS, signature-based, misuse behavior, network intrusion detection, snort

Computer-aided diagnostic scheme for lung nodule detection in digital chest radiographs by use of a multiple-template matching technique.

Abstract: We have been developing a computer-aided diagnostic (CAD) scheme to assist radiologists in improving the detection of pulmonary nodules in chest radiographs, because radiologists can miss as many as 30% of pulmonary nodules in routine clinical practice. A key to the successful clinical application of a CAD scheme is to ensure that there are only a small number of false positives that are incorrectly reported as nodules by the scheme. In order to significantly reduce the number of false positives in our CAD scheme, we developed, in this study, a multiple-template matching technique, in which a test candidate can be identified as a false positive and thus eliminated, if its largest cross-correlation value with non-nodule templates is larger than that with nodule templates. We describe the technique for determination of cross-correlation values for test candidates with nodule templates and non-nodule templates, the technique for creation of a large number of nodule templates and non-nodule templates, and the technique for removal of nodulelike non-nodule templates and non-nodulelike nodule templates, in order to achieve a good performance. In our study, a large number of false positives (44.3%) were removed with reduction of a very small number of true positives (2.3%) by use of the multiple-template matching technique. We believe that this technique can be used to significantly improve the performance of CAD schemes for lung nodule detection in chest radiographs.

Meaning of reliability in terms of correct and incorrect clinical decisions: the art of decision making is still alive.

Abstract: Examples using cut-off scores are presented to show the effects of measurement error (unreliability) on making correct and incorrect clinical decisions. These decisions are presented in terms of true positives (TPs), true negatives (TNs), false positives (FPs), and false negatives (FNs). A discussion of the difficulty in establishing reliability standards is presented.

What are the baselines for protein fold recognition

Abstract: Motivation: What constitutes a baseline level of success for protein fold recognition methods? As fold recognition benchmarks are often presented without any thought to the results that might be expected from a purely random set of predictions, an analysis of fold recognition baselines is long overdue. Given varying amounts of basic information about a protein—ranging from the length of the sequence to a knowledge of its secondary structure—to what extent can the fold be determined by intelligent guesswork? Can simple methods that make use of secondary structure information assign folds more accurately than purely random methods and could these methods be used to construct viable hierarchical classifications? Experiments performed: A number of rapid automatic methods which score similarities between protein domains were devised and tested. These methods ranged from those that incorporated no secondary structure information, such as measuring absolute differences in sequence lengths, to more complex alignments of secondary structure elements. Each method was assessed for accuracy by comparison with the Class Architecture Topology Homology (CATH) classification. Methods were rated against both a random baseline fold assignment method as a lower control and FSSP as an upper control. Similarity trees were constructed in order to evaluate the accuracy of optimum methods at producing a classification of structure. Results: Using a rigorous comparison of methods with CATH, the random fold assignment method set a lower baseline of 11% true positives allowing for 3% false positives and FSSP set an upper benchmark of 47% true positives at 3% false positives. The optimum secondary structure alignment method used here achieved 27% true positives at 3% false positives. Using a less rigorous Critical Assessment of Structure Prediction (CASP)-like sensitivity measurement the random assignment achieved 6%, FSSP—59% and the optimum secondary structure ∗ To whom correspondence should be addressed. alignment method—32%. Similarity trees produced by the optimum method illustrate that these methods cannot be used alone to produce a viable protein structural classification system. Conclusions: Simple methods that use perfect secondary structure information to assign folds cannot produce an accurate protein taxonomy, however they do provide useful baselines for fold recognition. In terms of a typical CASP assessment our results suggest that approximately 6% of targets with folds in the databases could be assigned correctly by randomly guessing, and as many as 32% could be recognised by trivial secondary structure comparison methods, given knowledge of their correct secondary structures.

False positives and false negatives in genome scans.

Abstract: It is emphasized that two types of errors are made in the testing of a hypothesis, false positive (type I) and false negative (type II). Genome-wide scans involving many markers give rise to the problem of multiple testing, which results in an increased number of false positives, thus necessitating a correction in the nominal significance level. While the literature has concentrated reasonably heavily on controlling false positives in genomic scans, the need to control false negatives has been largely neglected. This chapter highlights this need and attempts to strike a balance between the two error types. The need to develop alternative methods for discrimating between false positives and true positives is also stressed.

Method for building classifier models for event classes via phased rule induction

Abstract: A method for learning signatures of a target class using a sequential covering phased rule-induction The method balances recall and precision for the target class A first phase aims for high recall by inducing rules with high support and a reasonable level of accuracy A second phase improves the precision by learning rules to remove false positives in the collection of the records covered by the first phase rules, while keeping the overall recall at a desirable level The method constructs a mechanism to assign prediction probability scores to each classification decision The model includes a set of positive rules that predict presence of the target class, a set of negative rules that predict absence of the target class, and a set of prediction score values corresponding to each pair-wise combination of positive and negative rules The two-phase method is extensible to a multiphase approach

Determination of tumor markers in serum. Pitfalls and good practice.

Abstract: The determination of tumor markers may have consequences for the patients' treatment, which requires special attention to the analysis and to the expression of the results. In addition to the factors usually dealt with in the pre-analytical phase (identification, quality and storage of the sample) and in the analytical phase (interference, endogenous antibodies, hook effect), we must consider factors such as normal values (which depend heavily on the used techniques), threshold values defining other characteristics (sensitivity, specificity, positive and negative predictive value) and kinetics. Knowledge of the limitations of the tumor marker analysis, in particular tumor markers for non-cancerous diseases leading to possible increases (false positives), is also indispensable. All of this led to the definition of guidelines for good practice, including the design of quality control, rules for requesting, validation and interpretation of results, and the setting up of a serum library.

Image registration by aligning entropies

Abstract: Maximization of mutual information is a powerful method for registering images (and other data) captured with different sensors or under varying conditions, since the technique is robust to variations in the image formation process. On the other hand, the high level of robustness allows false positives when matching over a large search space and also makes it difficult to formulate an efficient search strategy for this case. We describe techniques to overcome these problems by aligning image entropies, which are robust to illumination variation and can be applied to multi-sensor registration. This results in a lower rate of false positives and a more efficient method to search an image for the matching position. The techniques are applied to real imagery and compared to methods based on mutual information and gradients to demonstrate their effectiveness.

False Positives in Genomic Map Assembly and Sequence Validation

Abstract: This paper outlines an algorithm for whole genome order restriction optical map assembly. The algorithm can run very reliably in polynomial time by exploiting a strict limit on the probability that two maps that appear to overlap are in fact unrelated (false positives). The main result of this paper is a tight bound on the false positive probability based on a careful model of the experimental errors in the maps found in practice. Using this false positive probability bound, we show that the probability of failure to compute the correct map can be limited to acceptable levels if the input map error rates satisfy certain sharply delineated conditions. Thus careful experimental design must be used to ensure that whole genome map assembly can be done quickly and reliably.

Tuning Cost-Sensitive Boosting and Its Application to Melanoma Diagnosis

Abstract: This paper investigates a methodology for effective model selection of cost-sensitive boosting algorithms. In many real situations, e.g. for automated medical diagnosis, it is crucial to tune the classification performance towards the sensitivity and specificity required by the user. To this purpose, for binary classification problems, we have designed a cost-sensitive variant of AdaBoost where (1) the model error function is weighted with separate costs for errors (false negative and false positives) in the two classes, and (2) the weights are updated differently for negatives and positives at each boosting step. Finally, (3) a practical search procedure allows to get into or as close as possible to the sensitivity and specificity constraints without an extensive tabulation of the ROC curve. This off-the-shelf methodology was applied for the automatic diagnosis of melanoma on a set of 152 skin lesions described by geometric and colorimetric features, out-performing, on the same data set, skilled dermatologists and a specialized automatic system based on a multiple classifier combination.

[Application of a computer-aided detection (CAD) system to digitalized mammograms for identifying microcalcifications].

Abstract: Purpose It is estimated that during mammographic screening programs radiologists fail to detect approximately 25% of breast cancers visible on retrospective review; this percentage rises to 50% if minimal signs are considered. Independent double reading is now strongly recommended as it allows to reduce the rate of false negative examinations by 5-15%. Recent technological progress has allowed to develop a number of computer-aided detection (CAD) systems. The aim of CAD is to help radiologists interpret lesions by serving as a second reader. In this study the authors developed and applied a CAD system to measure its ability to microcalcifications detect and compare its performance with that of a human observer. Material and methods The study was performed as part of the CALMA (computer-aided library for mammography) project of the Pisa section of the National Institute of Nuclear Physics. The aim of this project is to set up a large database of digital mammograms and to develop a CAD system. Our study series consisted of 802 mammograms - corresponding to 213 patients - digitalized between March and June 2000. We performed traditional mammography and then digitalized the mammograms using a CCD linear scanner (pixel size of 85 x 85 microm2, 12 bits). The images were evaluated by two radiologists with similar experience and then by the CAD system. This CAD system searches for microcalcifications by using ad hoc algorithms and an artificial neural network (Sanger type). Results The number of clusters in our database was 141 corresponding to 140 images; 692 images were non pathological. The CAD system identified a variable number of clusters depending on the threshold values. The threshold value is a number over which the probability of finding a lesion is highest. With thresholds of 0.13 and 0.14 the CAD system identified 140/141 clusters (99.3%); with a threshold of 0.15 it identified 139/141 clusters (98.6%); with a threshold of 0.16, 137/141 (97.2%); with a threshold of 0.18, 133/141 (94.3%); with thresholds of 0.18 and 0.20, 130/141 (92.2%). With threshold values of 0.13, 0.14, 0.15, 0.16 and 0.17 the system's sensitivity was greater than 82%, whereas with values of 0.18 and 0.20 it was greater than 80%. The number of false positive region of interest (ROI) / image was greater with low threshold values: in particular, thresholds of 0.13 and 0.14 yielded 16 false positives /image, thresholds of 0.15 and 0.16 yielded 9 and 7 false positives/image, and both 0.18 and 0.20 only 5/image. Discussion ROC curve shows how the use of high threshold values determined a very high specificity despite very low sensitivity rates. Conversely, low threshold values allowed to have a high sensitivity and a very low specificity. The best performance of our CAD system was obtained with threshold values at 0.15 and 0.16. In fact these thresholds resulted in a high sensitivity (greater than 82%) with an acceptable number of false positives/image, 9 and 7/image, respectively. It is not yet known how radiologists can deal with large numbers of false positives in screening programmes but in our opinion the most important feature of a good CAD system is a high sensitivity. Conclusions In the near future the use of CAD systems will be widespread and easier to apply to everyday practice above all in centers where digital mammography is performed. Mammograms could be directly shown to radiologists after the CAD system has selected the ROI and analysed the images. Thanks to its high sensitivity and despite its low specificity CAD represents a concrete aid for radiologists.

The Precautionary Principle and Electric and Magnetic Fields

Abstract: Current environmental regulation represents a paternalistic policy, more concerned to avoid false positives than false negatives, limiting opportunities for individuals to make choices between risk-avoidance and risk-taking alternatives. For example, many exposures to magnetic fields could be reduced at little or no cost but are not considered seriously, owing to the uncertainty of risk and the concern to avoid false positives. Even though precautionary approaches that focus on avoiding false negatives often do not lead to adverse economic consequences or irrational choices, such approaches usually are not taken. The value of autonomy and the proper role of governmental paternalism with respect to environmental policy need to be considered more carefully in environmental decision making.

DIAGT: Stata module to report summary statistics for diagnostic tests compared to true disease status

Abstract: diagt displays various summary statistics for a diagnostic test, compared to patients' true disease status, sensitivity, specificity, and predictive values, from a 2x2 table. diagti is the immediate version. #a #b #c #d are, respectively, the numbers of true positives (diseased subjects with correct positive test results), false negatives (disease, but negative test), false positives (no disease, but positive test) and true negatives (no disease, negative test). Version 2 of the software now includes an immediate version, diagti.

True and false positive peaks in genomewide scans: The long and the short of it.

Abstract: When performing a genome scan in linkage or linkage disequilibrium studies to detect loci underlying complex or quantitative traits, it is important to attempt to distinguish between true and false positives using the appropriate statistical methods. There has been some controversy in the literature regarding the use of the length of a positive peak, i.e., the length of a chromosome region displaying identity-by-descent in linkage studies among affected individuals or the length of a continuous chromosome region for which the test statistic is above a certain threshold. We show in this study, by reasoning and by simulation studies, that conditional on the strength of evidence for a locus affecting a trait of interest, i.e.. conditional on the peak height of a test statistic, there is no information in the length of the peak. Our finding has implications for linkage and association studies.

Using Priors to Improve Multiple Animal Carcinogenicity Tests

Abstract: This article reviews and summarizes methods for controlling false positives in animal carcinogenicity studies and promotes an alternative that incorporates historical control information via Bayesian methods. The Bayesian paradigm is used as a procedure generator; however, frequentist multisample, age-stratified exact trend tests are used in the ultimate analysis. Critical values for the exact tests are chosen to maximize total expected power, conditional on tumor totals, by using prior distributions. To control the risk of a false-positive finding for one or more tumor types, the sum of the individual critical levels is constrained to be less than a nominal familywise error rate, such as .05. The resulting tests give more power to tumor types with higher-than-expected tumor totals. We use historical control data from animal carcinogenicity studies obtained from a large pharmaceutical company to train and evaluate the tests. There is greatly enhanced power of the proposed method, with concurrent error rat...

Screening for speech and language impairment: a follow-up of true negatives and false positives.

Abstract: It is important to ascertain the validity of screening and case-finding procedures in terms of long-term outcome, particularly if they constitute the primary mode of accessing services. This study is a follow-up of a group of True Negatives and False Positives from the ‘original' study (Laing et al. 2000), which investigated the concurrent validity of a parent-led method and a formal screening method. Nine (29.03%) of the 31 children who were re-assessed fell below the cut-off point for referral and were identified as cases. In particular, as many as eight (57.14%) of the 14 false positives were referred.

Development of a multiple-template matching technique for removal of false positives in a computer-aided diagnostic scheme

Abstract: A problem in most current computer-aided diagnostic (CAD) scheme is the relatively large number of false positives that are incorrectly reported as nodules by the scheme. Therefore, in this study, we developed a multiple-templates matching technique to significantly reduce the number of false positives in our CAD scheme. With this technique applied to our CAD scheme for detection of pulmonary nodules in chest radiographs, we removed a large number of false positives (44.3%) with reduction of a small number of true positives (2.3%). We believe that this technique has the potential to significantly improve the performance of many different CAD schemes for detection of various lesions in medical images including nodules in chest radiographs, masses and microcalcifications in mammograms, nodules, colon polyps, liver tumors, and aneurysms in CT images.© (2001) COPYRIGHT SPIE--The International Society for Optical Engineering. Downloading of the abstract is permitted for personal use only.

Improvement of a Mammographic CAD System for Mass Detection

Abstract: A previously developed computerized scheme to detect masses has been further revised and several improvements were intended. Mammograms were digitized at a higher resolution with a mammographic laser scanner providing 12 bits. Some steps of the scheme, based on bilateral subtraction technique, were modified. Several new features were designed and a BPN neural network was used to reduce the number of false positives. Results obtained with the training set were encouraging, yielding a sensitivity of 85% and 1.54 mean number of false positives per image before applying false positive reduction. After applying false positive reduction, a sensitivity of 78.3% at a mean number of 0.4 false positives per image was obtained. The area under the AFROC curve was A1 = 0.808.

New scientific accuracy measure for performance evaluation of human-computer diagnostic systems

Abstract: This paper first presents a new scientific accuracy measure (denoted by G) for assessing/evaluating the performance of computer medical diagnostic (CMD) systems by incorporating the true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN) of human and computer's diagnoses with respect to each other. Based on G, a formula for computing a multi-parameter sensitivity vector S(G), with the assumption that the system parameter percentage variations are small, is then proposed. For a given set of parameter percentage errors, from the expression of S(G), we can compute the error bound of G and assess the reliability of the system with human and/or computer errors being taken into consideration. It has been demonstrated that the new measure G is capable of providing consistent performance evaluation of a CMD system in general. Based on the value of G, a CMD system can be classified as having 'good', 'fair', or 'poor' performance. Even though the proposed basic accuracy measure and its sensitivity study are derived based on the diagnosis using two diagnostic categories (positive and negative) compared by two observers (a human expert and a computer system), however, its methodology can be extended to CMD systems with multiple diagnostic categories and observers. The formulas for measuring the performance of such systems are discussed and present.

Stasticcal analysis of cow characteristics using fuzzy logic to detect oestrum and mastitis with high sensitivity and few false positives

Abstract: Predicted measurements of functions such as milk yield, temperature and conductivity are accorded upper and lower probability thresholds. Variations in actual measurements are subjected to fuzzy logic analysis, integrated and reverse transformed to give a clear 'true' or 'untrue' indication of mastitis. Oestrum is also indicated.