scispace - formally typeset
Search or ask a question
Journal ArticleDOI

Selecting thresholds of occurrence in the prediction of species distributions

01 Jun 2005-Ecography (Munksgaard International Publishers)-Vol. 28, Iss: 3, pp 385-393
TL;DR: Twelve approaches to determining thresholds were compared using two species in Europe and artificial neural networks, and the modelling results were assessed using four indices: sensitivity, specificity, overall prediction success and Cohen's kappa statistic.
Abstract: Transforming the results of species distribution modelling from probabilities of or suitabilities for species occurrence to presences/absences needs a specific threshold. Even though there are many approaches to determining thresholds, there is no comparative study. In this paper, twelve approaches were compared using two species in Europe and artificial neural networks, and the modelling results were assessed using four indices: sensitivity, specificity, overall prediction success and Cohen's kappa statistic. The results show that prevalence approach, average predicted probability/suitability approach, and three sensitivity-specificity-combined approaches, including sensitivity-specificity sum maximization approach, sensitivity-specificity equality approach and the approach based on the shortest distance to the top-left corner (0,1) in ROC plot, are the good ones. The commonly used kappa maximization approach is not as good as the afore-mentioned ones, and the fixed threshold approach is the worst one. We also recommend using datasets with prevalence of 50% to build models if possible since most optimization criteria might be satisfied or nearly satisfied at the same time, and therefore it's easier to find optimal thresholds in this situation.
Citations
More filters
Journal ArticleDOI
TL;DR: This work compared 16 modelling methods over 226 species from 6 regions of the world, creating the most comprehensive set of model comparisons to date and found that presence-only data were effective for modelling species' distributions for many species and regions.
Abstract: Prediction of species' distributions is central to diverse applications in ecology, evolution and conservation science. There is increasing electronic access to vast sets of occurrence records in museums and herbaria, yet little effective guidance on how best to use this information in the context of numerous approaches for modelling distributions. To meet this need, we compared 16 modelling methods over 226 species from 6 regions of the world, creating the most comprehensive set of model comparisons to date. We used presence-only data to fit models, and independent presence-absence data to evaluate the predictions. Along with well-established modelling methods such as generalised additive models and GARP and BIOCLIM, we explored methods that either have been developed recently or have rarely been applied to modelling species' distributions. These include machine-learning methods and community models, both of which have features that may make them particularly well suited to noisy or sparse information, as is typical of species' occurrence data. Presence-only data were effective for modelling species' distributions for many species and regions. The novel methods consistently outperformed more established methods. The results of our analysis are promising for the use of data from museums and herbaria, especially as methods suited to the noise inherent in such data improve.

7,589 citations


Cites methods from "Selecting thresholds of occurrence ..."

  • ...Liu et al. (2005) have demonstrated that other methods are more reliable for selecting thresholds from the training data (i....

    [...]

Journal ArticleDOI
TL;DR: In this article, the authors provide a theoretical explanation for the observed dependence of kappa on prevalence, and introduce an alternative measure of accuracy, the true skill statistic (TSS), which corrects for this dependence while still keeping all the advantages of Kappa.
Abstract: Summary 1In recent years the use of species distribution models by ecologists and conservation managers has increased considerably, along with an awareness of the need to provide accuracy assessment for predictions of such models. The kappa statistic is the most widely used measure for the performance of models generating presence–absence predictions, but several studies have criticized it for being inherently dependent on prevalence, and argued that this dependency introduces statistical artefacts to estimates of predictive accuracy. This criticism has been supported recently by computer simulations showing that kappa responds to the prevalence of the modelled species in a unimodal fashion. 2In this paper we provide a theoretical explanation for the observed dependence of kappa on prevalence, and introduce into ecology an alternative measure of accuracy, the true skill statistic (TSS), which corrects for this dependence while still keeping all the advantages of kappa. We also compare the responses of kappa and TSS to prevalence using empirical data, by modelling distribution patterns of 128 species of woody plant in Israel. 3The theoretical analysis shows that kappa responds in a unimodal fashion to variation in prevalence and that the level of prevalence that maximizes kappa depends on the ratio between sensitivity (the proportion of correctly predicted presences) and specificity (the proportion of correctly predicted absences). In contrast, TSS is independent of prevalence. 4When the two measures of accuracy were compared using empirical data, kappa showed a unimodal response to prevalence, in agreement with the theoretical analysis. TSS showed a decreasing linear response to prevalence, a result we interpret as reflecting true ecological phenomena rather than a statistical artefact. This interpretation is supported by the fact that a similar pattern was found for the area under the ROC curve, a measure known to be independent of prevalence. 5Synthesis and applications. Our results provide theoretical and empirical evidence that kappa, one of the most widely used measures of model performance in ecology, has serious limitations that make it unsuitable for such applications. The alternative we suggest, TSS, compensates for the shortcomings of kappa while keeping all of its advantages. We therefore recommend the TSS as a simple and intuitive measure for the performance of species distribution models when predictions are expressed as presence–absence maps.

3,518 citations

Journal ArticleDOI
TL;DR: The area under the receiver operating characteristic (ROC) curve, known as the AUC, is currently considered to be the standard method to assess the accuracy of predictive distribution models as discussed by the authors.
Abstract: The area under the receiver operating characteristic (ROC) curve, known as the AUC, is currently considered to be the standard method to assess the accuracy of predictive distribution models. It avoids the supposed subjectivity in the threshold selection process, when continuous probability derived scores are converted to a binary presence‐absence variable, by summarizing overall model performance over all possible thresholds. In this manuscript we review some of the features of this measure and bring into question its reliability as a comparative measure of accuracy between model results. We do not recommend using AUC for five reasons: (1) it ignores the predicted probability values and the goodness-of-fit of the model; (2) it summarises the test performance over regions of the ROC space in which one would rarely operate; (3) it weights omission and commission errors equally; (4) it does not give information about the spatial distribution of model errors; and, most importantly, (5) the total extent to which models are carried out highly influences the rate of well-predicted absences and the AUC scores.

2,711 citations


Cites background from "Selecting thresholds of occurrence ..."

  • ...Both thresholds are highly correlated and, more importantly, they also correlate with prevalence ( Liu et al ., 2005; Jimenez-Valverde & Lobo, 2007)....

    [...]

Journal ArticleDOI
TL;DR: A novel jackknife validation approach is developed and tested to assess the ability to predict species occurrence when fewer than 25 occurrence records are available and the minimum sample sizes required to yield useful predictions remain difficult to determine.
Abstract: Aim: Techniques that predict species potential distributions by combining observed occurrence records with environmental variables show much potential for application across a range of biogeographical analyses. Some of the most promising applications relate to species for which occurrence records are scarce, due to cryptic habits, locally restricted distributions or low sampling effort. However, the minimum sample sizes required to yield useful predictions remain difficult to determine. Here we developed and tested a novel jackknife validation approach to assess the ability to predict species occurrence when fewer than 25 occurrence records are available. Location: Madagascar. Methods: Models were developed and evaluated for 13 species of secretive leaf-tailed geckos (Uroplatus spp.) that are endemic to Madagascar, for which available sample sizes range from 4 to 23 occurrence localities (at 1 km2 grid resolution). Predictions were based on 20 environmental data layers and were generated using two modelling approaches: a method based on the principle of maximum entropy (Maxent) and a genetic algorithm (GARP). Results: We found high success rates and statistical significance in jackknife tests with sample sizes as low as five when the Maxent model was applied. Results for GARP at very low sample sizes (less than c. 10) were less good. When sample sizes were experimentally reduced for those species with the most records, variability among predictions using different combinations of localities demonstrated that models were greatly influenced by exactly which observations were included. Main conclusions: We emphasize that models developed using this approach with small sample sizes should be interpreted as identifying regions that have similar environmental conditions to where the species is known to occur, and not as predicting actual limits to the range of a species. The jackknife validation approach proposed here enables assessment of the predictive ability of models built using very small sample sizes, although use of this test with larger sample sizes may lead to overoptimistic estimates of predictive power. Our analyses demonstrate that geographical predictions developed from small numbers of occurrence records may be of great value, for example in targeting field surveys to accelerate the discovery of unknown populations and species. © 2007 The Authors.

2,647 citations


Cites background from "Selecting thresholds of occurrence ..."

  • ...Many different approaches have been employed for setting thresholds (Liu et al., 2005), yet most techniques depend on balancing false-positive and false-negative predictions, making them applicable only to cases in which absence data are available....

    [...]

Journal ArticleDOI
TL;DR: A detailed explanation of how MaxEnt works and a prospectus on modeling options are provided to enable users to make informed decisions when preparing data, choosing settings and interpreting output to highlight the need for making biologically motivated modeling decisions.
Abstract: The MaxEnt software package is one of the most popular tools for species distribution and environmental niche modeling, with over 1000 published applications since 2006. Its popularity is likely for two reasons: 1) MaxEnt typically outperforms other methods based on predictive accuracy and 2) the software is particularly easy to use. MaxEnt users must make a number of decisions about how they should select their input data and choose from a wide variety of settings in the software package to build models from these data. The underlying basis for making these decisions is unclear in many studies, and default settings are apparently chosen, even though alternative settings are often more appropriate. In this paper, we provide a detailed explanation of how MaxEnt works and a prospectus on modeling options to enable users to make informed decisions when preparing data, choosing settings and interpreting output. We explain how the choice of background samples reflects prior assumptions, how nonlinear functions of environmental variables (features) are created and selected, how to account for environmentally biased sampling, the interpretation of the various types of model output and the challenges for model evaluation. We demonstrate MaxEnt’s calculations using both simplified simulated data and occurrence data from South Africa on species of the flowering plant family Proteaceae. Throughout, we show how MaxEnt’s outputs vary in response to different settings to highlight the need for making biologically motivated modeling decisions.

2,370 citations

References
More filters
Journal ArticleDOI
TL;DR: A review of predictive habitat distribution modeling is presented, which shows that a wide array of models has been developed to cover aspects as diverse as biogeography, conservation biology, climate change research, and habitat or species management.

6,748 citations


"Selecting thresholds of occurrence ..." refers methods in this paper

  • ...generalized linear models (GLM), generalized additive models (GAM), classification and regression trees (CARTs), principal components analysis (PCA), artificial neural networks (ANNs) ( Guisan and Zimmermann 2000, Moisen and Frescino 2002, Guisan et al. 2002, Berg et al. 2004)....

    [...]

Journal ArticleDOI
TL;DR: Receiver-operating characteristic (ROC) plots provide a pure index of accuracy by demonstrating the limits of a test's ability to discriminate between alternative states of health over the complete spectrum of operating conditions.
Abstract: The clinical performance of a laboratory test can be described in terms of diagnostic accuracy, or the ability to correctly classify subjects into clinically relevant subgroups. Diagnostic accuracy refers to the quality of the information provided by the classification device and should be distinguished from the usefulness, or actual practical value, of the information. Receiver-operating characteristic (ROC) plots provide a pure index of accuracy by demonstrating the limits of a test's ability to discriminate between alternative states of health over the complete spectrum of operating conditions. Furthermore, ROC plots occupy a central or unifying position in the process of assessing and using diagnostic tools. Once the plot is generated, a user can readily go on to many other activities such as performing quantitative ROC analysis and comparisons of tests, using likelihood ratio to revise the probability of disease in individual subjects, selecting decision thresholds, using logistic-regression analysis, using discriminant-function analysis, or incorporating the tool into a clinical strategy by using decision analysis.

6,339 citations


"Selecting thresholds of occurrence ..." refers background in this paper

  • ...Zweig and Campbell (1993) suggested that if FPC > FNC, the threshold should favor specificity, while sensitivity should be favored if FNC > FPC (Fielding and Bell 1997)....

    [...]

Journal ArticleDOI
TL;DR: Thirteen recommendations are made to enable the objective selection of an error assessment technique for ecological presence/absence models and a new approach to estimating prediction error, which is based on the spatial characteristics of the errors, is proposed.
Abstract: Predicting the distribution of endangered species from habitat data is frequently perceived to be a useful technique. Models that predict the presence or absence of a species are normally judged by the number of prediction errors. These may be of two types: false positives and false negatives. Many of the prediction errors can be traced to ecological processes such as unsaturated habitat and species interactions. Consequently, if prediction errors are not placed in an ecological context the results of the model may be misleading. The simplest, and most widely used, measure of prediction accuracy is the number of correctly classified cases. There are other measures of prediction success that may be more appropriate. Strategies for assessing the causes and costs of these errors are discussed. A range of techniques for measuring error in presence/absence models, including some that are seldom used by ecologists (e.g. ROC plots and cost matrices), are described. A new approach to estimating prediction error, which is based on the spatial characteristics of the errors, is proposed. Thirteen recommendations are made to enable the objective selection of an error assessment technique for ecological presence/absence models.

6,044 citations


"Selecting thresholds of occurrence ..." refers background in this paper

  • ...Zweig and Campbell (1993) suggested that if FPC� /FNC, the threshold should favor specificity, while sensitivity should be favored if FNC � / FPC ( Fielding and Bell 1997 )....

    [...]

  • ... Fielding and Bell (1997) gave a comprehensive review (Manel et al. 2001)....

    [...]

  • ...The output is biased towards the larger of the two groups ( Fielding and Bell 1997, Cramer 2003), occupied sites and unoccupied sites....

    [...]

  • ...It should be noted that OPS can be deceptively high when frequencies of zeros and ones in binary data are very different ( Fielding and Bell 1997, Pearce and Ferrier 2000, Moisen and Frescino 2002)....

    [...]

  • ...Predicting species distributions is becoming increasingly important since it is relevant to resource assessment, environmental conservation and biodiversity management ( Fielding and Bell 1997, Manel et al. 1999, Austin 2002, D’heygere et al. 2003)....

    [...]

Journal ArticleDOI
TL;DR: ROC analysis is shown to be related in a direct and natural way to cost/benefit analysis of diagnostic decision making and the concepts of "average diagnostic cost" and "average net benefit" are developed and used to identify the optimal compromise among various kinds of diagnostic error.

5,590 citations


"Selecting thresholds of occurrence ..." refers background in this paper

  • ...negative and/or benefit for true positive or true negative ne ds to be taken into account, Metz's (1978) approach can be adopted because the cost of false positive and the be efit of true positive as well as prevalence were explicitly considered....

    [...]

Journal ArticleDOI
TL;DR: A series of papers prepared within the framework of an international workshop entitled: Advances in GLMs /GAMs modeling: from species distribution to environmental management, held in Riederalp, Switzerland, 6 � /11 August 2001 are introduced.

2,006 citations


"Selecting thresholds of occurrence ..." refers methods in this paper

  • ...generalized linear models (GLM), generalized additive models (GAM), classification and regression trees (CARTs), principal components analysis (PCA), artificial neural networks (ANNs) (Guisan and Zimmermann 2000, Moisen and Frescino 2002, Guisan et al. 2002, Berg et al. 2004)....

    [...]