https://www.cs.princeton.edu/~schapire/papers/ecolmod.pdf

Maximum entropy modeling of species geographic distributions

Prediction of species' distributions is central to diverse applications in ecology, evolution and conservation science. There is increasing electronic access to vast sets of occurrence records in museums and herbaria, yet little effective guidance on how best to use this information in the context of numerous approaches for modelling distributions. To meet this need, we compared 16 modelling methods over 226 species from 6 regions of the world, creating the most comprehensive set of model comparisons to date. We used presence-only data to fit models, and independent presence-absence data to evaluate the predictions. Along with well-established modelling methods such as generalised additive models and GARP and BIOCLIM, we explored methods that either have been developed recently or have rarely been applied to modelling species' distributions. These include machine-learning methods and community models, both of which have features that may make them particularly well suited to noisy or sparse information, as is typical of species' occurrence data. Presence-only data were effective for modelling species' distributions for many species and regions. The novel methods consistently outperformed more established methods. The results of our analysis are promising for the use of data from museums and herbaria, especially as methods suited to the noise inherent in such data improve.

/pdf/novel-methods-improve-prediction-of-species-distributions-3cmioqeu6v.pdf

Novel methods improve prediction of species' distributions from occurrence data

https://www.wsl.ch/staff/niklaus.zimmermann/papers/ecomod135_147.pdf

Predictive habitat distribution models in ecology

Accurate modeling of geographic distributions of species is crucial to various applications in ecology and conservation. The best performing techniques often require some parameter tuning, which may be prohibitively time-consuming to do separately for each species, or unreliable for small or biased datasets. Additionally, even with the abundance of good quality data, users interested in the application of species models need not have the statistical knowledge required for detailed tuning. In such cases, it is desirable to use "default settings", tuned and validated on diverse datasets. Maxent is a recently introduced modeling technique, achieving high predictive accuracy and enjoying several additional attractive properties. The performance of Maxent is influenced by a moderate number of parameters. The first contribution of this paper is the empirical tuning of these parameters. Since many datasets lack information about species absence, we present a tuning method that uses presence-only data. We evaluate our method on independently collected high-quality presence-absence data. In addition to tuning, we introduce several concepts that improve the predictive accuracy and running time of Maxent. We introduce "hinge features" that model more complex relationships in the training data; we describe a new logistic output format that gives an estimate of probability of presence; finally we explore "background sampling" strategies that cope with sample selection bias and decrease model-building time. Our evaluation, based on a diverse dataset of 226 species from 6 regions, shows: 1) default settings tuned on presence-only data achieve performance which is almost as good as if they had been tuned on the evaluation data itself; 2) hinge features substantially improve model performance; 3) logistic output improves model calibration, so that large differences in output values correspond better to large differences in suitability; 4) "target-group" background sampling can give much better predictive performance than random background sampling; 5) random background sampling results in a dramatic decrease in running time, with no decrease in model performance.

https://onlinelibrary.wiley.com/doi/pdf/10.1111/j.0906-7590.2008.5203.x

Modeling of species distributions with Maxent: new extensions and a comprehensive evaluation

Summary
1In recent years the use of species distribution models by ecologists and conservation managers has increased considerably, along with an awareness of the need to provide accuracy assessment for predictions of such models. The kappa statistic is the most widely used measure for the performance of models generating presence–absence predictions, but several studies have criticized it for being inherently dependent on prevalence, and argued that this dependency introduces statistical artefacts to estimates of predictive accuracy. This criticism has been supported recently by computer simulations showing that kappa responds to the prevalence of the modelled species in a unimodal fashion.
2In this paper we provide a theoretical explanation for the observed dependence of kappa on prevalence, and introduce into ecology an alternative measure of accuracy, the true skill statistic (TSS), which corrects for this dependence while still keeping all the advantages of kappa. We also compare the responses of kappa and TSS to prevalence using empirical data, by modelling distribution patterns of 128 species of woody plant in Israel.
3The theoretical analysis shows that kappa responds in a unimodal fashion to variation in prevalence and that the level of prevalence that maximizes kappa depends on the ratio between sensitivity (the proportion of correctly predicted presences) and specificity (the proportion of correctly predicted absences). In contrast, TSS is independent of prevalence.
4When the two measures of accuracy were compared using empirical data, kappa showed a unimodal response to prevalence, in agreement with the theoretical analysis. TSS showed a decreasing linear response to prevalence, a result we interpret as reflecting true ecological phenomena rather than a statistical artefact. This interpretation is supported by the fact that a similar pattern was found for the area under the ROC curve, a measure known to be independent of prevalence.
5Synthesis and applications. Our results provide theoretical and empirical evidence that kappa, one of the most widely used measures of model performance in ecology, has serious limitations that make it unsuitable for such applications. The alternative we suggest, TSS, compensates for the shortcomings of kappa while keeping all of its advantages. We therefore recommend the TSS as a simple and intuitive measure for the performance of species distribution models when predictions are expressed as presence–absence maps.

https://besjournals.onlinelibrary.wiley.com/doi/pdf/10.1111/j.1365-2664.2006.01214.x

Assessing the accuracy of species distribution models: prevalence, kappa and the true skill statistic (TSS)

Predicting the distribution of endangered species from habitat data is frequently perceived to be a useful technique. Models that predict the presence or absence of a species are normally judged by the number of prediction errors. These may be of two types: false positives and false negatives. Many of the prediction errors can be traced to ecological processes such as unsaturated habitat and species interactions. Consequently, if prediction errors are not placed in an ecological context the results of the model may be misleading. The simplest, and most widely used, measure of prediction accuracy is the number of correctly classified cases. There are other measures of prediction success that may be more appropriate. Strategies for assessing the causes and costs of these errors are discussed. A range of techniques for measuring error in presence/absence models, including some that are seldom used by ecologists (e.g. ROC plots and cost matrices), are described. A new approach to estimating prediction error, which is based on the spatial characteristics of the errors, is proposed. Thirteen recommendations are made to enable the objective selection of an error assessment technique for ecological presence/absence models.

https://www.cambridge.org/core/services/aop-cambridge-core/content/view/S0376892997000088

A review of methods for the assessment of prediction errors in conservation presence/absence models

Summer‐bom children did not perform as well as autumn‐born children on the APU Science Survey tests. There is bound to be some confounding of birthdate with length of schooling, but comparisons with foreign studies support the view that it is the age position of the pupil within the class that is the main explanatory factor for this performance difference. A hierarchical linear model (using HLM software) enabled evidence of a birthdate effect to be identified in the performance of pupils aged 11, 13 and 15. The effect is strong at the younger age but is still visible in the older age group. This paper urges teachers to acknowledge these artificial differences in performance when assigning pupils to different ability groupings related to National Assessment levels.

Are Summer‐born Children Disadvantaged? The Birthdate Effect in Education

Marking reliability is purported to be produced by having an effective community of practice. No experimental research has been identified which attempts to verify empirically the aspects of a community of practice that have been observed to produce marking reliability. This research outlines what that community of practice might entail and presents two experimental studies on the effects of particular aspects of community of practice on examiners' marking reliability. In the first study, the impact of exemplar work is investigated: examiners were provided with mark schemes and some examiners were provided with exemplar scripts and given feedback about the marking of those scripts. In the second study, the effects of discussion of the mark scheme are explored: all examiners received mark schemes and exemplar scripts, but some examiners did not attend a coordination meeting. Neither procedure (use of exemplar scripts or discussion between examiners) demonstrated an improvement in marking reliability, which...

What makes marking reliable? Experiments with UK examinations

CONTEXT Some medical courses in the UK have many more applicants than places and almost all applicants have the highest possible previous and predicted examination grades. The BioMedical Admissions Test (BMAT) was designed to assist in the student selection process specifically for a number of 'traditional' medical courses with clear pre-clinical and clinical phases and a strong focus on science teaching in the early years. It is intended to supplement the information provided by examination results, interviews and personal statements. This paper reports on the predictive validity of the BMAT and its predecessor, the Medical and Veterinary Admissions Test. METHODS Results from the earliest 4 years of the test (2000-2003) were matched to the pre-clinical examination results of those accepted onto the medical course at the University of Cambridge. Correlation and logistic regression analyses were performed for each cohort. RESULTS Section 2 of the test ('Scientific Knowledge') correlated more strongly with examination marks than did Section 1 ('Aptitude and Skills'). It also had a stronger relationship with the probability of achieving the highest examination class. CONCLUSIONS The BMAT and its predecessor demonstrate predictive validity for the pre-clinical years of the medical course at the University of Cambridge. The test identifies important differences in skills and knowledge between candidates, not shown by their previous attainment, which predict their examination performance. It is thus a valid source of additional admissions information for medical courses with a strong scientific emphasis when previous attainment is very high.

The predictive validity of the BioMedical Admissions Test for pre-clinical examination performance

This chapter introduces classification tree methodology as an alternative to methods such as discriminant analysis and logistic regression. The technique is illustrated using a small set of data on the species-habitat relationships of pronghorn deer. This is analysed using the package QUEST.

John Bell

Papers

A review of methods for the assessment of prediction errors in conservation presence/absence models

Are Summer‐born Children Disadvantaged? The Birthdate Effect in Education

What makes marking reliable? Experiments with UK examinations

The predictive validity of the BioMedical Admissions Test for pre-clinical examination performance

Tree-based methods