scholarly journals Prevalence, statistical thresholds, and accuracy assessment for species distribution models

Web Ecology ◽  
2013 ◽  
Vol 13 (1) ◽  
pp. 13-19 ◽  
Author(s):  
B. B. Hanberry ◽  
H. S. He
Keyword(s):  
Species Distribution ◽  
Species Distribution Models ◽  
Accuracy Assessment ◽  
Area Under The Curve ◽  
True Positive Rate ◽  
Distribution Models ◽  
True Negative ◽  
Entire Study ◽  
Predicted Probability ◽  
Positive Rate

Abstract. For species distribution models, species frequency is termed prevalence and prevalence in samples should be similar to natural species prevalence, for unbiased samples. However, modelers commonly adjust sampling prevalence, producing a modeling prevalence that has a different frequency of occurrences than sampling prevalence. The separate effects of (1) use of sampling prevalence compared to adjusted modeling prevalence and (2) modifications necessary in thresholds, which convert continuous probabilities to discrete presence or absence predictions, to account for prevalence, are unresolved issues. We examined effects of prevalence and thresholds and two types of pseudoabsences on model accuracy. Use of sampling prevalence produced similar models compared to use of adjusted modeling prevalences. Mean correlation between predicted probabilities of the least (0.33) and greatest modeling prevalence (0.83) was 0.86. Mean predicted probability values increased with increasing prevalence; therefore, unlike constant thresholds, varying threshold to match prevalence values was effective in holding true positive rate, true negative rate, and species prediction areas relatively constant for every modeling prevalence. The area under the curve (AUC) values appeared to be as informative as sensitivity and specificity, when using surveyed pseudoabsences as absent cases, but when the entire study area was coded, AUC values reflected the area of predicted presence as absent. Less frequent species had greater AUC values when pseudoabsences represented the study background. Modeling prevalence had a mild impact on species distribution models and accuracy assessment metrics when threshold varied with prevalence. Misinterpretation of AUC values is possible when AUC values are based on background absences, which correlate with frequency of species.

scholarly journals Goal-oriented evaluation of species distribution models’ accuracy and precision: True Skill Statistic profile and uncertainty maps

2015 ◽  
Author(s):  
Alejandro Ruete ◽  
Gerardo C Leynaud
Keyword(s):  
Species Distribution ◽  
Species Distribution Models ◽  
Detection Threshold ◽  
Area Under The Curve ◽  
Real Data ◽  
Complete Information ◽  
Alternative Methods ◽  
Model Errors ◽  
Distribution Models ◽  
Model Sets

The use of species distribution models’ (SDM) is limited by its performance in terms of accuracy, precision, or the spatial distribution of model errors. Despite the wide acceptance of some standard statistics used to evaluate SDM, there is currently a strong on-going debate as to their use. The “area under the curve” (AUC) is a popular measure used to evaluate SDMs; however, it does not provide complete information about model accuracy. The maximum True Skill Statistic (TSS) is another statistic that is gaining acceptance. However, evaluations of a model’s accuracy solely based on this statistic may also be misleading. We investigate the use of alternative methods to evaluate the performance of SDMs, to objectively compare among different modelling approaches. We evaluate the performance of SDMs fitted to simulated and real data by contrasting model predictions to additional validation datasets. We propose visualising TSS scores over the whole detection threshold range (TSS profile). We show how models with similarly good performance according to AUC, present very different results and may serve to different purposes. Also, a high maximum TSS may not guarantee accurate predictions and should be accompanied by the threshold where the maximum is reached (t*). We observe that the higher t* the better predicted observations correlate with confirmed observations. Also, SDM predictions should be accompanied with the corresponding uncertainty map to avoid misleading conclusions. Too high or too widely spread uncertainty on such maps would question the overall accuracy of the model. Whether the model is intended to detect all potential observation sites (sensitive model) or to accurately predict where confirmed observations could be found (specific model) sets a different performance targets to be achieved by the model. The approach proposed helps to discern which SDM may best suit the intended goals. Furthermore, the TSS profile helps i) to evaluate the overall performance of SDMs and compare among them, ii) to identify the main source of error, and iii) to select a detection threshold depending on the maps intended use.

scholarly journals Goal-oriented evaluation of species distribution models’ accuracy and precision: True Skill Statistic profile and uncertainty maps

2015 ◽  
Author(s):  
Alejandro Ruete ◽  
Gerardo C Leynaud
Keyword(s):  
Species Distribution ◽  
Species Distribution Models ◽  
Detection Threshold ◽  
Area Under The Curve ◽  
Real Data ◽  
Complete Information ◽  
Alternative Methods ◽  
Model Errors ◽  
Distribution Models ◽  
Model Sets

The use of species distribution models’ (SDM) is limited by its performance in terms of accuracy, precision, or the spatial distribution of model errors. Despite the wide acceptance of some standard statistics used to evaluate SDM, there is currently a strong on-going debate as to their use. The “area under the curve” (AUC) is a popular measure used to evaluate SDMs; however, it does not provide complete information about model accuracy. The maximum True Skill Statistic (TSS) is another statistic that is gaining acceptance. However, evaluations of a model’s accuracy solely based on this statistic may also be misleading. We investigate the use of alternative methods to evaluate the performance of SDMs, to objectively compare among different modelling approaches. We evaluate the performance of SDMs fitted to simulated and real data by contrasting model predictions to additional validation datasets. We propose visualising TSS scores over the whole detection threshold range (TSS profile). We show how models with similarly good performance according to AUC, present very different results and may serve to different purposes. Also, a high maximum TSS may not guarantee accurate predictions and should be accompanied by the threshold where the maximum is reached (t*). We observe that the higher t* the better predicted observations correlate with confirmed observations. Also, SDM predictions should be accompanied with the corresponding uncertainty map to avoid misleading conclusions. Too high or too widely spread uncertainty on such maps would question the overall accuracy of the model. Whether the model is intended to detect all potential observation sites (sensitive model) or to accurately predict where confirmed observations could be found (specific model) sets a different performance targets to be achieved by the model. The approach proposed helps to discern which SDM may best suit the intended goals. Furthermore, the TSS profile helps i) to evaluate the overall performance of SDMs and compare among them, ii) to identify the main source of error, and iii) to select a detection threshold depending on the maps intended use.

scholarly journals Iterative Models for Early Detection of Invasive Species across Spread Pathways

Forests ◽  
2019 ◽  
Vol 10 (2) ◽  
pp. 108 ◽  
Author(s):  
Gericke Cook ◽  
Catherine Jarnevich ◽  
Melissa Warden ◽  
Marla Downing ◽  
John Withrow ◽  
...  
Keyword(s):  
Invasive Species ◽  
Early Detection ◽  
Species Distribution ◽  
Gypsy Moth ◽  
Species Distribution Models ◽  
Survey Design ◽  
Area Under The Curve ◽  
Invasion Pathways ◽  
Distribution Models ◽  
Predicted Risk

Species distribution models can be used to direct early detection of invasive species, if they include proxies for invasion pathways. Due to the dynamic nature of invasion, these models violate assumptions of stationarity across space and time. To compensate for issues of stationarity, we iteratively update regionalized species distribution models annually for European gypsy moth (Lymantria dispar dispar) to target early detection surveys for the USDA APHIS gypsy moth program. We defined regions based on the distances from the invasion spread front where shifts in variable importance occurred and included models for the non-quarantine portion of the state of Maine, a short-range region, an intermediate region, and a long-range region. We considered variables that represented potential gypsy moth movement pathways within each region, including transportation networks, recreational activities, urban characteristics, and household movement data originating from gypsy moth infested areas (U.S. Postal Service address forwarding data). We updated the models annually, linked the models to an early detection survey design, and validated the models for the following year using predicted risk at new positive detection locations. Human-assisted pathways data, such as address forwarding, became increasingly important predictors of gypsy moth detection in the intermediate-range geographic model as more predictor data accumulated over time (relative importance = 5.9%, 17.36%, and 35.76% for 2015, 2016, and 2018, respectively). Receiver operating curves showed increasing performance for iterative annual models (area under the curve (AUC) = 0.63, 0.76, and 0.84 for 2014, 2015, and 2016 models, respectively), and boxplots of predicted risk each year showed increasing accuracy and precision of following year positive detection locations. The inclusion of human-assisted pathway predictors combined with the strategy of iterative modeling brings significant advantages to targeting early detection of invasive species. We present the first published example of iterative species distribution modeling for invasive species in an operational context.

scholarly journals A Robust Prediction Model for Species Distribution Using Bagging Ensembles with Deep Neural Networks

2021 ◽  
Vol 13 (8) ◽  
pp. 1495
Author(s):  
Jehyeok Rew ◽  
Yongjang Cho ◽  
Eenjun Hwang
Keyword(s):  
Neural Networks ◽  
Species Distribution ◽  
Best Practice ◽  
Deep Neural Networks ◽  
Species Distribution Models ◽  
Species Distribution Model ◽  
Environmental Data ◽  
Distribution Model ◽  
Distribution Models ◽  
Robust Prediction

Species distribution models have been used for various purposes, such as conserving species, discovering potential habitats, and obtaining evolutionary insights by predicting species occurrence. Many statistical and machine-learning-based approaches have been proposed to construct effective species distribution models, but with limited success due to spatial biases in presences and imbalanced presence-absences. We propose a novel species distribution model to address these problems based on bootstrap aggregating (bagging) ensembles of deep neural networks (DNNs). We first generate bootstraps considering presence-absence data on spatial balance to alleviate the bias problem. Then we construct DNNs using environmental data from presence and absence locations, and finally combine these into an ensemble model using three voting methods to improve prediction accuracy. Extensive experiments verified the proposed model’s effectiveness for species in South Korea using crowdsourced observations that have spatial biases. The proposed model achieved more accurate and robust prediction results than the current best practice models.

Ascertaining an efficient eligibility cut-off for extended Medicare items for eating disorders

2021 ◽  
pp. 103985622110286
Author(s):  
Tracey Wade ◽  
Jamie-Lee Pennesi ◽  
Yuan Zhou
Keyword(s):  
Eating Disorders ◽  
Eating Disorder ◽  
False Positive Rate ◽  
Area Under The Curve ◽  
Rate Sensitivity ◽  
True Positive Rate ◽  
Eating Disorder Examination Questionnaire ◽  
Eating Disorder Examination ◽  
Positive Rate ◽  
The Relationship

Objective: Currently eligibility for expanded Medicare items for eating disorders (excluding anorexia nervosa) require a score ⩾ 3 on the 22-item Eating Disorder Examination-Questionnaire (EDE-Q). We compared these EDE-Q “cases” with continuous scores on a validated 7-item version of the EDE-Q (EDE-Q7) to identify an EDE-Q7 cut-off commensurate to 3 on the EDE-Q. Methods: We utilised EDE-Q scores of female university students ( N = 337) at risk of developing an eating disorder. We used a receiver operating characteristic (ROC) curve to assess the relationship between the true-positive rate (sensitivity) and the false-positive rate (1-specificity) of cases ⩾ 3. Results: The area under the curve showed outstanding discrimination of 0.94 (95% CI: .92–.97). We examined two specific cut-off points on the EDE-Q7, which included 100% and 87% of true cases, respectively. Conclusion: Given the EDE-Q cut-off for Medicare is used in conjunction with other criteria, we suggest using the more permissive EDE-Q7 cut-off (⩾2.5) to replace use of the EDE-Q cut-off (⩾3) in eligibility assessments.

Sign in / Sign up

Export Citation Format

Share Document