ROBITT: a tool for assessing the risk-of-bias in studies of temporal trends in ecology

1. Aggregated species occurrence and abundance data from disparate sources are increasingly accessible to ecologists for the analysis of temporal trends in biodiversity. However, sampling biases relevant to any given research question are often poorly explored and infrequently reported; this has the potential to undermine statistical inference. In other disciplines, but particularly medicine, researchers are frequently required to complete “risk-of-bias” assessments to expose and document the potential for biases to undermine inference. The huge growth in available data, and recent controversies surrounding their use to infer temporal trends, indicate that similar tools are urgently needed in ecology.2. We introduce ROBITT, a structured tool for assessing the “Risk-Of-Bias In studies of Temporal Trends in ecology”. ROBITT has a similar format to its counterparts in other disciplines: it comprises signalling questions designed to elicit information on the potential for bias in key study domains. In answering these, users will define their inferential goal(s) and relevant statistical population. This information is used to assess potential sampling biases across domains relevant to the research question (e.g. geography, taxonomy, environment), and how these vary through time. If assessments indicate likely sampling biases, then the user must explain what mitigating action will be taken.3. Everything that users need to complete a ROBITT assessment is provided: the tool, a guidance document, and a worked example. Following other disciplines, the tool and guidance document were developed through a consensus-forming process across experts working in relevant areas of ecology and evidence synthesis.4. We propose that researchers should be strongly encouraged to include a ROBITT assessment as supplementary information when publishing studies of biodiversity trends. This will help researchers to structure their thinking, clearly acknowledge potential sampling issues, and provides an opportunity to describe data checks that might otherwise not be reported. ROBITT will also enable reviewers, editors, and readers to establish whether research conclusions are supported given a particular dataset combined with some analytical approach. In turn, it should strengthen evidence-based policy and practice, reduce differing interpretations of data, and provide a clearer picture of the uncertainties associated with our understanding of ecological reality.

Methodological Remarks Regarding Optimal Stopping Tasks and the Implications for Sampling Biases

This paper investigates a type of optimal stopping problem where options are presented in sequence and, once an option has been rejected, it is impossible to go back to it. With previous research finding mixed results of undersampling and oversampling biases on these kinds of optimal stopping tasks, the question remaining is what causes people to sample too much or too little compared to models of optimality? In two pilot studies and a main study, we explored task features that could lead to over- versus undersampling on number-based tasks. We found that, regardless of task features, there were no significant differences in human sampling rate across conditions. Nevertheless, we observed differences in sampling biases across conditions due to varying sampling rates of the optimal model. Our optimal model, like most models used for this type of optimal stopping problem, requires that researchers specify the mean and variance of a theoretical distribution, from which the options are generated. We show that different ways of specifying this generating distribution can lead to different model sampling rates, and consequently, differences in sampling biases. This highlights that a correct specification of the generating distribution is critical when investigating sampling biases on optimal stopping tasks.

Colonial history and global economics distort our understanding of deep-time biodiversity

Sampling variations in the fossil record distort estimates of past biodiversity. However, compilations of global fossil occurrences used in these analyses not only reflect the geological and spatial aspects of the fossil record, but also the historical collation of these data. Here, we demonstrate how the legacy of colonialism as well as socio-economic factors such as wealth, education and political stability impact research output in paleontology. Re- searchers in high or upper middle income countries contribute to 97% of fossil occurrence data, not only leading to spatial sampling biases but also generating a global power imbalance within the discipline. This work illustrates that our efforts to mitigate the effects of sampling biases to obtain a truly representative view of past biodiversity are not disconnected from the aim of diversifying our field.

Significant underestimation of radiative forcing by aerosol–cloud interactions derived from satellite-based methods

Satellite-based estimates of radiative forcing by aerosol–cloud interactions (RFaci) are consistently smaller than those from global models, hampering accurate projections of future climate change. Here we show that the discrepancy can be substantially reduced by correcting sampling biases induced by inherent limitations of satellite measurements, which tend to artificially discard the clouds with high cloud fraction. Those missed clouds exert a stronger cooling effect, and are more sensitive to aerosol perturbations. By accounting for the sampling biases, the magnitude of RFaci (from −0.38 to −0.59 W m−2) increases by 55 % globally (133 % over land and 33 % over ocean). Notably, the RFaci further increases to −1.09 W m−2 when switching total aerosol optical depth (AOD) to fine-mode AOD that is a better proxy for CCN than AOD. In contrast to previous weak satellite-based RFaci, the improved one substantially increases (especially over land), resolving a major difference with models.

Sampling bias and model choice in continuous phylogeography: Getting lost on a random walk

Phylogeographic inference allows reconstruction of past geographical spread of pathogens or living organisms by integrating genetic and geographic data. A popular model in continuous phylogeography—with location data provided in the form of latitude and longitude coordinates—describes spread as a Brownian motion (Brownian Motion Phylogeography, BMP) in continuous space and time, akin to similar models of continuous trait evolution. Here, we show that reconstructions using this model can be strongly affected by sampling biases, such as the lack of sampling from certain areas. As an attempt to reduce the effects of sampling bias on BMP, we consider the addition of sequence-free samples from under-sampled areas. While this approach alleviates the effects of sampling bias, in most scenarios this will not be a viable option due to the need for prior knowledge of an outbreak’s spatial distribution. We therefore consider an alternative model, the spatial Λ-Fleming-Viot process (ΛFV), which has recently gained popularity in population genetics. Despite the ΛFV’s robustness to sampling biases, we find that the different assumptions of the ΛFV and BMP models result in different applicabilities, with the ΛFV being more appropriate for scenarios of endemic spread, and BMP being more appropriate for recent outbreaks or colonizations.

Space-for-time inferences about range-edge dynamics of tree species can be influenced by sampling biases

Differences between the distributions of tree saplings and adults in geographic or niche space have been used to infer climate change effects on tree range dynamics. Previous studies have reported narrower latitudinal or climatic niche ranges of juvenile trees compared to adults, concluding that tree ranges are contracting, contradicting climate-based predictions. However, more comprehensive sampling of adult trees than juvenile trees in most regional forest inventories could potentially bias ontogenetic comparisons. Here we first report spatial simulations showing that reduced sampling intensity can result in underestimates of range and niche limits, but that resampling the same number of individuals of different life stages can eliminate this bias. We then re-analyzed the U.S. Forest Inventory and Analysis data, comparing the range and niche limits between adult trees and saplings of 92 tree species, both using the original data and two re-sampling procedures. Resampling aimed to reduce sampling biases by controlling for either sampling area or the number of individuals sampled. Overall, these resampling procedures had a major influence on the estimation of range limits, most often by reducing, eliminating, or even reversing the tendency in the original analyses for saplings to have broader distributions than adult trees. These results indicate that previous conclusions that the distributions of juvenile trees were contracting in response to climate change were potentially artefacts of sampling in the underlying data. More generally, sampling effects involved in the estimation of geographical ranges and environmental niche widths need to be taken into account in studies comparing different life stages, and also likely in other types of distribution comparisons.