Underestimation of Uncertainty in Statistical Regression of Environmental Models: Influence of Model Structure Uncertainty

Due to ever larger volumes of environmental data, environmental scientists can increasingly use machine learning to construct data-driven models of phenomena. Data-driven environmental models can provide useful information to society, but this requires that their uncertainties be understood. However, new conceptual tools are needed for this because existing approaches to assess the uncertainty of environmental models do so in terms of specific locations, such as model structure and parameter values. These locations are not informative for an assessment of the predictive uncertainty of data-driven models. Rather than the model structure or model parameters, we argue that it is the behavior of a data-driven model that should be subject to an assessment of uncertainty.In this paper, we present a novel framework that can be used to assess the uncertainty of data-driven environmental models. The framework uses argument analysis and focuses on epistemic uncertainty, i.e., uncertainty that is related to a lack of knowledge. It proceeds in three steps. The first step consists in reconstructing the justification of the assumption that the model used is fit for the predictive task at hand. Arguments for this justification may, for example, refer to sensitivity analyses and model performance on a validation dataset. In a second step, this justification is evaluated to identify how conclusively the fitness-for-purpose assumption is justified. In a third step, the epistemic uncertainty is assessed based on the evaluation of the arguments. Epistemic uncertainty emerges due to insufficient justification of the fitness-for-purpose assumption, i.e., if the model is less-than-maximally fit-for-purpose. This lack of justification translates to predictive uncertainty, or first-order uncertainty. Uncertainty also emerges if it is unclear how well the fitness-for-purpose assumption is justified. We refer to this uncertainty as &#8220;second-order uncertainty&#8221;. In other words, second-order uncertainty is uncertainty that researchers face when assessing first-order uncertainty.We illustrate how the framework is applied by discussing to a case study from environmental science in which data-driven models are used to make long-term projections of soil selenium concentrations. We highlight that in many applications, the lack of system understanding and the lack of transparency of machine learning can introduce a substantial level of second-order uncertainty. We close by sketching how the framework can inform uncertainty quantification.

Download Full-text

Impact of the models structure uncertainty in flood simulations: Simbach case study

10.5194/egusphere-egu2020-15216 ◽

2020 ◽

Author(s):

Qing Lin ◽

Jorge Leandro ◽

Markus Disse ◽

Daniel Sturm

Keyword(s):

Flash Flood ◽

Drainage System ◽

Model Structure ◽

Flood Simulation ◽

Hydraulic Models ◽

Uncertainty Factor ◽

Model Component ◽

Model Structure Uncertainty ◽

Flood Simulations ◽

Model Structures

The quantification of model structure uncertainty on hydraulic models is very important for flash flood simulations. The choice of an appropriate model structure complexity and assessment of the impacts due to infrastructure failure can have a huge impact on the simulation results. To assess the risk of flash floods, coupled hydraulic models, including 1D-sewer drainage and 2D-surface run-off models are required for urban areas because they include the bidirectional water exchange, which occurs between sewer and overland flow in a city [1]. By including various model components, we create different model structures. For example, modelling the inflow to the city with the 2D surface-runoff or with the delineated 1D model; including the sewer system or use a surrogate as an alternative; modifying the connectivity of manholes and pumps; or representing the drainage system failures during flood events. As the coupling pattern becomes complex, quantifying the model structure uncertainty is essential for the model structure evaluation. If one model component leads to higher model uncertainty, it is reasonable to conclude that the new component has a large impact in our model and therefore needs to be accounted for; if one component has a less impact in the overall uncertainty, then the model structure can be simplified, by removing that model component. In this study, we set up seven different model structures [2] for the German city of Simbach. By comparison with two inflow calculation types (1D-delineated inflow or 2D-catchment), the existence of drainage system and infrastructure failures, the Model Uncertainty Factor (MUF) is calculated to quantify the model structure uncertainties and further trade-off values with Parameter Uncertainty Factor (PUF) [3]. Finally, we can obtain a more efficient hydraulic model with the essential model structure for urban flash flood simulation. &#160; <ol>1. Leandro, J., Chen, A. S., Djordjevic, S., and Dragan, S. (2009). "A comparison of 1D/1D and 1D/2D coupled hydraulic models for urban flood simulation." Journal of Hydraulic Engineering-ASCE, 6(1):495-504.</ol> <ol>2. Leandro, J., Schumann, A., and Pfister, A. (2016). A step towards considering the spatial heterogeneity of urban, key features in urban hydrology flood modelling. J. Hydrol., Elsevier, 535 (4), 356-365.</ol> <ol>3. Van Zelm, R., Huijbregts, M.A.J. (2013). Quantifying the trade-off between parameter and model structure uncertainty in life cycle impact assessment, Environ. Sci. Technol., 47(16), pp. 9274-9280.</ol> &#160;

Download Full-text

Is modeling of biological wastewater treatment a mature technology?

Water Science & Technology ◽

10.2166/wst.2011.323 ◽

2011 ◽

Vol 63 (8) ◽

pp. 1739-1743 ◽

Cited By ~ 6

Author(s):

Willi Gujer

Keyword(s):

Wastewater Treatment ◽

Potential Model ◽

Biological Wastewater Treatment ◽

Full Potential ◽

Model Structure ◽

Hydraulic Models ◽

The Third ◽

First Case ◽

Model Structure Uncertainty ◽

Parameter Values

Based on three case studies it is demonstrated that the application of mathematical models in biological wastewater treatment has not yet reached its full potential. Model structure uncertainty and correlation of uncertain parameter values make up the first case. The combination of biokinetic and hydraulic models relates to the second case. The evolution of a full scale plant over its life expectancy is the frame for the third case. This paper was initially presented as a discussion starter and thus is raising questions rather than providing answers.

Download Full-text