Abstract. Hindcasting experiments (conducting a model forecast for a time period in which observational data is available) are rarely undertaken in the Integrated Assessment Model (IAM) community. When they are undertaken, the results are often evaluated using global aggregates or otherwise highly aggregated skill scores that mask deficiencies. We select a set of deviation based measures that can be applied at different spatial scales (regional versus global) to make evaluating the large number of variable-region combinations in IAMs more tractable. We also identify performance benchmarks for these measures, based on the statistics of the observational dataset, that allow a model to be evaluated in absolute terms rather than relative to the performance of other models at similar tasks. This is key in the integrated assessment community, where there often are not multiple models conducting hindcast experiments to allow for model intercomparison. The performance benchmarks serve a second purpose, providing information about the reasons a model may perform poorly on a given measure and therefore identifying opportunities for improvement. As a case study, the measures are applied to the results of a past hindcast experiment focusing on land allocation in the Global Change Assessment Model (GCAM) version 3.0. We find quantitative evidence that global aggregates alone are not sufficient for evaluating IAMs, such as GCAM, that require global supply to equal global demand at each time period. Additionally, the deviation measures examined in this work successfully identity parametric and structural changes that may improve land allocation decisions in GCAM. Future work will involve implementing the suggested improvements to the GCAM land allocation system identified by the measures in this work, using the measures to quantify performance improvement due to these changes, and, ideally, applying these measures to other sectors of GCAM and other land allocation models.
The role of biomaterials for the energy transition from the lens of a national integrated assessment model
