Reducing Simulation Input-Model Risk via Input Model Averaging

Input uncertainty is an aspect of simulation model risk that arises when the driving input distributions are derived or “fit” to real-world, historical data. Although there has been significant progress on quantifying and hedging against input uncertainty, there has been no direct attempt to reduce it via better input modeling. The meaning of “better” depends on the context and the objective: Our context is when (a) there are one or more families of parametric distributions that are plausible choices; (b) the real-world historical data are not expected to perfectly conform to any of them; and (c) our primary goal is to obtain higher-fidelity simulation output rather than to discover the “true” distribution. In this paper, we show that frequentist model averaging can be an effective way to create input models that better represent the true, unknown input distribution, thereby reducing model risk. Input model averaging builds from standard input modeling practice, is not computationally burdensome, requires no change in how the simulation is executed nor any follow-up experiments, and is available on the Comprehensive R Archive Network (CRAN). We provide theoretical and empirical support for our approach.

Download Full-text

Frequentist model averaging for threshold models

Annals of the Institute of Statistical Mathematics ◽

10.1007/s10463-017-0642-9 ◽

2018 ◽

Vol 71 (2) ◽

pp. 275-306 ◽

Cited By ~ 3

Author(s):

Yan Gao ◽

Xinyu Zhang ◽

Shouyang Wang ◽

Terence Tai-leung Chong ◽

Guohua Zou

Keyword(s):

Model Averaging ◽

Threshold Models ◽

Frequentist Model Averaging

Download Full-text

Publication and Identification Biases in Measuring the Intertemporal Substitution of Labor Supply

10.31222/osf.io/nshqx ◽

2021 ◽

Author(s):

Tomas Havranek ◽

Roman Horvath ◽

Ali Elminejad

Keyword(s):

Labor Supply ◽

Publication Bias ◽

Model Uncertainty ◽

Model Averaging ◽

Extensive Margin ◽

Real Business Cycle ◽

Intertemporal Substitution ◽

The Mean ◽

Frisch Elasticity ◽

Frequentist Model Averaging

The intertemporal substitution (Frisch) elasticity of labor supply governs the predictions of real business cycle models and models of taxation. We show that, for the extensive margin elasticity, two biases conspire to systematically produce large positive estimates when the elasticity is in fact zero. Among 723 estimates in 36 studies, the mean reported elasticity is 0.5. One half of that number is due to publication bias: larger estimates are reported preferentially. The other half is due to identification bias: studies with less exogenous time variation in wages report larger elasticities. Net of the biases, the literature implies a zero mean elasticity and, with 95% confidence, is inconsistent with calibrations above 0.25. To derive these results we collect 23 variables that reflect the context in which the elasticity was obtained, use nonlinear techniques to correct for publication bias, and employ Bayesian and frequentist model averaging to address model uncertainty.

Download Full-text

On the Identification of Modeler Communities

International Journal of Information System Modeling and Design ◽

10.4018/ijismd.2014040102 ◽

2014 ◽

Vol 5 (2) ◽

pp. 22-40 ◽

Cited By ~ 1

Author(s):

Dirk van der Linden ◽

Stijn J.B.A. Hoppenbrouwers ◽

Henderik A. Proper

Keyword(s):

Conceptual Understanding ◽

Real World ◽

Empirical Support ◽

Enterprise Modeling ◽

Modeling Language ◽

Science Students ◽

Modeling Process ◽

Psychometric Data ◽

Computing Science ◽

Insight Into

The authors discuss the use and challenges of identifying communities with shared semantics in Enterprise Modeling (EM). People tend to understand modeling meta-concepts (i.e., a modeling language's constructs or types) in a certain way and can be grouped by this conceptual understanding. Having an insight into the typical communities and their composition (e.g., what kind of people constitute such a semantic community) can make it easier to predict how a conceptual modeler with a certain background will generally understand the meta-concepts s/he uses, which is useful for e.g., validating model semantics and improving the efficiency of the modeling process itself. The authors have observed that in practice decisions to group people based on certain shared properties are often made, but are rarely backed up by empirical data demonstrating their supposed efficacy. The authors demonstrate the use of psychometric data from two studies involving experienced (enterprise) modeling practitioners and computing science students to find such communities. The authors also discuss the challenge that arises in finding common real-world factors shared between their members to identify them by and conclude that there is no empirical support for commonly used (and often implicit) grouping properties such as similar background, focus and modeling language.

Download Full-text

Fuzzy-Based Multivariate Analysis for Input Modeling of Risk Assessment in Wind Farm Projects

Algorithms ◽

10.3390/a13120325 ◽

2020 ◽

Vol 13 (12) ◽

pp. 325

Author(s):

Emad Mohamed ◽

Parinaz Jafari ◽

Simaan AbouRizk

Keyword(s):

Risk Assessment ◽

Monte Carlo Simulation ◽

Monte Carlo ◽

Wind Farm ◽

Historical Data ◽

Expert Knowledge ◽

Bivariate Distribution ◽

Expert Elicitation ◽

Illustrative Case ◽

Input Modeling

Currently, input modeling for Monte Carlo simulation (MSC) is performed either by fitting a probability distribution to historical data or using expert elicitation methods when historical data are limited. These approaches, however, are not suitable for wind farm construction, where—although lacking in historical data—large amounts of subjective knowledge describing the impacts of risk factors are available. Existing approaches are also limited by their inability to consider a risk factor’s impact on cost and schedule as dependent. This paper is proposing a methodology to enhance input modeling in Monte Carlo risk assessment of wind farm projects based on fuzzy set theory and multivariate modeling. In the proposed method, subjective expert knowledge is quantified using fuzzy logic and is used to determine the parameters of a marginal generalized Beta distribution. Then, the correlation between the cost and schedule impact is determined and fit jointly into a bivariate distribution using copulas. To evaluate the feasibility of the proposed methodology and to demonstrate its main features, the method was applied to an illustrative case study, and sensitivity analysis and face validation were used to evaluate the method. The results demonstrated that the proposed approach provides a reliable method for enhancing input modeling in Monte Carlo simulation (MCS).

Download Full-text

A general framework for frequentist model averaging

Science China Mathematics ◽

10.1007/s11425-018-9403-x ◽

2019 ◽

Vol 62 (2) ◽

pp. 205-226

Author(s):

Priyam Mitra ◽

Heng Lian ◽

Ritwik Mitra ◽

Hua Liang ◽

Min-ge Xie

Keyword(s):

General Framework ◽

Model Averaging ◽

Frequentist Model Averaging

Download Full-text

Model Averaging and Its Use in Economics

Journal of Economic Literature ◽

10.1257/jel.20191385 ◽

2020 ◽

Vol 58 (3) ◽

pp. 644-719 ◽

Cited By ~ 3

Author(s):

Mark F. J. Steel

Keyword(s):

Model Uncertainty ◽

Bayesian Model Averaging ◽

Model Averaging ◽

Linear Regression Models ◽

Bayesian Procedures ◽

Wide Range ◽

Computational Resources ◽

Production Modeling ◽

Frequentist Model Averaging

The method of model averaging has become an important tool to deal with model uncertainty, for example in situations where a large amount of different theories exist, as are common in economics. Model averaging is a natural and formal response to model uncertainty in a Bayesian framework, and most of the paper deals with Bayesian model averaging. The important role of the prior assumptions in these Bayesian procedures is highlighted. In addition, frequentist model averaging methods are also discussed. Numerical techniques to implement these methods are explained, and I point the reader to some freely available computational resources. The main focus is on uncertainty regarding the choice of covariates in normal linear regression models, but the paper also covers other, more challenging, settings, with particular emphasis on sampling models commonly used in economics. Applications of model averaging in economics are reviewed and discussed in a wide range of areas including growth economics, production modeling, finance and forecasting macroeconomic quantities. (JEL C11, C15, C20, C52, O47).

Download Full-text