“A Bias Recognized Is a Bias Sterilized”: The Effects of a Bias in Forecast Evaluation

Are traditional tests of forecast evaluation well behaved when the competing (nested) model is biased? No, they are not. In this paper, we show analytically and via simulations that, under the null hypothesis of no encompassing, a bias in the nested model may severely distort the size properties of traditional out-of-sample tests in economic forecasting. Not surprisingly, these size distortions depend on the magnitude of the bias and the persistency of the additional predictors. We consider two different cases: (i) There is both in-sample and out-of-sample bias in the nested model. (ii) The bias is present exclusively out-of-sample. To address the former case, we propose a modified encompassing test (MENC-NEW) robust to a bias in the null model. Akin to the ENC-NEW statistic, the asymptotic distribution of our test is a functional of stochastic integrals of quadratic Brownian motions. While this distribution is not pivotal, we can easily estimate the nuisance parameters. To address the second case, we derive the new asymptotic distribution of the ENC-NEW, showing that critical values may differ remarkably. Our Monte Carlo simulations reveal that the MENC-NEW (and the ENC-NEW with adjusted critical values) is reasonably well-sized even when the ENC-NEW (with standard critical values) exhibits rejections rates three times higher than the nominal size.

Human Factor Risk Modeling for Shipyard Operation by Mapping Fuzzy Fault Tree into Bayesian Network

The operational activities conducted in a shipyard are exposed to high risk associated with human factors. To investigate human factors involved in shipyard operational accidents, a double-nested model was proposed in the present study. The modified human factor analysis classification system (HFACS) was applied to identify the human factors involved in the accidents, the results of which were then converted into diverse components of a fault tree and, as a result, a single-level nested model was established. For the development of a double-nested model, the structured fault tree was mapped into a Bayesian network (BN), which can be simulated with the obtained prior probabilities of parent nodes and the conditional probability table by fuzzy theory and expert elicitation. Finally, the developed BN model is simulated for various scenarios to analyze the identified human factors by means of structural analysis, path dependencies and sensitivity analysis. The general interpretation of these analysis verify the effectiveness of the proposed methodology to evaluate the human factor risks involved in operational accidents in a shipyard.

A nested model for tuberculosis: Combining within-host and between-host processes in a single framework

Tuberculosis (TB) is among the 10 top causes of deaths worldwide, and one-quarter of the world population hosts latent TB pathogens. Therefore, avoiding the emergence of drug-resistant strains has become a central issue in TB control. In this work, we propose a nested model for TB transmission and control, wherein both within-host and between-host dynamics are modeled. We use the model to compare the effects of three types of antibiotic treatment protocols and combinations thereof in an in silico population. For a fixed value of antibiotics clearance rate and relative efficacy against resistant strains, the oscillating intermittent protocol, pure or combined, is the most effective against the sensitive strains. However, this protocol also creates a selective advantage for the resistant strains, returning the worst result in comparison to the other protocols. We suggest that nested models should be further developed, since they might be able to inform decision-makers regarding the optimal TB control protocols to be applied under the specific parameters and other epidemiological factors in different populations.

We need to change how we compute RMSEA for nested model comparisons in structural equation modeling

Comparison of nested models is common in applications of structural equation modeling (SEM). When two models are nested, model comparison can be done via a chi-square difference test or by comparing indices of approximate fit. The advantage of fit indices is that they permit some amount of misspecification in the additional constraints imposed on the model, which is a more realistic scenario. The most popular index of approximate fit is the root mean square error of approximation (RMSEA). In this article, we argue that the dominant way of comparing RMSEA values for two nested models, which is simply taking their difference, is problematic and will often mask misfit. We instead advocate computing the RMSEA associated with the chi-square difference test. We are not the first to propose this idea, and we review numerous methodological articles that have suggested it. Nonetheless, these articles appear to have had little impact on actual practice. The modification of current practice that we call for may be particularly needed in the context of measurement invariance assessment. We illustrate the difference between the current approach and our advocated approach on three examples, where two involve multiple-group and longitudinal measurement invariance assessment and the third involves comparisons of models with different numbers of factors. We conclude with a discussion of limitations and future research directions.

Are large scale models useful? A case of nested model domains for assessing current and future stream runoff and sediments

<p>Continental and global dynamic hydrological models have emerged recently as tools for e.g. flood forecasting, large-scale climate impact analyses, and estimation of time-dynamic water fluxes into sea basins. One such tool is a dynamic process-based rainfall-runoff and water quality model Hydrological Predictions for Environment (HYPE). We present and compare historical simulations of runoff, soil moisture, aridity, and sediment concentrations for three nested model domains using global, continental (Europe), and national (Sweden) catchment-based HYPE applications. Future impacts on hydrological variables from changing climate were then assessed using the global and continental HYPE applications with ensembles based on 3 CMIP5 global climate models (GCMs).</p><p>Simulated historical sediment concentrations varied considerably among the nested models in spatial patterns while runoff values were more similar. Regardless of the variation, the global model was able to provide information on climate change impacts comparable to those from the continental and national models for hydrological indicators. Output variables that were calibrated, e.g. runoff, were shown to result in more reliable and consistent projected changes among the different model scales than derived variables such as the actual aridity index. The comparison was carried out for ensemble averages as well as individual GCMs to illustrate the variability and the need for robust assessments.</p><p>Global hydrological models are shown to be valuable tools for e.g. first screenings of climate change effects and detection of spatial patterns and can be useful to provide information on current and future hydrological states at various domains. The challenge is (1) in deciding when we should use the large-scale models and (2) in interpreting the results, considering the uncertainty of the model results and quality of data especially at the global scale. Comparison across nested domains demonstrates the significance of scale which needs to be considered when interpreting the impacts alongside with model performance.</p><p>Bartosova et al, 2021: Large-scale hydrological and sediment modeling in nested domains under current and changing climate. Accepted to Special Issue Journal of Hydraulic Engineering.</p>

Streamflow Predictions in a Small Urban–Rural Watershed: The Effects of Radar Rainfall Resolution and Urban Rainfall–Runoff Dynamics

The authors predicted streamflow in an urban–rural watershed using a nested regional–local modeling approach for the community of Manchester, Iowa, which is downstream of a largely rural watershed. The nested model coupled the hillslope-link model (HLM), used to simulate the upstream rural basins, and XPSWMM, which was used to simulate the more complex rainfall–runoff dynamics and surface and subsurface drainage in the urban areas, making it capable of producing flood maps at the street level. By integrating these models built for different purposes, we enabled fast and accurate simulation of hydrological processes in the rural basins while also modeling the flows in an urban environment. Using the model, we investigated how the spatial and temporal resolution of radar rainfall inputs can affect the modeled streamflow. We used a combination of three radar rainfall products to capture the uncertainty of rainfall estimation in the model results. Our nested model was able to simulate the hydrographs and timing and duration above the threshold known to result in nuisance flooding in Manchester. The spatiotemporal resolution the radar rainfall input to the model impacted the streamflow outputs of the regional, local, and nested models differently depending on the storm event.

Recurrent Nested Model for Sequence Generation

Depth has been shown beneficial to neural network models. In this paper, we make an attempt to make the encoder-decoder model deeper for sequence generation. We propose a module that can be plugged into the middle between the encoder and decoder to increase the depth of the whole model. The proposed module follows a nested structure, which is divided into blocks with each block containing several recurrent transition steps. To reduce the training difficulty and preserve the necessary information for the decoder during transitions, inter-block connections and intra-block connections are constructed in our model. The inter-block connections provide the thought vectors from the current block to all the subsequent blocks. The intra-block connections connect all the hidden states entering the current block to the current transition step. The advantages of our model are illustrated on the image captioning and code captioning tasks.