scholarly journals A wavelet-based approach to streamflow event identification and modeled timing error evaluation

2021 ◽  
Vol 25 (5) ◽  
pp. 2599-2615
Author(s):  
Erin Towler ◽  
James L. McCreight
Keyword(s):  
Wavelet Transform ◽  
Model Evaluation ◽  
Statistical Significance ◽  
Model Development ◽  
Model Performance ◽  
Water Model ◽  
Discharge Data ◽  
Event Identification ◽  
Timing Errors ◽  
Stream Discharge

Abstract. Streamflow timing errors (in the units of time) are rarely explicitly evaluated but are useful for model evaluation and development. Wavelet-based approaches have been shown to reliably quantify timing errors in streamflow simulations but have not been applied in a systematic way that is suitable for model evaluation. This paper provides a step-by-step methodology that objectively identifies events, and then estimates timing errors for those events, in a way that can be applied to large-sample, high-resolution predictions. Step 1 applies the wavelet transform to the observations and uses statistical significance to identify observed events. Step 2 utilizes the cross-wavelet transform to calculate the timing errors for the events identified in step 1; this includes the diagnostic of model event hits, and timing errors are only assessed for hits. The methodology is illustrated using real and simulated stream discharge data from several locations to highlight key method features. The method groups event timing errors by dominant timescales, which can be used to identify the potential processes contributing to the timing errors and the associated model development needs. For instance, timing errors that are associated with the diurnal melt cycle are identified. The method is also useful for documenting and evaluating model performance in terms of defined standards. This is illustrated by showing the version-over-version performance of the National Water Model (NWM) in terms of timing errors.

scholarly journals A Wavelet-Based Approach to Streamflow Event Identification and Modeled Timing Error Evaluation

2020 ◽  
Author(s):  
Erin Towler ◽  
James L. McCreight
Keyword(s):  
Wavelet Transform ◽  
Model Evaluation ◽  
Statistical Significance ◽  
Model Development ◽  
Model Performance ◽  
Water Model ◽  
Timing Error ◽  
Discharge Data ◽  
Event Identification ◽  
Timing Errors

Abstract. Streamflow timing errors (in the units of time) are rarely explicitly evaluated, but are useful for model evaluation and development. Wavelet-based approaches have been shown to reliably quantify timing errors in streamflow simulations, but have not been applied in a systematic way that is suitable for model evaluation. This paper provides a step-by-step methodology that objectively identifies events, and then estimates timing errors for those events, in a way that can be applied to large-sample, high-resolution predictions. Step 1 applies the wavelet transform to the observations, and uses statistical significance to identify observed events. Step 2 utilizes the cross-wavelet transform to calculate the timing errors for the events identified in Step 1. The approach also includes a quantification of the confidence in the timing error estimates. The methodology is illustrated using real and simulated stream discharge data from several locations to highlight key method features. The method groups event timing errors by dominant timescales, which can be used to identify the potential processes contributing to the timing errors and the associated model development needs. For instance, timing errors that are associated with the diurnal melt cycle are identified. The method is also useful for documenting and evaluating model performance in terms of defined standards. This is illustrated by showing version-over-version performance of the National Water Model (NWM) in terms of timing errors.

scholarly journals Evaluating Climate Models with an African Lens

2018 ◽  
Vol 99 (2) ◽  
pp. 313-336 ◽  
Author(s):  
Rachel James ◽  
Richard Washington ◽  
Babatunde Abiodun ◽  
Gillian Kay ◽  
Joseph Mutemi ◽  
...  
Keyword(s):  
Model Evaluation ◽  
Climate Models ◽  
Regional Climate ◽  
Model Development ◽  
Model Performance ◽  
Modes Of Variability ◽  
Important Region ◽  
Weather And Climate ◽  
African Perspective ◽  
Local Weather

Abstract Climate models are becoming evermore complex and increasingly relied upon to inform climate change adaptation. Yet progress in model development is lagging behind in many of the regions that need the information most, including in Africa. Targeted model development for Africa is crucial and so too is targeted model evaluation. Assessment of model performance in specific regions often follows a “validation” approach, focusing on mean biases, but if models are to be improved, it is important to understand how they simulate regional climate dynamics: to move from validation to process-based evaluation. This evaluation may be different for every region and requires local weather and climate expertise: a “one size fits all” approach could overlook important, region-specific phenomena. So which are the important processes in African regions? And how might they be evaluated? This paper addresses these questions, drawing on the expertise of a team of scientists from Central, East, southern, and West Africa. For each region, the current understanding of climate models is reviewed, and an example of targeted evaluation is provided, including analysis of moist circulations, teleconnections, and modes of variability. A pan-African perspective is also considered, to examine processes operating between regions. The analysis is based on the Met Office Unified Model, but it uses diagnostics that might be applied to other models. These examples are intended to prompt further discussion among climate modelers and African scientists about how to best evaluate models with an African lens, and promote the development of a model evaluation hub for Africa, to fast track understanding of model behavior for this important continent.

scholarly journals Prognostic factors and prediction models for acute aortic dissection: a systematic review

BMJ Open ◽  
2021 ◽  
Vol 11 (2) ◽  
pp. e042435
Author(s):  
Yan Ren ◽  
Shiyao Huang ◽  
Qianrui Li ◽  
Chunrong Liu ◽  
Ling Li ◽  
...  
Keyword(s):  
Prognostic Factors ◽  
Aortic Dissection ◽  
Acute Aortic Dissection ◽  
Prediction Models ◽  
Statistical Significance ◽  
Model Development ◽  
Model Performance ◽  
Model Studies ◽  
Significance Threshold ◽  
Cohort Study Design

ObjectiveOur study aimed to systematically review the methodological characteristics of studies that identified prognostic factors or developed or validated models for predicting mortalities among patients with acute aortic dissection (AAD), which would inform future work.Design/settingA methodological review of published studies.MethodsWe searched PubMed and EMBASE from inception to June 2020 for studies about prognostic factors or prediction models on mortality among patients with AAD. Two reviewers independently collected the information about methodological characteristics. We also documented the information about the performance of the prognostic factors or prediction models.ResultsThirty-two studies were included, of which 18 evaluated the performance of prognostic factors, and 14 developed or validated prediction models. Of the 32 studies, 23 (72%) were single-centre studies, 22 (69%) used data from electronic medical records, 19 (59%) chose retrospective cohort study design, 26 (81%) did not report missing predictor data and 5 (16%) that reported missing predictor data used complete-case analysis. Among the 14 prediction model studies, only 3 (21%) had the event per variable over 20, and only 5 (36%) reported both discrimination and calibration statistics. Among model development studies, 3 (27%) did not report statistical methods, 3 (27%) exclusively used statistical significance threshold for selecting predictors and 7 (64%) did not report the methods for handling continuous predictors. Most prediction models were considered at high risk of bias. The performance of prognostic factors showed varying discrimination (AUC 0.58 to 0.95), and the performance of prediction models also varied substantially (AUC 0.49 to 0.91). Only six studies reported calibration statistic.ConclusionsThe methods used for prognostic studies on mortality among patients with AAD—including prediction models or prognostic factor studies—were suboptimal, and the model performance highly varied. Substantial efforts are warranted to improve the use of the methods in this population.

scholarly journals Evaluation of a Distributed Streamflow Forecast Model at Multiple Watershed Scales

Water ◽  
2020 ◽  
Vol 12 (5) ◽  
pp. 1279
Author(s):  
Tyler Madsen ◽  
Kristie Franz ◽  
Terri Hogue
Keyword(s):  
Local Population ◽  
Model Performance ◽  
Forecast Model ◽  
Hydrologic Model ◽  
Laboratory Research ◽  
Discharge Data ◽  
Distributed Hydrologic Model ◽  
Verification Period ◽  
Silver Spring

Demand for reliable estimates of streamflow has increased as society becomes more susceptible to climatic extremes such as droughts and flooding, especially at small scales where local population centers and infrastructure can be affected by rapidly occurring events. In the current study, the Hydrology Laboratory-Research Distributed Hydrologic Model (HL-RDHM) (NOAA/NWS, Silver Spring, MD, USA) was used to explore the accuracy of a distributed hydrologic model to simulate discharge at watershed scales ranging from 20 to 2500 km2. The model was calibrated and validated using observed discharge data at the basin outlets, and discharge at uncalibrated subbasin locations was evaluated. Two precipitation products with nominal spatial resolutions of 12.5 km and 4 km were tested to characterize the role of input resolution on the discharge simulations. In general, model performance decreased as basin size decreased. When sub-basin area was less than 250 km2 or 20–40% of the total watershed area, model performance dropped below the defined acceptable levels. Simulations forced with the lower resolution precipitation product had better model evaluation statistics; for example, the Nash–Sutcliffe efficiency (NSE) scores ranged from 0.50 to 0.67 for the verification period for basin outlets, compared to scores that ranged from 0.33 to 0.52 for the higher spatial resolution forcing.

scholarly journals The Influence of Flow Regime on Ecological Quality, Bird Diversity, and Shellfish Fisheries in a Lowland Mediterranean River and Its Coastal Area

Water ◽  
2019 ◽  
Vol 11 (5) ◽  
pp. 918 ◽  
Author(s):  
Oscar Belmar ◽  
Carles Ibáñez ◽  
Ana Forner ◽  
Nuno Caiola
Keyword(s):  
Aquatic Vegetation ◽  
Environmental Flows ◽  
Ecological Status ◽  
Model Performance ◽  
Habitat Characteristics ◽  
Bird Diversity ◽  
Ecological Quality ◽  
Anthropogenic Pressures ◽  
Discharge Data ◽  
Mantis Shrimp

Designing environmental flows in lowland river sections and estuaries is a challenge for researchers and managers, given their complexity and their importance, both for nature conservation and economy. The Ebro River and its delta belong to a Mediterranean area with marked anthropogenic pressures. This study presents an assessment of the relationships between mean flows (discharges) computed at different time scales and (i) ecological quality based on fish populations in the lower Ebro, (ii) bird populations, and (iii) two shellfish fishery species of socioeconomic importance (prawn, or Penaeus kerathurus, and mantis shrimp, or Squilla mantis). Daily discharge data from 2000 to 2015 were used for analyses. Mean annual discharge was able to explain the variation in fish-based ecological quality, and model performance increased when aquatic vegetation was incorporated. Our results indicate that a good ecological status cannot be reached only through changes on discharge, and that habitat characteristics, such as the coverage of macrophytes, must be taken into account. In addition, among the different bird groups identified in our study area, predators were related to river discharge. This was likely due to its influence on available resources. Finally, prawn and mantis shrimp productivity were influenced up to a certain degree by discharge and physicochemical variables, as inputs from rivers constitute major sources of nutrients in oligotrophic environments such as the Mediterranean Sea. Such outcomes allowed revisiting the environmental flow regimes designed for the study area, which provides information for water management in this or in other similar Mediterranean zones.

scholarly journals IDENTIFYING KEY FACTORS OF COMMUTER TRAIN SERVICE QUALITY:AN EMPIRICAL ANALYSIS FOR DHAKA CITY

2019 ◽  
Vol 31 (2) ◽  
Author(s):  
Anika Nowshin Mowrin ◽  
Md. Hadiuzzaman ◽  
Saurav Barua ◽  
Md. Mizanur Rahman
Keyword(s):  
Traffic Congestion ◽  
Fuzzy Inference ◽  
Confusion Matrix ◽  
Model Development ◽  
Model Performance ◽  
Total Sample ◽  
Dhaka City ◽  
Anfis Model ◽  
Regular User ◽  
Inference System

Commuter train is a viable alternative to road transport to ease the traffic congestion which requires appropriate planning by concerned authorities. The research is aimed to assess passengers’ perception about commuter train service running in areas near Dhaka city. An Adaptive Neuro Fuzzy Inference System (ANFIS) model has been developed to evaluate service quality (SQ) of commuter train. Field survey data has been conducted among 802 respondents who were the regular user of commuter train and 12 attributes have been selected for model development. ANFIS was developed by the training and then tested by 80% and 20% of the total sample respectively. After that, model performance has been evaluated by (i) Confusion Matrix (ii) Root Mean Square Error (RMSE) and attributes are ranked based on their relative importance. The proposed ANFIS model has 61.50% accuracy in training and 47.80% accuracy in testing.  From the results, it is found that 'Bogie condition', 'Cleanliness', ‘Female harassment’, 'Behavior of staff' and 'Toilet facility' are the most significant attributes. This indicates that some necessary measures should be taken immediately to recover the effects of these attributes to improve the SQ of commuter train. 

scholarly journals Putting Psychology to the Test: Rethinking Model Evaluation Through Benchmarking and Prediction

2021 ◽  
Vol 4 (3) ◽  
pp. 251524592110268
Author(s):  
Roberta Rocca ◽  
Tal Yarkoni
Keyword(s):  
Machine Learning ◽  
Performance Evaluation ◽  
Model Evaluation ◽  
Evaluation Criteria ◽  
Model Performance ◽  
Scientific Models ◽  
Practical Utility ◽  
Model Performance Evaluation ◽  
Unseen Data ◽  
Reliable Model

Consensus on standards for evaluating models and theories is an integral part of every science. Nonetheless, in psychology, relatively little focus has been placed on defining reliable communal metrics to assess model performance. Evaluation practices are often idiosyncratic and are affected by a number of shortcomings (e.g., failure to assess models’ ability to generalize to unseen data) that make it difficult to discriminate between good and bad models. Drawing inspiration from fields such as machine learning and statistical genetics, we argue in favor of introducing common benchmarks as a means of overcoming the lack of reliable model evaluation criteria currently observed in psychology. We discuss a number of principles benchmarks should satisfy to achieve maximal utility, identify concrete steps the community could take to promote the development of such benchmarks, and address a number of potential pitfalls and concerns that may arise in the course of implementation. We argue that reaching consensus on common evaluation benchmarks will foster cumulative progress in psychology and encourage researchers to place heavier emphasis on the practical utility of scientific models.

scholarly journals Using Model Performance to Assess the Representativeness of Data for Model Development and Calibration in Financial Institutions

Risks ◽  
2021 ◽  
Vol 9 (11) ◽  
pp. 204
Author(s):  
Chamay Kruger ◽  
Willem Daniel Schutte ◽  
Tanja Verster
Keyword(s):  
Case Studies ◽  
Model Development ◽  
Model Performance ◽  
Data Set ◽  
Pooled Data ◽  
External Data ◽  
First Case ◽  
Multivariate Prediction ◽  
Credit Data

This paper proposes a methodology that utilises model performance as a metric to assess the representativeness of external or pooled data when it is used by banks in regulatory model development and calibration. There is currently no formal methodology to assess representativeness. The paper provides a review of existing regulatory literature on the requirements of assessing representativeness and emphasises that both qualitative and quantitative aspects need to be considered. We present a novel methodology and apply it to two case studies. We compared our methodology with the Multivariate Prediction Accuracy Index. The first case study investigates whether a pooled data source from Global Credit Data (GCD) is representative when considering the enrichment of internal data with pooled data in the development of a regulatory loss given default (LGD) model. The second case study differs from the first by illustrating which other countries in the pooled data set could be representative when enriching internal data during the development of a LGD model. Using these case studies as examples, our proposed methodology provides users with a generalised framework to identify subsets of the external data that are representative of their Country’s or bank’s data, making the results general and universally applicable.

scholarly journals Heterogeneous effects of hospital competition on inpatient expenses: an empirical analysis of diseases grouping basing on conditions’ complexity and urgency

2021 ◽  
Vol 21 (1) ◽  
Author(s):  
Liyong Lu ◽  
Xiaojun Lin ◽  
Jay Pan
Keyword(s):  
Regression Model ◽  
Patient Flow ◽  
Statistical Significance ◽  
Healthcare Delivery ◽  
Hospital Competition ◽  
Process Of Care ◽  
Healthcare Delivery System ◽  
Discharge Data ◽  
Heterogeneous Effects ◽  
Hospital Administrative

Abstract Background Multiple pro-competition policies were implemented during the new round of healthcare reform in China. Differences in conditions’ complexity and urgency across diseases associating with various degrees of information asymmetry and choice autonomy in the process of care provision, would lead to heterogeneous effects of competition on healthcare expenses. However, there are limited studies to explore it. This study aims to examine the heterogeneous effects of hospital competition on inpatient expenses basing on disease grouping according to conditions’ complexity and urgency. Methods Collecting information from discharge data of inpatients and hospital administrative data of Sichuan province in China, we selected representative diseases. K-means clustering was used to group the selected diseases and Herfindahl-Hirschman Index (HHI) was calculated based on the predicted patient flow to measure the hospital competition. The log-linear multivariate regression model was used to examine the heterogeneous effects of hospital competition on inpatient expenses. Results We selected 19 representative diseases with significant burdens (more than 1.1 million hospitalizations). The selected diseases were divided into three groups, including diseases with highly complex conditions, diseases with urgent conditions, and diseases with less complex and less urgent conditions. For diseases with highly complex conditions and diseases with urgent conditions, the estimated coefficients of HHI are mixed in the direction and statistical significance in the identical regression model at the 5% level. For diseases with less complex and less urgent conditions, the coefficients of HHI are all positive, and almost all of them significant at the 5% level. Conclusions We found heterogeneous effects of hospital competition on inpatient expenses across disease groups: hospital competition does not play an ideal role in reducing inpatient expenses for diseases with highly complex conditions and diseases with urgent conditions, but it has a significant effect in reducing inpatient expenses of diseases with less complex and less urgent conditions. Our study offers implications that the differences in condition’s complexity and urgency among diseases would lead to different impacts of hospital competition, which would be given full consideration when designing the pro-competition policy in the healthcare delivery system to achieve the desired goal.

Sign in / Sign up

Export Citation Format

Share Document