Integrating Bayesian Calibration, Bias Correction, and Machine Learning for the 2014 Sandia Verification and Validation Challenge Problem

2016 ◽  
Vol 1 (1) ◽  
Author(s):  
Wei Li ◽  
Shishi Chen ◽  
Zhen Jiang ◽  
Daniel W. Apley ◽  
Zhenzhou Lu ◽  
...  
Keyword(s):  
Machine Learning ◽  
Predictive Model ◽  
Bias Correction ◽  
Verification And Validation ◽  
Model Parameters ◽  
Loading Conditions ◽  
Multiple Sources ◽  
Bayesian Calibration ◽  
Sources Of Uncertainty ◽  
Machine Learning Model

This paper describes an integrated Bayesian calibration, bias correction, and machine learning approach to the validation challenge problem posed at the Sandia Verification and Validation Challenge Workshop, May 7–9, 2014. Three main challenges are recognized as: I—identification of unknown model parameters; II—quantification of multiple sources of uncertainty; and III—validation assessment when there are no direct experimental measurements associated with one of the quantities of interest (QoIs), i.e., the von Mises stress. This paper addresses these challenges as follows. For challenge I, sensitivity analysis is conducted to select model parameters that have significant impact on the model predictions for the displacement, and then a modular Bayesian approach is performed to calibrate the selected model parameters using experimental displacement data from lab tests under the “pressure only” loading conditions. Challenge II is addressed using a Bayesian model calibration and bias correction approach. For improving predictions of displacement under “pressure plus liquid” loading conditions, a spatial random process (SRP) based model bias correction approach is applied to develop a refined predictive model using experimental displacement data from field tests. For challenge III, the underlying relationship between stress and displacement is identified by training a machine learning model on the simulation data generated from the supplied tank model. Final predictions of stress are made via the machine learning model and using predictions of displacements from the bias-corrected predictive model. The proposed approach not only allows the quantification of multiple sources of uncertainty and errors in the given computer models, but also is able to combine multiple sources of information to improve model performance predictions in untested domains.

P1060USING ARTIFICIAL INTELLIGENCE TO PREDICT HOME THERAPY CANDIDATES

2020 ◽  
Vol 35 (Supplement_3) ◽  
Author(s):  
Jerry Yu ◽  
Andrew Long ◽  
Maria Hanson ◽  
Aleetha Ellis ◽  
Michael Macarthur ◽  
...  
Keyword(s):  
Machine Learning ◽  
Predictive Model ◽  
Feedback Loop ◽  
Area Under The Curve ◽  
Patient Characteristics ◽  
Training Data ◽  
Quality Improvement Initiative ◽  
Home Therapy ◽  
Test Dataset ◽  
Machine Learning Model

Abstract Background and Aims There are many benefits for performing dialysis at home including more flexibility and more frequent treatments. A possible barrier to election of home therapy (HT) by in-center patients is a lack of adequate HT education. To aid efficient education efforts, a predictive model was developed to help identify patients who are more likely to switch from in-center and succeed on HT. Method We developed a model using machine learning to predict which patients who are treated in-center without prior HT history are most likely to switch to HT in the next 90 days and stay on HT for at least 90 days. Training data was extracted from 2016–2019 for approximately 300,000 patients. We randomly sampled one in-center treatment date per patient and determined if the patient would switch and succeed on HT. The input features consisted of treatment vitals, laboratories, absence history, comprehensive assessments, facility information, county-level housing, and patient characteristics. Patients were excluded if they had less than 30 days on dialysis due to lack of data. A machine learning model (XGBoost classifier) was deployed monthly in a pilot with a team of HT educators to investigate the model’s utility for identifying HT candidates. Results There were approximately 1,200 patients starting a home therapy per month in a large dialysis provider, with approximately one-third being in-center patients. The prevalence of switching and succeeding to HT in this population was 2.54%. The predictive model achieved an area under the curve of 0.87, sensitivity of 0.77, and a specificity of 0.80 on a hold-out test dataset. The pilot was successfully executed for several months and two major lessons were learned: 1) some patients who reappeared on each month’s list should be removed from the list after expressing no interest in HT, and 2) a data collection mechanism should be put in place to capture the reasons why patients are not interested in HT. Conclusion This quality-improvement initiative demonstrates that predictive modeling can be used to identify patients likely to switch and succeed on home therapy. Integration of the model in existing workflows requires creating a feedback loop which can help improve future worklists.

Updating Predictive Models: Calibration, Bias Correction and Identifiability

2010 ◽  
Author(s):  
Paul D. Arendt ◽  
Wei Chen ◽  
Daniel W. Apley
Keyword(s):  
Computer Model ◽  
Bayesian Approach ◽  
Bias Correction ◽  
Process Model ◽  
Model Updating ◽  
Future Research ◽  
Model Parameters ◽  
Sources Of Uncertainty ◽  
Uncertain Model ◽  
Many Sources

Model updating, which utilizes mathematical means to combine model simulations with physical observations for improving model predictions, has been viewed as an integral part of a model validation process. While calibration is often used to “tune” uncertain model parameters, bias-correction has been used to capture model inadequacy due to a lack of knowledge of the physics of a problem. While both sources of uncertainty co-exist, these two techniques are often implemented separately in model updating. This paper examines existing approaches to model updating and presents a modular Bayesian approach as a comprehensive framework that accounts for many sources of uncertainty in a typical model updating process and provides stochastic predictions for the purpose of design. In addition to the uncertainty in the computer model parameters and the computer model itself, this framework accounts for the experimental uncertainty and the uncertainty due to the lack of data in both computer simulations and physical experiments using the Gaussian process model. Several challenges are apparent in the implementation of the modular Bayesian approach. We argue that distinguishing between uncertain model parameters (calibration) and systematic inadequacies (bias correction) is often quite challenging due to an identifiability issue. We present several explanations and examples of this issue and bring up the needs of future research in distinguishing between the two sources of uncertainty.

scholarly journals Machine Learning Model for GSM BSC Control Plane Units

2019 ◽  
Vol 8 (6S) ◽  
pp. 219-223
Keyword(s):  
Machine Learning ◽  
Back Propagation ◽  
Back Propagation Neural Network ◽  
Model Parameters ◽  
Large Set ◽  
Data Set ◽  
Wide Acceptance ◽  
Machine Learning Model ◽  
Machine Learning Approach ◽  
Accuracy Of Prediction

At maximum traffic intensity i.e. during the busy hour, the GSM BSC signalling units (BSU) measured CPU load will be at its peak. The BSUs CPU load is a function of the number of transceivers (TRXs) mapped to it and hence the volume of offered traffic being handled by the unit. The unit CPU load is also a function of the nature of the offered load, i.e. with the same volume of offered load, the CPU load with the nominal traffic profile would be different as compared to some other arbitrary traffic profile. To manage future traffic growth, a model to estimate the BSU unit CPU load is an essential need. In recent times, using Machine Learning (ML) to develop such a model is an approach that has gained wide acceptance. Since, the estimation of CPU load is difficult as it depends on large set of parameters, machine learning approach is more scalable. In this paper, we describe a back-propagation neural network model that was developed to estimate the BSU unit CPU load. We describe the model parameters and choices and implementation architecture, and estimate its accuracy of prediction, based on an evaluation data set. We also discuss alternative ML architectures and compare their relative prediction accuracies, to the primary ML model

A machine learning approach to the observation operator for satellite radiance data assimilation

2021 ◽  
Author(s):  
Jianyu Liang ◽  
Koji Terasaki ◽  
Takemasa Miyoshi
Keyword(s):  
Machine Learning ◽  
Data Assimilation ◽  
Bias Correction ◽  
Reference System ◽  
Weather Prediction ◽  
Learning Model ◽  
Correction Procedure ◽  
Observation Operator ◽  
Machine Learning Model ◽  
Physically Based

<p>The ‘observation operator’ is essential in data assimilation (DA) to derive the model equivalent of the observations from the model variables. For satellite radiance observations, it is usually based on complex radiative transfer model (RTM) with a bias correction procedure. Therefore, it usually takes time to start using new satellite data after launching the satellites. Here we take advantage of the recent fast development of machine learning (ML) which is good at finding the complex relationships within data. ML can potentially be used as the ‘observation operator’ to reveal the relationships between the model variables and the observations without knowing their physical relationships. In this study, we test with the numerical weather prediction system composed of the Nonhydrostatic Icosahedral Atmospheric Model (NICAM) and the Local Ensemble Transform Kalman Filter (LETKF). We focus on the satellite microwave brightness temperature (BT) from the Advanced Microwave Sounding Unit-A (AMSU-A). Conventional observations and AMSU-A data were assimilated every 6 hours. The reference DA system employed the observation operator based on the RTTOV and an online bias correction method.</p><p>We used this reference system to generate 1-month data to train the machine learning model. Since the reference system includes running a physically-based RTM, we implicitly used the information from RTM for training the ML model in this study, although in our future research we will explore methods without the use of RTM. The machine learning model is artificial neural networks with 5 fully connected layers. The input of the ML model includes the NICAM model variables and predictors for bias correction, and the output of the ML model is the corresponding satellite BT in 3 channels from 5 satellites. Next, we ran the DA cycle for the same month the following year to test the performance of the ML model. Two experiments were conducted. The control experiment (CTRL) was performed with the reference system. In the test experiment (TEST), the ML model was used as the observation operator and there is no separate bias correction procedure since the training includes biased differences between the model and observation. The results showed no significant bias of the simulated BT by the ML model. Using the ECMWF global atmospheric reanalysis (ERA-interim) as a benchmark to evaluate the analysis accuracy, the global-mean RMSE, bias, and ensemble spread for temperature in TEST are 2% higher, 4% higher, and 1% lower respectively than those in CTRL. The result is encouraging since our ML can emulate the RTM. The limitation of our study is that we rely on the physically-based RTM in the reference DA system, which is used for training the ML model. This is the first result and still preliminary. We are currently considering other methods to train the ML model without using the RTM at all.</p>

A Sequential Calibration and Validation Framework for Model Parameter Updating and Bias Correction

2020 ◽  
Author(s):  
Chen Jiang ◽  
Yixuan Liu ◽  
Zhen Hu ◽  
Zissimos P. Mourelatos ◽  
David Gorsich ◽  
...  
Keyword(s):  
Model Validation ◽  
Bias Correction ◽  
Model Parameters ◽  
Model Bias ◽  
Bayesian Calibration ◽  
Prior Distributions ◽  
Model Parameter ◽  
Calibration And Validation ◽  
Validation Framework ◽  
Uncertain Model

Abstract Model parameter updating and bias correction plays an essential role in improving the validity of Modeling and Simulation (M&S) in engineering design and analysis. However, it is observed that the existing methods may either be misled by potentially wrong information if the computer model cannot adequately capture the underlying true physics, or be affected by the prior distributions of the unknown model parameters. In this paper, a sequential model calibration and validation (SeCAV) framework is proposed to improve the efficacy of both model parameter updating and model bias correction, where the model validation and Bayesian calibration are implemented in a sequential manner. In each iteration, the model validation assessment is employed as a filter to select the best experimental data for Bayesian calibration, and to update the prior distributions of uncertain model parameters for the next iteration. The calibrated parameters are then integrated with model bias correction to improve the prediction accuracy of the M&S. A mathematical example is employed to demonstrate the advantages of the SeCAV method.

scholarly journals Privacy-Preserving Gaussian Process Regression – A Modular Approach to the Application of Homomorphic Encryption

2020 ◽  
Vol 34 (04) ◽  
pp. 3866-3873
Author(s):  
Peter Fenner ◽  
Edward Pyzer-Knapp
Keyword(s):  
Machine Learning ◽  
Gaussian Process ◽  
Data Privacy ◽  
Homomorphic Encryption ◽  
Gaussian Process Regression ◽  
Machine Learning Algorithms ◽  
Modular Approach ◽  
Multiple Sources ◽  
Machine Learning Model ◽  
Evaluation Of Data

Much of machine learning relies on the use of large amounts of data to train models to make predictions. When this data comes from multiple sources, for example when evaluation of data against a machine learning model is offered as a service, there can be privacy issues and legal concerns over the sharing of data. Fully homomorphic encryption (FHE) allows data to be computed on whilst encrypted, which can provide a solution to the problem of data privacy. However, FHE is both slow and restrictive, so existing algorithms must be manipulated to make them work efficiently under the FHE paradigm. Some commonly used machine learning algorithms, such as Gaussian process regression, are poorly suited to FHE and cannot be manipulated to work both efficiently and accurately. In this paper, we show that a modular approach, which applies FHE to only the sensitive steps of a workflow that need protection, allows one party to make predictions on their data using a Gaussian process regression model built from another party's data, without either party gaining access to the other's data, in a way which is both accurate and efficient. This construction is, to our knowledge, the first example of an effectively encrypted Gaussian process.

scholarly journals Machine learning for observation bias correction with application to dust storm data assimilation

2019 ◽  
Vol 19 (15) ◽  
pp. 10009-10026 ◽  
Author(s):  
Jianbing Jin ◽  
Hai Xiang Lin ◽  
Arjo Segers ◽  
Yu Xie ◽  
Arnold Heemink
Keyword(s):  
Machine Learning ◽  
Data Assimilation ◽  
Dust Storm ◽  
Bias Correction ◽  
Dust Emission ◽  
Learning Model ◽  
Life Cycles ◽  
Dust Aerosols ◽  
Machine Learning Model ◽  
Observation Bias

Abstract. Data assimilation algorithms rely on a basic assumption of an unbiased observation error. However, the presence of inconsistent measurements with nontrivial biases or inseparable baselines is unavoidable in practice. Assimilation analysis might diverge from reality since the data assimilation itself cannot distinguish whether the differences between model simulations and observations are due to the biased observations or model deficiencies. Unfortunately, modeling of observation biases or baselines which show strong spatiotemporal variability is a challenging task. In this study, we report how data-driven machine learning can be used to perform observation bias correction for data assimilation through a real application, which is the dust emission inversion using PM10 observations. PM10 observations are considered unbiased; however, a bias correction is necessary if they are used as a proxy for dust during dust storms since they actually represent a sum of dust particles and non-dust aerosols. Two observation bias correction methods have been designed in order to use PM10 measurements as proxy for the dust storm loads under severe dust conditions. The first one is the conventional chemistry transport model (CTM) that simulates life cycles of non-dust aerosols. The other one is the machine-learning model that describes the relations between the regular PM10 and other air quality measurements. The latter is trained by learning using 2 years of historical samples. The machine-learning-based non-dust model is shown to be in better agreement with observations compared to the CTM. The dust emission inversion tests have been performed, through assimilating either the raw measurements or the bias-corrected dust observations using either the CTM or machine-learning model. The emission field, surface dust concentration, and forecast skill are evaluated. The worst case is when we directly assimilate the original observations. The forecasts driven by the a posteriori emission in this case even result in larger errors than the reference prediction. This shows the necessities of bias correction in data assimilation. The best results are obtained when using the machine-learning model for bias correction, with the existing measurements used more precisely and the resulting forecasts close to reality.

scholarly journals Learning with Previously Unseen Features

2017 ◽  
Author(s):  
Yuan Shi ◽  
Craig A. Knoblock
Keyword(s):  
Machine Learning ◽  
Joint Distribution ◽  
Learning Systems ◽  
Test Time ◽  
Model Parameters ◽  
Training Set ◽  
Open Environment ◽  
Novel Approach ◽  
Machine Learning Model ◽  
Classification Tasks

We study the problem of improving a machine learning model by identifying and using features that are not in the training set. This is applicable to machine learning systems deployed in an open environment. For example, a prediction model built on a set of sensors may be improved when it has access to new and relevant sensors at test time. To effectively use new features, we propose a novel approach that learns a model over both the original and new features, with the goal of making the joint distribution of features and predicted labels similar to that in the training set. Our approach can naturally leverage labels associated with these new features when they are accessible. We present an efficient optimization algorithm for learning the model parameters and empirically evaluate the approach on several regression and classification tasks. Experimental results show that our approach can achieve on average 11.2% improvement over baselines.

scholarly journals Personalised prediction of daily eczema severity scores using a mechanistic machine learning model

2020 ◽  
Author(s):  
Guillem Hurault ◽  
Elisa Domínguez-Hüttinger ◽  
Sinéad M. Langan ◽  
Hywel C. Williams ◽  
Reiko J. Tanaka
Keyword(s):  
Machine Learning ◽  
Predictive Model ◽  
Treatment Strategies ◽  
Learning Model ◽  
Patient Specific ◽  
Topical Steroids ◽  
Individual Level ◽  
Machine Learning Model ◽  
Severity Scores ◽  
Personalised Treatment

ABSTRACTBackgroundAtopic dermatitis (AD) is a chronic inflammatory skin disease with periods of flares and remission. Designing personalised treatment strategies for AD is challenging, given the apparent unpredictability and large variation in AD symptoms and treatment responses within and across individuals. Better prediction of AD severity over time for individual patients could help to select optimum timing and type of treatment for improving disease control.ObjectiveWe aimed to develop a mechanistic machine learning model that predicts the patient-specific evolution of AD severity scores on a daily basis.MethodsWe designed a probabilistic predictive model and trained it using Bayesian inference with the longitudinal data from two published clinical studies. The data consisted of daily recordings of AD severity scores and treatments used by 59 and 334 AD children over 6 months and 16 weeks, respectively. Internal and external validation of the predictive model was conducted in a forward-chaining setting.ResultsOur model was able to predict future severity scores at the individual level and improved chance-level forecast by 60%. Heterogeneous patterns in severity trajectories were captured with patient-specific parameters such as the short-term persistence of AD severity and responsiveness to topical steroids, calcineurin inhibitors and step-up treatment.ConclusionOur proof of principle model successfully predicted the daily evolution of AD severity scores at an individual level, and could inform the design of personalised treatment strategies that can be tested in future studies.

Sign in / Sign up

Export Citation Format

Share Document