scholarly journals Evaluating disease prediction models using a cohort whose covariate distribution differs from that of the target population

2017 ◽  
Vol 28 (1) ◽  
pp. 309-320 ◽  
Author(s):  
Scott Powers ◽  
Valerie McGuire ◽  
Leslie Bernstein ◽  
Alison J Canchola ◽  
Alice S Whittemore
Keyword(s):  
Prediction Models ◽  
Risk Model ◽  
Disease Incidence ◽  
Model Performance ◽  
Target Population ◽  
R Package ◽  
Model Assessment ◽  
Weighted Estimates ◽  
Cancer Prediction ◽  
Model Predictions

Personal predictive models for disease development play important roles in chronic disease prevention. The performance of these models is evaluated by applying them to the baseline covariates of participants in external cohort studies, with model predictions compared to subjects' subsequent disease incidence. However, the covariate distribution among participants in a validation cohort may differ from that of the population for which the model will be used. Since estimates of predictive model performance depend on the distribution of covariates among the subjects to which it is applied, such differences can cause misleading estimates of model performance in the target population. We propose a method for addressing this problem by weighting the cohort subjects to make their covariate distribution better match that of the target population. Simulations show that the method provides accurate estimates of model performance in the target population, while un-weighted estimates may not. We illustrate the method by applying it to evaluate an ovarian cancer prediction model targeted to US women, using cohort data from participants in the California Teachers Study. The methods can be implemented using open-source code for public use as the R-package RMAP (Risk Model Assessment Package) available at http://stanford.edu/~ggong/rmap/ .

Application of 17 Contrast-Induced Acute Kidney Injury Risk Prediction Models

2020 ◽  
Vol 10 (3) ◽  
pp. 162-174
Author(s):  
Levent Serif ◽  
George Chalikias ◽  
Matthaios Didagelos ◽  
Dimitrios Stakos ◽  
Petros Kikas ◽  
...  
Keyword(s):  
Acute Kidney Injury ◽  
Predictive Value ◽  
Injury Risk ◽  
Prediction Models ◽  
Risk Model ◽  
Model Performance ◽  
Kidney Injury ◽  
Risk Scores ◽  
True Negative ◽  
C Statistic

Introduction: Contrast-induced acute kidney injury (CI-AKI) is a frequent complication of percutaneous coronary interventions (PCI). Various groups have developed and validated risk scores for CI-AKI. Although the majority of these risk scores achieve an adequate accuracy, their usability in clinical practice is limited and greatly debated. Objective: With the present study, we aimed to prospectively assess the diagnostic performance of recently published CI-AKI risk scores (up to 2018) in a cohort of patients undergoing PCI. Methods: We enrolled 1,247 consecutive patients (80% men, mean age 62 ± 10 years) treated with elective or urgent PCI. For each patient, we calculated the individual CI-AKI risk score based on 17 different risk models. CI-AKI was defined as an increase of ≥25% (liberal) or ≥0.5 mg/dL (strict) in pre-PCI serum creatinine 48 h after PCI. Results: CI-AKI definition and, therefore, CI-AKI incidence have a significant impact on risk model performance (median negative predictive value increased from 85 to 99%; median c-statistic increased from 0.516 to 0.603 using more strict definition criteria). All of the 17 published models were characterized by a weak-to-moderate discriminating ability mainly based on the identification of “true-negative” cases (median positive predictive value 19% with liberal criterion and 3% with strict criterion). In none of the models, c-statistic was >0.800 with either CI-AKI definition. Novel, different combinations of the >35 independent variables used in the published models either by down- or by up-scaling did not result in significant improvement in predictive performance. Conclusions: The predictive ability of all models was similar and only modest, derived mainly by identifying true-negative cases. A new approach is probably needed by adding novel markers or periprocedural characteristics.

scholarly journals Prediction of COVID-19 severity using laboratory findings on admission: informative values, thresholds, ML model performance

BMJ Open ◽  
2021 ◽  
Vol 11 (2) ◽  
pp. e044500
Author(s):  
Yauhen Statsenko ◽  
Fatmah Al Zahmi ◽  
Tetiana Habuza ◽  
Klaus Neidl-Van Gorkom ◽  
Nazar Zaki
Keyword(s):  
Lymphocyte Count ◽  
Prediction Models ◽  
Model Performance ◽  
Area Under The Curve ◽  
Predictive Biomarkers ◽  
Supervised Machine Learning ◽  
Study Cohort ◽  
Threshold Values ◽  
Laboratory Findings ◽  
Study Results

BackgroundDespite the necessity, there is no reliable biomarker to predict disease severity and prognosis of patients with COVID-19. The currently published prediction models are not fully applicable to clinical use.ObjectivesTo identify predictive biomarkers of COVID-19 severity and to justify their threshold values for the stratification of the risk of deterioration that would require transferring to the intensive care unit (ICU).MethodsThe study cohort (560 subjects) included all consecutive patients admitted to Dubai Mediclinic Parkview Hospital from February to May 2020 with COVID-19 confirmed by the PCR. The challenge of finding the cut-off thresholds was the unbalanced dataset (eg, the disproportion in the number of 72 patients admitted to ICU vs 488 non-severe cases). Therefore, we customised supervised machine learning (ML) algorithm in terms of threshold value used to predict worsening.ResultsWith the default thresholds returned by the ML estimator, the performance of the models was low. It was improved by setting the cut-off level to the 25th percentile for lymphocyte count and the 75th percentile for other features. The study justified the following threshold values of the laboratory tests done on admission: lymphocyte count <2.59×109/L, and the upper levels for total bilirubin 11.9 μmol/L, alanine aminotransferase 43 U/L, aspartate aminotransferase 32 U/L, D-dimer 0.7 mg/L, activated partial thromboplastin time (aPTT) 39.9 s, creatine kinase 247 U/L, C reactive protein (CRP) 14.3 mg/L, lactate dehydrogenase 246 U/L, troponin 0.037 ng/mL, ferritin 498 ng/mL and fibrinogen 446 mg/dL.ConclusionThe performance of the neural network trained with top valuable tests (aPTT, CRP and fibrinogen) is admissible (area under the curve (AUC) 0.86; 95% CI 0.486 to 0.884; p<0.001) and comparable with the model trained with all the tests (AUC 0.90; 95% CI 0.812 to 0.902; p<0.001). Free online tool at https://med-predict.com illustrates the study results.

scholarly journals Enabling deeper learning on big data for materials informatics applications

2021 ◽  
Vol 11 (1) ◽  
Author(s):  
Dipendra Jha ◽  
Vishu Gupta ◽  
Logan Ward ◽  
Zijiang Yang ◽  
Christopher Wolverton ◽  
...  
Keyword(s):  
Neural Networks ◽  
Big Data ◽  
Deep Learning ◽  
Deep Neural Networks ◽  
Materials Science ◽  
Prediction Models ◽  
Model Performance ◽  
Materials Informatics ◽  
Learning Framework ◽  
Significant Attention

AbstractThe application of machine learning (ML) techniques in materials science has attracted significant attention in recent years, due to their impressive ability to efficiently extract data-driven linkages from various input materials representations to their output properties. While the application of traditional ML techniques has become quite ubiquitous, there have been limited applications of more advanced deep learning (DL) techniques, primarily because big materials datasets are relatively rare. Given the demonstrated potential and advantages of DL and the increasing availability of big materials datasets, it is attractive to go for deeper neural networks in a bid to boost model performance, but in reality, it leads to performance degradation due to the vanishing gradient problem. In this paper, we address the question of how to enable deeper learning for cases where big materials data is available. Here, we present a general deep learning framework based on Individual Residual learning (IRNet) composed of very deep neural networks that can work with any vector-based materials representation as input to build accurate property prediction models. We find that the proposed IRNet models can not only successfully alleviate the vanishing gradient problem and enable deeper learning, but also lead to significantly (up to 47%) better model accuracy as compared to plain deep neural networks and traditional ML techniques for a given input materials representation in the presence of big data.

scholarly journals Modelling Exposure by Spraying Activities—Status and Future Needs

2021 ◽  
Vol 18 (15) ◽  
pp. 7737
Author(s):  
Stefan Hahn ◽  
Jessica Meyer ◽  
Michael Roitzsch ◽  
Christiaan Delmaar ◽  
Wolfgang Koch ◽  
...  
Keyword(s):  
Current Knowledge ◽  
Model Performance ◽  
Relevant Information ◽  
Systematic Evaluation ◽  
Model Accuracy ◽  
Efficient Manner ◽  
User Perspective ◽  
Model Predictions ◽  
Spray Applications

Spray applications enable a uniform distribution of substances on surfaces in a highly efficient manner, and thus can be found at workplaces as well as in consumer environments. A systematic literature review on modelling exposure by spraying activities has been conducted and status and further needs have been discussed with experts at a symposium. This review summarizes the current knowledge about models and their level of conservatism and accuracy. We found that extraction of relevant information on model performance for spraying from published studies and interpretation of model accuracy proved to be challenging, as the studies often accounted for only a small part of potential spray applications. To achieve a better quality of exposure estimates in the future, more systematic evaluation of models is beneficial, taking into account a representative variety of spray equipment and application patterns. Model predictions could be improved by more accurate consideration of variation in spray equipment. Inter-model harmonization with regard to spray input parameters and appropriate grouping of spray exposure situations is recommended. From a user perspective, a platform or database with information on different spraying equipment and techniques and agreed standard parameters for specific spraying scenarios from different regulations may be useful.

scholarly journals Vaccine refusal and the endgame: walking the last mile first

2013 ◽  
Vol 368 (1623) ◽  
pp. 20120148 ◽  
Author(s):  
Diane S. Saint-Victor ◽  
Saad B. Omer
Keyword(s):  
Disease Control ◽  
Disease Incidence ◽  
Target Population ◽  
Special Issue ◽  
Disease Eradication ◽  
Vaccine Refusal ◽  
Last Mile ◽  
Low Levels

As multiple papers within this special issue illustrate, the dynamics of disease eradication are different from disease control. When it comes to disease eradication, ‘the last mile is longest’. For social and ecological reasons such as vaccine refusal, further ending incidence of a disease when it has reached low levels is frequently complex. Issues of non-compliance within a target population often influence the outcome of disease eradication efforts. Past eradication efforts confronted such obstacles towards the tail end of the campaign, when disease incidence was lowest. This article provides a comparison of non-compliance within polio, measles and smallpox campaigns, demonstrating the tendency of vaccine refusal to rise as disease incidence falls. In order to overcome one of the most intractable challenges to eradication, future disease eradication efforts must prioritize vaccine refusal from the start, i.e. ‘walk the last mile first’.

Abstract 003: Personalizing the Intensity of Blood Pressure Control: Modeling the Heterogeneity of Risks and Benefits From the SPRINT Trial

2017 ◽  
Vol 10 (suppl_3) ◽  
Author(s):  
Krishna K Patel ◽  
Suzanne V Arnold ◽  
Paul S Chan ◽  
Yuanyuan Tang ◽  
Yashashwi Pokharel ◽  
...  
Keyword(s):  
Blood Pressure ◽  
Cardiovascular Risk ◽  
Prediction Models ◽  
Risk Model ◽  
Treatment Strategies ◽  
Intensive Treatment ◽  
Major Adverse Cardiovascular Events ◽  
Diabetic Patients ◽  
High Cardiovascular Risk ◽  
Serious Adverse Events

Introduction: In SPRINT (Systolic blood PRessure INtervention Trial), non-diabetic patients with hypertension at high cardiovascular risk treated with intensive blood pressure (BP) control (<120mmHg) had fewer major adverse cardiovascular events (MACE) and all-cause deaths but higher rates of serious adverse events (SAE) compared with patients treated with standard BP control (<140mmHg). However, the degree of benefit or harm for an individual patient could vary due to heterogeneity in treatment effect. Methods: Using patient-level data from SPRINT, we developed predictive models for benefit (freedom from death or MACE) and harm (increased SAE) to allow for individualized BP treatment goals based on projected risk-benefit for each patient. Interactions between candidate variable and treatment were evaluated in the models to identify differential treatment effects. We performed 10 fold cross-validation for both the models. Results: Among 9361 patients, 8606 (92%) patients had no MACE or death event (benefit) and 3529 (38%) patients had a SAE (harm) over a median follow-up of 3.3 years. The benefit model showed good discrimination (c-index= 0.72; cross-validated c-index= 0.72) with treatment interactions of age, sex, and baseline systolic BP (Figure A), with more benefit of intensive BP treatment in patients who are older, male, and have lower baseline SBP. The SAE risk model showed moderate discrimination (c-index=0.66; cross-validated c-index= 0.65) with a treatment interaction of baseline renal function (Figure B), indicating less harm of intensive treatment in patients with a higher baseline creatinine. The mean predicted absolute benefit of intensive BP treatment was of 2.2% ± 2.5% compared with standard treatment, but ranged from 10.7% lower benefit to 17% greater benefit in individual patients. Similarly, mean predicted absolute harm with intensive treatment was 1.0% ± 1.9%, but ranged from 15.9% lesser harm to 4.9% more harm. Conclusion: Among non-diabetic patients with hypertension at high cardiovascular risk, we developed prediction models using basic clinical data that can identify patients with higher likelihood of benefit vs. harm with BP treatment strategies. These models could be used to tailor the treatment approach based on the projected risk and benefit for each unique patient.

Forecasting the risk at infractions: an ensemble comparison of machine learning approach

2021 ◽  
Vol ahead-of-print (ahead-of-print) ◽  
Author(s):  
Lei Li ◽  
Desheng Wu
Keyword(s):  
Machine Learning ◽  
Prediction Models ◽  
Short Term Memory ◽  
Model Performance ◽  
Large Data ◽  
Support Vector ◽  
Learning Approaches ◽  
Content Type ◽  
Day To Day Operations ◽  
Prediction Approach

PurposeThe infraction of securities regulations (ISRs) of listed firms in their day-to-day operations and management has become one of common problems. This paper proposed several machine learning approaches to forecast the risk at infractions of listed corporates to solve financial problems that are not effective and precise in supervision.Design/methodology/approachThe overall proposed research framework designed for forecasting the infractions (ISRs) include data collection and cleaning, feature engineering, data split, prediction approach application and model performance evaluation. We select Logistic Regression, Naïve Bayes, Random Forest, Support Vector Machines, Artificial Neural Network and Long Short-Term Memory Networks (LSTMs) as ISRs prediction models.FindingsThe research results show that prediction performance of proposed models with the prior infractions provides a significant improvement of the ISRs than those without prior, especially for large sample set. The results also indicate when judging whether a company has infractions, we should pay attention to novel artificial intelligence methods, previous infractions of the company, and large data sets.Originality/valueThe findings could be utilized to address the problems of identifying listed corporates' ISRs at hand to a certain degree. Overall, results elucidate the value of the prior infraction of securities regulations (ISRs). This shows the importance of including more data sources when constructing distress models and not only focus on building increasingly more complex models on the same data. This is also beneficial to the regulatory authorities.

Cardiovascular risk model performance in women with and without hypertensive disorders of pregnancy

Heart ◽  
2018 ◽  
Vol 105 (4) ◽  
pp. 330-336 ◽  
Author(s):  
Veerle Dam ◽  
N Charlotte Onland-Moret ◽  
W M Monique Verschuren ◽  
Jolanda M A Boer ◽  
Laura Benschop ◽  
...  
Keyword(s):  
Risk Model ◽  
Model Performance ◽  
Predictive Performance ◽  
Baseline Risk ◽  
Hypertensive Disorders Of Pregnancy ◽  
Cvd Risk ◽  
Hypertensive Disorders ◽  
Framingham Risk ◽  
History Of ◽  
Score Model

ObjectivesCompare the predictive performance of Framingham Risk Score (FRS), Pooled Cohort Equations (PCEs) and Systematic COronary Risk Evaluation (SCORE) model between women with and without a history of hypertensive disorders of pregnancy (hHDP) and determine the effects of recalibration and refitting on predictive performance.MethodsWe included 29 751 women, 6302 with hHDP and 17 369 without. We assessed whether models accurately predicted observed 10-year cardiovascular disease (CVD) risk (calibration) and whether they accurately distinguished between women developing CVD during follow-up and not (discrimination), separately for women with and without hHDP. We also recalibrated (updating intercept and slope) and refitted (recalculating coefficients) the models.ResultsOriginal FRS and PCEs overpredicted 10-year CVD risks, with expected:observed (E:O) ratios ranging from 1.51 (for FRS in women with hHDP) to 2.29 (for PCEs in women without hHDP), while E:O ratios were close to 1 for SCORE. Overprediction attenuated slightly after recalibration for FRS and PCEs in both hHDP groups. Discrimination was reasonable for all models, with C-statistics ranging from 0.70-0.81 (women with hHDP) and 0.72–0.74 (women without hHDP). C-statistics improved slightly after refitting 0.71–0.83 (with hHDP) and 0.73–0.80 (without hHDP). The E:O ratio of the original PCE model was statistically significantly better in women with hHDP compared with women without hHDP.ConclusionsSCORE performed best in terms of both calibration and discrimination, while FRS and PCEs overpredicted risk in women with and without hHDP, but improved after recalibrating and refitting the models. No separate model for women with hHDP seems necessary, despite their higher baseline risk.

Assessment of risk model performance

Absolute Risk ◽  
2017 ◽  
pp. 75-100
Author(s):  
Ruth M. Pfeiffer ◽  
Mitchell H. Gail
Keyword(s):  
Risk Model ◽  
Model Performance

scholarly journals Feasibility and Evaluation of a Large-Scale External Validation Approach for Patient-Level Prediction in an International Data Network: Validation of models predicting stroke in female patients newly diagnosed with atrial fibrillation.

2020 ◽  
Author(s):  
Jenna Marie Reps ◽  
Ross Williams ◽  
Seng Chan You ◽  
Thomas Falconer ◽  
Evan Minty ◽  
...  
Keyword(s):  
Atrial Fibrillation ◽  
Large Scale ◽  
Data Science ◽  
Prediction Models ◽  
External Validation ◽  
Scale Up ◽  
R Package ◽  
Prognostic Models ◽  
Healthcare Data ◽  
Patient Level

Abstract Objective: To demonstrate how the Observational Healthcare Data Science and Informatics (OHDSI) collaborative network and standardization can be utilized to scale-up external validation of patient-level prediction models by enabling validation across a large number of heterogeneous observational healthcare datasets.Materials & Methods: Five previously published prognostic models (ATRIA, CHADS2, CHADS2VASC, Q-Stroke and Framingham) that predict future risk of stroke in patients with atrial fibrillation were replicated using the OHDSI frameworks. A network study was run that enabled the five models to be externally validated across nine observational healthcare datasets spanning three countries and five independent sites. Results: The five existing models were able to be integrated into the OHDSI framework for patient-level prediction and they obtained mean c-statistics ranging between 0.57-0.63 across the 6 databases with sufficient data to predict stroke within 1 year of initial atrial fibrillation diagnosis for females with atrial fibrillation. This was comparable with existing validation studies. The validation network study was run across nine datasets within 60 days once the models were replicated. An R package for the study was published at https://github.com/OHDSI/StudyProtocolSandbox/tree/master/ExistingStrokeRiskExternalValidation.Discussion: This study demonstrates the ability to scale up external validation of patient-level prediction models using a collaboration of researchers and a data standardization that enable models to be readily shared across data sites. External validation is necessary to understand the transportability or reproducibility of a prediction model, but without collaborative approaches it can take three or more years for a model to be validated by one independent researcher. Conclusion : In this paper we show it is possible to both scale-up and speed-up external validation by showing how validation can be done across multiple databases in less than 2 months. We recommend that researchers developing new prediction models use the OHDSI network to externally validate their models.

Sign in / Sign up

Export Citation Format

Share Document