scholarly journals Review and evaluation of performance measures for survival prediction models in external validation settings

2017 ◽  
Vol 17 (1) ◽  
Author(s):  
M. Shafiqur Rahman ◽  
Gareth Ambler ◽  
Babak Choodari-Oskooei ◽  
Rumana Z. Omar
Keyword(s):  
Performance Measures ◽  
Prediction Models ◽  
External Validation ◽  
Survival Prediction ◽  
Evaluation Of Performance

scholarly journals Bi-Centric Independent Validation of Outcome Prediction after Radioembolization of Primary and Secondary Liver Cancer

2021 ◽  
Vol 10 (16) ◽  
pp. 3668
Author(s):  
Matthias Philipp Fabritius ◽  
Max Seidensticker ◽  
Johannes Rueckel ◽  
Constanze Heinze ◽  
Maciej Pech ◽  
...  
Keyword(s):  
Patient Selection ◽  
Outcome Prediction ◽  
Prediction Models ◽  
External Validation ◽  
Cox Proportional Hazards ◽  
Brier Score ◽  
Survival Prediction ◽  
Liver Malignancies ◽  
Prognostic Performance ◽  
Hepatic Reserve

Background: Yttrium-90 radioembolization (RE) plays an important role in the treatment of liver malignancies. Optimal patient selection is crucial for an effective and safe treatment. In this study, we aim to validate the prognostic performance of a previously established random survival forest (RSF) with an external validation cohort from a different national center. Furthermore, we compare outcome prediction models with different established metrics. Methods: A previously established RSF model, trained on a consecutive cohort of 366 patients who had received RE due to primary or secondary liver tumor at a national center (center 1), was used to predict the outcome of an independent consecutive cohort of 202 patients from a different national center (center 2) and vice versa. Prognostic performance was evaluated using the concordance index (C-index) and the integrated Brier score (IBS). The prognostic importance of designated baseline parameters was measured with the minimal depth concept, and the influence on the predicted outcome was analyzed with accumulated local effects plots. RSF values were compared to conventional cox proportional hazards models in terms of C-index and IBS. Results: The established RSF model achieved a C-index of 0.67 for center 2, comparable to the results obtained for center 1, which it was trained on (0.66). The RSF model trained on center 2 achieved a C-index of 0.68 on center 2 data and 0.66 on center 1 data. CPH models showed comparable results on both cohorts, with C-index ranging from 0.68 to 0.72. IBS validation showed more differentiated results depending on which cohort was trained on and which cohort was predicted (range: 0.08 to 0.20). Baseline cholinesterase was the most important variable for survival prediction. Conclusion: The previously developed predictive RSF model was successfully validated with an independent external cohort. C-index and IBS are suitable metrics to compare outcome prediction models, with IBS showing more differentiated results. The findings corroborate that survival after RE is critically determined by functional hepatic reserve and thus baseline liver function should play a key role in patient selection.

scholarly journals Genetic co-expression networks contribute to creating predictive model and exploring novel biomarkers for the prognosis of breast cancer

2021 ◽  
Vol 11 (1) ◽  
Author(s):  
Chi-Ming Chu ◽  
Huan-Ming Hsu ◽  
Chi-Wen Chang ◽  
Yuan-Kuei Li ◽  
Yu-Jia Chang ◽  
...  
Keyword(s):  
Breast Cancer ◽  
Prediction Models ◽  
Reference Model ◽  
External Validation ◽  
Risk Scores ◽  
Consistent Finding ◽  
Hazard Ratios ◽  
Genetic Predictors ◽  
Novel Biomarkers ◽  
Cox Proportional Hazard Regression

AbstractGenetic co-expression network (GCN) analysis augments the understanding of breast cancer (BC). We aimed to propose GCN-based modeling for BC relapse-free survival (RFS) prediction and to discover novel biomarkers. We used GCN and Cox proportional hazard regression to create various prediction models using mRNA microarray of 920 tumors and conduct external validation using independent data of 1056 tumors. GCNs of 34 identified candidate genes were plotted in various sizes. Compared to the reference model, the genetic predictors selected from bigger GCNs composed better prediction models. The prediction accuracy and AUC of 3 ~ 15-year RFS are 71.0–81.4% and 74.6–78% respectively (rfm, ACC 63.2–65.5%, AUC 61.9–74.9%). The hazard ratios of risk scores of developing relapse ranged from 1.89 ~ 3.32 (p < 10–8) over all models under the control of the node status. External validation showed the consistent finding. We found top 12 co-expressed genes are relative new or novel biomarkers that have not been explored in BC prognosis or other cancers until this decade. GCN-based modeling creates better prediction models and facilitates novel genes exploration on BC prognosis.

scholarly journals Comparing the characteristics and predicting the survival of patients with head and neck melanoma versus body melanoma: a population-based study

BMC Cancer ◽  
2021 ◽  
Vol 21 (1) ◽  
Author(s):  
Yuxin Ding ◽  
Runyi Jiang ◽  
Yuhong Chen ◽  
Jing Jing ◽  
Xiaoshuang Yang ◽  
...  
Keyword(s):  
Head And Neck ◽  
Health Management ◽  
External Validation ◽  
Area Under The Curve ◽  
Population Based ◽  
Survival Prediction ◽  
Cancer Specific Survival ◽  
Cox Models ◽  
Lentigo Maligna Melanoma ◽  
Clinical Tools

Abstract Background Previous studies reported cutaneous melanoma in head and neck (HNM) differed from those in other regions (body melanoma, BM). Individualized tools to predict the survival of patients with HNM or BM remain insufficient. We aimed at comparing the characteristics of HNM and BM, developing and validating nomograms for predicting the survival of patients with HNM or BM. Methods The information of patients with HNM or BM from 2004 to 2015 was obtained from the Surveillance, Epidemiology, and End Results (SEER) database. The HNM group and BM group were randomly divided into training and validation cohorts. We used the Kaplan-Meier method and multivariate Cox models to identify independent prognostic factors. Nomograms were developed via the rms and dynnom packages, and were measured by the concordance index (C-index), the area under the curve (AUC) of the receiver operating characteristic (ROC) curve and calibration plots. Results Of 70,605 patients acquired, 21% had HNM and 79% had BM. The HNM group contained more older patients, male sex and lentigo maligna melanoma, and more frequently had thicker tumors and metastases than the BM group. The 5-year cancer-specific survival (CSS) and overall survival (OS) rates were 88.1 ± 0.3% and 74.4 ± 0.4% in the HNM group and 92.5 ± 0.1% and 85.8 ± 0.2% in the BM group, respectively. Eight variables (age, sex, histology, thickness, ulceration, stage, metastases, and surgery) were identified to construct nomograms of CSS and OS for patients with HNM or BM. Additionally, four dynamic nomograms were available on web. The internal and external validation of each nomogram showed high C-index values (0.785–0.896) and AUC values (0.81–0.925), and the calibration plots showed great consistency. Conclusions The characteristics of HNM and BM are heterogeneous. We constructed and validated four nomograms for predicting the 3-, 5- and 10-year CSS and OS probabilities of patients with HNM or BM. These nomograms can serve as practical clinical tools for survival prediction and individual health management.

scholarly journals Predicting hospital admissions from individual patient data (IPD): an applied example to explore key elements driving external validity

BMJ Open ◽  
2021 ◽  
Vol 11 (8) ◽  
pp. e045572
Author(s):  
Andreas Daniel Meid ◽  
Ana Isabel Gonzalez-Gonzalez ◽  
Truc Sophia Dinh ◽  
Jeanet Blom ◽  
Marjan van den Akker ◽  
...  
Keyword(s):  
Prognostic Model ◽  
Prediction Models ◽  
Hospital Admissions ◽  
Absolute Risk ◽  
External Validation ◽  
Baseline Risk ◽  
Study Heterogeneity ◽  
Related Quality ◽  
Individual Participant ◽  
Risk Predictor

ObjectiveTo explore factors that potentially impact external validation performance while developing and validating a prognostic model for hospital admissions (HAs) in complex older general practice patients.Study design and settingUsing individual participant data from four cluster-randomised trials conducted in the Netherlands and Germany, we used logistic regression to develop a prognostic model to predict all-cause HAs within a 6-month follow-up period. A stratified intercept was used to account for heterogeneity in baseline risk between the studies. The model was validated both internally and by using internal-external cross-validation (IECV).ResultsPrior HAs, physical components of the health-related quality of life comorbidity index, and medication-related variables were used in the final model. While achieving moderate discriminatory performance, internal bootstrap validation revealed a pronounced risk of overfitting. The results of the IECV, in which calibration was highly variable even after accounting for between-study heterogeneity, agreed with this finding. Heterogeneity was equally reflected in differing baseline risk, predictor effects and absolute risk predictions.ConclusionsPredictor effect heterogeneity and differing baseline risk can explain the limited external performance of HA prediction models. With such drivers known, model adjustments in external validation settings (eg, intercept recalibration, complete updating) can be applied more purposefully.Trial registration numberPROSPERO id: CRD42018088129.

scholarly journals Development and validation of prediction models for mechanical ventilation, renal replacement therapy, and readmission in COVID-19 patients

2021 ◽  
Author(s):  
Victor Alfonso Rodriguez ◽  
Shreyas Bhave ◽  
Ruijun Chen ◽  
Chao Pang ◽  
George Hripcsak ◽  
...  
Keyword(s):  
At Risk ◽  
Mechanical Ventilation ◽  
Renal Replacement Therapy ◽  
Receiver Operating Characteristic ◽  
Replacement Therapy ◽  
Operating Characteristic ◽  
Prediction Models ◽  
External Validation ◽  
Electronic Health Record Data ◽  
Receiver Operating

Abstract Objective Coronavirus disease 2019 (COVID-19) patients are at risk for resource-intensive outcomes including mechanical ventilation (MV), renal replacement therapy (RRT), and readmission. Accurate outcome prognostication could facilitate hospital resource allocation. We develop and validate predictive models for each outcome using retrospective electronic health record data for COVID-19 patients treated between March 2 and May 6, 2020. Materials and Methods For each outcome, we trained 3 classes of prediction models using clinical data for a cohort of SARS-CoV-2 (severe acute respiratory syndrome coronavirus 2)–positive patients (n = 2256). Cross-validation was used to select the best-performing models per the areas under the receiver-operating characteristic and precision-recall curves. Models were validated using a held-out cohort (n = 855). We measured each model’s calibration and evaluated feature importances to interpret model output. Results The predictive performance for our selected models on the held-out cohort was as follows: area under the receiver-operating characteristic curve—MV 0.743 (95% CI, 0.682-0.812), RRT 0.847 (95% CI, 0.772-0.936), readmission 0.871 (95% CI, 0.830-0.917); area under the precision-recall curve—MV 0.137 (95% CI, 0.047-0.175), RRT 0.325 (95% CI, 0.117-0.497), readmission 0.504 (95% CI, 0.388-0.604). Predictions were well calibrated, and the most important features within each model were consistent with clinical intuition. Discussion Our models produce performant, well-calibrated, and interpretable predictions for COVID-19 patients at risk for the target outcomes. They demonstrate the potential to accurately estimate outcome prognosis in resource-constrained care sites managing COVID-19 patients. Conclusions We develop and validate prognostic models targeting MV, RRT, and readmission for hospitalized COVID-19 patients which produce accurate, interpretable predictions. Additional external validation studies are needed to further verify the generalizability of our results.

scholarly journals Insights into the molecular properties underlying antibacterial activity of prenylated (iso)flavonoids against MRSA

2021 ◽  
Vol 11 (1) ◽  
Author(s):  
Sylvia Kalli ◽  
Carla Araya-Cloutier ◽  
Jos Hageman ◽  
Jean-Paul Vincken
Keyword(s):  
Prediction Models ◽  
External Validation ◽  
Qsar Model ◽  
Prediction Errors ◽  
Activity Data ◽  
Gram Positive Bacteria ◽  
Level Of Activity ◽  
Formal Charge ◽  
Validation Set

AbstractHigh resistance towards traditional antibiotics has urged the development of new, natural therapeutics against methicillin-resistant Staphylococcus aureus (MRSA). Prenylated (iso)flavonoids, present mainly in the Fabaceae, can serve as promising candidates. Herein, the anti-MRSA properties of 23 prenylated (iso)flavonoids were assessed in-vitro. The di-prenylated (iso)flavonoids, glabrol (flavanone) and 6,8-diprenyl genistein (isoflavone), together with the mono-prenylated, 4′-O-methyl glabridin (isoflavan), were the most active anti-MRSA compounds (Minimum Inhibitory Concentrations (MIC) ≤ 10 µg/mL, 30 µM). The in-house activity data was complemented with literature data to yield an extended, curated dataset of 67 molecules for the development of robust in-silico prediction models. A QSAR model having a good fit (R2adj 0.61), low average prediction errors and a good predictive power (Q2) for the training (4% and Q2LOO 0.57, respectively) and the test set (5% and Q2test 0.75, respectively) was obtained. Furthermore, the model predicted well the activity of an external validation set (on average 5% prediction errors), as well as the level of activity (low, moderate, high) of prenylated (iso)flavonoids against other Gram-positive bacteria. For the first time, the importance of formal charge, besides hydrophobic volume and hydrogen-bonding, in the anti-MRSA activity was highlighted, thereby suggesting potentially different modes of action of the different prenylated (iso)flavonoids.

scholarly journals Models for predicting treatment efficacy of antiepileptic drugs and prognosis of treatment withdrawal in epilepsy patients

2021 ◽  
Vol 3 (1) ◽  
Author(s):  
Shijun Yang ◽  
Bin Wang ◽  
Xiong Han
Keyword(s):  
Antiepileptic Drugs ◽  
Statistical Models ◽  
Prediction Models ◽  
External Validation ◽  
Machine Learning Algorithms ◽  
Treatment Withdrawal ◽  
Patient Treatment ◽  
Recent Developments ◽  
Progression Of Disease ◽  
Patients With Epilepsy

AbstractAlthough antiepileptic drugs (AEDs) are the most effective treatment for epilepsy, 30–40% of patients with epilepsy would develop drug-refractory epilepsy. An accurate, preliminary prediction of the efficacy of AEDs has great clinical significance for patient treatment and prognosis. Some studies have developed statistical models and machine-learning algorithms (MLAs) to predict the efficacy of AEDs treatment and the progression of disease after treatment withdrawal, in order to provide assistance for making clinical decisions in the aim of precise, personalized treatment. The field of prediction models with statistical models and MLAs is attracting growing interest and is developing rapidly. What’s more, more and more studies focus on the external validation of the existing model. In this review, we will give a brief overview of recent developments in this discipline.

Sign in / Sign up

Export Citation Format

Share Document