scholarly journals External validation of six existing multivariable clinical prediction models for short-term mortality in patients undergoing lung resection

2020 ◽  
Author(s):  
Marcus Taylor ◽  
Bartłomiej Szafron ◽  
Glen P Martin ◽  
Udo Abah ◽  
Matthew Smith ◽  
...  
Keyword(s):  
Receiver Operating Characteristic ◽  
Operating Characteristic ◽  
Prediction Models ◽  
Characteristic Curve ◽  
Lung Resection ◽  
External Validation ◽  
Clinical Prediction ◽  
National Guidelines ◽  
Clinical Prediction Models ◽  
Receiver Operating

Abstract OBJECTIVES National guidelines advocate the use of clinical prediction models to estimate perioperative mortality for patients undergoing lung resection. Several models have been developed that may potentially be useful but contemporary external validation studies are lacking. The aim of this study was to validate existing models in a multicentre patient cohort. METHODS The Thoracoscore, Modified Thoracoscore, Eurolung, Modified Eurolung, European Society Objective Score and Brunelli models were validated using a database of 6600 patients who underwent lung resection between 2012 and 2018. Models were validated for in-hospital or 30-day mortality (depending on intended outcome of each model) and also for 90-day mortality. Model calibration (calibration intercept, calibration slope, observed to expected ratio and calibration plots) and discrimination (area under receiver operating characteristic curve) were assessed as measures of model performance. RESULTS Mean age was 66.8 years (±10.9 years) and 49.7% (n = 3281) of patients were male. In-hospital, 30-day, perioperative (in-hospital or 30-day) and 90-day mortality were 1.5% (n = 99), 1.4% (n = 93), 1.8% (n = 121) and 3.1% (n = 204), respectively. Model area under the receiver operating characteristic curves ranged from 0.67 to 0.73. Calibration was inadequate in five models and mortality was significantly overestimated in five models. No model was able to adequately predict 90-day mortality. CONCLUSIONS Five of the validated models were poorly calibrated and had inadequate discriminatory ability. The modified Eurolung model demonstrated adequate statistical performance but lacked clinical validity. Development of accurate models that can be used to estimate the contemporary risk of lung resection is required.

scholarly journals External Validations of Cardiovascular Clinical Prediction Models: A Large-Scale Review of the Literature

2021 ◽  
Author(s):  
Benjamin S. Wessler ◽  
Jason Nelson ◽  
Jinny G. Park ◽  
Hannah McGinnes ◽  
Gaurav Gulati ◽  
...  
Keyword(s):  
Receiver Operating Characteristic Curve ◽  
Receiver Operating Characteristic ◽  
Operating Characteristic ◽  
Prediction Models ◽  
Characteristic Curve ◽  
External Validation ◽  
Clinical Prediction ◽  
Clinical Prediction Models ◽  
Operating Characteristic Curve ◽  
Performance Heterogeneity

Background: There are many clinical prediction models (CPMs) available to inform treatment decisions for patients with cardiovascular disease. However, the extent to which they have been externally tested, and how well they generally perform has not been broadly evaluated. Methods: A SCOPUS citation search was run on March 22, 2017 to identify external validations of cardiovascular CPMs in the Tufts Predictive Analytics and Comparative Effectiveness CPM Registry. We assessed the extent of external validation, performance heterogeneity across databases, and explored factors associated with model performance, including a global assessment of the clinical relatedness between the derivation and validation data. Results: We identified 2030 external validations of 1382 CPMs. Eight hundred seven (58%) of the CPMs in the Registry have never been externally validated. On average, there were 1.5 validations per CPM (range, 0–94). The median external validation area under the receiver operating characteristic curve was 0.73 (25th–75th percentile [interquartile range (IQR)], 0.66–0.79), representing a median percent decrease in discrimination of −11.1% (IQR, −32.4% to +2.7%) compared with performance on derivation data. 81% (n=1333) of validations reporting area under the receiver operating characteristic curve showed discrimination below that reported in the derivation dataset. 53% (n=983) of the validations report some measure of CPM calibration. For CPMs evaluated more than once, there was typically a large range of performance. Of 1702 validations classified by relatedness, the percent change in discrimination was −3.7% (IQR, −13.2 to 3.1) for closely related validations (n=123), −9.0 (IQR, −27.6 to 3.9) for related validations (n=862), and −17.2% (IQR, −42.3 to 0) for distantly related validations (n=717; P <0.001). Conclusions: Many published cardiovascular CPMs have never been externally validated, and for those that have, apparent performance during development is often overly optimistic. A single external validation appears insufficient to broadly understand the performance heterogeneity across different settings.

scholarly journals Development and validation of prediction models for mechanical ventilation, renal replacement therapy, and readmission in COVID-19 patients

2021 ◽  
Author(s):  
Victor Alfonso Rodriguez ◽  
Shreyas Bhave ◽  
Ruijun Chen ◽  
Chao Pang ◽  
George Hripcsak ◽  
...  
Keyword(s):  
At Risk ◽  
Mechanical Ventilation ◽  
Renal Replacement Therapy ◽  
Receiver Operating Characteristic ◽  
Replacement Therapy ◽  
Operating Characteristic ◽  
Prediction Models ◽  
External Validation ◽  
Electronic Health Record Data ◽  
Receiver Operating

Abstract Objective Coronavirus disease 2019 (COVID-19) patients are at risk for resource-intensive outcomes including mechanical ventilation (MV), renal replacement therapy (RRT), and readmission. Accurate outcome prognostication could facilitate hospital resource allocation. We develop and validate predictive models for each outcome using retrospective electronic health record data for COVID-19 patients treated between March 2 and May 6, 2020. Materials and Methods For each outcome, we trained 3 classes of prediction models using clinical data for a cohort of SARS-CoV-2 (severe acute respiratory syndrome coronavirus 2)–positive patients (n = 2256). Cross-validation was used to select the best-performing models per the areas under the receiver-operating characteristic and precision-recall curves. Models were validated using a held-out cohort (n = 855). We measured each model’s calibration and evaluated feature importances to interpret model output. Results The predictive performance for our selected models on the held-out cohort was as follows: area under the receiver-operating characteristic curve—MV 0.743 (95% CI, 0.682-0.812), RRT 0.847 (95% CI, 0.772-0.936), readmission 0.871 (95% CI, 0.830-0.917); area under the precision-recall curve—MV 0.137 (95% CI, 0.047-0.175), RRT 0.325 (95% CI, 0.117-0.497), readmission 0.504 (95% CI, 0.388-0.604). Predictions were well calibrated, and the most important features within each model were consistent with clinical intuition. Discussion Our models produce performant, well-calibrated, and interpretable predictions for COVID-19 patients at risk for the target outcomes. They demonstrate the potential to accurately estimate outcome prognosis in resource-constrained care sites managing COVID-19 patients. Conclusions We develop and validate prognostic models targeting MV, RRT, and readmission for hospitalized COVID-19 patients which produce accurate, interpretable predictions. Additional external validation studies are needed to further verify the generalizability of our results.

scholarly journals External Validation of Five Scores to Predict Stroke-Associated Pneumonia and the Role of Selected Blood Biomarkers

Stroke ◽  
2021 ◽  
Vol 52 (1) ◽  
pp. 325-330
Author(s):  
Benjamin Hotter ◽  
Sarah Hoffmann ◽  
Lena Ulm ◽  
Christian Meisel ◽  
Alejandro Bustamante ◽  
...  
Keyword(s):  
Receiver Operating Characteristic Curve ◽  
Receiver Operating Characteristic ◽  
Literature Search ◽  
Operating Characteristic ◽  
Prediction Models ◽  
Characteristic Curve ◽  
Data Set ◽  
Biomarker Analysis ◽  
Operating Characteristic Curve ◽  
Receiver Operating

Background and Purpose: Several clinical scoring systems as well as biomarkers have been proposed to predict stroke-associated pneumonia (SAP). We aimed to externally and competitively validate SAP scores and hypothesized that 5 selected biomarkers would improve performance of these scores. Methods: We pooled the clinical data of 2 acute stroke studies with identical data assessment: STRAWINSKI and PREDICT. Biomarkers (ultrasensitive procalcitonin; mid-regional proadrenomedullin; mid-regional proatrionatriuretic peptide; ultrasensitive copeptin; C-terminal proendothelin) were measured from hospital admission serum samples. A literature search was performed to identify SAP prediction scores. We then calculated multivariate regression models with the individual scores and the biomarkers. Areas under receiver operating characteristic curves were used to compare discrimination of these scores and models. Results: The combined cohort consisted of 683 cases, of which 573 had available backup samples to perform the biomarker analysis. Literature search identified 9 SAP prediction scores. Our data set enabled us to calculate 5 of these scores. The scores had area under receiver operating characteristic curve of 0.543 to 0.651 for physician determined SAP, 0.574 to 0.685 for probable and 0.689 to 0.811 for definite SAP according to Pneumonia in Stroke Consensus group criteria. Multivariate models of the scores with biomarkers improved virtually all predictions, but mostly in the range of an area under receiver operating characteristic curve delta of 0.05. Conclusions: All SAP prediction scores identified patients who would develop SAP with fair to strong capabilities, with better discrimination when stricter criteria for SAP diagnosis were applied. The selected biomarkers provided only limited added predictive value, currently not warranting addition of these markers to prediction models. Registration: URL: https://www.clinicaltrials.gov . Unique identifier: NCT01264549 and NCT01079728.

scholarly journals A Nomogram to Predict Patients with Obstructive Coronary Artery Disease: Development and Validation

2021 ◽  
Author(s):  
Lihong Lai
Keyword(s):  
Risk Factors ◽  
Operating Characteristic ◽  
Prediction Models ◽  
Characteristic Curve ◽  
Information Criterion ◽  
Clinical Prediction ◽  
Clinical Prediction Models ◽  
Net Reclassification Improvement ◽  
Artery Disease ◽  
Integrated Discrimination Improvement

Objective: To develop and validate clinical prediction models for the development of a nomogram to estimate the probability of patients having coronary artery disease (CAD).Methods and Results: A total of 1,025 patients referred for coronary angiography were included in a retrospective, single-center study. Randomly, 720 patients (70%) were selected as the development group and the other patients were selected as the validation group. Multivariate logistic regression analysis showed that the seven risk factors age, sex, systolic blood pressure, lipoprotein-associated phospholipase A2, type of angina, hypertension, and diabetes were significant for diagnosis of CAD, from which we established model A. We established model B with the risk factors age, sex, height, systolic blood pressure, low-density lipoprotein cholesterol, lipoprotein-associated phospholipase A2, type of angina, hypertension, and diabetes via the Akaike information criterion. The risk factors from the original Framingham Risk Score were used for model C. From comparison of the areas under the receiver operating characteristic curve, net reclassification improvement, and integrated discrimination improvement of models A, B, and C, we chose model B to develop the nomogram because of its fitness in discrimination, calibration, and clinical efficiency. The nomogram for diagnosis of CAD could be used easily and conveniently. Conclusion: An individualized clinical prediction model for patients with CAD allowed an accurate estimation in Chinese populations. The Akaike information criterion is a better method in screening risk factors. The net reclassification improvement and integrated discrimination improvement are better than the area under the receiver operating characteristic curve in discrimination. Decision curve analysis can be used to evaluate the efficiency of clinical prediction models.

A validated novel preoperative index to predict the extent of intraperitoneal contamination in patients with acute abdominal pathology: A cohort study

2019 ◽  
Vol 30 (7-8) ◽  
pp. 221-228
Author(s):  
Shahab Hajibandeh ◽  
Shahin Hajibandeh ◽  
Nicholas Hobbs ◽  
Jigar Shah ◽  
Matthew Harris ◽  
...  
Keyword(s):  
Receiver Operating Characteristic Curve ◽  
Receiver Operating Characteristic ◽  
Operating Characteristic ◽  
Validation Cohort ◽  
Characteristic Curve ◽  
Curve Analysis ◽  
Emergency Laparotomy ◽  
Contamination Index ◽  
Operating Characteristic Curve ◽  
Receiver Operating

Aims To investigate whether an intraperitoneal contamination index (ICI) derived from combined preoperative levels of C-reactive protein, lactate, neutrophils, lymphocytes and albumin could predict the extent of intraperitoneal contamination in patients with acute abdominal pathology. Methods Patients aged over 18 who underwent emergency laparotomy for acute abdominal pathology between January 2014 and October 2018 were randomly divided into primary and validation cohorts. The proposed intraperitoneal contamination index was calculated for each patient in each cohort. Receiver operating characteristic curve analysis was performed to determine discrimination of the index and cut-off values of preoperative intraperitoneal contamination index that could predict the extent of intraperitoneal contamination. Results Overall, 468 patients were included in this study; 234 in the primary cohort and 234 in the validation cohort. The analyses identified intraperitoneal contamination index of 24.77 and 24.32 as cut-off values for purulent contamination in the primary cohort (area under the curve (AUC): 0.73, P < 0.0001; sensitivity: 84%, specificity: 60%) and validation cohort (AUC: 0.83, P < 0.0001; sensitivity: 91%, specificity: 69%), respectively. Receiver operating characteristic curve analysis also identified intraperitoneal contamination index of 33.70 and 33.41 as cut-off values for feculent contamination in the primary cohort (AUC: 0.78, P < 0.0001; sensitivity: 87%, specificity: 64%) and validation cohort (AUC: 0.79, P < 0.0001; sensitivity: 86%, specificity: 73%), respectively. Conclusions As a predictive measure which is derived purely from biomarkers, intraperitoneal contamination index may be accurate enough to predict the extent of intraperitoneal contamination in patients with acute abdominal pathology and to facilitate decision-making together with clinical and radiological findings.

scholarly journals MicroRNAs-1299, -126-3p and -30e-3p as Potential Diagnostic Biomarkers for Prediabetes

Diagnostics ◽  
2021 ◽  
Vol 11 (6) ◽  
pp. 949
Author(s):  
Cecil J. Weale ◽  
Don M. Matshazi ◽  
Saarah F. G. Davids ◽  
Shanel Raghubeer ◽  
Rajiv T. Erasmus ◽  
...  
Keyword(s):  
Receiver Operating Characteristic ◽  
Operating Characteristic ◽  
Characteristic Curve ◽  
Predictive Ability ◽  
Roc Curves ◽  
Tolerance Test ◽  
Cross Sectional Study ◽  
Oral Glucose ◽  
Cross Sectional ◽  
Receiver Operating

This cross-sectional study investigated the association of miR-1299, -126-3p and -30e-3p with and their diagnostic capability for dysglycaemia in 1273 (men, n = 345) South Africans, aged >20 years. Glycaemic status was assessed by oral glucose tolerance test (OGTT). Whole blood microRNA (miRNA) expressions were assessed using TaqMan-based reverse transcription quantitative-PCR (RT-qPCR). Receiver operating characteristic (ROC) curves assessed the ability of each miRNA to discriminate dysglycaemia, while multivariable logistic regression analyses linked expression with dysglycaemia. In all, 207 (16.2%) and 94 (7.4%) participants had prediabetes and type 2 diabetes mellitus (T2DM), respectively. All three miRNAs were significantly highly expressed in individuals with prediabetes compared to normotolerant patients, p < 0.001. miR-30e-3p and miR-126-3p were also significantly more expressed in T2DM versus normotolerant patients, p < 0.001. In multivariable logistic regressions, the three miRNAs were consistently and continuously associated with prediabetes, while only miR-126-3p was associated with T2DM. The ROC analysis indicated all three miRNAs had a significant overall predictive ability to diagnose prediabetes, diabetes and the combination of both (dysglycaemia), with the area under the receiver operating characteristic curve (AUC) being significantly higher for miR-126-3p in prediabetes. For prediabetes diagnosis, miR-126-3p (AUC = 0.760) outperformed HbA1c (AUC = 0.695), p = 0.042. These results suggest that miR-1299, -126-3p and -30e-3p are associated with prediabetes, and measuring miR-126-3p could potentially contribute to diabetes risk screening strategies.

A conditional approach for the receiver operating characteristic curve construction to evaluate diagnostic test performance in a family-matched case–control design

2021 ◽  
pp. 096228022199595
Author(s):  
Yalda Zarnegarnia ◽  
Shari Messinger
Keyword(s):  
Receiver Operating Characteristic Curve ◽  
Receiver Operating Characteristic ◽  
Operating Characteristic ◽  
Characteristic Curve ◽  
Case Control ◽  
Paired Data ◽  
Characteristic Curves ◽  
Receiver Operating Characteristic Curves ◽  
Operating Characteristic Curve ◽  
Receiver Operating

Receiver operating characteristic curves are widely used in medical research to illustrate biomarker performance in binary classification, particularly with respect to disease or health status. Study designs that include related subjects, such as siblings, usually have common environmental or genetic factors giving rise to correlated biomarker data. The design could be used to improve detection of biomarkers informative of increased risk, allowing initiation of treatment to stop or slow disease progression. Available methods for receiver operating characteristic construction do not take advantage of correlation inherent in this design to improve biomarker performance. This paper will briefly review some developed methods for receiver operating characteristic curve estimation in settings with correlated data from case–control designs and will discuss the limitations of current methods for analyzing correlated familial paired data. An alternative approach using conditional receiver operating characteristic curves will be demonstrated. The proposed approach will use information about correlation among biomarker values, producing conditional receiver operating characteristic curves that evaluate the ability of a biomarker to discriminate between affected and unaffected subjects in a familial paired design.

scholarly journals Deciphering the Interactions of Bioactive Compounds in Selected Traditional Medicinal Plants against Alzheimer’s Diseases via Pharmacophore Modeling, Auto-QSAR, and Molecular Docking Approaches

Molecules ◽  
2021 ◽  
Vol 26 (7) ◽  
pp. 1996
Author(s):  
 Oluwafemi Adeleke Ojo ◽  
Adebola Busola Ojo ◽  
Charles Okolie ◽  
Mary-Ann Chinyere Nwakama ◽  
Matthew Iyobhebhe ◽  
...  
Keyword(s):  
Medicinal Plants ◽  
Receiver Operating Characteristic ◽  
Binding Affinity ◽  
Operating Characteristic ◽  
Characteristic Curve ◽  
Screening Method ◽  
Pharmacophore Modeling ◽  
Structure Identification ◽  
Mao Activity ◽  
Receiver Operating

Neurodegenerative diseases, for example Alzheimer’s, are perceived as driven by hereditary, cellular, and multifaceted biochemical actions. Numerous plant products, for example flavonoids, are documented in studies for having the ability to pass the blood-brain barrier and moderate the development of such illnesses. Computer-aided drug design (CADD) has achieved importance in the drug discovery world; innovative developments in the aspects of structure identification and characterization, bio-computational science, and molecular biology have added to the preparation of new medications towards these ailments. In this study we evaluated nine flavonoid compounds identified from three medicinal plants, namely T. diversifolia, B. sapida, and I. gabonensis for their inhibitory role on acetylcholinesterase (AChE), butyrylcholinesterase (BChE) and monoamine oxidase (MAO) activity, using pharmacophore modeling, auto-QSAR prediction, and molecular studies, in comparison with standard drugs. The results indicated that the pharmacophore models produced from structures of AChE, BChE and MAO could identify the active compounds, with a recuperation rate of the actives found near 100% in the complete ranked decoy database. Moreso, the robustness of the virtual screening method was accessed by well-established methods including enrichment factor (EF), receiver operating characteristic curve (ROC), Boltzmann-enhanced discrimination of receiver operating characteristic (BEDROC), and area under accumulation curve (AUAC). Most notably, the compounds’ pIC50 values were predicted by a machine learning-based model generated by the AutoQSAR algorithm. The generated model was validated to affirm its predictive model. The best models achieved for AChE, BChE and MAO were models kpls_radial_17 (R2 = 0.86 and Q2 = 0.73), pls_38 (R2 = 0.77 and Q2 = 0.72), kpls_desc_44 (R2 = 0.81 and Q2 = 0.81) and these externally validated models were utilized to predict the bioactivities of the lead compounds. The binding affinity results of the ligands against the three selected targets revealed that luteolin displayed the highest affinity score of −9.60 kcal/mol, closely followed by apigenin and ellagic acid with docking scores of −9.60 and −9.53 kcal/mol, respectively. The least binding affinity was attained by gallic acid (−6.30 kcal/mol). The docking scores of our standards were −10.40 and −7.93 kcal/mol for donepezil and galanthamine, respectively. The toxicity prediction revealed that none of the flavonoids presented toxicity and they all had good absorption parameters for the analyzed targets. Hence, these compounds can be considered as likely leads for drug improvement against the same.

Sign in / Sign up

Export Citation Format

Share Document