Model-Based ROC Curve: Examining the Effect of Case Mix and Model Calibration on the ROC Plot

Background The performance of risk prediction models is often characterized in terms of discrimination and calibration. The receiver-operating characteristic (ROC) curve is widely used for evaluating model discrimination. However, when comparing ROC curves across different samples, the effect of case mix makes the interpretation of discrepancies difficult. Further, compared with model discrimination, evaluating model calibration has not received the same level of attention. Current methods for examining model calibration require specification of smoothing or grouping factors. Methods We introduce the “model-based” ROC curve (mROC) to assess model calibration and the effect of case mix during external validation. The mROC curve is the ROC curve that should be observed if the prediction model is calibrated in the external population. We show that calibration-in-the-large and the equivalence of mROC and ROC curves are together sufficient conditions for the model to be calibrated. Based on this, we propose a novel statistical test for calibration that, unlike current methods, does not require any subjective specification of smoothing or grouping factors. Results Through a stylized example, we demonstrate how mROC separates the effect of case mix and model miscalibration when externally validating a risk prediction model. We present the results of simulation studies that confirm the properties of the new calibration test. A case study on predicting the risk of acute exacerbations of chronic obstructive pulmonary disease puts the developments in a practical context. R code for the implementation of this method is provided. Conclusion mROC can easily be constructed and used to interpret the effect of case mix and calibration on the ROC plot. Given the popularity of ROC curves among applied investigators, this framework can further promote assessment of model calibration. Highlights Compared with examining model discrimination, examining model calibration has not received the same level of attention among investigators who develop or examine risk prediction models. This article introduces the model-based ROC (mROC) curve as the basis for graphical and statistical examination of model calibration on the ROC plot. This article introduces a formal statistical test based on mROC for examining model calibration that does not require arbitrary smoothing or grouping factors. Investigators who develop or validate risk prediction models can now also use the popular ROC plot for examining model calibration, as a critical but often neglected component in predictive analytics.

NELA.03Laparoscopy in Emergency Surgery is Associated with Lower Risk-Adjusted Mortality

Aims To compare risk-adjusted outcomes after emergency intestinal surgery by operative approach. Methods Data from December 2013-November 2018 was retrieved from the NELA national database. Complete data on 102,154 patients with P-POSSUM was available, and 47,667 had NELA score. AUROC curves were calculated to assess model discrimination (c-statistic), and calibration plots to visualise agreement between predicted and observed mortality. Standardised Mortality Ratio's (SMR) were calculated for the total cohort and by operative approach. Operative approach was divided into: laparotomy, completed laparoscopically, converted to open and lap assisted. Results Both P-POSSUM and NELA score displayed good discrimination for total cohort and by operative approach (P-POSSUM c-statistic=0.801-0.815; NELA score c-statistic=0.851-0.880). Calibration plots demonstrated that P-POSSUM was highly accurate up to 20% mortality, after which it substantially over-predicted mortality. NELA score was highly accurate up to 25% mortality after which it slightly under-predicted. Overall SMR of observed vs expected deaths was 0.77 using P-POSSUM, 0.8 for laparotomy and 0.46 for laparoscopy. Restricting cases to < 10% predicted mortality (n = 65,000), overall SMR improved (0.9) and was considerably lower for cases completed laparoscopically (0.41) compared to open (0.97). Using NELA scores of < 10% predicted mortality (n = 27,000) had similar overall SMR (0.96), with cases completed laparoscopically displaying much lower SMR (0.61) compared to laparotomy (1.0). Conclusions SMR's calculated using P-POSSUM and NELA score have demonstrated that laparoscopy has significantly lower observed vs expected mortality rate compared to laparotomy. This raises the question of why laparoscopy is associated with reduced mortality and should operative approach be included in risk models?

Left Atrioventricular Coupling Index to Predict Incident Heart Failure: The Multi-Ethnic Study of Atherosclerosis

Background: Although left atrial (LA) and left ventricular (LV) structural and functional parameters have independent prognostic value as predictors of heart failure (HF), the close physiological relationship between the LA and LV suggest that the assessment of LA/LV coupling could better reflect left atrioventricular dysfunction and be a better predictor of HF.Aim: We investigated the prognostic value of a left atrioventricular coupling index (LACI), measured by cardiovascular magnetic resonance (CMR), as well as change in LACI to predict incident HF in the Multi-Ethnic Study of Atherosclerosis (MESA).Materials and Methods: In the MESA, 2,250 study participants, free of clinically recognized HF and cardiovascular disease (CVD) at baseline, had LACI assessed by CMR imaging at baseline (Exam 1, 2000–2002), and 10 years later (Exam 5, 2010–2012). Left atrioventricular coupling index was defined as the ratio of LA to LV end-diastolic volumes. Univariable and multivariable Cox proportional hazard models were used to evaluate the associations of LACI and average annualized change in LACI (ΔLACI) with incident HF after adjustment for traditional MESA-HF risk factors. The incremental risk prediction was calculated using C-statistic, categorical net reclassification index (NRI) and integrative discrimination index (IDI).Results: Among the 2,250 participants (mean age 59.3 ± 9.3 years and 47.6% male participants), 50 incident HF events occurred over 6.8 ± 1.3 years after the second CMR exam. After adjustment, greater LACI and ΔLACI were independently associated with HF (adjusted HR 1.44, 95% CI [1.25–1.66] and adjusted HR 1.55, 95% CI [1.30–1.85], respectively; both p < 0.0001). Adjusted models for LACI showed significant improvement in model discrimination and reclassification compared to currently used HF risk score model for predicting HF incidence (C-statistic: 0.81 vs. 0.77; NRI = 0.411; IDI = 0.043). After adjustment, ΔLACI showed also significant improvement in model discrimination compared to the multivariable model with traditional MESA-HF risk factors for predicting incident HF (C-statistic: 0.82 vs. 0.77; NRI = 0.491; IDI = 0.058).Conclusions: In a multi-ethnic population, atrioventricular coupling (LACI), and coupling change (ΔLACI) are independently associated with incident HF. Both have incremental prognostic value for predicting HF events over traditional HF risk factors.

Left atrioventricular coupling index to predict incident heart failure: the multi-ethnic study of atherosclerosis (MESA)

Funding Acknowledgements Type of funding sources: None. BACKGROUND Although left atrial (LA) and left ventricular (LV) structural and functional parameters have independent prognostic value as predictors of HF, the close physiological relationship between LA and LV suggest that the assessment of LA/LV coupling could better reflect left atrioventricular dysfunction and be a better predictor of heart failure (HF). PURPOSE We investigated the prognostic value of a left atrioventricular coupling index (LACI), measured by cardiovascular magnetic resonance (CMR), as well as change in LACI to predict incident HF in the Multi-Ethnic Study of Atherosclerosis (MESA). METHODS In the MESA, 2,250 study participants, free of clinically recognized HF and cardiovascular disease at baseline, had LACI assessed by CMR imaging at baseline (Exam 1, 2000–2002), and ten years later (Exam 5, 2010–2012). LACI was defined as the ratio of LA to LV end-diastolic volumes. Univariable and multivariable Cox proportional hazard models were used to evaluate the associations of LACI and average annualized change in LACI (ΔLACI) with incident HF after adjustment on traditional HF risk factors. The incremental risk prediction was calculated using C-statistic, categorical net reclassification index (NRI) and integrative discrimination index (IDI). RESULTS Among the 2,250 participants (mean age 59.3 ± 9.3 years and 47.6% male participants), 50 incident HF events occurred over 6.8 ± 1.3 years after the second CMR exam. After adjustment, greater LACI and ΔLACI were independently associated with HF (adjusted HR 1.44, 95% CI [1.25-1.66] and adjusted HR 1.55, 95% CI [1.30-1.85], respectively; both p < 0.0001). Adjusted models for LACI showed significant improvement in model discrimination and reclassification compared to currently used HF risk score model for predicting HF incidence (C-statistic: 0.81 vs. 0.77; NRI = 0.411; IDI = 0.043). After adjustment, ΔLACI showed also significant improvement in model discrimination compared to the multivariable model with traditional HF risk factors for predicting incident HF (C-statistic: 0.82 vs. 0.77; NRI = 0.491; IDI = 0.058). CONCLUSIONS In a multi-ethnic population, atrioventricular coupling (LACI) and coupling change (ΔLACI) are independently associated with incident HF. Both have incremental prognostic value for predicting HF over traditional HF risk factors.

Cellular connectomes as arbiters of local circuit models in the cerebral cortex

With the availability of cellular-resolution connectivity maps, connectomes, from the mammalian nervous system, it is in question how informative such massive connectomic data can be for the distinction of local circuit models in the mammalian cerebral cortex. Here, we investigated whether cellular-resolution connectomic data can in principle allow model discrimination for local circuit modules in layer 4 of mouse primary somatosensory cortex. We used approximate Bayesian model selection based on a set of simple connectome statistics to compute the posterior probability over proposed models given a to-be-measured connectome. We find that the distinction of the investigated local cortical models is faithfully possible based on purely structural connectomic data with an accuracy of more than 90%, and that such distinction is stable against substantial errors in the connectome measurement. Furthermore, mapping a fraction of only 10% of the local connectome is sufficient for connectome-based model distinction under realistic experimental constraints. Together, these results show for a concrete local circuit example that connectomic data allows model selection in the cerebral cortex and define the experimental strategy for obtaining such connectomic data.

Optimal design of experiments for effective modeling of atrazine degradation in soils

<p>The natural degradation pathways of the herbicide atrazine (AT) are highly complex. These pathways involve the metabolic activity of several bacterial guilds (that use AT as a source of carbon, nitrogen or both) and abiotic degradation mechanisms. The co-occurrence of multiple degradation pathways, combined with challenges in quantifying bacterial guilds and relevant intermediate metabolites, has led to the development of competing model formulations, which all represent valid descriptions of the fate of AT. A proper understanding of the fate of this complex compound is needed to develop effective management and mitigation strategies.</p><p>Here, we propose a model discrimination process in combination with prospective optimal design of experiments. We performed Monte-Carlo simulations using a first-order model that reflects a simple reaction chain of complete AT degradation and a set of Monod-based model variants that consider different bacterial consortia and degradation pathways. We used a Bayesian statistical analysis of these simulation ensembles to simulate virtual degradation experiments and chemical analysis strategies, thus obtaining predictions on the utility of experiments to deliver conclusive data for model discrimination. To do so, we defined different experimental protocols including a combination of: i) the metabolites to measure (AT, metabolites and CO<sub>2</sub>), ii) sampling frequency (sampling every day, every two days and every four days), iii) features difficult to quantify (specific bacterial guilds). As a statistical metric to measure the conclusiveness of these virtual experiments, we used the so-called energy distance.</p><p>Our results show that simulated AT degradation pathways following first-order reaction chains can be clearly distinguished from simulations using Monod-based models. Within the Monod-based models, we detected three clusters of models that differ in the number of bacterial guilds involved in AT degradation. Experimental designs considering main AT metabolites and sampling frequencies of once every two or four days at durations of 50 or 100 days provided the most informative data to discriminate models. Including measurements of bacterial guilds only slightly improved model discrimination. Our study highlights that environmental fate studies should prioritize measuring metabolites to elucidate active AT degradation pathways in soil and identify robust model formulations supporting risk assessment and mitigation strategies. </p>

Predicting specific abilities after disabling stroke: Development and validation of prognostic models

Background Predicting specific abilities (e.g. walk and talk) to provide a functional profile six months after disabling stroke could help patients/families prepare for the consequences of stroke and facilitate involvement in treatment decision-making. Aim To develop new statistical models to predict specific abilities six months after stroke and test their performance in an independent cohort of patients with disabling stroke. Methods We developed models to predict six specific abilities (to be independent, walk, talk, eat normally, live without major anxiety/depression, and to live at home) using data from seven large multicenter stroke trials with multivariable logistic regression. We included 13,117 participants recruited within three days of hospital admission. We assessed model discrimination and derived optimal cut-off values using four statistical methods. We validated the models in an independent single-center cohort of patients ( n = 403) with disabling stroke. We assessed model discrimination and calibration and reported the performance of our models at the statistically derived cut-off values. Results All six models had good discrimination in external validation (AUC 0.78–0.84). Four models (predicting to walk, eat normally, live without major anxiety/depression, live at home) calibrated well. Models had sensitivities between 45.0 and 97.9% and specificities between 21.6 and 96.5%. Conclusions We have developed statistical models to predict specific abilities and demonstrated that these models perform reasonably well in an independent cohort of disabling stroke patients. To aid decision-making regarding treatments, further evaluation of our models is required.

Chance of live birth: a nationwide, registry-based cohort study

STUDY QUESTION Does the sequence of prior pregnancy events (pregnancy losses, live births, ectopic pregnancies, molar pregnancy and still birth), obstetric complications and maternal age affect chance of live birth in the next pregnancy and are prior events predictive for the outcome? SUMMARY ANSWER The sequence of pregnancy outcomes is significantly associated with chance of live birth; however, pregnancy history and age are insufficient to predict the outcome of an individual woman’s next pregnancy. WHAT IS KNOWN ALREADY Adverse pregnancy outcomes decrease the chance of live birth in the next pregnancy, whereas the impact of prior live births is less clear. STUDY DESIGN, SIZE, DURATION Nationwide, registry-based cohort study of 1 285 230 women with a total of 2 722 441 pregnancies from 1977 to 2017. PARTICIPANTS/MATERIALS, SETTING, METHODS All women living in Denmark in the study period with at least one pregnancy in either the Danish Medical Birth Registry or the Danish National Patient Registry. Data were analysed using logistic regression with a robust covariance model to account for women with more than one pregnancy. Model discrimination and calibration were ascertained using 20% of the women in the cohort randomly selected as an internal validation set. MAIN RESULTS AND THE ROLE OF CHANCE Obstetric complications, still birth, ectopic pregnancies and pregnancy losses had a negative effect on the chance of live birth in the next pregnancy. Consecutive, identical pregnancy outcomes (pregnancy losses, live births or ectopic pregnancies) immediately preceding the next pregnancy had a larger impact than the total number of any outcome. Model discrimination was modest (C-index = 0.60, positive predictive value = 0.45), but the models were well calibrated. LIMITATIONS, REASONS FOR CAUTION While prior pregnancy outcomes and their sequence significantly influenced the chance of live birth, the discriminative abilities of the predictive models demonstrate clearly that pregnancy history and maternal age are insufficient to reliably predict the outcome of a given pregnancy. WIDER IMPLICATIONS OF THE FINDINGS Prior pregnancy history has a significant impact on the chance of live birth in the next pregnancy. However, the results emphasize that only taking age and number of losses into account does not predict if a pregnancy will end as a live birth or not. A better understanding of biological determinants for pregnancy outcomes is urgently needed. STUDY FUNDING/COMPETING INTEREST(S) The work was supported by the Novo Nordisk Foundation, Ole Kirk Foundation and Rigshospitalet’s Research Foundation. The authors have no financial relationships that could appear to have influenced the work. TRIAL REGISTRATION NUMBER N/A.