Reviewing the use and quality of machine learning in developing clinical prediction models for cardiovascular disease

2021 ◽  
pp. postgradmedj-2020-139352
Author(s):  
Simon Allan ◽  
Raphael Olaiya ◽  
Rasan Burhan

Cardiovascular disease (CVD) is one of the leading causes of death across the world. CVD can lead to angina, heart attacks, heart failure, strokes, and eventually, death; among many other serious conditions. The early intervention with those at a higher risk of developing CVD, typically with statin treatment, leads to better health outcomes. For this reason, clinical prediction models (CPMs) have been developed to identify those at a high risk of developing CVD so that treatment can begin at an earlier stage. Currently, CPMs are built around statistical analysis of factors linked to developing CVD, such as body mass index and family history. The emerging field of machine learning (ML) in healthcare, using computer algorithms that learn from a dataset without explicit programming, has the potential to outperform the CPMs available today. ML has already shown exciting progress in the detection of skin malignancies, bone fractures and many other medical conditions. In this review, we will analyse and explain the CPMs currently in use with comparisons to their developing ML counterparts. We have found that although the newest non-ML CPMs are effective, ML-based approaches consistently outperform them. However, improvements to the literature need to be made before ML should be implemented over current CPMs.

2021 ◽  
Vol 36 (Supplement_1) ◽  
Author(s):  
Enrico Favaro ◽  
Roberta Lazzarin ◽  
Daniela Cremasco ◽  
Erika Pierobon ◽  
Marta Guizzo ◽  
...  

Abstract Background and Aims The modern development of the black box approach in clinical nephrology is inconceivable without a logical theory of renal function and a comprehension of anatomical architecture of the kidney, in health and disease: this is the undisputed contribution offered by Malpighi, Oliver and Trueta starting from the seventeenth century. The machine learning model for the prediction of acute kidney injury, progression of renal failure and tubulointerstitial nephritis is a good example of how different knowledge about kidney are an indispensable tool for the interpretation of model itself. Method Historical data were collected from literature, textbooks, encyclopedias, scientific periodicals and laboratory experimental data concerning these three authors. Results The Italian Marcello Malpighi (1628-1694), born in Crevalcore near Bologna, was Professor of anatomy at Bologna, Pisa and Messina. The historic description of the pulmonary capillaries was made in his second epistle to Borelli published in 1661 and intitled De pulmonibus, by means of the frog as “the microscope of nature” (Fig. 1). It is the first description of capillaries in any circulation. William Harvey in De motu cordis in 1628 (year of publication the same of date of birth of Italian anatomist!) could not see the capillary vessels. This thriumphant discovery will serve for the next reconnaissance of characteristic renal rete mirabile.in the corpuscle of Malpighi, lying within the capsule of Bowman. Jean Redman Oliver (1889-1976), a pathologist born and raised in Northern California, was able to bridge the gap between the nephron and collecting system through meticulous dissections, hand drawn illustrations and experiments which underpin our current understanding of renal anatomy and physiology. In the skillful lecture “When is the kidney not a kidney?” (1949) Oliver summarizes his far-sighted vision on renal physiology and disease in the following sentence: the Kidney in health, if you will, but the Nephrons in disease. Because, the “nephron” like the “kidney” is an abstraction that must be qualified in terms of its various parts, its cellular components and the molecular mechanisms involved in each discrete activity (Fig. 2). The Catalan surgeon Josep Trueta I Raspall (1897-1977) was born in the Poblenou neighborhood of Barcelona. His impact of pioneering and visionary contribution to the changes in renal circulation for the pathogenesis of acute kidney injury was pivotal for history of renal physiology. “The kidney has two potential circulatory circulations. Blood may pass either almost exclusively through one or other of two pathways, or to a varying degree through both”. (Studies of the Renal Circulation, published in 1947). Now this diversion of blood from cortex to the less resistant medullary circulation is known with the eponym Trueta shunt. Conclusion The black box approach to the kidney diseases should be considered by practitioners as a further tool to help to inform model update in many clinical setting. The number of machine learning clinical prediction models being published is rising, as new fields of application are being explored in medicine (Fig. 3). A challenge in the clinical nephrology is to explore the “kidney machine” during each therapeutic diagnostic procedure. Always, the intriguing relationship between the set of nephrological syndromes and kidney diseases cannot disregard the precious notions the specific organization of kidney microcirculation, fruit of many scientific contributions of the work by Malpighi, Oliver and Trueta (Fig. 3).


2021 ◽  
Author(s):  
Gaurav Gulati ◽  
Riley J Brazil ◽  
Jason Nelson ◽  
David van Klaveren ◽  
Christine M. Lundquist ◽  
...  

AbstractBackgroundClinical prediction models (CPMs) are used to inform treatment decisions for the primary prevention of cardiovascular disease. We aimed to assess the performance of such CPMs in fully independent cohorts.Methods and Results63 models predicting outcomes for patients at risk of cardiovascular disease from the Tufts PACE CPM Registry were selected for external validation on publicly available data from up to 4 broadly inclusive primary prevention clinical trials. For each CPM-trial pair, we assessed model discrimination, calibration, and net benefit. Results were stratified based on the relatedness of derivation and validation cohorts, and net benefit was reassessed after updating model intercept, slope, or complete re-estimation. The median c statistic of the CPMs decreased from 0.77 (IQR 0.72-0.78) in the derivation cohorts to 0.63 (IQR 0.58-0.66) when externally validated. The validation c-statistic was higher when derivation and validation cohorts were considered related than when they were distantly related (0.67 vs 0.60, p < 0.001). The calibration slope was also higher in related cohorts than distantly related cohorts (0.69 vs 0.58, p < 0.001). Net benefit analysis suggested substantial likelihood of harm when models were externally applied, but this likelihood decreased after model updating.ConclusionsDiscrimination and calibration decrease significantly when CPMs for primary prevention of cardiovascular disease are tested in external populations, particularly when the population is only distantly related to the derivation population. Poorly calibrated predictions lead to poor decision making. Model updating can reduce the likelihood of harmful decision making, and is needed to realize the full potential of risk-based decision making in new settings.


2020 ◽  
Vol 35 (1) ◽  
pp. 100-116 ◽  
Author(s):  
M B Ratna ◽  
S Bhattacharya ◽  
B Abdulrahim ◽  
D J McLernon

Abstract STUDY QUESTION What are the best-quality clinical prediction models in IVF (including ICSI) treatment to inform clinicians and their patients of their chance of success? SUMMARY ANSWER The review recommends the McLernon post-treatment model for predicting the cumulative chance of live birth over and up to six complete cycles of IVF. WHAT IS KNOWN ALREADY Prediction models in IVF have not found widespread use in routine clinical practice. This could be due to their limited predictive accuracy and clinical utility. A previous systematic review of IVF prediction models, published a decade ago and which has never been updated, did not assess the methodological quality of existing models nor provided recommendations for the best-quality models for use in clinical practice. STUDY DESIGN, SIZE, DURATION The electronic databases OVID MEDLINE, OVID EMBASE and Cochrane library were searched systematically for primary articles published from 1978 to January 2019 using search terms on the development and/or validation (internal and external) of models in predicting pregnancy or live birth. No language or any other restrictions were applied. PARTICIPANTS/MATERIALS, SETTING, METHODS The PRISMA flowchart was used for the inclusion of studies after screening. All studies reporting on the development and/or validation of IVF prediction models were included. Articles reporting on women who had any treatment elements involving donor eggs or sperm and surrogacy were excluded. The CHARMS checklist was used to extract and critically appraise the methodological quality of the included articles. We evaluated models’ performance by assessing their c-statistics and plots of calibration in studies and assessed correct reporting by calculating the percentage of the TRIPOD 22 checklist items met in each study. MAIN RESULTS AND THE ROLE OF CHANCE We identified 33 publications reporting on 35 prediction models. Seventeen articles had been published since the last systematic review. The quality of models has improved over time with regard to clinical relevance, methodological rigour and utility. The percentage of TRIPOD score for all included studies ranged from 29 to 95%, and the c-statistics of all externally validated studies ranged between 0.55 and 0.77. Most of the models predicted the chance of pregnancy/live birth for a single fresh cycle. Six models aimed to predict the chance of pregnancy/live birth per individual treatment cycle, and three predicted more clinically relevant outcomes such as cumulative pregnancy/live birth. The McLernon (pre- and post-treatment) models predict the cumulative chance of live birth over multiple complete cycles of IVF per woman where a complete cycle includes all fresh and frozen embryo transfers from the same episode of ovarian stimulation. McLernon models were developed using national UK data and had the highest TRIPOD score, and the post-treatment model performed best on external validation. LIMITATIONS, REASONS FOR CAUTION To assess the reporting quality of all included studies, we used the TRIPOD checklist, but many of the earlier IVF prediction models were developed and validated before the formal TRIPOD reporting was published in 2015. It should also be noted that two of the authors of this systematic review are authors of the McLernon model article. However, we feel we have conducted our review and made our recommendations using a fair and transparent systematic approach. WIDER IMPLICATIONS OF THE FINDINGS This study provides a comprehensive picture of the evolving quality of IVF prediction models. Clinicians should use the most appropriate model to suit their patients’ needs. We recommend the McLernon post-treatment model as a counselling tool to inform couples of their predicted chance of success over and up to six complete cycles. However, it requires further external validation to assess applicability in countries with different IVF practices and policies. STUDY FUNDING/COMPETING INTEREST(S) The study was funded by the Elphinstone Scholarship Scheme and the Assisted Reproduction Unit, University of Aberdeen. Both D.J.M. and S.B. are authors of the McLernon model article and S.B. is Editor in Chief of Human Reproduction Open. They have completed and submitted the ICMJE forms for Disclosure of potential Conflicts of Interest. The other co-authors have no conflicts of interest to declare. REGISTRATION NUMBER N/A


2016 ◽  
Vol 9 (2 suppl 1) ◽  
pp. S8-S15 ◽  
Author(s):  
Jessica K. Paulus ◽  
Benjamin S. Wessler ◽  
Christine Lundquist ◽  
Lana L.Y. Lai ◽  
Gowri Raman ◽  
...  

Endocrine ◽  
2021 ◽  
Author(s):  
Olivier Zanier ◽  
Matteo Zoli ◽  
Victor E. Staartjes ◽  
Federica Guaraldi ◽  
Sofia Asioli ◽  
...  

Abstract Purpose Biochemical remission (BR), gross total resection (GTR), and intraoperative cerebrospinal fluid (CSF) leaks are important metrics in transsphenoidal surgery for acromegaly, and prediction of their likelihood using machine learning would be clinically advantageous. We aim to develop and externally validate clinical prediction models for outcomes after transsphenoidal surgery for acromegaly. Methods Using data from two registries, we develop and externally validate machine learning models for GTR, BR, and CSF leaks after endoscopic transsphenoidal surgery in acromegalic patients. For the model development a registry from Bologna, Italy was used. External validation was then performed using data from Zurich, Switzerland. Gender, age, prior surgery, as well as Hardy and Knosp classification were used as input features. Discrimination and calibration metrics were assessed. Results The derivation cohort consisted of 307 patients (43.3% male; mean [SD] age, 47.2 [12.7] years). GTR was achieved in 226 (73.6%) and BR in 245 (79.8%) patients. In the external validation cohort with 46 patients, 31 (75.6%) achieved GTR and 31 (77.5%) achieved BR. Area under the curve (AUC) at external validation was 0.75 (95% confidence interval: 0.59–0.88) for GTR, 0.63 (0.40–0.82) for BR, as well as 0.77 (0.62–0.91) for intraoperative CSF leaks. While prior surgery was the most important variable for prediction of GTR, age, and Hardy grading contributed most to the predictions of BR and CSF leaks, respectively. Conclusions Gross total resection, biochemical remission, and CSF leaks remain hard to predict, but machine learning offers potential in helping to tailor surgical therapy. We demonstrate the feasibility of developing and externally validating clinical prediction models for these outcomes after surgery for acromegaly and lay the groundwork for development of a multicenter model with more robust generalization.


2021 ◽  
Author(s):  
Thomas Stojanov ◽  
Linda Modler ◽  
Andreas M. Müller ◽  
Soheila Aghlmandi ◽  
Christian Appenzeller-Herzog ◽  
...  

Abstract BackgroundPost-operative shoulder stiffness (POSS) is one of the most frequent complications after arthroscopic rotator cuff repair (ARCR). Factors specifying clinical prediction models for the occurrence of POSS should rely on the literature and expert assessment. Our objective was to map prognostic factors for the occurrence of POSS in patients after an ARCR.MethodsLongitudinal studies of ARCR reporting prognostic factors for the occurrence of POSS with an endpoint of at least 6 months were included. We systematically searched Embase, Medline, and Scopus for articles published between January 1, 2014 and February 12, 2020 and screened cited and citing literature of eligible records and identified reviews. The risk of bias of included studies and the quality of evidence were assessed using the Quality in Prognosis Studies tool and an adapted Grading of Recommendations, Assessment, Development and Evaluations framework. A database was implemented to report the results of individual studies. The review was registered on PROSPERO (CRD42020199257).ResultsSeven cohort studies including 23 257 patients were included after screening 5013 records. POSS prevalence ranged from 0.51% to 8.75% with an endpoint ranging from 6 to 24 months. Due to scarcity of data, no meta-analysis could be performed. Overall risk of bias and quality of evidence was deemed high and low or very low, respectively. Twenty-two potential prognostic factors were identified. Increased age and male sex emerged as protective factors against POSS. Additional factors were reported but do require further analyses to determine their prognostic value.DiscussionAvailable evidence pointed to male sex and increased age as probable protective factors against POSS after ARCR. To establish a reliable pre-specified set of factors for clinical prediction models, our review results require complementation with an expert's opinion.


Author(s):  
Benjamin S Wessler ◽  
Lana Lai YH ◽  
Whitney Kramer ◽  
Michael Cangelosi ◽  
Gowri Raman ◽  
...  

Background: Clinical prediction models (CPMs) estimate the probability of clinical outcomes and hold the potential to improve decision making and individualize care. For patients with cardiovascular disease (CVD) there are numerous CPMs available though the extent of this literature is not well described. Methods and Results: We conducted a systematic review for articles containing CPMs for CVD published between January 1990 through May 2012. CVD includes coronary artery disease (CAD), congestive heart failure (CHF), arrhythmias, stroke, venous thromboembolism (VTE) and peripheral vascular disease (PVD). We created a novel database and characterized CPMs based on the stage of development, population under study, performance, covariates, and predicted outcomes. We included articles that describe newly developed CPMs that predict the risk of developing an outcome (prognostic models) or the probability of a specific diagnosis (diagnostic models). There are 796 models included in this database representing 31 distinct index conditions. 717 (90%) are de novo CPMs, 21 (3%) are CPM recalibrations, and 58 (7%) are CPM adaptations. There are 215 CPMs for patients with CAD, 168 CPMs for population samples at risk for incident CVD, and 79 models for patients with CHF (Figure). De novo CPMs predicting mortality were most commonly published for patients with known CAD (98 models) followed by HF (63 models) and stroke (24 models). There are 77 distinct index/ outcome (I/O) pairings and models are roughly evenly split between those predicting short term outcomes (< 3 months) and those predicting long term outcomes (< 6 months). There are 41 diagnostic CPMs included in this database, most commonly predicting diagnoses of CAD (11 models), VTE (10 models), and acute coronary syndrome (5 models). Of the de novo models in this database 450 (63%) report a c-statistic and 259 (36%) report either the Hosmer-Lemeshow statistic or show a calibration plot. Conclusions: There is an abundance of CPMs available for many CVD conditions, with substantial redundancy in the literature. The comparative performance of these models, the consistency of effects and risk estimates across models and the actual and potential clinical impact of this body of literature is poorly understood.


PLoS ONE ◽  
2018 ◽  
Vol 13 (9) ◽  
pp. e0202685 ◽  
Author(s):  
Christian A. Bannister ◽  
Julian P. Halcox ◽  
Craig J. Currie ◽  
Alun Preece ◽  
Irena Spasić

2019 ◽  
Vol 110 ◽  
pp. 12-22 ◽  
Author(s):  
Evangelia Christodoulou ◽  
Jie Ma ◽  
Gary S. Collins ◽  
Ewout W. Steyerberg ◽  
Jan Y. Verbakel ◽  
...  

Sign in / Sign up

Export Citation Format

Share Document