Assessing the suitability of general practice electronic health records for clinical prediction model development: a data quality assessment

Abstract Background The use of general practice electronic health records (EHRs) for research purposes is in its infancy in Australia. Given these data were collected for clinical purposes, questions remain around data quality and whether these data are suitable for use in prediction model development. In this study we assess the quality of data recorded in 201,462 patient EHRs from 483 Australian general practices to determine its usefulness in the development of a clinical prediction model for total knee replacement (TKR) surgery in patients with osteoarthritis (OA). Methods Variables to be used in model development were assessed for completeness and plausibility. Accuracy for the outcome and competing risk were assessed through record level linkage with two gold standard national registries, Australian Orthopaedic Association National Joint Replacement Registry (AOANJRR) and National Death Index (NDI). The validity of the EHR data was tested using participant characteristics from the 2014–15 Australian National Health Survey (NHS). Results There were substantial missing data for body mass index and weight gain between early adulthood and middle age. TKR and death were recorded with good accuracy, however, year of TKR, year of death and side of TKR were poorly recorded. Patient characteristics recorded in the EHR were comparable to participant characteristics from the NHS, except for OA medication and metastatic solid tumour. Conclusions In this study, data relating to the outcome, competing risk and two predictors were unfit for prediction model development. This study highlights the need for more accurate and complete recording of patient data within EHRs if these data are to be used to develop clinical prediction models. Data linkage with other gold standard data sets/registries may in the meantime help overcome some of the current data quality challenges in general practice EHRs when developing prediction models.

Download Full-text

Trends in the conduct and reporting of clinical prediction model development and validation: a systematic review

10.1101/2021.10.22.21265374 ◽

2021 ◽

Author(s):

Cynthia Yang ◽

Jan A. Kors ◽

Solomon Ioannou ◽

Luis H. John ◽

Aniek F. Markus ◽

...

Keyword(s):

Systematic Review ◽

Prediction Model ◽

Prediction Models ◽

External Validation ◽

Model Development ◽

Clinical Prediction ◽

Clinical Prediction Model ◽

Final Model ◽

Development And Validation ◽

Over Time

Objectives This systematic review aims to provide further insights into the conduct and reporting of clinical prediction model development and validation over time. We focus on assessing the reporting of information necessary to enable external validation by other investigators. Materials and Methods We searched Embase, Medline, Web-of-Science, Cochrane Library and Google Scholar to identify studies that developed one or more multivariable prognostic prediction models using electronic health record (EHR) data published in the period 2009-2019. Results We identified 422 studies that developed a total of 579 clinical prediction models using EHR data. We observed a steep increase over the years in the number of developed models. The percentage of models externally validated in the same paper remained at around 10%. Throughout 2009-2019, for both the target population and the outcome definitions, code lists were provided for less than 20% of the models. For about half of the models that were developed using regression analysis, the final model was not completely presented. Discussion Overall, we observed limited improvement over time in the conduct and reporting of clinical prediction model development and validation. In particular, the prediction problem definition was often not clearly reported, and the final model was often not completely presented. Conclusion Improvement in the reporting of information necessary to enable external validation by other investigators is still urgently needed to increase clinical adoption of developed models.

Download Full-text

A review of statistical updating methods for clinical prediction models

Statistical Methods in Medical Research ◽

10.1177/0962280215626466 ◽

2016 ◽

Vol 27 (1) ◽

pp. 185-197 ◽

Cited By ~ 27

Author(s):

Ting-Li Su ◽

Thomas Jaki ◽

Graeme L Hickey ◽

Iain Buchan ◽

Matthew Sperrin

Keyword(s):

Prediction Model ◽

Prediction Models ◽

Healthcare Outcomes ◽

Clinical Prediction ◽

Clinical Prediction Models ◽

Clinical Prediction Model ◽

The United Kingdom ◽

Single Strategy ◽

Using Data ◽

Dynamic Updating

A clinical prediction model is a tool for predicting healthcare outcomes, usually within a specific population and context. A common approach is to develop a new clinical prediction model for each population and context; however, this wastes potentially useful historical information. A better approach is to update or incorporate the existing clinical prediction models already developed for use in similar contexts or populations. In addition, clinical prediction models commonly become miscalibrated over time, and need replacing or updating. In this article, we review a range of approaches for re-using and updating clinical prediction models; these fall in into three main categories: simple coefficient updating, combining multiple previous clinical prediction models in a meta-model and dynamic updating of models. We evaluated the performance (discrimination and calibration) of the different strategies using data on mortality following cardiac surgery in the United Kingdom: We found that no single strategy performed sufficiently well to be used to the exclusion of the others. In conclusion, useful tools exist for updating existing clinical prediction models to a new population or context, and these should be implemented rather than developing a new clinical prediction model from scratch, using a breadth of complementary statistical methods.

Download Full-text

Development and external validation of a prognostic multivariable model on admission for hospitalized patients with COVID-19

10.1101/2020.03.28.20045997 ◽

2020 ◽

Cited By ~ 18

Author(s):

Jianfeng Xie ◽

Daniel Hungerford ◽

Hui Chen ◽

Simon T Abrams ◽

Shusheng Li ◽

...

Keyword(s):

Lactate Dehydrogenase ◽

Prediction Model ◽

Lymphocyte Count ◽

Prediction Models ◽

External Validation ◽

Model Development ◽

Added Value ◽

Clinical Prediction ◽

Healthcare Resources ◽

D Dimer

SummaryBackgroundCOVID-19 pandemic has developed rapidly and the ability to stratify the most vulnerable patients is vital. However, routinely used severity scoring systems are often low on diagnosis, even in non-survivors. Therefore, clinical prediction models for mortality are urgently required.MethodsWe developed and internally validated a multivariable logistic regression model to predict inpatient mortality in COVID-19 positive patients using data collected retrospectively from Tongji Hospital, Wuhan (299 patients). External validation was conducted using a retrospective cohort from Jinyintan Hospital, Wuhan (145 patients). Nine variables commonly measured in these acute settings were considered for model development, including age, biomarkers and comorbidities. Backwards stepwise selection and bootstrap resampling were used for model development and internal validation. We assessed discrimination via the C statistic, and calibration using calibration-in-the-large, calibration slopes and plots.FindingsThe final model included age, lymphocyte count, lactate dehydrogenase and SpO2 as independent predictors of mortality. Discrimination of the model was excellent in both internal (c=0·89) and external (c=0·98) validation. Internal calibration was excellent (calibration slope=1). External validation showed some over-prediction of risk in low-risk individuals and under-prediction of risk in high-risk individuals prior to recalibration. Recalibration of the intercept and slope led to excellent performance of the model in independent data.InterpretationCOVID-19 is a new disease and behaves differently from common critical illnesses. This study provides a new prediction model to identify patients with lethal COVID-19. Its practical reliance on commonly available parameters should improve usage of limited healthcare resources and patient survival rate.FundingThis study was supported by following funding: Key Research and Development Plan of Jiangsu Province (BE2018743 and BE2019749), National Institute for Health Research (NIHR) (PDF-2018-11-ST2-006), British Heart Foundation (BHF) (PG/16/65/32313) and Liverpool University Hospitals NHS Foundation Trust in UK.Research in contextEvidence before this studySince the outbreak of COVID-19, there has been a pressing need for development of a prognostic tool that is easy for clinicians to use. Recently, a Lancet publication showed that in a cohort of 191 patients with COVID-19, age, SOFA score and D-dimer measurements were associated with mortality. No other publication involving prognostic factors or models has been identified to date.Added value of this studyIn our cohorts of 444 patients from two hospitals, SOFA scores were low in the majority of patients on admission. The relevance of D-dimer could not be verified, as it is not included in routine laboratory tests. In this study, we have established a multivariable clinical prediction model using a development cohort of 299 patients from one hospital. After backwards selection, four variables, including age, lymphocyte count, lactate dehydrogenase and SpO2 remained in the model to predict mortality. This has been validated internally and externally with a cohort of 145 patients from a different hospital. Discrimination of the model was excellent in both internal (c=0·89) and external (c=0·98) validation. Calibration plots showed excellent agreement between predicted and observed probabilities of mortality after recalibration of the model to account for underlying differences in the risk profile of the datasets. This demonstrated that the model is able to make reliable predictions in patients from different hospitals. In addition, these variables agree with pathological mechanisms and the model is easy to use in all types of clinical settings.Implication of all the available evidenceAfter further external validation in different countries the model will enable better risk stratification and more targeted management of patients with COVID-19. With the nomogram, this model that is based on readily available parameters can help clinicians to stratify COVID-19 patients on diagnosis to use limited healthcare resources effectively and improve patient outcome.

Download Full-text

Statistical Development and Validation of Clinical Prediction Models

Anesthesiology ◽

10.1097/aln.0000000000003871 ◽

2021 ◽

Author(s):

Steven J. Staffa ◽

David Zurakowski

Keyword(s):

Prediction Model ◽

Prediction Models ◽

External Validation ◽

Model Development ◽

Predictive Ability ◽

Binary Outcome ◽

Clinical Prediction ◽

Clinical Prediction Models ◽

Surgery Research ◽

Development And Validation

Summary Clinical prediction models in anesthesia and surgery research have many clinical applications including preoperative risk stratification with implications for clinical utility in decision-making, resource utilization, and costs. It is imperative that predictive algorithms and multivariable models are validated in a suitable and comprehensive way in order to establish the robustness of the model in terms of accuracy, predictive ability, reliability, and generalizability. The purpose of this article is to educate anesthesia researchers at an introductory level on important statistical concepts involved with development and validation of multivariable prediction models for a binary outcome. Methods covered include assessments of discrimination and calibration through internal and external validation. An anesthesia research publication is examined to illustrate the process and presentation of multivariable prediction model development and validation for a binary outcome. Properly assessing the statistical and clinical validity of a multivariable prediction model is essential for reassuring the generalizability and reproducibility of the published tool.

Download Full-text

Localized Osteosarcoma in Upper Egypt: A Clinical Prediction Model

Case Medical Research ◽

10.31525/ct1-nct03839095 ◽

2000 ◽

Author(s):

Keyword(s):

Prediction Model ◽

Clinical Prediction ◽

Clinical Prediction Model ◽

Upper Egypt

Download Full-text

Minimum sample size for external validation of a clinical prediction model with a binary outcome

Statistics in Medicine ◽

10.1002/sim.9025 ◽

2021 ◽

Author(s):

Richard D. Riley ◽

Thomas P. A. Debray ◽

Gary S. Collins ◽

Lucinda Archer ◽

Joie Ensor ◽

...

Keyword(s):

Prediction Model ◽

Sample Size ◽

External Validation ◽

Minimum Sample Size ◽

Binary Outcome ◽

Clinical Prediction ◽

Clinical Prediction Model ◽

Minimum Sample

Download Full-text

Develop a Clinical Prediction Model for Postoperative Cognitive Dysfunction after Major Noncardiac Surgery in Elderly Patients: A Protocol for a Prospective Observational Study

Gerontology ◽

10.1159/000517511 ◽

2021 ◽

pp. 1-8

Author(s):

Yang Shen ◽

Xianchen Li ◽

Junyan Yao

Keyword(s):

Observational Study ◽

Prediction Model ◽

Cognitive Decline ◽

Cognitive Dysfunction ◽

Prospective Observational Study ◽

Postoperative Cognitive Dysfunction ◽

Noncardiac Surgery ◽

Clinical Prediction ◽

Clinical Prediction Model ◽

Postoperative Cognitive Decline

Perioperative neurocognitive disorders (PNDs) refer to cognitive decline identified in the preoperative or postoperative period. It has been reported that the incidence of postoperative neurocognitive impairment after noncardiac surgery in patients older than 65 at 1 week was 25.8∼41.4%, and at 3 months 9.9∼12.7%. PNDs will last months or even develop to permanent dementia, leading to prolonged hospital stays, reduced quality of life, and increased mortality within 1 year. Despite the high incidence and poor prognosis of PNDs in the aged population, no effective clinical prediction model has been established to predict postoperative cognitive decline preoperatively. To develop a clinical prediction model for postoperative neurocognitive dysfunction, a prospective observational study (Clinical trial registration number: ChiCTR2000036304) will be performed in the Shanghai General Hospital during January 2021 to October 2022. A sample size of 675 patients aged >65 years old, male or female, and scheduled for elective major noncardiac surgery will be recruited. A battery of neuropsychological tests will be used to test the cognitive function of patients at 1 week, 1 month, and 3 months postoperatively. We will evaluate the associations of PNDs with a bunch of candidate predictors including general characteristics of patients, blood biomarkers, indices associated with anesthesia and surgery, retinal nerve-fiber layer thickness, and frailty index to develop the clinical prediction model by using multiple logistic regression analysis and least absolute shrinkage and the selection operator (LASSO) method. The <i>k</i>-fold cross-validation method will be utilized to validate the clinical prediction model. In conclusion, this study was aimed to develop a clinical prediction model for postoperative cognitive dysfunction of old patients. It is anticipated that the knowledge gained from this study will facilitate clinical decision-making for anesthetists and surgeons managing the aged patients undergoing noncardiac surgery.

Download Full-text

Predicting severe pneumonia in the emergency department: a global study of the Pediatric Emergency Research Networks (PERN)—study protocol

BMJ Open ◽

10.1136/bmjopen-2020-041093 ◽

2020 ◽

Vol 10 (12) ◽

pp. e041093

Author(s):

Todd Adam Florin ◽

Daniel Joseph Tancredi ◽

Lilliam Ambroggio ◽

Franz E Babl ◽

Stuart R Dalziel ◽

...

Keyword(s):

Emergency Department ◽

Prediction Model ◽

Severe Disease ◽

Pediatric Emergency ◽

Global Network ◽

Clinical Prediction ◽

Emergency Research ◽

Research Networks ◽

Clinical Prediction Model ◽

Severity Scores

IntroductionPneumonia is a frequent and costly cause of emergency department (ED) visits and hospitalisations in children. There are no evidence-based, validated tools to assist physicians in management and disposition decisions for children presenting to the ED with community-acquired pneumonia (CAP). The objective of this study is to develop a clinical prediction model to accurately stratify children with CAP who are at risk for low, moderate and severe disease across a global network of EDs.Methods and analysisThis study is a prospective cohort study enrolling up to 4700 children with CAP at EDs at ~80 member sites of the Pediatric Emergency Research Networks (PERN; https://pern-global.com/). We will include children aged 3 months to <14 years with a clinical diagnosis of CAP. We will exclude children with hospital admissions within 7 days prior to the study visit, hospital-acquired pneumonias or chronic complex conditions. Clinical, laboratory and imaging data from the ED visit and hospitalisations within 7 days will be collected. A follow-up telephone or text survey will be completed 7–14 days after the visit. The primary outcome is a three-tier composite of disease severity. Ordinal logistic regression, assuming a partial proportional odds specification, and recursive partitioning will be used to develop the risk stratification models.Ethics and disseminationThis study will result in a clinical prediction model to accurately identify risk of severe disease on presentation to the ED. Ethics approval was obtained for all sites included in the study. Cincinnati Children’s Hospital Institutional Review Board (IRB) serves as the central IRB for most US sites. Informed consent will be obtained from all participants. Results will be disseminated through international conferences and peer-reviewed publications. This study overcomes limitations of prior pneumonia severity scores by allowing for broad generalisability of findings, which can be actively implemented after model development and validation.

Download Full-text