Completeness of reporting of clinical prediction models developed using supervised machine learning: a systematic review

Background While many studies have consistently found incomplete reporting of regression-based prediction model studies, evidence is lacking for machine learning-based prediction model studies. We aim to systematically review the adherence of Machine Learning (ML)-based prediction model studies to the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) Statement. Methods We included articles reporting on development or external validation of a multivariable prediction model (either diagnostic or prognostic) developed using supervised ML for individualized predictions across all medical fields. We searched PubMed from 1 January 2018 to 31 December 2019. Data extraction was performed using the 22-item checklist for reporting of prediction model studies (www.TRIPOD-statement.org). We measured the overall adherence per article and per TRIPOD item. Results Our search identified 24,814 articles, of which 152 articles were included: 94 (61.8%) prognostic and 58 (38.2%) diagnostic prediction model studies. Overall, articles adhered to a median of 38.7% (IQR 31.0–46.4%) of TRIPOD items. No article fully adhered to complete reporting of the abstract and very few reported the flow of participants (3.9%, 95% CI 1.8 to 8.3), appropriate title (4.6%, 95% CI 2.2 to 9.2), blinding of predictors (4.6%, 95% CI 2.2 to 9.2), model specification (5.2%, 95% CI 2.4 to 10.8), and model’s predictive performance (5.9%, 95% CI 3.1 to 10.9). There was often complete reporting of source of data (98.0%, 95% CI 94.4 to 99.3) and interpretation of the results (94.7%, 95% CI 90.0 to 97.3). Conclusion Similar to prediction model studies developed using conventional regression-based techniques, the completeness of reporting is poor. Essential information to decide to use the model (i.e. model specification and its performance) is rarely reported. However, some items and sub-items of TRIPOD might be less suitable for ML-based prediction model studies and thus, TRIPOD requires extensions. Overall, there is an urgent need to improve the reporting quality and usability of research to avoid research waste. Systematic review registration PROSPERO, CRD42019161764.

Radiology reporting in rectal cancer using MRI: adherence to national template for structured reporting

Background In 2014, a national workshop program was initiated and a reporting template and manual for rectal cancer primary staging using magnetic resonance imaging (MRI) was introduced and made available by the national Swedish Colorectal Cancer Registry. Purpose To evaluate the effect of the national template program by identify if there was a gap between the content in Swedish MRI reports from 2016 and the national reporting template from 2014. The aim was to explore and compare differences in content in reporting practice in different hospitals in relation to the national reporting template, with focus on: (i) identifying any implementational differences in reporting styles; and (ii) evaluating if reporting completeness vary based on such implementational differences. Material and Methods A total of 250 MRI reports from 10 hospitals in four healthcare regions in Sweden were collected. Reports were analyzed using qualitative content analysis with a deductive thematic coding scheme based on the national reporting template. Results Three different implemented reporting styles were identified with variations of content coverage in relation to the template: (i) standardized and structured protocol (reporting style A); (ii) standardized semi-structured free-text (reporting style B); and (iii) regular free-text (reporting style C). The relative completeness of reporting practice of rectal cancer staging in relation to the national reporting template were 92.9% for reporting style A, 77.5% for reporting style B, and 63.9% for reporting style C. Conclusion The implementation of template-based reporting according to reporting style A is a key factor to conform to evidence-based practice for rectal cancer reporting using MRI.

Capture - recapture based study on the completeness of smear positive pulmonary tuberculosis reporting in southwest Iran during 2016

Background Tuberculosis (TB) is one of the ten leading causes of death in infectious diseases and one of the ten leading causes of death in the world. For any TB control program, a valid surveillance is essential. In order to assess the status of the assessment, the quality of the record and the completeness of reporting should be assessed. The purpose of this study was to investigate the completeness of smear positive pulmonary tuberculosis reporting in Ahvaz, south west of Iran. Methods This cross-sectional study was conducted in 2016 in Ahvaz, southwest Iran. The study was conducted through a three-source Capture recapture method by collecting laboratory, hospital, physician prescription data; including patient referral to the health care center, prescriptions of patients receiving anti-tuberculosis drugs and prescriptions of medical TB diagnostic laboratories, and laboratory prescriptions. Percentage, mean and standard deviation were used to describe the variables. Data analysis was performed using log-linear model in Rcapture package R software. Results Generally, 134 new cases of smear-positive pulmonary tuberculosis patients were reported through three sources from urban and rural regions during 2016. Pulmonary tuberculosis was reported through three sources from urban and rural regions during 2016. The most common age group was 25 to 44 years and 79.1% of the patient were man. The overall prevalence of new cases of smear-positive pulmonary tuberculosis was in persons that lived urban areas (97.8%). The completeness of reporting the disease estimated by log-linear model was 87.5% and the incidence rate was estimated to be 11.8 disease per 100,000 persons. Completeness of reporting of laboratory, hospital and physician resources were 79%, 30% and 16.3%, respectively. Conclusions The present study shows the necessity of evaluating the quality, completeness and linkage between data. Linking between data sources can improve the accuracy and completeness of TB surveillance.

A Systematic Review of Cost-Effectiveness Analyses of Novel Agents in the Treatment of Multiple Myeloma

Background: Novel therapies for multiple myeloma (MM) promise to improve outcomes but are also associated with substantial increasing costs. Evidence regarding cost-effectiveness of novel treatments is necessary, but a comprehensive up-to-date overview of the cost-effectiveness evidence of novel treatments is currently lacking. Methods: We searched Embase, Medline via Ovid, Web of Science and EconLIT ProQuest to identify all cost-effectiveness evaluations of novel pharmacological treatment of MM reporting cost per quality-adjusted life year (QALY) and cost per life year (LY) gained since 2005. Quality and completeness of reporting was assessed using the Consolidated Health Economic Evaluation Reporting Standards. Results: We identified 13 economic evaluations, comprising 32 comparisons. Our results show that novel agents generate additional LYs (range: 0.311–3.85) and QALYs (range: 0.1–2.85) compared to backbone regimens and 0.02 to 1.10 LYs and 0.01 to 0.91 QALYs for comparisons between regimens containing two novel agents. Lifetime healthcare costs ranged from USD 60,413 to 1,434,937 per patient. The cost-effectiveness ratios per QALY gained ranged from dominating to USD 1,369,062 for novel agents compared with backbone therapies and from dominating to USD 618,018 for comparisons between novel agents. Conclusions: Cost-effectiveness ratios of novel agents were generally above current willingness-to-pay thresholds. To ensure access, cost-effectiveness should be improved or cost-effectiveness ratios above current thresholds should be accepted.

Salmonella in Animal Feeds: A Scoping Review

The objective of this study was to describe the volume and nature of published literature on Salmonella in animal feeds using a formal scoping review methodology. A structured search followed by eligibility screening resulted in the identification of 547 relevant studies, encompassing studies conducted in the fields in which animal feeds are grown (15 studies), the manufacturing sector (106), during transportation (11), in the retail sector (15), and on-farm (226), with the sector not described for 204 studies. The most common study purposes were to estimate the prevalence of Salmonella in animal feeds (372 studies) and to identify serovars (195). The serovars that were found in animal feeds included serovars associated with human illness, with animal illness, and with serovars identified in food (livestock and poultry) intended for human consumption. There were 120 intervention studies and 83 studies conducted to evaluate potential risk factors. Within intervention and risk factor studies, there may be sufficient depth to warrant synthesis research in the areas of heat interventions, fermentation and ensiling, organic acids, season, and geographic region. Some deficiencies were identified in the completeness of reporting of key features in the relevant studies.

Evolutions in rectal cancer MRI staging and risk stratification in The Netherlands

Purpose To analyze how the MRI reporting of rectal cancer has evolved (following guideline updates) in The Netherlands. Methods Retrospective analysis of 712 patients (2011–2018) from 8 teaching hospitals in The Netherlands with available original radiological staging reports that were re-evaluated by a dedicated MR expert using updated guideline criteria. Original reports were classified as “free-text,” “semi-structured,” or “template” and completeness of reporting was documented. Patients were categorized as low versus high risk, first based on the original reports (high risk = cT3-4, cN+, and/or cMRF+) and then based on the expert re-evaluations (high risk = cT3cd-4, cN+, MRF+, and/or EMVI+). Evolutions over time were studied by splitting the inclusion period in 3 equal time periods. Results A significant increase in template reporting was observed (from 1.6 to 17.6–29.6%; p < 0.001), along with a significant increase in the reporting of cT-substage, number of N+ and extramesorectal nodes, MRF invasion and tumor-MRF distance, EMVI, anal sphincter involvement, and tumor morphology and circumference. Expert re-evaluation changed the risk classification from high to low risk in 18.0% of cases and from low to high risk in 1.7% (total 19.7%). In the majority (17.9%) of these cases, the changed risk classification was likely (at least in part) related to use of updated guideline criteria, which mainly led to a reduction in high-risk cT-stage and nodal downstaging. Conclusion Updated concepts of risk stratification have increasingly been adopted, accompanied by an increase in template reporting and improved completeness of reporting. Use of updated guideline criteria resulted in considerable downstaging (of mainly high-risk cT-stage and nodal stage). Graphic abstract

Endocrine Disrupting Chemicals and Risk of Testicular Cancer A Systematic Review and Meta-analysis

The incidence of many hormone-dependent diseases, including testicular cancer, have sharply increased in all high-income countries during the 20th century. This is not fully explained by established risk factors. Concurrent, increasing exposure to antiandrogenic environmental endocrine disrupting chemicals (EDCs) in fetal life may partially explain this trend. This systematic review assessed available evidence regarding the association between environmental EDC exposure and risk of testicular cancer (seminomas and non-seminomas). Following PRISMA guidelines, a search of English peer-reviewed literature published prior to December 14 th, 2020, in the databases PubMed and Embase® was performed. Among the 279 identified records, 19 were eligible for quality assessment and 10 for further meta-analysis. The completeness of reporting was high across papers, but over 50% were considered subject to potential risk of bias. Mean age at diagnosis was 31.9 years. None considered effects of EDCs multipollutant mixtures. The meta-analyses showed that maternal exposure to combined EDCs was associated with a higher risk of testicular cancer in male offspring (summary RRs: 2.16, (95% CI:1.78-2.62); 1.93 (95% CI:1.49-2.48); 2.78 (95% CI:2.27-3.41) for all, seminoma, non-seminoma respectively). Similarly, high maternal exposures to grouped organochlorines and organo-halogens were associated with higher risk of seminoma and non-seminoma in the offspring. Summary estimates related to postnatal adult male EDC exposures were inconsistent.Maternal but not postnatal adult male, EDC exposures were consistently associated with a higher risk of testicular cancer, particularly risk of non-seminomas. However, the quality of studies was mixed and considering the fields complexity, more prospective studies of prenatal EDC multipollutant mixture exposures and testicular cancer are needed.

Completeness of reporting of clinical prediction models developed using supervised machine learning: A systematic review

ABSTRACT Objective. While many studies have consistently found incomplete reporting of regression-based prediction model studies, evidence is lacking for machine learning-based prediction model studies. Our aim is to systematically review the adherence of Machine Learning (ML)-based prediction model studies to the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) Statement. Study design and setting: We included articles reporting on development or external validation of a multivariable prediction model (either diagnostic or prognostic) developed using supervised ML for individualized predictions across all medical fields (PROSPERO, CRD42019161764). We searched PubMed from 1 January 2018 to 31 December 2019. Data extraction was performed using the 22-item checklist for reporting of prediction model studies (www.TRIPOD-statement.org). We measured the overall adherence per article and per TRIPOD item. Results: Our search identified 24 814 articles, of which 152 articles were included: 94 (61.8%) prognostic and 58 (38.2%) diagnostic prediction model studies. Overall, articles adhered to a median of 38.7% (IQR 31.0-46.4) of TRIPOD items. No articles fully adhered to complete reporting of the abstract and very few reported the flow of participants (3.9%, 95% CI 1.8 to 8.3), appropriate title (4.6%, 95% CI 2.2 to 9.2), blinding of predictors (4.6%, 95% CI 2.2 to 9.2), model specification (5.2%, 95% CI 2.4 to 10.8), and model's predictive performance (5.9%, 95% CI 3.1 to 10.9). There was often complete reporting of source of data (98.0%, 95% CI 94.4 to 99.3) and interpretation of the results (94.7%, 95% CI 90.0 to 97.3). Conclusion. Similar to studies using conventional statistical techniques, the completeness of reporting is poor. Essential information to decide to use the model (i.e. model specification and its performance) is rarely reported. However, some items and sub-items of TRIPOD might be less suitable for ML-based prediction model studies and thus, TRIPOD requires extensions. Overall, there is an urgent need to improve the reporting quality and usability of research to avoid research waste.