Inconsistencies are common between non-Cochrane systematic reviews and their protocols registered in the International prospective register of systematic reviews (PROSPERO) but were seldom explained

Abstract Background: Protocols of systematic reviews allow for planning and documentation of review methods and thus improve the transparency of the reviews process. However, pre-registration of a protocol is not enough, the author also need to follow it. PROSPERO is an open-access online database for the registration of non-Cochrane systematic reviews. The purpose of this study is to compare published non-Cochrane reviews with their pre-registered protocols on PROSPERO to determine what changes, if any, have been made, and how likely these changes are to impact the quality of systematic review. Methods: This is a retrospective comparative study. We searched for protocols on PROSPERO platform that were registered in 2018 and then selected the protocols that full text have been published as of January 1st 2019. Published full texts were identified through the protocol's final publication citation. Two authors independently compared and identified changes between protocols and systematic reviews and then evaluated the impact (improve, reduce, or unclear) of these changes on the reporting or methodology quality of reviews. Descriptive statistics of percentage (%) and frequency (n) were conducted. Results: We identified 39 pairs, all of which exhibited changes. “Search strategy”(92%, n=36), “data extraction”(90%, n=35), “data synthesis”(77%, n=30), “outcome”(64%, n=24), and “subgroup analysis”(64%, n=24) all showed significant changes. All changes to only one review were considered to improve the reporting or methodology quality, and the remaining 97% of reviews (n=38) contained changes that were considered to reduce the methodology or reporting quality or that had an unclear impact on systematic reviews. Conclusions: Changes between the non-Cochrane systematic reviews and their protocols recorded on PROSPERO were widespread. Some of the changes reduced the methodology or reporting quality of systematic reviews or had an unclear impact. Measures should be taken to further improve the transparency of the non-Cochrane systematic reviews. Adding a new item in updated “Preferred Reporting Items for Systematic reviews and Meta-Analyses” (PRISMA) and “Meta-analysis of Observational Studies in Epidemiology” (MOOSE) to guide reporting and explaining the changes, as well as advising peer reviewers (and editors) to check the reviews against the protocols are two suggested fundamental solutions.

Download Full-text

Inconsistencies are common between non-Cochrane systematic reviews and their protocols registered in the International prospective register of systematic reviews (PROSPERO) but were seldom explained

10.21203/rs.2.18725/v1 ◽

2019 ◽

Author(s):

Kaiyan Hu ◽

Ting Zhang ◽

Weiyi Zhang ◽

Qi Zhou ◽

Joey S.W. Kwong ◽

...

Keyword(s):

Systematic Review ◽

Open Access ◽

Comparative Study ◽

Systematic Reviews ◽

Reporting Quality ◽

Fundamental Solutions ◽

Cochrane Reviews ◽

Retrospective Comparative Study ◽

The Impact

Abstract Background: Protocols of systematic reviews allow for planning and documentation of review methods, and thus improve the transparency of reviews process. However, pre-registered a protocol is not enough, the author also need to follow it. PROSPERO is an open-access online facility for the registration of non-Cochrane systematic reviews. The purpose of our research is to determined what changed were made between non-Cochrane reviews and their protocols in PROSPERO and how likely these changes impacted the quality of systematic review. Method: In this retrospective comparative study we electronically searched for protocols and their corresponding systematic reviews in the PROSPERO platform that were “completed and published” from January to December, 2018. Two reviewers independently identified and classified changes between the protocols and systematic reviews then evaluated the impact (improve/reduce/unclear)of these change on the reporting/methodology quality of reviews. Frequency (n), percentage (%) were used to analyze the number of changes categorically in each review and the distribution of different impact caused by these changes. Results: We identified 39 pre-registered protocols and their reviews, all of which exhibited alterations. All changes to only one review are considered to improve the reporting/methodology quality, and remaining 97% of reviews (n=38) contain changes that are categorically considered to reduce the methodology/reporting quality or that have an unclear impact on reviews. Conclusions: Differences between the non-Cochrane reviews and their protocols recorded in PROSPERO are widespread, and there have been many changes having an unclear impact on the quality of reviews. Guiding the author to report and explain the differences between protocol and reviews or even requiring authors to so at the level of journal are two fundamental solutions to further improve the transparency of the non-Cochrane reviews.

Download Full-text

Quality of Systematic Reviews of the Foods with Function Claims in Japan: Comparative Before- and After-Evaluation of Verification Reports by the Consumer Affairs Agency

Nutrients ◽

10.3390/nu11071583 ◽

2019 ◽

Vol 11 (7) ◽

pp. 1583 ◽

Cited By ~ 1

Author(s):

Kamioka ◽

Tsutani ◽

Origasa ◽

Yoshizaki ◽

Kitayuguchi ◽

...

Keyword(s):

Systematic Reviews ◽

Data Extraction ◽

Meta Analysis ◽

Scientific Evidence ◽

Reporting Quality ◽

Health Claims ◽

Scientific Quality ◽

Before And After ◽

Product Effectiveness

Background: In Japan, a new type of foods with health claims, called Foods with Function Claims (FFC), was introduced in April 2015 in order to make more products available that are clearly labeled with certain health functions. Regarding substantiating product effectiveness, scientific evidence for the proposed function claims must be explained by systematic reviews (SRs), but the quality of SRs was not clear. The objectives of this review were to assess the quality of SRs based on the FFC registered on the Consumer Affairs Agency (CAA) website in Japan, and to determine whether the CAA’s verification report in 2016 was associated with improvement in the quality of SRs. Methods: We evaluated the reporting quality of each SR by the AMSTAR checklist on methodological quality. We searched the database from 1 April to 31 October 2015 as the before-SR and from 1 July 2017 to 31 January 2018 as the after-SR. Results: Among the 104 SRs reviewed, 96 final products were included: 51 (53.1%) were supplements, 42 (43.8%) were processed foods without supplements, and 3 (3.1%) were fresh foods. Of the 104 SRs, 92 (88.5%) were qualitative reviews (i.e., without meta-analysis) and 12 (11.5%) performed a meta-analysis. The average quality score of before-SRs and after-SRs was 6.2 ± 1.8 and 5.0 ± 1.9, respectively, a statistically significant decrease (p < 0.001). Conclusion: Overall, the methodology and reporting quality of after-SRs based on the FFC were poorer than those of before-SRs. In particular, there were very poor descriptions and/or implementations of study selection and data extraction, search strategy, evaluation methods for risk of bias, assessment of publication bias, and formulating conclusions based on methodological rigor and scientific quality of the included studies.

Download Full-text

Reporting quality of systematic review abstracts in articles hand and wrist pathology: a review

Australasian Journal of Plastic Surgery ◽

10.34239/ajops.v2n2.141 ◽

2019 ◽

Vol 2 (2) ◽

pp. 50-57

Author(s):

Amanda Yang Shen ◽

Robert S Ware ◽

Tom J O'Donohoe ◽

Jason Wasiak

Keyword(s):

Systematic Review ◽

Systematic Reviews ◽

English Language ◽

Meta Analysis ◽

Reporting Quality ◽

Cochrane Library ◽

Annual Basis ◽

Cochrane Reviews ◽

Abstract Reporting

Background: An increasing number of systematic reviews are published on an annual basis. Although perusal of the full text of articles is preferable, abstracts are sometimes relied upon to guide clinical decisions. Despite this, the abstracts of systematic reviews have historically been poorly reported. We evaluated the reporting quality of systematic review abstracts within hand and wrist pathology literature. Methods: We searched MEDLINE®, EMBASE and Cochrane Library from inception to December 2017 for systematic reviews in hand and wrist pathology using the 12-item PRISMA-A checklist to assess abstract reporting quality. Results: A total of 114 abstracts were included. Most related to fracture (38%) or arthritis (17%) management. Forty-seven systematic reviews (41%) included meta-analysis. Mean PRISMA-A score was 3.6/12 with Cochrane reviews having the highest mean score and hand-specific journals having the lowest. Abstracts longer than 300 words (mean difference [MD]: 1.43, 95% CI [0.74, 2.13]; p <0.001) and systematic reviews with meta-analysis (MD: 0.64, 95% CI [0.05, 1.22]; p = 0.034) were associated with higher scores. Unstructured abstracts were associated with lower scores (MD: –0.65, 95% CI [–1.28, –0.02]; p = 0.044). A limitation of this study is the possible exclusion of relevant studies that were not published in the English language. Conclusion: Abstracts of systematic reviews pertaining to hand and wrist pathology have been suboptimally reported as assessed by the PRISMA-A checklist. Improvements in reporting quality could be achieved by endorsement of PRISMA-A guidelines by authors and journals, and reducing constraints on abstract length.

Download Full-text

Factors underpinning discrepant findings from reviews on the same topic: A systematic analysis of Cochrane and non-Cochrane reviews

10.1101/574046 ◽

2019 ◽

Author(s):

Claudia Hacke ◽

David Nunan

Keyword(s):

Systematic Reviews ◽

Methodological Quality ◽

Data Extraction ◽

Meta Analysis ◽

List Type ◽

Eligibility Criteria ◽

Cochrane Reviews ◽

And Performance ◽

Meta Analyses

AbstractObjectiveTo explore factors underpinning discrepancies in reported pooled effect estimates from Cochrane and non-Cochrane systematic reviews answering the same question.Study Design and SettingWe observed discrepant pooled effects in 23 out of 24 pairs of meta-analyses from Cochrane and non-Cochrane systematic reviews answering the same question. Here we present the results of a systematic assessment of methodological quality and factors that explain the observed quantitative discrepancies. Methodological quality of each review was assessed using AMSTAR (Assessing the Methodological Quality of Systematic Reviews). Matched pairs were contrasted at the macro- (review methodology), meso- (application of methodology) and micro- (data extraction) level and reasons for differences were derived.ResultsAll Cochrane reviews had high methodological quality (AMSTAR 8-11), whereas the majority (87.5%) of non-Cochrane reviews were classified as moderate (AMSTAR 4-7). Only one pair included exactly the same studies for their respective meta-analyses but there was still a discrepancy in the pooled estimate due to differences in data extraction. One pair did not include any study of its match and for one pair the same effect estimates were reported despite inclusion of different studies. The remaining pairs included at least one study in their match. Due to insufficient reporting (predominantly affecting non-Cochrane reviews) we were only able to completely ascertain the reasons for discrepancies in all included studies for 9/24 (37.5%) pairs. Across all pairs, differences in pre-defined methods (macro-level) including search strategy, eligibility criteria and performance of dual screening could possibly explain mismatches in included studies. Study selection procedures (meso-level) including disagreements in the interpretation of pre-defined eligibility criteria (14 matches) were identified as reasons underpinning discrepant review findings. Comparison of data extraction from primary studies (micro-level) was not possible in 13/24 pairs as a result of the non-Cochrane review providing insufficient details of the studies included in their meta-analyses. Two out of 24 pairs completely agreed on the numerical data presented for the same studies in their respective meta-analysis. Both review types provided sufficient information to check the accuracy of data extraction for 8 pairs (45 studies) where there were discrepancies. An assessment of 50% (22 studies) of these showed that reasons for differences in extracted data could be identified in 15 studies. We found examples for both types of review where data presented were discrepant from that given in the source study without a plausible explanation.ConclusionMethodological and author judgements and performance are key aspects underpinning poor overlap of included studies and discrepancies in reported pooled effect estimates between topic-matched reviews. Though caution must be taken when extrapolating, our findings raise the question as to what extent the entire meta-analysis evidence-base accurately reflects the available primary research both in terms of volume and data. Reinforcing awareness of the application of guidelines for systematic reviews and meta-analyses may help mitigate some of the key issues identified in our analysis.What is new?Key findings Non-Cochrane reviews were of a lower overall methodological quality compared with Cochrane reviews. Discrepant results of meta-analyses on the same topic can be attributed to differences in included studies based on review author decision, judgements and performance at different stages of the review process.What this adds to what was known?This study provides the most robust analysis to date of the potential methodological factors underpinning discrepant review findings between matched meta-analyses answering the same question. Assessing differences between reviews at the macro-, meso-, and micro-levels is a useful method to identify reasons for discrepant meta-analyses at key stages of the review process.What is the implication and what should change now?There is a need for a standardised approach to performing matched-pair analysis of meta-analyses and systematic reviews answering the same question. Our paper provides a base for this that can be refined by replication and expert consensus.

Download Full-text

Systematic Reviews and Meta-analysis Published in Indexed Portuguese Medical Journals: Time Trends and Critical Appraisal.

10.21203/rs.3.rs-354547/v1 ◽

2021 ◽

Author(s):

Luísa Prada ◽

Ana Prada ◽

Miguel Antunes ◽

Ricardo Fernandes ◽

João Costa ◽

...

Keyword(s):

Systematic Reviews ◽

Critical Appraisal ◽

Data Extraction ◽

Evidence Synthesis ◽

Meta Analysis ◽

Measurement Tool ◽

Medical Journals ◽

Electronic Search ◽

Potential Impact

Abstract Introduction:Over the last years, the number of systematic reviews published is steadily increasing due to the global interest in this type of evidence synthesis. However, little is known about the characteristics of this research published in Portuguese medical journals. This study aims to evaluate the publication trends and overall quality of these systematic reviews.Material and Methods:Systematic reviews were identified through an electronic search up to August 2020, targeting Portuguese Medical journals indexed in MEDLINE. Systematic reviews selection and data extraction were done independently by three authors. The overall quality critical appraisal using the A MeaSurement Tool to Assess systematic Reviews (AMSTAR II) was independently assessed by three authors. Disagreements were solved by consensus.Results:Seventy systematic reviews published in 5 Portuguese medical journals were included. Most (n=57; 81,4%) were systematic reviews without meta-analysis. Until 2010, the number of systematic reviews per year increased. Since then, the number of reviews published has not remained stable and no less than 3 SRs were published per year. According to the systematic reviews’ typology, most have been predominantly conducted to assess the effectiveness of health interventions (n=28; 40,0%). General and Internal Medicine (n=26; 37,1%) was the most addressed field. Most systematic reviews (n=45; 64,3%) were rated as being of “critically low-quality”.Conclusions:There were consistent flaws in the methodological quality report of the systematic reviews included, particularly in establishing a prior protocol and not assessing the potential impact of the risk of bias on the results.Through the years, the number of systematic reviews published increased, yet their quality is suboptimal. There is a need to improve the reporting of systematic reviews in Portuguese medical journals, which can be achieved by better adherence to quality checklists/tools.Systematic review registration: INPLASY202090105

Download Full-text

Outcomes of digital biomarker-based interventions: protocol for a systematic review of systematic reviews (Preprint)

10.2196/preprints.28204 ◽

2021 ◽

Author(s):

Hossein Motahari-Nezhad ◽

Márta Péntek ◽

László Gulácsi ◽

Zsombor Zrubka

Keyword(s):

Clinical Outcomes ◽

Systematic Reviews ◽

Meta Analysis ◽

Cochrane Library ◽

Level Of Evidence ◽

Clinical Benefits ◽

The Cochrane Library ◽

The Impact ◽

Digital Biomarkers

BACKGROUND Digital biomarkers are defined as objective, quantifiable physiological and behavioral data that are collected and measured by means of digital devices such as portables, wearables, implantables or digestibles. For their widespread adoption in publicly financed healthcare systems, it is important to understand how their benefits translate into improved patient outcomes, which is essential for demonstrating their value. OBJECTIVE To assess the quality and strength of evidence of the impact of digital biomarkers on clinical outcomes compared to interventions without digital biomarkers, reported in systematic reviews. METHODS A comprehensive search for 2019-2020 will be conducted in the PubMed and the Cochrane Library using keywords related to digital biomarkers and a filter for systematic reviews. Original full-text English publications of systematic reviews comparing clinical outcomes of interventions with and without digital biomarkers via meta-analysis will be included. The AMSTAR-2 tool will be used to assess the methodological quality of reviews. To assess the quality of evidence, we will evaluate systematic reviews using the GRADE tool. To detect the possible presence of reporting bias, we will record whether the protocol of the systematic reviews was published before the start of the study. A qualitative summary of results by digital biomarker technology and outcome will be provided. RESULTS This protocol was submitted before data collection. The next steps in this review will be initiated after the protocol is accepted for publication. CONCLUSIONS Our study will provide a comprehensive summary of the highest level of evidence available on digital biomarker interventions. Our results will help identify clinical areas where the use of digital biomarkers leads to favorable clinical outcomes. In addition, our findings will highlight areas of evidence gaps where the clinical benefits of digital biomarkers have not yet been demonstrated.

Download Full-text

The collaboration and reporting quality of social welfare systematic reviews in the Campbell Collaboration online library

Health and Quality of Life Outcomes ◽

10.1186/s12955-019-1241-7 ◽

2019 ◽

Vol 17 (1) ◽

Author(s):

Li-li Wei ◽

Jing Zhang ◽

Ying Yang ◽

Hao-Yu Cao ◽

Ke-hu Yang ◽

...

Keyword(s):

Social Network ◽

Social Welfare ◽

Systematic Reviews ◽

Meta Analysis ◽

Reporting Quality ◽

Reporting Rate ◽

Campbell Collaboration ◽

The Social ◽

Online Library

Abstract Background To analyze the collaboration and reporting quality of the systematic reviews of social welfare in the Campbell collaboration online library. Methods The Campbell collaboration online library was searched for systematic reviews of social welfare and the basic information extracted in order to assess the reporting quality of systematic reviews using a MOOSE checklist. BICOMS-2 and UCINET software were used to produce the social network, and Comprehensive Meta Analysis (Version 2) and STATA 13.0 were used to analyze the related data. Results Fifty-seven systematic reviews of social welfare were included. Twenty-eight items of the included social welfare systematic reviews were rated as complete (≥70%). There were significant differences between ≤2013 and ≥ 2014 in five items. These differences were as follows: research published by one organization or more than one organization in one item, more than three authors or less than four authors in two items, and one country or more than one country in six items. It’s completed about researches with more than one organization, three authors or more than one country. Some items were found to have a low reporting rate of studies published before 2014, by one organization, with less than four authors or one country, respectively. The social network of authors and organizations showed good collaboration. Conclusions Some items could be further improved with regard to the rate of reporting systematic reviews of social welfare in the Campbell collaboration online library. This could improve the overall quality of social welfare systematic reviews.

Download Full-text

The Impact of Financial Incentives on Physical Activity: A Systematic Review and Meta-Analysis

American Journal of Health Promotion ◽

10.1177/0890117120940133 ◽

2020 ◽

pp. 089011712094013

Author(s):

My-Linh Nguyen Luong ◽

Michelle Hall ◽

Kim L. Bennell ◽

Jessica Kasza ◽

Anthony Harris ◽

...

Keyword(s):

Physical Activity ◽

Financial Incentives ◽

Leisure Time ◽

Data Extraction ◽

Meta Analysis ◽

Random Effects Models ◽

Walking Behavior ◽

The Impact ◽

Incentive Strategy

Objective: To evaluate the effects of financial incentives on physical activity (PA). Data Sources: MEDLINE, Embase, 7 other databases, and 2 trial registries until July 17, 2019. Study Inclusion and Exclusion Criteria: Randomized controlled trials with adults aged ≥18 years assessing the effect of financial incentives on PA. Any comparator was eligible provided the only difference between groups was the incentive strategy. Data Extraction: Two independent reviewers extracted data and assessed study quality. Of 5765 records identified, 57 records (51 unique trials; n = 17 773 participants) were included. Data Synthesis: Random-effects models pooling data for each of the 5 PA domains. Results: Financial incentives increase leisure time PA (gym or class attendance; standardized mean difference [95% CI], 0.46 [0.28-0.63], n = 5057) and walking behavior (steps walked; 0.25 [0.13-0.36], n = 3254). No change in total minutes of PA (0.52 [−0.09 to 1.12], n = 968), kilocalories expended (0.19 [−0.06 to 0.44], n = 247), or the proportion of participants meeting PA guidelines (risk ratio [95% CI] 1.53 [0.53-4.44], n = 650) postintervention was observed. After intervention has ceased, incentives sustain a slight increase in leisure time PA (0.10 [0.02-0.18], n = 2678) and walking behavior (0.11 [0.00-0.22], n = 2425). Conclusions: Incentives probably improve leisure time PA and walking at intervention end, and small improvements may be sustained over time once incentives have ceased. They lead to little or no difference in kilocalories expended or minutes of PA. It is uncertain whether incentives change the likelihood of meeting PA guidelines because the certainty of the evidence is low.

Download Full-text

A systematic review to explore the impact of anxiety on quality of life and treatment outcomes in rheumatoid arthritis

British Journal of General Practice ◽

10.3399/bjgp18x697181 ◽

2018 ◽

Vol 68 (suppl 1) ◽

pp. bjgp18X697181

Author(s):

Annabelle Machin ◽

Randula Haththotuwa ◽

Opeyemi Babatunde ◽

Ian Scott ◽

Nadia Corp ◽

...

Keyword(s):

Quality Of Life ◽

Rheumatoid Arthritis ◽

Systematic Review ◽

Treatment Outcomes ◽

Treatment Response ◽

Data Extraction ◽

Meta Analysis ◽

Past Research ◽

The Impact

BackgroundRheumatoid arthritis (RA) is the commonest inflammatory arthritis. Comorbid anxiety and depression are common, but under-recognised and under-treated. Depression in RA has been linked to reduced quality of life and treatment response and increased mortality. Past research into co-morbid mood problems in RA has mainly focused on the prevalence and impact of depression.AimTo complete a systematic review to explore the impact of anxiety on QOL and treatment outcomes in RA.MethodThe systematic review has been registered on PROSPERO. Databases (Web of Science, PsycINFO, CINAHL, Embase and Medline) have been systematically searched for relevant studies. Titles and abstracts have been independently screened by two reviewers. Data extraction and quality appraisal of studies will be carried out using a customized and piloted data extraction tool and the Newcastle-Ottawa Scale. Disagreements will be resolved through discussion or by a third reviewer. A narrative synthesis framework will be used to develop a theory of how anxiety impacts on QOL and treatment response and a meta-analysis performed.ResultsThe search has identified 6404 articles. After title screening, 284 abstracts have been reviewed and 48 full texts are being reviewed for inclusion. Full text review is ongoing, and results of the analysis will be presented.ConclusionHighlighting the impact of anxiety on outcomes in RA may suggest areas for future implementation studies to improve patient outcomes, potentially reducing associated morbidity and mortality.

Download Full-text

Impact and use of reviews and ‘overviews of reviews’ to inform clinical practice guideline recommendations: protocol for a methods study

BMJ Open ◽

10.1136/bmjopen-2019-031442 ◽

2020 ◽

Vol 10 (1) ◽

pp. e031442

Author(s):

Carole Lunny ◽

Cynthia Ramasubbu ◽

Savannah Gerrish ◽

Tracy Liu ◽

Douglas M Salzwedel ◽

...

Keyword(s):

Clinical Practice ◽

Systematic Reviews ◽

Clinical Practice Guideline ◽

Practice Guideline ◽

Data Extraction ◽

Meta Analysis ◽

Assessment Development ◽

Evaluation Approach ◽

Meta Analyses ◽

The Impact

IntroductionGuidelines are systematically developed recommendations to assist practitioner and patient decisions about treatments for clinical conditions. High quality and comprehensive systematic reviews and ‘overviews of systematic reviews’ (overviews) represent the best available evidence. Many guideline developers, such as the WHO and the Australian National Health and Medical Research Council, recommend the use of these research syntheses to underpin guideline recommendations. We aim to evaluate the impact and use of systematic reviews with and without pairwise meta-analysis or network meta-analyses (NMAs) and overviews in clinical practice guideline (CPG) recommendations.Methods and analysisCPGs will be retrieved from Turning Research Into Practice and Epistemonikos (2017–2018). The retrieved citations will be sorted randomly and then screened sequentially by two independent reviewers until 50 CPGs have been identified. We will include CPGs that provide at least two explicit recommendations for the management of any clinical condition. We will assess whether reviews or overviews were cited in a recommendation as part of the development process for guidelines. Data extraction will be done independently by two authors and compared. We will assess the risk of bias by examining how each guideline developed clinical recommendations. We will calculate the number and frequency of citations of reviews with or without pairwise meta-analysis, reviews with NMAs and overviews, and whether they were systematically or non-systematically developed. Results will be described, tabulated and categorised based on review type (reviews or overviews). CPGs reporting the use of the Grading of Recommendations, Assessment, Development and Evaluation approach will be compared with those using a different system, and pharmacological versus non-pharmacological CPGs will be compared.Ethics and disseminationNo ethics approval is required. We will present at the Cochrane Colloquium and the Guidelines International Network conference.

Download Full-text