scholarly journals Reporting of methods to prepare, pilot and perform data extraction in systematic reviews: analysis of a sample of 152 Cochrane and non-Cochrane reviews

2021 ◽  
Vol 21 (1) ◽  
Author(s):  
Roland Brian Büchter ◽  
Alina Weise ◽  
Dawid Pieper
Keyword(s):  
Systematic Reviews ◽  
Data Extraction ◽  
Extraction Methods ◽  
Data Errors ◽  
Cochrane Reviews ◽  
Study Selection ◽  
Data Extraction Form ◽  
Current Sample ◽  
Automation Tools ◽  
Insight Into

Abstract Background Previous research on data extraction methods in systematic reviews has focused on single aspects of the process. We aimed to provide a deeper insight into these methods by analysing a current sample of reviews. Methods We included systematic reviews of health interventions in humans published in English. We analysed 75 Cochrane reviews from May and June 2020 and a random sample of non-Cochrane reviews published in the same period and retrieved from Medline. We linked reviews with protocols and study registrations. We collected information on preparing, piloting, and performing data extraction and on use of software to assist review conduct (automation tools). Data were extracted by one author, with 20% extracted in duplicate. Data were analysed descriptively. Results Of the 152 included reviews, 77 reported use of a standardized extraction form (51%); 42 provided information on the type of form used (28%); 24 on piloting (16%); 58 on what data was collected (38%); 133 on the extraction method (88%); 107 on resolving disagreements (70%); 103 on methods to obtain additional data or information (68%); 52 on procedures to avoid data errors (34%); and 47 on methods to deal with multiple study reports (31%). Items were more frequently reported in Cochrane than non-Cochrane reviews. The data extraction form used was published in 10 reviews (7%). Use of software was rarely reported except for statistical analysis software and use of RevMan and GRADEpro GDT in Cochrane reviews. Covidence was the most frequent automation tool used: 18 reviews used it for study selection (12%) and 9 for data extraction (6%). Conclusions Reporting of data extraction methods in systematic reviews is limited, especially in non-Cochrane reviews. This includes core items of data extraction such as methods used to manage disagreements. Few reviews currently use software to assist data extraction and review conduct. Our results can serve as a baseline to assess the uptake of such tools in future analyses.

PP65 Methods Applied For Systematic Reviews Of Economic Evaluations In Health Technology Assessment

2019 ◽  
Vol 35 (S1) ◽  
pp. 49-50
Author(s):  
Miriam Luhnen ◽  
Barbara Prediger ◽  
Edmund A.M. Neugebauer ◽  
Tim Mathes
Keyword(s):  
Health Technology Assessment ◽  
Systematic Reviews ◽  
Technology Assessment ◽  
Data Extraction ◽  
Reporting Quality ◽  
Health Technology ◽  
Economic Evaluations ◽  
Study Selection ◽  
Data Extraction Form ◽  
Literature Search Strategy

IntroductionWhen making decisions in health care, it is essential to consider economic evidence about an intervention. The objective of this study was to analyze the methods applied for systematic reviews of economic evaluations in Health Technology Assessment (HTA) and to identify common challenges.MethodsWe manually searched the webpages of HTA organizations and included HTA-reports published since 2015. Prerequisites for inclusion were the conduct of a systematic review of economic evaluations in at least one electronic database and the use of the English, German, French, or Spanish language. Methodological features were extracted in standardized tables. We prepared descriptive statistical (e.g., median, range) measures to describe the applied methods. Data were synthesized in a structured narrative way.ResultsEighty-three reports were included in the analysis. We identified inexplicable heterogeneity, particularly concerning literature search strategy, data extraction, assessment of quality, and applicability. Furthermore, process steps were often missing or reported in a nontransparent way. The use of a standardized data extraction form was indicated in one-third of reports (32 percent). Fifty-four percent of authors systematically appraised included studies. In 10 percent of reports, the applicability of included studies was assessed. Involvement of two reviewers was rarely reported for the study selection (43 percent), data extraction (28 percent), and quality assessment (39 percent).ConclusionsThe methods applied for systematic reviews of economic evaluations in HTA and their reporting quality are very heterogeneous. Efforts toward a detailed, standardized guidance for the preparation of systematic reviews of economic evaluations definitely seem necessary. A general harmonization and improvement of the applied methodology would increase their value for decision makers.

SYSTEMATIC REVIEWS OF ECONOMIC EVALUATIONS IN HEALTH TECHNOLOGY ASSESSMENT: A REVIEW OF CHARACTERISTICS AND APPLIED METHODS

2018 ◽  
Vol 34 (6) ◽  
pp. 537-546 ◽  
Author(s):  
Miriam Luhnen ◽  
Barbara Prediger ◽  
Edmund A.M. Neugebauer ◽  
Tim Mathes
Keyword(s):  
Systematic Reviews ◽  
Search Strategy ◽  
Data Extraction ◽  
Reporting Quality ◽  
Decision Makers ◽  
Economic Evaluations ◽  
Web Pages ◽  
Study Selection ◽  
Data Extraction Form ◽  
Literature Search Strategy

Objectives:When making decisions in health care, it is essential to consider economic evidence about an intervention. The objective of this study was to analyze the methods applied for systematic reviews of health economic evaluations (SR-HEs) in HTA and to identify common challenges.Methods:We manually searched the Web pages of HTA organizations and included HTA-reports published since 2015. Prerequisites for inclusion were the conduct of an SR-HE in at least one electronic database and the use of the English, German, French, or Spanish language. Methodological features were extracted in standardized tables. We prepared descriptive statistical (e.g., median, range) measures to describe the applied methods. Data were synthesized in a structured narrative way.Results:Eighty-three reports were included in the analysis. We identified inexplicable heterogeneity, particularly concerning literature search strategy, data extraction, assessment of quality, and applicability. Furthermore, process steps were often missing or reported in a nontransparent way. The use of a standardized data extraction form was indicated in one-third of reports (32 percent). Fifty-four percent of authors systematically appraised included studies. In 10 percent of reports, the applicability of included studies was assessed. Involvement of two reviewers was rarely reported for the study selection (43 percent), data extraction (28 percent), and quality assessment (39 percent).Conclusions:The methods applied for SR-HEs in HTA and their reporting quality are very heterogeneous. Efforts toward a detailed, standardized guidance for the preparation of SR-HEs definitely seem necessary. A general harmonization and improvement of the applied methodology would increase the value of SR-HE for decision makers.

scholarly journals Reporting of the Role of the Expert Searcher in Cochrane Reviews

2006 ◽  
Vol 1 (4) ◽  
pp. 3 ◽  
Author(s):  
Li Zhang ◽  
Margaret Sampson ◽  
Jessie McGowan
Keyword(s):  
Systematic Reviews ◽  
Search Strategy ◽  
Data Extraction ◽  
Cochrane Library ◽  
Eligibility Criteria ◽  
Statistical Decision ◽  
Cochrane Reviews ◽  
Statistical Decisions ◽  
Search Quality

Introduction - This study applied the principles of evidence based information practice to clarify the role of information specialists and librarians in the preparation of Cochrane systematic reviews and to determine whether information specialists impact the quality of searching in Cochrane systematic reviews. Objectives - This research project sought to determine how the contribution of the person responsible for searching in the preparation of Cochrane systematic reviews was reported; whether the contribution was recognized through authorship or acknowledgement; the qualifications of the searcher; and the association between the type of contributorship and characteristics of the search strategy, assessability, and the presence of certain types of errors. Methods - Data sources: The Cochrane Database of Systematic Reviews, The Cochrane Library 3 (2002). Inclusion criteria: The study included systematic reviews that met the following criteria: one or more sections of the Cochrane Highly Sensitive Search Strategy were utilised, primary studies were either randomised controlled trials (RCTs) or quasi-RCTs, and included and excluded studies were clearly identified. Data extraction: Two librarians assessed the searches for errors, establishing consensus on discordant ratings. Results - Of the 169 reviews screened for this project, 105 met all eligibility criteria. Authors fulfilled the searching role in 41.9% of reviews studied, acknowledged persons or groups in 13.3%, a combination in 9.5%, and the role was not reported in 35.2% of reviews. For the 78 reviews in which meta-analyses were performed, the positions of those responsible for statistical decisions were examined for comparative purposes. The statistical role was performed by an author in 47.4% of cases and unreported in the same number of cases. Insufficient analyzable data was obtained regarding professional qualifications (3/105 for searching, 2/78 for statistical decisions). Search quality was assessed for 66 searches across 74 reviews. In general, it was more possible to assess the search quality when the searcher role was reported. An association was found between the reporting of searcher role and the presence of a consequential error. There was no association between the number of consequential errors and how the contribution of the searcher was reported. Conclusions - Qualifications of the persons responsible for searching and statistical decision-making were poorly reported in Cochrane reviews, but more complete role reporting is associated with greater assessability of searches and fewer substantive errors in search strategies.

Pharmacological Treatment for Obesity in Adults: An Umbrella Review

2020 ◽  
Vol 54 (7) ◽  
pp. 691-705 ◽  
Author(s):  
Hanan Khalil ◽  
Laura Ellwood ◽  
Heidi Lord ◽  
Ritin Fernandez
Keyword(s):  
Weight Loss ◽  
Systematic Reviews ◽  
Data Extraction ◽  
Fasting Blood Glucose ◽  
Pharmacological Therapy ◽  
Umbrella Review ◽  
Data Synthesis ◽  
Study Selection ◽  
The Us ◽  
Key Terms

Objective: To synthesize the evidence from systematic reviews of clinical trials investigating the effectiveness of pharmacological therapies approved by the Australian Therapeutic Goods Administration and the US Food and Drug Administration for the management of obesity in adults. Data Sources: A 3-step literature search of the MEDLINE, EMBASE, CINAHL, and PubMed databases was conducted between March and May 2019. The key terms used were obesity, pharmacological therapy, antiobesity agent, antiobesity medication, weight loss, and systematic review. Study Selection and Data Extraction: Systematic reviews that evaluated the effectiveness of pharmacological therapies for the management of obesity in patients with a body mass index of or greater than 25 kg/m2. Data Synthesis: Nine systematic reviews involving three pharmacotherapies, liraglutide, orlistat, and naltrexone-bupropion were identified. The results indicate that the pharmacotherapies reduced weight when compared with placebo. Orlistat was effective in significantly reducing fasting blood glucose, HbA1c, total cholesterol, triglycerides, and systolic and diastolic blood pressure. All reviews discussed the presence or risk of gastrointestinal adverse effects including diarrhea, vomiting, and nausea related to orlistat and liraglutide. Relevance to Patient Care and Clinical Practice: This umbrella review compares the efficacy and safety of antiobesity medications for reducing weight and a discussion on their weight loss and metabolic control to guide clinicians when prescribing medications for obesity. Conclusions: All pharmacological therapies included in this review are superior to placebo in reducing weight. Clinicians should consider patient comorbidities and risk of adverse events when recommending medications for weight loss.

scholarly journals Quality assessment of systematic reviews and meta-analyses that examine preventive antibiotic uses and management practices designed to prevent disease in livestock

2019 ◽  
Vol 20 (2) ◽  
pp. 305-318
Author(s):  
Rachael Vriezen ◽  
Jan M. Sargeant ◽  
Ellen Vriezen ◽  
Charlotte B. Winder ◽  
Annette M. O'Connor
Keyword(s):  
Quality Assessment ◽  
Systematic Reviews ◽  
Management Practices ◽  
Disease Risk ◽  
Data Extraction ◽  
A Priori ◽  
Antibiotic Use ◽  
Study Selection ◽  
Meta Analyses

AbstractTo implement effective stewardship in food animal production, it is essential that producers and veterinarians are aware of preventive interventions to reduce illness in livestock. Systematic reviews and meta-analyses (SR/MA) provide transparent, replicable, and quality-assessed overviews. At present, it is unknown how many SR/MA evaluate preventive antibiotic use or management practices aimed at reducing disease risk in animal agriculture. Further, the quality of existing reviews is unknown. Our aim was to identify reviews investigating these topics and to provide an assessment of their quality. Thirty-eight relevant reviews were identified. Quality assessment was based on the AMSTAR 2 framework for the critical appraisal of systematic reviews. The quality of most of the reviews captured was classified as critically low (84.2%, n = 32/38), and only a small percentage of the evaluated reviews did not contain critical weaknesses (7.9%, n = 3/38). Particularly, a small number of reviews reported the development of an a priori protocol (15.8%, n = 6/38), and few reviews stated that key review steps were conducted in duplicate (study selection/screening: 26.3%, n = 10/38; data extraction: 15.8%, n = 6/38). The development of high-quality reviews summarizing evidence on approaches to antibiotic reduction is essential, and thus greater adherence to quality conduct guidelines for synthesis research is crucial.

scholarly journals Search for unpublished data by systematic reviewers: an audit

BMJ Open ◽  
2017 ◽  
Vol 7 (10) ◽  
pp. e017737 ◽  
Author(s):  
Hedyeh Ziai ◽  
Rujun Zhang ◽  
An-Wen Chan ◽  
Nav Persaud
Keyword(s):  
Publication Bias ◽  
Systematic Reviews ◽  
Data Extraction ◽  
Grey Literature ◽  
Cochrane Library ◽  
Eligibility Criteria ◽  
Ovid Medline ◽  
Cochrane Reviews ◽  
Unpublished Studies ◽  
Cochrane Database

ObjectivesWe audited a selection of systematic reviews published in 2013 and reported on the proportion of reviews that researched for unpublished data, included unpublished data in analysis and assessed for publication bias.DesignAudit of systematic reviews.Data sourcesWe searched PubMed and Ovid MEDLINE In-Process & Other Non-Indexed Citations between 1 January 2013 and 31 December 2013 for the following journals:Journal of the American Medical Association,The British Medical Journal,Lancet,Annals of Internal Medicineand theCochrane Database of Systematic Reviews. We also searched the Cochrane Library and included 100 randomly selected Cochrane reviews.Eligibility criteriaSystematic reviews published in 2013 in the selected journals were included. Methodological reviews were excluded.Data extraction and synthesisTwo reviewers independently reviewed each included systematic review. The following data were extracted: whether the review searched for grey literature or unpublished data, the sources searched, whether unpublished data were included in analysis, whether publication bias was assessed and whether there was evidence of publication bias.Main findings203 reviews were included for analysis. 36% (73/203) of studies did not describe any attempt to obtain unpublished studies or to search grey literature. 89% (116/130) of studies that sought unpublished data found them. 33% (68/203) of studies included an assessment of publication bias, and 40% (27/68) of these found evidence of publication bias.ConclusionA significant fraction of systematic reviews included in our study did not search for unpublished data. Publication bias may be present in almost half the published systematic reviews that assessed for it. Exclusion of unpublished data may lead to biased estimates of efficacy or safety in systematic reviews.

scholarly journals A new method for testing reproducibility in systematic reviews was developed, but needs more testing

2021 ◽  
Vol 21 (1) ◽  
Author(s):  
Dawid Pieper ◽  
Simone Heß ◽  
Clovis Mariano Faggion
Keyword(s):  
Random Sample ◽  
Systematic Reviews ◽  
Data Extraction ◽  
Risk Of Bias ◽  
New Method ◽  
Literature Searches ◽  
Study Selection ◽  
The Future ◽  
Number Of Citations

Abstract Background To develop and test an approach to test reproducibility of SRs. Methods Case study. We have developed an approach to test reproducibility retrospectively while focusing on the whole conduct of an SR instead of single steps of it. We replicated the literature searches and drew a 25% random sample followed by study selection, data extraction, and risk of bias (ROB) assessments performed by two reviewers independently. These results were compared narratively with the original review. Results We were not able to fully reproduce the original search resulting in minor differences in the number of citations retrieved. The biggest disagreements were found in study selection. The most difficult section to be reproduced was the RoB assessment due to the lack of reporting clear criteria to support the judgement of RoB ratings, although agreement was still found to be satisfactory. Conclusion Our approach as well as other approaches needs to undergo testing and comparison in the future as the area of testing for reproducibility of SRs is still in its infancy.

scholarly journals Methods used to meta-analyse results from interrupted time series studies: A methodological systematic review protocol

F1000Research ◽  
2020 ◽  
Vol 9 ◽  
pp. 110
Author(s):  
Elizabeth Korevaar ◽  
Amalia Karahalios ◽  
Andrew B. Forbes ◽  
Simon L. Turner ◽  
Steve McDonald ◽  
...  
Keyword(s):  
Public Health ◽  
Time Series ◽  
Systematic Reviews ◽  
Data Extraction ◽  
Meta Analysis ◽  
Interrupted Time Series ◽  
Analysis Methods ◽  
Study Selection ◽  
The Individual ◽  
Meta Analyses

Background: Systematic reviews are used to inform healthcare decision making. In reviews that aim to examine the effects of organisational, policy change or public health interventions, or exposures, evidence from interrupted time series (ITS) studies may be included. A core component of many systematic reviews is meta-analysis, which is the statistical synthesis of results across studies. There is currently a lack of guidance informing the choice of meta-analysis methods for combining results from ITS studies, and there have been no studies examining the meta-analysis methods used in practice. This study therefore aims to describe current meta-analysis methods used in a cohort of reviews of ITS studies. Methods: We will identify the 100 most recent reviews (published between 1 January 2000 and 11 October 2019) that include meta-analyses of ITS studies from a search of eight electronic databases covering several disciplines (public health, psychology, education, economics). Study selection will be undertaken independently by two authors. Data extraction will be undertaken by one author, and for a random sample of the reviews, two authors. From eligible reviews we will extract details at the review level including discipline, type of interruption and any tools used to assess the risk of bias / methodological quality of included ITS studies; at the meta-analytic level we will extract type of outcome, effect measure(s), meta-analytic methods, and any methods used to re-analyse the individual ITS studies. Descriptive statistics will be used to summarise the data. Conclusions: This review will describe the methods used to meta-analyse results from ITS studies. Results from this review will inform future methods research examining how different meta-analysis methods perform, and ultimately, the development of guidance.

scholarly journals Publication practices of sub-Saharan African Cochrane authors: a bibliometric study

BMJ Open ◽  
2021 ◽  
Vol 11 (9) ◽  
pp. e051839
Author(s):  
Lawrence Mbuagbaw ◽  
Anel Schoonees ◽  
Joy Oliver ◽  
Dachi Arikpo ◽  
Solange Durão ◽  
...  
Keyword(s):  
Systematic Reviews ◽  
Data Extraction ◽  
Sub Saharan Africa ◽  
Bibliometric Study ◽  
Test Accuracy ◽  
Eligibility Criteria ◽  
Cochrane Reviews ◽  
Publication Practices ◽  
Sub Saharan ◽  
Overviews Of Systematic Reviews

IntroductionCochrane Africa (https://africa.cochrane.org/) aims to increase Cochrane reviews addressing high priority questions in sub-Saharan Africa (SSA). Researchers residing in SSA, despite often drawing on Cochrane methods, training or resources, conduct and publish systematic reviews outside of Cochrane. Our objective was to investigate the extent to which Cochrane authors from SSA publish Cochrane and non-Cochrane reviews.MethodsWe conducted a bibliometric study of systematic reviews and overviews of systematic reviews from SSA, first by identifying SSA Cochrane authors, then retrieving their first and last author systematic reviews and overviews from PubMed (2008 to April 2019) and using descriptive analyses to investigate the country of origin, types of reviews and trends in publishing Cochrane and non-Cochrane systematic reviews over time. To be eligible, a review had to have predetermined objectives, eligibility criteria, at least two databases searched, data extraction, quality assessment and a first or last author with a SSA affiliation.ResultsWe identified 657 Cochrane authors and 757 eligible systematic reviews. Most authors were from South Africa (n=332; 51%), followed by Nigeria (n=126; 19%). Three-quarters of the reviews (71%) were systematic reviews of interventions. The intervention reviews were more likely to be Cochrane reviews (60.3% vs 39.7%). Conversely, the overviews (23.8% vs 76.2%), qualitative reviews (14.8% vs 85.2%), diagnostic test accuracy reviews (16.1% vs 83.9%) and the ‘other’ reviews (11.1% vs 88.9%) were more likely to be non-Cochrane reviews. During the study period, the number of non-Cochrane reviews increased more than the number of Cochrane reviews. About a quarter of the reviews covered infectious disease topics.ConclusionCochrane authors from SSA are increasingly publishing a diverse variety of systematic reviews and overviews of systematic reviews, often opting for non-Cochrane journals.

scholarly journals Inconsistencies are common between non-Cochrane systematic reviews and their protocols registered in the International prospective register of systematic reviews (PROSPERO) but were seldom explained

2020 ◽  
Author(s):  
Kaiyan Hu ◽  
Ting Zhang ◽  
Weiyi Zhang ◽  
Qi Zhou ◽  
Mengyao Jiang New ◽  
...  
Keyword(s):  
Systematic Reviews ◽  
Data Extraction ◽  
Meta Analysis ◽  
Reporting Quality ◽  
Fundamental Solutions ◽  
Cochrane Reviews ◽  
Or Methodology ◽  
Retrospective Comparative Study ◽  
The Impact

Abstract Background: Protocols of systematic reviews allow for planning and documentation of review methods and thus improve the transparency of the reviews process. However, pre-registration of a protocol is not enough, the author also need to follow it. PROSPERO is an open-access online database for the registration of non-Cochrane systematic reviews. The purpose of this study is to compare published non-Cochrane reviews with their pre-registered protocols on PROSPERO to determine what changes, if any, have been made, and how likely these changes are to impact the quality of systematic review. Methods: This is a retrospective comparative study. We searched for protocols on PROSPERO platform that were registered in 2018 and then selected the protocols that full text have been published as of January 1st 2019. Published full texts were identified through the protocol's final publication citation. Two authors independently compared and identified changes between protocols and systematic reviews and then evaluated the impact (improve, reduce, or unclear) of these changes on the reporting or methodology quality of reviews. Descriptive statistics of percentage (%) and frequency (n) were conducted. Results: We identified 39 pairs, all of which exhibited changes. “Search strategy”(92%, n=36), “data extraction”(90%, n=35), “data synthesis”(77%, n=30), “outcome”(64%, n=24), and “subgroup analysis”(64%, n=24) all showed significant changes. All changes to only one review were considered to improve the reporting or methodology quality, and the remaining 97% of reviews (n=38) contained changes that were considered to reduce the methodology or reporting quality or that had an unclear impact on systematic reviews. Conclusions: Changes between the non-Cochrane systematic reviews and their protocols recorded on PROSPERO were widespread. Some of the changes reduced the methodology or reporting quality of systematic reviews or had an unclear impact. Measures should be taken to further improve the transparency of the non-Cochrane systematic reviews. Adding a new item in updated “Preferred Reporting Items for Systematic reviews and Meta-Analyses” (PRISMA) and “Meta-analysis of Observational Studies in Epidemiology” (MOOSE) to guide reporting and explaining the changes, as well as advising peer reviewers (and editors) to check the reviews against the protocols are two suggested fundamental solutions.

Sign in / Sign up

Export Citation Format

Share Document