Improving peer review of systematic reviews by involving librarians and information specialists: protocol for a randomized controlled trial

Background: Problems continue to exist with the reporting of and risk of bias in search methods and strategies in systematic reviews and related review types. Peer reviewers who are not familiar with what is required to transparently and fully report a search may not be prepared to review the search components of systematic reviews, nor may they know what is likely to introduce bias into a search. Librarians and information specialists, who have expertise in searching, may offer specialized knowledge that would help improve systematic review search reporting and lessen risk of bias, but they are underutilized as methodological peer reviewers.Methods: This study will evaluate the effect of adding librarians and information specialists as methodological peer reviewers on the quality of search reporting and risk of bias in systematic review searches. The study will be a pragmatic randomized controlled trial using 150 systematic review manuscripts submitted to BMJ and BMJ Open as the unit of randomization. Manuscripts that report on completed systematic reviews and related review types and have been sent for peer review are eligible. For each manuscript randomized to the intervention, a librarian/information specialist will be invited as an additional peer reviewer using standard practices for each journal. First revision manuscripts will be assessed in duplicate for reporting quality and risk of bias, using adherence to 4 items from PRISMA-S and assessors’ judgements on 4 signaling questions from ROBIS Domain 2, respectively. Identifying information from the manuscripts will be removed prior to assessment.Discussion: The primary outcomes for this study are quality of reporting as indicated by differences in the proportion of adequately reported searches in first revision manuscripts between intervention and control groups and risk of bias as indicated by differences in the proportions of first revision manuscripts with high, low, and unclear bias. If the intervention demonstrates an effect on search reporting or bias, this may indicate a need for journal editors to work with librarians and information specialists as methodological peer reviewers.Trial registration: This trial was registered on the Open Science Framework on June 17, 2021 at https://doi.org/10.17605/OSF.IO/W4CK2.

Download Full-text

Improving peer review of systematic reviews by involving librarians and information specialists: protocol for a randomized controlled trial

Trials ◽

10.1186/s13063-021-05738-z ◽

2021 ◽

Vol 22 (1) ◽

Author(s):

Melissa L. Rethlefsen ◽

Sara Schroter ◽

Lex M. Bouter ◽

David Moher ◽

Ana Patricia Ayala ◽

...

Keyword(s):

Systematic Review ◽

Randomized Controlled Trial ◽

Peer Review ◽

Systematic Reviews ◽

Controlled Trial ◽

Reporting Quality ◽

Risk Of Bias ◽

Randomized Controlled ◽

Peer Reviewers

Abstract Background Problems continue to exist with the reporting quality and risk of bias in search methods and strategies in systematic reviews and related review types. Peer reviewers who are not familiar with what is required to transparently and fully report a search may not be prepared to review the search components of systematic reviews, nor may they know what is likely to introduce bias into a search. Librarians and information specialists, who have expertise in searching, may offer specialized knowledge that would help improve systematic review search reporting and lessen risk of bias, but they are underutilized as methodological peer reviewers. Methods This study will evaluate the effect of adding librarians and information specialists as methodological peer reviewers on the quality of search reporting and risk of bias in systematic review searches. The study will be a pragmatic randomized controlled trial using 150 systematic review manuscripts submitted to BMJ and BMJ Open as the unit of randomization. Manuscripts that report on completed systematic reviews and related review types and have been sent for peer review are eligible. For each manuscript randomized to the intervention, a librarian/information specialist will be invited as an additional peer reviewer using standard practices for each journal. First revision manuscripts will be assessed in duplicate for reporting quality and risk of bias, using adherence to 4 items from PRISMA-S and assessors’ judgements on 4 signaling questions from ROBIS Domain 2, respectively. Identifying information from the manuscripts will be removed prior to assessment. Discussion The primary outcomes for this study are quality of reporting as indicated by differences in the proportion of adequately reported searches in first revision manuscripts between intervention and control groups and risk of bias as indicated by differences in the proportions of first revision manuscripts with high, low, and unclear bias. If the intervention demonstrates an effect on search reporting or bias, this may indicate a need for journal editors to work with librarians and information specialists as methodological peer reviewers. Trial registration Open Science Framework. Registered on June 17, 2021, at 10.17605/OSF.IO/W4CK2.

Download Full-text

Biomedical conferences’ author instructions rarely mention guidelines for reporting abstracts of trials and systematic reviews

Journal of Comparative Effectiveness Research ◽

10.2217/cer-2019-0158 ◽

2020 ◽

Vol 9 (2) ◽

pp. 83-91

Author(s):

Lenko Saric ◽

Svjetlana Dosenovic ◽

Jakov Mihanovic ◽

Livia Puljak

Keyword(s):

Systematic Review ◽

Randomized Controlled Trial ◽

Systematic Reviews ◽

Controlled Trial ◽

Reporting Guidelines ◽

Randomized Controlled ◽

Meta Analyses

Aim: To analyze whether instructions for authors of biomedical conference abstracts mention guidelines for writing randomized controlled trial and systematic review abstracts and to evaluate reasons for their absence from instructions. Materials & methods: We analyzed instructions for authors of biomedical conferences advertized in 2019 and assessed whether they mentioned Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Abstracts and Consolidated Standards of Reporting Trials for Abstracts guidelines. We surveyed contact persons from abstract/publication committees of selected conferences to analyze why relevant guidelines were missing. Results: Instructions for abstracts were available for 819 conferences. Only two (0.2%) had reporting instructions for randomized controlled trial/systematic review authors. Almost half of the contacted conference organizers whose response we received were not aware of Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Abstracts and Consolidated Standards of Reporting Trials for Abstracts guidelines. Conclusion: Conference organizers do not require and are not familiar enough with reporting guidelines.

Download Full-text

There were large discrepancies in risk of bias tool judgments when a randomized controlled trial appeared in more than one systematic review

Journal of Clinical Epidemiology ◽

10.1016/j.jclinepi.2016.08.012 ◽

2017 ◽

Vol 81 ◽

pp. 72-76 ◽

Cited By ~ 22

Author(s):

Vanessa M.B. Jordan ◽

Sarah F. Lensen ◽

Cynthia M. Farquhar

Keyword(s):

Systematic Review ◽

Randomized Controlled Trial ◽

Controlled Trial ◽

Risk Of Bias ◽

Randomized Controlled

Download Full-text

Consistency in the Analysis and Reporting of Primary End Points in Oncology Randomized Controlled Trials From Registration to Publication: A Systematic Review

Journal of Clinical Oncology ◽

10.1200/jco.2011.37.0890 ◽

2012 ◽

Vol 30 (2) ◽

pp. 210-216 ◽

Cited By ~ 55

Author(s):

Benoit You ◽

Hui K. Gan ◽

Gregory Pond ◽

Eric X. Chen

Keyword(s):

Systematic Review ◽

Randomized Clinical Trials ◽

Sample Size Calculation ◽

Trial Registration ◽

Controlled Trials ◽

Quality Of Reporting ◽

Randomized Controlled ◽

Peer Reviewers ◽

Over Time

Purpose To improve the quality of reporting of randomized clinical trials (RCTs), international registries for RCTs and guidelines for primary end point (PEP) analysis were established. The objectives of this systematic review were to evaluate concordance of PEP between publication and the corresponding registry and to assess intrapublication consistency in PEP reporting. Methods All adult oncology RCTs in solid tumors published in 10 journals between 2005 and 2009 were reviewed. Registration information was extracted from international trial registries. Results A total 366 RCTs were identified. Trial registration was found for 215 trials, and the rate increased from 43% in 2005 to 82% in 2009 (P < .001). There were 134 RCTs with clearly defined PEPs in registry, with the rate increasing from 15% to 67% (P < .001). PEP differed between registration and final publication in 14% trials with clearly defined PEPs. Reporting issues in methodology were found in 15% of RCTs, mainly because of inadequate reporting of PEP or sample size calculation. Problems with the interpretation of trial results were found in 22% publications, mostly resulting from negative superiority studies being interpreted as showing equivalence. Conclusion The rates of trial registration and of trials with clearly defined PEPs have improved over time; however, 14% of these trials reported a different PEP in the final publication. Intrapublication inconsistencies in PEP reporting are frequent. Our findings highlight the need for investigators, peer reviewers, and readers to exercise increased awareness and scrutiny of reporting outcomes of oncology RCTs.

Download Full-text