Peer review reduces spin in PCORI research reports

Abstract Background The Patient-Centered Outcomes Research Institute (PCORI) is obligated to peer review and to post publicly “Final Research Reports” of all funded projects. PCORI peer review emphasizes adherence to PCORI’s Methodology Standards and principles of ethical scientific communication. During the peer review process, reviewers and editors seek to ensure that results are presented objectively and interpreted appropriately, e.g., free of spin. Methods Two independent raters assessed PCORI peer review feedback sent to authors. We calculated the proportion of reports in which spin was identified during peer review, and the types of spin identified. We included reports submitted by April 2018 with at least one associated journal article. The same raters then assessed whether authors addressed reviewers’ comments about spin. The raters also assessed whether spin identified during PCORI peer review was present in related journal articles. Results We included 64 PCORI-funded projects. Peer reviewers or editors identified spin in 55/64 (86%) submitted research reports. Types of spin included reporting bias (46/55; 84%), inappropriate interpretation (40/55; 73%), inappropriate extrapolation of results (15/55; 27%), and inappropriate attribution of causality (5/55; 9%). Authors addressed comments about spin related to 47/55 (85%) of the reports. Of 110 associated journal articles, PCORI comments about spin were potentially applicable to 44/110 (40%) articles, of which 27/44 (61%) contained the same spin that was identified in the PCORI research report. The proportion of articles with spin was similar for articles accepted before and after PCORI peer review (63% vs 58%). Discussion Just as spin is common in journal articles and press releases, we found that most reports submitted to PCORI included spin. While most spin was mitigated during the funder’s peer review process, we found no evidence that review of PCORI reports influenced spin in journal articles. Funders could explore interventions aimed at reducing spin in published articles of studies they support.

Download Full-text

Peer review reduces spin in PCORI research reports

10.31222/osf.io/xknhp ◽

2021 ◽

Author(s):

Evan Mayo-Wilson ◽

Meredith Phillips ◽

Avonne Connor ◽

Kelly Vander Ley ◽

Kevin Naaman ◽

...

Keyword(s):

Peer Review ◽

Outcomes Research ◽

Reporting Bias ◽

Research Report ◽

Journal Articles ◽

Patient Centered ◽

Reporting Practices ◽

Before And After ◽

Publishing Journal ◽

Interpretation Of Results

BACKGROUNDThe Patient-Centered Outcomes Research Institute (PCORI) is obligated to peer review and to post publicly “Final Research Reports” of all funded projects. Peer review emphasizes adherence to PCORI’s Methodology Standards. As part of meeting these standards, reviewers and editors seek to minimize “spin,” defined elsewhere as “reporting practices that distort the interpretation of results and mislead readers.” METHODSTwo independent raters assessed the prevalence of spin in reports submitted for PCORI peer review by April 2018. We then assessed whether authors addressed comments about spin received during peer review. Because investigators who submit research reports before publishing journal articles might incorporate PCORI’s feedback in their journal articles, we also assessed whether spin identified during PCORI peer review was present in related journal articles.RESULTSWe included 64 projects funded by PCORI. Spin was identified during peer review in 55/64 (86%) submitted research reports. Types of spin included reporting bias (46/55; 84%), inappropriate interpretation (40/55; 73%), inappropriate extrapolation of results (15/55; 27%), and inappropriate attribution of causality (5/55; 9%). Authors addressed comments about spin in 47/55 (85%) of the revised and accepted research reports.Journal articles associated with 21/55 (38%) research reports contained spin that was also identified in the research report. PCORI comments about spin were potentially applicable to 44/110 journal articles with results. Of these, 27/44 (61%) contained spin that was also identified in the PCORI research report. The proportion of articles with spin was similar for those accepted before and after PCORI peer review.DISCUSSIONMost reports submitted to PCORI included spin, which was mitigated during peer review. We found no evidence that peer review of PCORI research reports affected spin in journal articles. Some journal articles contained spin even when authors removed spin from their PCORI research reports.When both are available, PCORI research reports might be more useful to systematic reviewers and other stakeholders compared with journal articles about the same study because they include less reporting bias and other types of spin.

Download Full-text

Evaluating, Improving, and Appreciating Peer Review at IJTMB

International Journal of Therapeutic Massage & Bodywork Research Education & Practice ◽

10.3822/ijtmb.v14i1.637 ◽

2021 ◽

Vol 14 (1) ◽

pp. 1-3

Author(s):

Ann Blair Kennedy, LMT, BCTMB, DrPH

Keyword(s):

Peer Review ◽

Review Process ◽

Peer Review Process ◽

Scientific Publishing ◽

Publication Process ◽

Improve Efficiency ◽

Peer Reviewers

Peer review is a mainstay of scientific publishing and, while peer reviewers and scientists report satisfaction with the process, peer review has not been without criticism. Within this editorial, the peer review process at the IJTMB is defined and explained. Further, seven steps are identified by the editors as a way to improve efficiency of the peer review and publication process. Those seven steps are: 1) Ask authors to submit possible reviewers; 2) Ask reviewers to update profiles; 3) Ask reviewers to “refer a friend”; 4) Thank reviewers regularly; 5) Ask published authors to review for the Journal; 6) Reduce the length of time to accept peer review invitation; and 7) Reduce requested time to complete peer review. We believe these small requests and changes can have a big effect on the quality of reviews and speed in which manuscripts are published. This manuscript will present instructions for completing peer review profiles. Finally, we more formally recognize and thank peer reviewers from 2018–2020.

Download Full-text

The effects of an editor serving as one of the reviewers during the peer-review process

F1000Research ◽

10.12688/f1000research.8452.1 ◽

2016 ◽

Vol 5 ◽

pp. 683 ◽

Cited By ~ 1

Author(s):

Marco Giordan ◽

Attila Csikasz-Nagy ◽

Andrew M. Collings ◽

Federico Vaggi

Keyword(s):

Peer Review ◽

Review Process ◽

Peer Review Process ◽

Final Decision ◽

Scientific Journals ◽

Scientific Publications ◽

Peer Reviewers ◽

Decision Type ◽

Number Of Citations ◽

Decision Times

BackgroundPublishing in scientific journals is one of the most important ways in which scientists disseminate research to their peers and to the wider public. Pre-publication peer review underpins this process, but peer review is subject to various criticisms and is under pressure from growth in the number of scientific publications.MethodsHere we examine an element of the editorial process ateLife, in which the Reviewing Editor usually serves as one of the referees, to see what effect this has on decision times, decision type, and the number of citations. We analysed a dataset of 8,905 research submissions toeLifesince June 2012, of which 2,750 were sent for peer review, using R and Python to perform the statistical analysis.ResultsThe Reviewing Editor serving as one of the peer reviewers results in faster decision times on average, with the time to final decision ten days faster for accepted submissions (n=1,405) and 5 days faster for papers that were rejected after peer review (n=1,099). There was no effect on whether submissions were accepted or rejected, and a very small (but significant) effect on citation rates for published articles where the Reviewing Editor served as one of the peer reviewers.ConclusionsAn important aspect ofeLife’s peer-review process is shown to be effective, given that decision times are faster when the Reviewing Editor serves as a reviewer. Other journals hoping to improve decision times could consider adopting a similar approach.

Download Full-text

Metascience on Peer Review: Testing the Effects of a Study’s Originality and Statistical Significance in a Field Experiment

10.31222/osf.io/dkuhs ◽

2019 ◽

Cited By ~ 1

Author(s):

Malte Elson ◽

Markus Huff ◽

Sonja Utz

Keyword(s):

Field Experiment ◽

Peer Review ◽

Gold Standard ◽

Primary Outcome ◽

Review Process ◽

Peer Review Process ◽

Statistical Significance ◽

Selection Method ◽

Psychological Theories ◽

Peer Reviewers

Peer review has become the gold standard in scientific publishing as a selection method and a refinement scheme for research reports. However, despite its pervasiveness and conferred importance, relatively little empirical research has been conducted to document its effectiveness. Further, there is evidence that factors other than a submission’s merits can substantially influence peer reviewers’ evaluations. We report the results of a metascientific field experiment on the effect of the originality of a study and the statistical significance of its primary outcome on reviewers’ evaluations. The general aim of this experiment, which was carried out in the peer-review process for a conference, was to demonstrate the feasibility and value of metascientific experiments on the peer-review process and thereby encourage research that will lead to understanding its mechanisms and determinants, effectively contextualizing it in psychological theories of various biases, and developing practical procedures to increase its utility.

Download Full-text

High adoption rates in a new preprint service from Springer Nature and Research Square

Septentrio Conference Series ◽

10.7557/5.4897 ◽

2019 ◽

Author(s):

Damian Pattinson

Keyword(s):

Peer Review ◽

Real Time ◽

Biomedical Research ◽

Review Process ◽

Positive Impact ◽

Peer Review Process ◽

Manuscript Reviewer ◽

Peer Reviewers ◽

Gates Foundation

In recent years, funders have increased their support for early sharing of biomedical research through the use of preprints. For most, such as the COAlitionS group of funders (ASAPbio 2019) and the Gates foundation, this takes the form of active encouragement, while for others, it is mandated. But despite these motivations, few authors are routinely depositing their work as a preprint before submitting to a journal. Some journals have started offering authors the option of posting their work early at the point at which it is submitted for review. These include PLOS, who offer a link to BiorXiv, the Cell journals, who offer SSRN posting through ‘Sneak Peak’, and Nature Communications, who offer posting to any preprint and a link from the journal page called ‘Under Consideration’. Uptake has ranged from 3% for the Nature pilot, to 18% for PLOS (The Official Plos Blog 2018). In order to encourage more researchers to post their work early, we have been offering authors who submit to BMC Series titles the opportunity to post their work as a preprint on Research Square, a new platform that lets authors share and improve their research. To encourage participation, authors are offered a greater amount of control and transparency over the peer review process if they opt in. First, they are given a detailed peer review timeline which updates in real time every time an event occurs on their manuscript (reviewer invited, reviewer accepts etc). Second, they are encouraged to share their preprint with colleagues, who are able to post comments on the paper. These comments are sent to the editor when they are making their decision. Third, authors can suggest potential peer reviewers, recommendations which are also passed onto the editor to vet and invite. Together, these incentives have had a positive impact on authors choosing to post a preprint. Among the journals that offer this service, the average opt-in rate is 40%. This translates to over 3,000 manuscripts (as of July 2019) that have been posted to Research Square since the launch of the service in October 2018. In this talk I will demonstrate the functionality of Research Square, and provide demographic and discipline data on which areas are most and least likely to post.

Download Full-text

The effects of an editor serving as one of the reviewers during the peer-review process

F1000Research ◽

10.12688/f1000research.8452.2 ◽

2016 ◽

Vol 5 ◽

pp. 683 ◽

Cited By ~ 2

Author(s):

Marco Giordan ◽

Attila Csikasz-Nagy ◽

Andrew M. Collings ◽

Federico Vaggi

Keyword(s):

Peer Review ◽

Review Process ◽

Peer Review Process ◽

Final Decision ◽

Scientific Journals ◽

Scientific Publications ◽

Peer Reviewers ◽

Decision Type ◽

Number Of Citations ◽

Decision Times

BackgroundPublishing in scientific journals is one of the most important ways in which scientists disseminate research to their peers and to the wider public. Pre-publication peer review underpins this process, but peer review is subject to various criticisms and is under pressure from growth in the number of scientific publications.MethodsHere we examine an element of the editorial process ateLife, in which the Reviewing Editor usually serves as one of the referees, to see what effect this has on decision times, decision type, and the number of citations. We analysed a dataset of 8,905 research submissions toeLifesince June 2012, of which 2,747 were sent for peer review. This subset of 2747 papers was then analysed in detail. ResultsThe Reviewing Editor serving as one of the peer reviewers results in faster decision times on average, with the time to final decision ten days faster for accepted submissions (n=1,405) and five days faster for papers that were rejected after peer review (n=1,099). Moreover, editors acting as reviewers had no effect on whether submissions were accepted or rejected, and a very small (but significant) effect on citation rates.ConclusionsAn important aspect ofeLife’s peer-review process is shown to be effective, given that decision times are faster when the Reviewing Editor serves as a reviewer. Other journals hoping to improve decision times could consider adopting a similar approach.

Download Full-text

Publication practices during the COVID-19 pandemic: Biomedical preprints and peer-reviewed literature

10.1101/2021.01.21.427563 ◽

2021 ◽

Author(s):

Yulia V. Sevryugina ◽

Andrew J. Dicks

Keyword(s):

Review Process ◽

Peer Review Process ◽

Journal Article ◽

Life Sciences ◽

Open Science ◽

Journal Articles ◽

Biomedical Sciences ◽

Online Journal ◽

Publication Practices ◽

Comprehensive Study

AbstractThe coronavirus pandemic introduced many changes to our society, and deeply affected the established in biomedical sciences publication practices. In this article, we present a comprehensive study of the changes in scholarly publication landscape for biomedical sciences during the COVID-19 pandemic, with special emphasis on preprints posted on bioRxiv and medRxiv servers. We observe the emergence of a new category of preprint authors working in the fields of immunology, microbiology, infectious diseases, and epidemiology, who extensively used preprint platforms during the pandemic for sharing their immediate findings. The majority of these findings were works-in-progress unfitting for a prompt acceptance by refereed journals. The COVID-19 preprints that became peer-reviewed journal articles were often submitted to journals concurrently with the posting on a preprint server, and the entire publication cycle, from preprint to the online journal article, took on average 63 days. This included an expedited peer-review process of 43 days and journal’s production stage of 15 days, however there was a wide variation in publication delays between journals. Only one third of COVID-19 preprints posted during the first nine months of the pandemic appeared as peer-reviewed journal articles. These journal articles display high Altmetric Attention Scores further emphasizing a significance of COVID-19 research during 2020. This article will be relevant to editors, publishers, open science enthusiasts, and anyone interested in changes that the 2020 crisis transpired to publication practices and a culture of preprints in life sciences.

Download Full-text

Metascience on Peer Review: Testing the Effects of a Study’s Originality and Statistical Significance in a Field Experiment

10.31234/osf.io/gyds8 ◽

2019 ◽

Author(s):

Malte Elson ◽

Markus Huff ◽

Sonja Utz

Keyword(s):

Field Experiment ◽

Peer Review ◽

Gold Standard ◽

Primary Outcome ◽

Review Process ◽

Peer Review Process ◽

Statistical Significance ◽

Selection Method ◽

Psychological Theories ◽

Peer Reviewers

Download Full-text

A brief guide to the science and art of writing manuscripts in biomedicine

Journal of Translational Medicine ◽

10.1186/s12967-020-02596-2 ◽

2020 ◽

Vol 18 (1) ◽

Author(s):

Diego A. Forero ◽

Sandra Lopez-Leon ◽

George Perry

Keyword(s):

Biological Sciences ◽

Review Process ◽

Peer Review Process ◽

Scientific Journals ◽

Journal Articles ◽

Science And Art ◽

Multiple Dimensions ◽

Peer Reviewers ◽

Research Findings ◽

Writing Major

Abstract Publishing articles in international scientific journals is the primary method for the communication of validated research findings and ideas. Journal articles are commonly used as a major input for evaluations of researchers and institutions. Few articles have been published previously about the different aspects needed for writing high-quality articles. In this manuscript, we provide an updated and brief guide for the multiple dimensions needed for writing manuscripts in the health and biological sciences, from current, international and interdisciplinary perspectives and from our expertise as authors, peer reviewers and editors. We provide key suggestions for writing major sections of the manuscript (e.g. title, abstract, introduction, methods, results and discussion), for submitting the manuscript and bring an overview of the peer review process and of the post-publication impact of the articles.

Download Full-text

Point and Click Data: An Assessment of Editorial Perceptions and Recommendations for the Peer-Review Process in the New Data Frontier

Journal of Information Systems ◽

10.2308/isys-51980 ◽

2017 ◽

Vol 33 (1) ◽

pp. 129-144 ◽

Cited By ~ 3

Author(s):

Jay C. Thibodeau ◽

L. Tyler Williams ◽

Annie L. Witte

Keyword(s):

Peer Review ◽

Data Streams ◽

Best Practice ◽

Review Process ◽

Peer Review Process ◽

Data Integrity ◽

Data Availability ◽

Peer Reviewers ◽

Online Labor ◽

New Research

ABSTRACT In the new research frontier of data availability, this study develops guidelines to aid accounting academicians as they seek to evidence data integrity proactively in the peer-review process. To that end, we explore data integrity issues associated with two emerging data streams that are gaining prominence in the accounting literature: online labor markets and social media sources. We provide rich detail surrounding academic thought about these data platforms through interview data collected from a sample of former senior journal editors and survey data collected from a sample of peer reviewers. We then propound a set of best practice considerations that are designed to mitigate the perceived risks identified by our assessment.

Download Full-text