Peer Review in the Funding of Research in Higher Education: The Australian Experience

2001 ◽  
Vol 23 (4) ◽  
pp. 343-364 ◽  
Author(s):  
Upali W. Jayasinghe ◽  
Herbert W. Marsh ◽  
Nigel Bond
Keyword(s):  
Peer Review ◽  
Research Council ◽  
Review Process ◽  
Peer Review Process ◽  
Interrater Agreement ◽  
Negative Effects ◽  
External Reviewer ◽  
Peer Reviews ◽  
Gender And Age ◽  
Australian Experience

In this article we evaluate the peer review process used to fund Australian university research across all disciplines. Peer reviews of research proposals (2,989 proposals, 6,233 external reviewers) submitted to the Australian Research Council (ARC) are related to characteristics of the researchers and of external reviewers. The reliability of the peer reviews was disappointingly low (interrater agreement of .53 for researcher ratings based on an average of 4.3 external reviewers per proposal). The gender and age of a researcher and the number of researchers on a research team did not affect the probability that funding would be granted, but professors were more likely to be funded than nonprofessors. Australian external reviewers gave lower ratings than did non-Australian reviewers, particularly those from North America. The number of external reviewers for each proposal and the number of proposals assessed by each external reviewer had small negative effects on ratings. Researcher-nominated external reviewers (those chosen by the authors of a research proposal) gave higher, less-reliable ratings than did panel-nominated external reviewers chosen by the ARC. To improve the reliability of peer reviews, we offer the following recommendations: (a) Researcher-nominated reviewers should not be used; (b) there should be more reviews per proposal; and (c) a smaller number of more highly selected reviewers should perform most of the reviews within each subdiscipline, thereby providing greater control over error associated with individual reviewers.

Disagreement and Agonistic Chance in Peer Review

2021 ◽  
pp. 016224392110260
Author(s):  
Lambros Roumbanis
Keyword(s):  
Peer Review ◽  
Empirical Analysis ◽  
Research Council ◽  
Review Process ◽  
Peer Review Process ◽  
Research Projects ◽  
Engineering Sciences ◽  
Strong Disagreement ◽  
Grant Proposals ◽  
The Empirical Analysis

The purpose of grant peer review is to identify the most excellent and promising research projects. However, sociologists of science and STS scholars have shown that peer review tends to promote solid low-risk projects at the expense of more original and innovative projects that often come with higher risk. It has also been shown that the review process is affected by significant measures of chance. Against this background, the aim of this study is to theorize the notions of academic judgment and agonistic chance and to present and analyze situations in which expert reviewers are faced with the challenge of trying to decide which grant proposals to select when there is strong disagreement. The empirical analysis is based on ethnographic observations of ten panel groups at the Swedish Research Council in the areas of natural and engineering sciences. By focusing on disagreement, the study provides a more in-depth understanding of how agonistic chance creeps into the peer-review process and becomes part of the consensus that is created.

scholarly journals Open Peer Review in Scientific Publishing: A Web Mining Study of PeerJ Authors and Reviewers

2017 ◽  
Vol 1 (4) ◽  
pp. 60-80 ◽  
Author(s):  
Peiling Wang ◽  
Sukjin You ◽  
Rath Manasa ◽  
Dietmar Wolfram
Keyword(s):  
Peer Review ◽  
Web Mining ◽  
Review Process ◽  
Peer Review Process ◽  
Open Science ◽  
Open Access Journal ◽  
Peer Reviews ◽  
Current Characteristics ◽  
Review Reports ◽  
Analytical Tools

AbstractPurposeTo understand how authors and reviewers are accepting and embracing Open Peer Review (OPR), one of the newest innovations in the Open Science movement.Design/methodology/approachThis research collected and analyzed data from the Open Access journal PeerJ over its first three years (2013–2016). Web data were scraped, cleaned, and structured using several Web tools and programs. The structured data were imported into a relational database. Data analyses were conducted using analytical tools as well as programs developed by the researchers.FindingsPeerJ, which supports optional OPR, has a broad international representation of authors and referees. Approximately 73.89% of articles provide full review histories. Of the articles with published review histories, 17.61% had identities of all reviewers and 52.57% had at least one signed reviewer. In total, 43.23% of all reviews were signed. The observed proportions of signed reviews have been relatively stable over the period since the Journal’s inception.Research limitationsThis research is constrained by the availability of the peer review history data. Some peer reviews were not available when the authors opted out of publishing their review histories. The anonymity of reviewers made it impossible to give an accurate count of reviewers who contributed to the review process.Practical implicationsThese findings shed light on the current characteristics of OPR. Given the policy that authors are encouraged to make their articles’ review history public and referees are encouraged to sign their review reports, the three years of PeerJ review data demonstrate that there is still some reluctance by authors to make their reviews public and by reviewers to identify themselves.Originality/valueThis is the first study to closely examine PeerJ as an example of an OPR model journal. As Open Science moves further towards open research, OPR is a final and critical component. Research in this area must identify the best policies and paths towards a transparent and open peer review process for scientific communication.

scholarly journals The role and significance of peer-review in Russian and foreign library and information science journals: a comparative analysis

2021 ◽  
Author(s):  
V.  N. Gureyev ◽  
N.  A. Mazov
Keyword(s):  
Peer Review ◽  
Information Science ◽  
Review Process ◽  
Peer Review Process ◽  
Library And Information Science ◽  
Review System ◽  
Academic Journals ◽  
Peer Reviews ◽  
Peer Reviewers

The paper summarizes experience of the authors as peer-reviewers of more than 100 manuscripts in twelve Russian and foreign academic journals on Library and Information Science in the last seven years. Prepared peer-reviews were used for making a list of the most usual critical and special comments for each manuscript that were subsequently structured for the conducted analyzes. Typical issues accompanying the peer-review process are shown. Significant differences between the results of peer-review in Russian and foreign journals are detected: although the initial quality of newly submitted manuscripts is approximately equal, the final published versions in foreign journals addressed all critical and the majority of minor reviewers’ comments, while in Russian journals more than one third of final versions were published with critical gaps. We conclude about low interest in high quality peer reviews among both authors and editors-in-chief in Russian journals. Despite the limitations of the samples, the obtained findings can be useful when evaluating the current peer-review system in Russian academic journals on Library and Information Science.

scholarly journals Students' Peer-Review in Modelling Exercises

10.28945/2632 ◽  
2003 ◽  
Author(s):  
Guttorm Sindre ◽  
Daniel Moody ◽  
Terje Brasethvik ◽  
Arne Solvberg
Keyword(s):  
Information Systems ◽  
Peer Review ◽  
Student Performance ◽  
Student Motivation ◽  
Review Process ◽  
Peer Review Process ◽  
Learning Experience ◽  
Teaching Staff ◽  
Peer Reviews

This paper reports on the 2002 experiences with students’ peer-review of modeling exercises in a third year Information Systems course. While the peer-reviews did yield some positive and promising results, there were also some problems, and the goals set when introducing peer reviews were not fully achieved. Based on an analysis of student performance and perceptions, main causes for the problems were found to be lacking student motivation combined with unclear demands from teaching staff. For the 2003 offering of the course, the peer review process has been changed to remedy these problems, and this seems to have improved the learning experience for the students.

scholarly journals Ensuring Quality and Status: Peer Review Practices in Kriterium, A Portal for Quality-Marked Monographs and Edited Volumes in Swedish SSH

2021 ◽  
Vol 6 ◽  
Author(s):  
Björn Hammarfelt ◽  
Isak Hammar ◽  
Helena Francke
Keyword(s):  
Peer Review ◽  
Review Process ◽  
Peer Review Process ◽  
The Novel ◽  
Journal Articles ◽  
Important Indicator ◽  
Peer Reviews ◽  
Publication Patterns ◽  
Novel Approach

Although established forms of peer review are often criticized for being slow, secretive, and even unfair, they are repeatedly mentioned by academics as the most important indicator of quality in scholarly publishing. In many countries, the peer review of books is a less codified practice than that of journal articles or conference papers, and the processes and actors involved are far from uniform. In Sweden, the review process of books has seldom been formalized. However, more formal peer review of books has been identified as a response to the increasing importance placed on streamlined peer-reviewed publishing of journal articles in English, which has been described as a direct challenge to more pluralistic publication patterns found particularly in the humanities. In this study, we focus on a novel approach to book review, Kriterium, where an independent portal maintained by academic institutions oversees the reviewing of academic books. The portal administers peer reviews, providing a mark of quality through a process which involves reviewers, an academic coordinator, and an editorial board. The paper studies how this process functions in practice by exploring materials concerning 24 scholarly books reviewed within Kriterium. Our analysis specifically targets tensions identified in the process of reviewing books with a focus on three main themes, namely the intended audience, the edited volume, and the novel role of the academic coordinator. Moreover, we find that the two main aims of the portal–quality enhancement (making research better) and certification (displaying that research is of high quality)–are recurrent in deliberations made in the peer review process. Consequently, we argue that reviewing procedures and criteria of quality are negotiated within a broader discussion where more traditional forms of publishing are challenged by new standards and evaluation practices.

scholarly journals Peer Review of Datasets: When, Why, and How

2015 ◽  
Vol 96 (2) ◽  
pp. 191-201 ◽  
Author(s):  
Matthew S. Mayernik ◽  
Sarah Callaghan ◽  
Roland Leigh ◽  
Jonathan Tedds ◽  
Steven Worley
Keyword(s):  
Peer Review ◽  
Review Process ◽  
Peer Review Process ◽  
Strong Support ◽  
Central Place ◽  
Academic Community ◽  
Research Quality ◽  
Data Repositories ◽  
Peer Reviews

Abstract Peer review holds a central place within the scientific communication system. Traditionally, research quality has been assessed by peer review of journal articles, conference proceedings, and books. There is strong support for the peer review process within the academic community, with scholars contributing peer reviews with little formal reward. Reviewing is seen as a contribution to the community as well as an opportunity to polish and refine understanding of the cutting edge of research. This paper discusses the applicability of the peer review process for assessing and ensuring the quality of datasets. Establishing the quality of datasets is a multifaceted task that encompasses many automated and manual processes. Adding research data into the publication and peer review queues will increase the stress on the scientific publishing system, but if done with forethought will also increase the trustworthiness and value of individual datasets, strengthen the findings based on cited datasets, and increase the transparency and traceability of data and publications. This paper discusses issues related to data peer review—in particular, the peer review processes, needs, and challenges related to the following scenarios: 1) data analyzed in traditional scientific articles, 2) data articles published in traditional scientific journals, 3) data submitted to open access data repositories, and 4) datasets published via articles in data journals.

scholarly journals Open Peer Reviews and Blind Peer Reviews: Merits and Demerits

2018 ◽  
Vol 23 (4) ◽  
pp. 426-427
Author(s):  
Saira Afzal
Keyword(s):  
Quality Control ◽  
Peer Review ◽  
Review Process ◽  
Control Mechanism ◽  
Peer Review Process ◽  
Scientific Quality ◽  
Peer Reviews ◽  
Advantages And Disadvantages ◽  
Quality Assurance Mechanism ◽  
Quality Control Mechanism

The peer review process is essentially the quality control mechanism. Scientific discoveries and advancements have far reaching implications especially in health and medical publications. The quality assurance mechanism in medical journals has to be stringent and flawless. The peer review systems are continuously being criticized, debated and updated. It may be open peer review or blind peer review, both have advantages and disadvantages. Open peer review is performed for scientific quality after publication. It is also known as transparent peer review and public peer review.

scholarly journals Peer Review of Manuscripts: A Valuable yet Neglected Educational Tool for Early-Career Researchers

2019 ◽  
Vol 2019 ◽  
pp. 1-9 ◽  
Author(s):  
Robert McNair ◽  
Hai Anh Le Phuong ◽  
Levente Cseri ◽  
Gyorgy Szekely
Keyword(s):  
Peer Review ◽  
Review Process ◽  
Peer Review Process ◽  
Early Career ◽  
High Quality Research ◽  
Peer Reviews ◽  
Early Career Researchers ◽  
Peer Reviewers ◽  
Survey Results ◽  
Number Of Publications

With the number of publications being all-time high, academic peer review is imperative to ensure high-quality research content. The wider involvement of postgraduate, early-career researchers (ECRs) has been proposed on several platforms to address the unsustainability of the peer review process caused by a lack of peer reviewers. A survey involving 1203 academics and ECRs in ten countries revealed their attitudes towards the involvement of ECRs in the peer review process. The trends and motives were identified, with emphasis on the peer review being an oft-neglected tool in the skill development of ECRs. In light of the survey results, the transferrable skills that ECRs acquire from performing peer reviews at a crucial stage in their career development are systematically explored. The findings call for further engagement of ECRs in the peer review process under supervisory mentoring.

Sign in / Sign up

Export Citation Format

Share Document