Peer review and competition in the Art Exhibition Game

To investigate the effect of competitive incentives under peer review, we designed a novel experimental setup called the Art Exhibition Game. We present experimental evidence of how competition introduces both positive and negative effects when creative artifacts are evaluated and selected by peer review. Competition proved to be a double-edged sword: on the one hand, it fosters innovation and product diversity, but on the other hand, it also leads to more unfair reviews and to a lower level of agreement between reviewers. Moreover, an external validation of the quality of peer reviews during the laboratory experiment, based on 23,627 online evaluations on Amazon Mechanical Turk, shows that competition does not significantly increase the level of creativity. Furthermore, the higher rejection rate under competitive conditions does not improve the average quality of published contributions, because more high-quality work is also rejected. Overall, our results could explain why many ground-breaking studies in science end up in lower-tier journals. Differences and similarities between the Art Exhibition Game and scholarly peer review are discussed and the implications for the design of new incentive systems for scientists are explained.

Download Full-text

A Study of Medical Board Peer Reviews in Nevada

Journal of Medical Regulation ◽

10.30770/2572-1852-96.1.20 ◽

2010 ◽

Vol 96 (1) ◽

pp. 20-29

Author(s):

Jerry C. Calvanese

Keyword(s):

Peer Review ◽

Board Members ◽

Study Objective ◽

Clark County ◽

Peer Reviews ◽

Medical Examiners ◽

Washoe County ◽

Total Average Cost ◽

Total Average

ABSTRACT Study Objective: The purpose of this study was to obtain data on various characteristics of peer reviews. These reviews were performed for the Nevada State Board of Medical Examiners (NSBME) to assess physician licensees' negligence and/or incompetence. It was hoped that this data could help identify and define certain characteristics of peer reviews. Methods: This study examined two years of data collected on peer reviews. The complaints were initially screened by a medical reviewer and/or a committee composed of Board members to assess the need for a peer review. Data was then collected from the peer reviews performed. The data included costs, specialty of the peer reviewer, location of the peer reviewer, and timeliness of the peer reviews. Results: During the two-year study, 102 peer reviews were evaluated. Sixty-nine percent of the peer-reviewed complaints originated from civil malpractice cases and 15% originated from complaints made by patients. Eighty percent of the complaint physicians were located in Clark County and 12% were located in Washoe County. Sixty-one percent of the physicians who performed the peer reviews were located in Washoe County and 24% were located in Clark County. Twelve percent of the complaint physicians were in practice in the state for 5 years or less, 40% from 6 to 10 years, 20% from 11 to 15 years, 16% from 16 to 20 years, and 13% were in practice 21 years or more. Forty-seven percent of the complaint physicians had three or less total complaints filed with the Board, 10% had four to six complaints, 17% had 7 to 10 complaints, and 26% had 11 or more complaints. The overall quality of peer reviews was judged to be good or excellent in 96% of the reviews. A finding of malpractice was found in 42% of the reviews ordered by the medical reviewer and in 15% ordered by the Investigative Committees. There was a finding of malpractice in 38% of the overall total of peer reviews. The total average cost of a peer review was $791. In 47% of the peer reviews requested, materials were sent from the Board to the peer reviewer within 60 days of the original request and 33% took more than 120 days for the request to be sent. In 48% of the reviews, the total time for the peer review to be performed by the peer reviewer was less than 60 days. Twenty seven percent of the peer reviews took more than 120 days to be returned. Conclusion: Further data is needed to draw meaningful conclusions from certain peer review characteristics reported in this study. However, useful data was obtained regarding timeliness in sending out peer review materials, total times for the peer reviews, and costs.

Download Full-text

Implementing Teaching Portfolios and Peer Reviews in Tax Courses

Journal of the American Taxation Association ◽

10.2308/jata.1999.21.2.95 ◽

1999 ◽

Vol 21 (2) ◽

pp. 95-107 ◽

Cited By ~ 3

Author(s):

Michael J. Calegari ◽

Gregory G. Geisler ◽

Ernest R. Larkins

Keyword(s):

Peer Review ◽

Public University ◽

Written Feedback ◽

Writing Assignments ◽

Peer Reviews ◽

Extant Literature ◽

Research Publications ◽

Teaching Portfolio ◽

Course Syllabus

Extant literature suggests that the process of constructing a teaching portfolio can identify areas to improve, motivate positive changes, and elevate the importance of teaching in academe. This study describes the experience of the tax faculty at a public university in using teaching portfolios and peer reviews to improve the quality of the first two tax courses. The type of teaching portfolio used in this project consists of a course syllabus and a reflective statement that documents the rationale for all components of a course (i.e., lectures, projects, exams, writing assignments, presentations, etc.). The peer review aspect involves written feedback from a colleague on this teaching portfolio. Though research publications are usually subject to extensive peer review, teaching generally is not. Like research, however, teaching can be evaluated and ultimately improved through peer review. Thus, this study can provide valuable guidance to tax professors attempting to improve their courses.

Download Full-text

Opening and Reusing Transparent Peer Reviews with Automatic Article Annotation

Publications ◽

10.3390/publications7010013 ◽

2019 ◽

Vol 7 (1) ◽

pp. 13 ◽

Cited By ~ 2

Author(s):

Afshin Sadeghi ◽

Sarven Capadisli ◽

Johannes Wilm ◽

Christoph Lange ◽

Philipp Mayr

Keyword(s):

Peer Review ◽

Data Representation ◽

Third Party ◽

Tool Support ◽

Scientific Publications ◽

Peer Reviews ◽

Fine Grained ◽

Unstructured Text ◽

Semantic Markup

An increasing number of scientific publications are created in open and transparent peer review models: a submission is published first, and then reviewers are invited, or a submission is reviewed in a closed environment but then these reviews are published with the final article, or combinations of these. Reasons for open peer review include giving better credit to reviewers, and enabling readers to better appraise the quality of a publication. In most cases, the full, unstructured text of an open review is published next to the full, unstructured text of the article reviewed. This approach prevents human readers from getting a quick impression of the quality of parts of an article, and it does not easily support secondary exploitation, e.g., for scientometrics on reviews. While document formats have been proposed for publishing structured articles including reviews, integrated tool support for entire open peer review workflows resulting in such documents is still scarce. We present AR-Annotator, the Automatic Article and Review Annotator which employs a semantic information model of an article and its reviews, using semantic markup and unique identifiers for all entities of interest. The fine-grained article structure is not only exposed to authors and reviewers but also preserved in the published version. We publish articles and their reviews in a Linked Data representation and thus maximise their reusability by third party applications. We demonstrate this reusability by running quality-related queries against the structured representation of articles and their reviews.

Download Full-text

The role and significance of peer-review in Russian and foreign library and information science journals: a comparative analysis

Science Editor and Publisher ◽

10.24069/sep-21-03 ◽

2021 ◽

Author(s):

V. N. Gureyev ◽

N. A. Mazov

Keyword(s):

Peer Review ◽

Information Science ◽

Review Process ◽

Peer Review Process ◽

Library And Information Science ◽

Review System ◽

Academic Journals ◽

Peer Reviews ◽

Peer Reviewers

The paper summarizes experience of the authors as peer-reviewers of more than 100 manuscripts in twelve Russian and foreign academic journals on Library and Information Science in the last seven years. Prepared peer-reviews were used for making a list of the most usual critical and special comments for each manuscript that were subsequently structured for the conducted analyzes. Typical issues accompanying the peer-review process are shown. Significant differences between the results of peer-review in Russian and foreign journals are detected: although the initial quality of newly submitted manuscripts is approximately equal, the final published versions in foreign journals addressed all critical and the majority of minor reviewers’ comments, while in Russian journals more than one third of final versions were published with critical gaps. We conclude about low interest in high quality peer reviews among both authors and editors-in-chief in Russian journals. Despite the limitations of the samples, the obtained findings can be useful when evaluating the current peer-review system in Russian academic journals on Library and Information Science.

Download Full-text

Supporting robust, rigorous, and reliable reviewing as the cornerstone of our profession: Introducing a competency framework for peer review

Industrial and Organizational Psychology ◽

10.1017/iop.2019.121 ◽

2020 ◽

Vol 13 (1) ◽

pp. 1-27 ◽

Cited By ~ 4

Author(s):

Tine Köhler ◽

M. Gloria González-Morales ◽

George C. Banks ◽

Ernest H. O’Boyle ◽

Joseph A. Allen ◽

...

Keyword(s):

Peer Review ◽

Competency Framework ◽

Academic Publishing ◽

Formal Training ◽

Peer Reviews ◽

University Departments ◽

Peer Reviewing ◽

Definition Of ◽

Quality Of Research

AbstractPeer review is a critical component toward facilitating a robust science in industrial and organizational (I-O) psychology. Peer review exists beyond academic publishing in organizations, university departments, grant agencies, classrooms, and many more work contexts. Reviewers are responsible for judging the quality of research conducted and submitted for evaluation. Furthermore, they are responsible for treating authors and their work with respect, in a supportive and developmental manner. Given its central role in our profession, it is curious that we do not have formalized review guidelines or standards and that most of us never receive formal training in peer reviewing. To support this endeavor, we are proposing a competency framework for peer review. The purpose of the competency framework is to provide a definition of excellent peer reviewing and guidelines to reviewers for which types of behaviors will lead to good peer reviews. By defining these competencies, we create clarity around expectations for peer review, standards for good peer reviews, and opportunities for training the behaviors required to deliver good peer reviews. We further discuss how the competency framework can be used to improve peer reviewing and suggest additional steps forward that involve suggestions for how stakeholders can get involved in fostering high-quality peer reviewing.

Download Full-text

Effects of Experimental Interventions to Improve the Biomedical Peer‐Review Process: A Systematic Review and Meta‐Analysis

Journal of the American Heart Association ◽

10.1161/jaha.120.019903 ◽

2021 ◽

Author(s):

Mario Gaudino ◽

N. Bryce Robinson ◽

Antonino Di Franco ◽

Irbaz Hameed ◽

Ajita Naik ◽

...

Keyword(s):

Peer Review ◽

Randomized Trials ◽

Review Process ◽

Peer Review Process ◽

Meta Analysis ◽

Rejection Rate ◽

Unique Identifier ◽

The Individual ◽

The Cost

Background Quality of the peer‐review process has been tested only in small studies. We describe and summarize the randomized trials that investigated interventions aimed at improving peer‐review process of biomedical manuscripts. Methods and Results All randomized trials comparing different peer‐review interventions at author‐, reviewer‐, and/or editor‐level were included. Differences between traditional and intervention‐modified peer‐review processes were pooled as standardized mean difference (SMD) in quality based on the definitions used in the individual studies. Main outcomes assessed were quality and duration of the peer‐review process. Five‐hundred and seventy‐five studies were retrieved, eventually yielding 24 randomized trials. Eight studies evaluated the effect of interventions at author‐level, 16 at reviewer‐level, and 3 at editor‐level. Three studies investigated interventions at multiple levels. The effects of the interventions were reported as mean change in review quality, duration of the peer‐review process, acceptance/rejection rate, manuscript quality, and number of errors detected in 13, 11, 5, 4, and 3 studies, respectively. At network meta‐analysis, reviewer‐level interventions were associated with a significant improvement in review quality (SMD, 0.20 [0.06 to 0.33]), at the cost of increased duration of the review process (SMD, 0.15 [0.01 to 0.29]), except for reviewer blinding. Author‐ and editor‐level interventions did not significantly impact peer‐review quality and duration (respectively, SMD, 0.17 [−0.16 to 0.51] and SMD, 0.19 [−0.40 to 0.79] for quality, and SMD, 0.17 [−0.16 to 0.51] and SMD, 0.19 [−0.40 to 0.79] for duration). Conclusions Modifications of the traditional peer‐review process at reviewer‐level are associated with improved quality, at the price of longer duration. Further studies are needed. Registration URL: https://www.crd.york.ac.uk/prospero ; Unique identifier: CRD42020187910.

Download Full-text

Peer Review of Datasets: When, Why, and How

Bulletin of the American Meteorological Society ◽

10.1175/bams-d-13-00083.1 ◽

2015 ◽

Vol 96 (2) ◽

pp. 191-201 ◽

Cited By ~ 22

Author(s):

Matthew S. Mayernik ◽

Sarah Callaghan ◽

Roland Leigh ◽

Jonathan Tedds ◽

Steven Worley

Keyword(s):

Peer Review ◽

Review Process ◽

Peer Review Process ◽

Strong Support ◽

Central Place ◽

Academic Community ◽

Research Quality ◽

Data Repositories ◽

Peer Reviews

Abstract Peer review holds a central place within the scientific communication system. Traditionally, research quality has been assessed by peer review of journal articles, conference proceedings, and books. There is strong support for the peer review process within the academic community, with scholars contributing peer reviews with little formal reward. Reviewing is seen as a contribution to the community as well as an opportunity to polish and refine understanding of the cutting edge of research. This paper discusses the applicability of the peer review process for assessing and ensuring the quality of datasets. Establishing the quality of datasets is a multifaceted task that encompasses many automated and manual processes. Adding research data into the publication and peer review queues will increase the stress on the scientific publishing system, but if done with forethought will also increase the trustworthiness and value of individual datasets, strengthen the findings based on cited datasets, and increase the transparency and traceability of data and publications. This paper discusses issues related to data peer review—in particular, the peer review processes, needs, and challenges related to the following scenarios: 1) data analyzed in traditional scientific articles, 2) data articles published in traditional scientific journals, 3) data submitted to open access data repositories, and 4) datasets published via articles in data journals.

Download Full-text

Evaluating AVADAR Mobile Health Intervention Using a Multi-prong Approach

10.21203/rs.3.rs-149157/v1 ◽

2021 ◽

Author(s):

Godwin Ubong Akpan ◽

Johnson Ticha ◽

Fiona Lau ◽

Reuben Ngofa ◽

Diallo Mamadou ◽

...

Keyword(s):

Peer Review ◽

Mobile Health ◽

Health Workers ◽

Digital Health ◽

Health Intervention ◽

African Countries ◽

Peer Reviews ◽

Acute Flaccid Paralysis Surveillance ◽

Afp Surveillance

Abstract Background The Auto-Visual AFP Detection and Reporting (AVADAR) digital health intervention programme is a programme that was introduced to Africa in 2016. The programme adopts the use of the AVADAR SMS – based smartphone application (app) in community-based AFP surveillance activities in order to enhance the detection and reporting of AFP (polio) cases and improve AFP surveillance quality. As at 2020, the AVADAR application is being used in 11 African countries. The need to conduct regular and relevant evaluations of the AVADAR programme is very essential towards improving polio eradication programme performance and effectiveness in Africa. Hence, this study aimed to review and evaluate the quality of the AFP cases reported through the AVADAR intervention and as well evaluate the documentation process of AVADAR alerts and investigations, and the assimilation of AFP cases found via AVADAR into the national databases. Methods This study reviewed and evaluated the quality of AVADAR-involved AFP case reporting and documentation process in 7 of the 11 African countries implementing the AVADAR programme (Cameroon, Chad, the DRC, Liberia, Mali, Niger, and South Sudan). Case validations of all AFP cases reported via AVADAR app, iterations of methods used for peer reviewing AVADAR reporting and documentation, informal interview of community informants (CIs) and health workers (HWs), as well as the development of interactive dashboard to showcase the results of peer reviews, were the approaches used for the review and evaluation process. Results Thirty-nine districts, cutting across the participating 7 African countries were selected for the study. A total of 581 AFP cases were reviewed in the selected districts; of which 496 AFP cases were physically seen with 384 cases confirmed as true AFP cases by the peer reviewers. Thematic findings obtained the interview with CIs and HWs identified key areas (communication, multi-disease reporting, and periodic evaluation) that needs to be improved in the AVADAR surveillance system. Also, the interactive dashboard gave a summary of the peer review outcomes at few glances. Conclusions The findings of the AVADAR AFP peer reviews revealed the app’s efficacy in reporting AFP cases and improving surveillance indicators at district level. However, its documentation at health facility level needs to be re-emphasized and improved via a systematic accountability framework implementation for the actors in the reporting cycle. In order to significantly improve AFP surveillance, we recommend on-going commitment to improve knowledge and collaboration between all AVADAR surveillance reporting teams involved in identifying children presenting with AFP. Keywords: AVADAR, Mobile Health, Peer Review, Acute Flaccid Paralysis, surveillance , Africa

Download Full-text

Analysis of the Impact of Employee Share Ownership on the Quality of Governance

10.4018/978-1-7998-8557-3.ch009 ◽

2022 ◽

pp. 160-188

Author(s):

Wafae Nada Nejjar

Keyword(s):

Organizational Performance ◽

Good Governance ◽

Stock Exchange ◽

Negative Effects ◽

Share Ownership ◽

Positive Effects ◽

Quality Of Governance ◽

The One ◽

The Impact

This research assesses the “positive or negative” effects of employee share ownership on good governance in Morocco. It focuses on a lagging but constantly evolving phenomenon in Morocco due, on the one hand, to the lack of awareness of its positive effects on both individual and organizational performance and, on the other hand, on the significant delay in Moroccan legislation regarding this subject. The authors propose an index to measure good governance that they use in order to test research hypotheses. The quantitative study examines all companies listed on the Casablanca Stock Exchange over an observation period from 2015 to 2020. Through this research, the authors demonstrate the favorable effects of employee share ownership on good governance.

Download Full-text

PENGARUH INDEPENDENSI DAN KOMPETENSI AUDITOR TERHADAP KUALITAS AUDIT DALAM MENDETEKSI KECURANGAN DAN KEKELIRUAN PELAPORAN KEUANGAN (Studi Empiris Pada Auditor KAP Di Semarang)

Jurnal Dinamika Sosial Budaya ◽

10.26623/jdsb.v21i2.1721 ◽

2019 ◽

Vol 21 (2) ◽

pp. 121

Author(s):

WAWAN - SETIAWAN ◽

WILLYANTO KARTIKO KUSUMO

Keyword(s):

Decision Making ◽

Peer Review ◽

Audit Quality ◽

The Other ◽

Public Accounting ◽

Financial Independence ◽

Personal Stress ◽

The One ◽

Positive Effect

Public Accounting Profession ( auditors ) are like " two eyes sword ", on the one hand the auditor must consider the credibility and ethics of the profession, but on the other hand also had to face the pressure of clients in a variety of decision-making auditors. If the auditor is not able to resist pressure from clients such as personal stress, emotional or the financial independence of auditors has been reduced and can affect the quality of the audit. This study aims to analyze the influence of empirical evidence and prove the experience, knowledge, long associated with the client, the pressure from the client, the auditor's study of colleagues ( peer review), and non-audit services provided by the Firm to audit quality. The samples are 79 respondents is 18 auditors contained in the firm in Semarang. As for answering the research hypotheses using multiple regression analysis, after testing the classical assumptions. Based on the results of this study concluded that the experience in performing the audit, the auditor's knowledge as well as a study of co-auditors (peer review) has a positive effect on audit quality. So the depth and breadth of knowledge of an auditor as well as more experience in auditing also the presence of a fellow peer auditors , the better the quality of audits conducted.

Download Full-text