scholarly journals Single- or double-blind review? A field study of system preference, reliability, bias, and validity

2021 ◽  
Author(s):  
Timothy Joseph Pleskac ◽  
Ellie Kyung ◽  
Gretchen B. Chapman ◽  
Oleg Urminsky
Keyword(s):  
Field Study ◽  
Scientific Work ◽  
Annual Conference ◽  
High Stakes ◽  
Double Blind ◽  
Anna Karenina ◽  
Blind Review ◽  
Subsequent Publication ◽  
Blind Reviews ◽  
Single Blind

Many scientists experience a practice-preference gap about peer review. Single-blind review---where authors' identities are revealed to reviewers---is often used for evaluation. Yet, double-blind review---where authors' identities are concealed---is seen as more fair. To understand this gap, we compared both systems in a high-stakes field study: submissions to the Society for Judgment and Decision Making’s annual conference, the leading international conference on this topic. Each submission received both review types. Reviewers were randomly assigned to the review system and submissions. Selected conference talks were evaluated for quality, popularity, and subsequent publication status. We assessed the two systems on reliability, bias, and validity. On reliability, while both systems had moderate reliability, agreement was higher on what constituted a poor submission than a strong one (Anna Karenina Principle). On bias, double-blind reviews showed a slight bias against submission by women (Matilda Effect), while single-blind reviews showed a preference for submissions with senior co-authors (Matthew Effect). On validity, neither system predicted talk quality or popularity, but both predicted publication status. Author characteristics did not consistently predict outcomes. Thus, we suggest the costs of single-blind review do not outweigh its benefits. Yet, double-blind review is also not a perfect solution. We propose an equitable approach for selecting scientific work may be an informed lottery: use double-blind review to identify submissions of merit, then randomly choose from this set.

scholarly journals The cases for and against double-blind reviews

PeerJ ◽  
2019 ◽  
Vol 7 ◽  
pp. e6702 ◽  
Author(s):  
Amelia R. Cox ◽  
Robert Montgomerie
Keyword(s):  
Gender Bias ◽  
Behavioral Ecology ◽  
Peer Review Process ◽  
Impact Factors ◽  
Double Blind ◽  
Blind Review ◽  
Female Authors ◽  
Female Authorship ◽  
Blind Reviews ◽  
Single Blind

To date, the majority of authors on scientific publications have been men. While much of this gender bias can be explained by historic sexism and discrimination, there is concern that women may still be disadvantaged by the peer review process if reviewers’ biases lead them to reject publications with female authors more often. One potential solution to this perceived gender bias in the reviewing process is for journals to adopt double-blind reviews whereby neither the authors nor the reviewers are aware of each other’s identity and gender. To test the efficacy of double-blind reviews in one behavioral ecology journal (Behavioral Ecology, BE), we assigned gender to every authorship of every paper published for 2010–2018 in that journal compared to four other journals with single-blind reviews but similar subject matter and impact factors. While female authorships comprised only 35% of the total in all journals, the double-blind journal (BE) did not have more female authorships than its single-blind counterparts. Interestingly, the incidence of female authorship is higher at behavioral ecology journals (BE and Behavioral Ecology and Sociobiology) than in the ornithology journals (Auk, Condor, Ibis) for papers on all topics as well as those on birds. These analyses suggest that double-blind review does not currently increase the incidence of female authorship in the journals studied here. We conclude, at least for these journals, that double-blind review no longer benefits female authors and we discuss the pros and cons of the double-blind reviewing process based on our findings.

scholarly journals Double blind reviews in software engineering venues: Practicability, promises and perils

2016 ◽  
Author(s):  
Moritz Beller ◽  
Alberto Bacchelli
Keyword(s):  
Software Engineering ◽  
Scientific Method ◽  
Review Process ◽  
Peer Review Process ◽  
Double Blind ◽  
Blind Review ◽  
Blind Reviews ◽  
Hidden Biases ◽  
Benefits And Costs ◽  
Single Blind

The peer review process is central to the scientific method, the advancement and spread of research as well as crucial for individual careers. However, the single blind review process currently used in most Software Engineering (SE) venues is susceptible towards apparent and hidden biases, since reviewers know the identity of authors. In this paper, we perform a study on the benefits and costs that are associated with introducing double-blind reviews in SE venues. We surveyed the SE community’s opinion, interviewed experts on double-blind reviewing, and estimated the likelihood of reviewers being able to guess the authors. Our results indicate that double-blind reviewing could be introduced in large SE conferences at lower-than-generally believed costs and that the majority of the SE community is in favor of introducing it.

The Case For and Against Double-blind Reviews

2018 ◽  
Author(s):  
Amelia R Cox ◽  
Robert Montgomerie
Keyword(s):  
Peer Review ◽  
Gender Bias ◽  
Behavioral Ecology ◽  
Peer Review Process ◽  
Double Blind ◽  
Blind Review ◽  
Female Authors ◽  
Female Authorship ◽  
Blind Reviews ◽  
Single Blind

To date, the majority of authors on scientific publications have been men. While much of this gender bias can be explained by historic sexism and discrimination, there is concern that women may still be disadvantaged by the peer review process if reviewers' unconscious biases lead them to reject publications with female authors more often. One potential solution to this perceived gender bias in the reviewing process is for journals to adopt double-blind reviews whereby neither the authors nor the reviewers are aware of each other's identities and genders. To test the efficacy of double-blind reviews, we assigned gender to every authorship of every paper published in 5 different journals with different peer review processes (double-blind vs. single blind) and subject matter (birds vs. behavioral ecology) from 2010-2018 (n = 4865 papers). While female authorships comprised only 35% of the total, the double-blind journal Behavioral Ecology did not have more female authorships than its single-blind counterparts. Interestingly, the incidence of female authorship is higher at behavioral ecology journals (Behavioral Ecology and Behavioral Ecology and Sociobiology) than in the ornithology journals (Auk, Condor, Ibis), for papers on all topics as well as those on birds. These analyses suggest that double-blind review does not currently increase the incidence of female authorship in the journals studied here. We conclude, at least for these journals, that double-blind review does not benefit female authors and may, in the long run, be detrimental.

Does double-blind peer review impact gender authorship trends? An evaluation of two leading neurosurgical journals from 2010 to 2019

2020 ◽  
pp. 1-9
Author(s):  
Uma V. Mahajan ◽  
Harsh Wadhwa ◽  
Parastou Fatemi ◽  
Samantha Xu ◽  
Judy Shan ◽  
...  
Keyword(s):  
Peer Review ◽  
Board Certification ◽  
Current Status ◽  
Double Blind ◽  
Blind Review ◽  
Female Authorship ◽  
Before And After ◽  
Single Blind ◽  
Single Versus ◽  
Status Of Women

OBJECTIVEPublications are key for advancement within academia. Although women are underrepresented in academic neurosurgery, the rates of women entering residency, achieving board certification, and publishing papers are increasing. The goal of this study was to assess the current status of women in academic neurosurgery publications. Specifically, this study sought to 1) survey female authorship rates in the Journal of Neurosurgery (JNS [not including JNS: Spine or JNS: Pediatrics]) and Neurosurgery from 2010 to 2019; 2) analyze whether double-blind peer review (started in Neurosurgery in 2011) altered female authorship rates relative to single-blind review (JNS); and 3) evaluate how female authorship rates compared with the number of women entering neurosurgery residency and obtaining neurosurgery board certification.METHODSGenders of the first and last authors for JNS and Neurosurgery articles from 2010 to 2019 were obtained. Data were also gathered on the number and percentage of women entering neurosurgery residency and women obtaining American Board of Neurological Surgeons (ABNS) certification between 2010 and 2019.RESULTSWomen accounted for 13.4% (n = 570) of first authors and 6.8% (n = 240) of last authors in JNS and Neurosurgery publications. No difference in rates of women publishing existed between the two journals (first authors: 13.0% JNS vs 13.9% Neurosurgery, p = 0.29; last authors: 7.3% JNS vs 6.0% Neurosurgery, p = 0.25). No difference existed between women first or last authors in Neurosurgery before and after initiation of double-blind review (p = 0.066). Significant concordance existed between the gender of first and last authors: in publications with a woman last author, the odds of the first author being a woman was increased by twofold (OR 2.14 [95% CI 1.43–3.13], p = 0.0001). Women represented a lower proportion of authors of invited papers (8.6% of first authors and 3.1% of last authors were women) compared with noninvited papers (14.1% of first authors and 7.4% of last authors were women) (first authors: OR 0.576 [95% CI 0.410–0.794], p = 0.0004; last authors: OR 0.407 [95% CI 0.198–0.751], p = 0.001). The proportion of women US last authors (7.4%) mirrors the percentage of board-certified women neurosurgeons (5.4% in 2010 and 6.8% in 2019), while the percentage of women US first authors (14.3%) is less than that for women entering neurosurgical residency (11.2% in 2009 and 23.6% in 2018).CONCLUSIONSThis is the first report of female authorship in the neurosurgical literature. The authors found that single- versus double-blind peer review did not impact female authorship rates at two top neurosurgical journals.

scholarly journals Peer review blinding practices of highly ranked dental journals: analysis and discussion

BDJ ◽  
2021 ◽  
Author(s):  
Clovis Mariano Faggion Jr
Keyword(s):  
Logistic Regression ◽  
Peer Review ◽  
Odds Ratio ◽  
Review Process ◽  
Peer Review Process ◽  
Linear Regression Analysis ◽  
Double Blind ◽  
Lower Odds ◽  
Blind Review ◽  
Single Blind

AbstractObjectives To evaluate the type of peer review blinding used in highly ranked dental journals and to discuss the influence of the blinding approaches on the peer review process.Methods All 91 dental journals classified by impact factor (IF) had their websites scrutinised for the type of peer review blinding used for submissions. If the information was not reported, the journals were contacted to obtain the information. Linear and logistic regression were applied to evaluate the association between type of peer review blinding and IF.Results The selected journals reported the following peer review blinding approaches: single-blind (N = 36, 39.6%), double-blind (N = 46, 50.5%), transparent (N = 2, 2.2%) and open (N = 1, 1.1%). Information from six (6.6%) journals was not available. A linear regression analysis demonstrated that journals with lower IFs were associated with double-blind review (p = 0.001). A logistic regression suggested lower odds of association between single-blind peer review and journals with IFs below a threshold of 2 (odds ratio 0.157, confidence interval 0.059 to 0.417, p <0.001).Conclusions The majority of highly ranked dental journals had single- and double-blind peer review; journals with higher IFs presented single-blind peer review and those with lower IFs reported double-blind peer review.

scholarly journals Peer review declaration

2021 ◽  
Vol 2104 (1) ◽  
pp. 011002
Keyword(s):  
Peer Review ◽  
Management System ◽  
Review Process ◽  
Acceptance Rate ◽  
Review System ◽  
Double Blind ◽  
Contact Person ◽  
Blind Review ◽  
Content Review ◽  
Single Blind

All conference organizers/editors are required to declare details about their peer review. Therefore, please provide the following information: • Type of peer review: Single-blind / Double-blind / Triple-blind / Open / Other (please describe) The SNPF 2021 article review process was carried out using a single-blind review system. The number of papers submitted was 84 articles. The number of SNPF 2021 reviewers is 14 people. One reviewer reviewed one article. The review process was done via email and or OCS. The article review results were returned to the authors for revision within a certain period of time. The author sent the revised results, and the plagiarism checked results of the article. The editor then rechecked the revision result. If suitable, it would be forwarded to the editor, either for plagiarism, language, or template. • Conference submission management system: OCS (http://snpfmotogpe.ulm.ac.id/ocs/index.php/snpf/2021) • The number of submissions received: There were 84 articles submitted to SNPF 2021 • The number of submissions sent for review: There were 84 articles reviewed by reviewers of SNPF 2021. • The number of submissions accepted: There were 40 articles accepted for recommendation/publication to JPCS IOP Publishing. • Acceptance Rate (Number of Submissions Accepted / Number of Submissions Received X 100): 47,61% • The average number of reviews per paper: One article was reviewed four times: content review 2x, language review 1x, and template review 1x. • The total number of reviewers involved: There were 14 reviewers (content review), 10 editors (content and template review), and 4 people (language review) • Any additional info on the review process: The author sent the revised article along with the similarity check (maximum 20%). A similarity check was also carried out using Turnitin (maximum 20%) when an article was declared fit for publication. So, the similarity check was done two times. • Contact person for queries: +628975586104 (Misbah) Universitas Lambung Mangkurat [email protected]

scholarly journals Peer review declaration

2021 ◽  
Vol 2106 (1) ◽  
pp. 011002
Keyword(s):  
Peer Review ◽  
Management System ◽  
Review Process ◽  
Acceptance Rate ◽  
Review System ◽  
Double Blind ◽  
Contact Person ◽  
Blind Review ◽  
Scientific Standards ◽  
Single Blind

All papers published in this volume of Journal of Physics: Conference Series have been peer reviewed through processes administered by the Editors. Reviews were conducted by expert referees to the professional and scientific standards expected of a proceedings journal published by IOP Publishing. • Type of peer review: Single-blind / Double-blind / Triple-blind / Open / Other (please describe) The ICMSS 2021 article review process was carried out using a single-blind review system. One reviewer reviewed 1 article. The number of papers submitted was 52 articles. The number of ICMSS 2021 reviewers is 8 people. 1 article reviewed by 1 reviewer. The review process is done via email and or OCS. The results of the review of the article are returned to the author for revision within a certain period of time. The author sends the revised results and the plagiarism check results of the article. The editor then rechecks the revision result. If it is suitable, it will be forwarded to the editor, whether it is checked for plagiarism, language, or template. • Conference submission management system: OCS (https://conference.ulm.ac.id/index.php/icmss/icmss/) • Number of submissions received: There are 50 articles submitted to ICMSS 2021 • Number of submissions sent for review: There are 50 articles reviewed by reviewers ICMSS 2021. • Number of submissions accepted: 31 articles were accepted for recommendation/publication to JPCS IOP Publishing. • Acceptance Rate (Number of Submissions Accepted / Number of Submissions Received X 100): 62% • Average number of reviews per paper: One article was reviewed 4 times, reviewed content twice, reviews related to language 1x, and review template 1x. • Total number of reviewers involved: There are 8 reviewers (review content), There are 5 editors (review content and templates), and 6 people (review language) • Any additional info on review process: The author sends the revised article along with the similarity check (maximum 20%). When an article is declared fit for publication, a similarity check is also carried out using Turnitin (maximum 20%). So, the similarity check is done 2 times. • Contact person for queries: +628975586104 (Misbah) [email protected] Universitas Lambung Mangkurat

scholarly journals Double-blind review in software engineering venues

2016 ◽  
Author(s):  
Moritz Beller ◽  
Alberto Bacchelli
Keyword(s):  
Side Effects ◽  
Software Engineering ◽  
Scientific Method ◽  
Review Process ◽  
Peer Review Process ◽  
Double Blind ◽  
Blind Review ◽  
Hidden Biases ◽  
Benefits And Costs ◽  
Single Blind

The peer review process is central to the scientific method, the advancement and spread of research, as well as crucial for individual careers. However, the single-blind review mode currently used in most Software Engineering (SE) venues is susceptible to apparent and hidden biases, since reviewers know the identity of authors. We perform a study on the benefits and costs that are associated with introducing double- blind review in SE venues. We surveyed the SE community’s opinion and interviewed experts on double-blind reviewing. Our results indicate that the costs, mostly logistic challenges and side effects, outnumber its benefits and mostly regard difficulty for authors in blinding papers, for reviewers in understanding the increment with respect to previous work from the same authors, and for organizers to manage a complex transition. While the surveyed community largely consents on the costs of DBR, only less than one-third disagree with a switch to DBR for SE journals, all SE conferences, and, in particular, ICSE; the analysis of a survey with authors of submitted papers at ICSE 2016 run by the program chairs of that edition corroborates our result.

scholarly journals Double-blind review in software engineering venues

2016 ◽  
Author(s):  
Moritz Beller ◽  
Alberto Bacchelli
Keyword(s):  
Side Effects ◽  
Software Engineering ◽  
Scientific Method ◽  
Review Process ◽  
Peer Review Process ◽  
Double Blind ◽  
Blind Review ◽  
Hidden Biases ◽  
Benefits And Costs ◽  
Single Blind

The peer review process is central to the scientific method, the advancement and spread of research, as well as crucial for individual careers. However, the single-blind review mode currently used in most Software Engineering (SE) venues is susceptible to apparent and hidden biases, since reviewers know the identity of authors. We perform a study on the benefits and costs that are associated with introducing double- blind review in SE venues. We surveyed the SE community’s opinion and interviewed experts on double-blind reviewing. Our results indicate that the costs, mostly logistic challenges and side effects, outnumber its benefits and mostly regard difficulty for authors in blinding papers, for reviewers in understanding the increment with respect to previous work from the same authors, and for organizers to manage a complex transition. While the surveyed community largely consents on the costs of DBR, only less than one-third disagree with a switch to DBR for SE journals, all SE conferences, and, in particular, ICSE; the analysis of a survey with authors of submitted papers at ICSE 2016 run by the program chairs of that edition corroborates our result.

scholarly journals Peer review declaration

2022 ◽  
Vol 2155 (1) ◽  
pp. 011002
Keyword(s):  
Peer Review ◽  
Nuclear Physics ◽  
Review Process ◽  
Program Committee ◽  
Double Blind ◽  
Editorial Committee ◽  
Blind Review ◽  
Check System ◽  
Paper Abstract ◽  
Single Blind

The Organiser and/or the Editor(s) are required to declare details about their peer review processes. Therefore, please provide the following information: • Type of peer review: Single-blind / Double-blind / Triple-blind / Open / Other Single-blind review Single-anonymous: authors’ identities are known to the reviewers, reviewers’ identities are hidden from authors; • Describe criteria used by Reviewers when accepting/declining papers. Was there the opportunity to resubmit articles after revisions? The following criteria were applied: 1. Quality assessment Significance, novelty, correctness Special attention was paid to repetition and Plagiarism. 2. Technical Criteria Clarity of expression; readability and discussion of concepts Sufficient discussion of the context of the work, and suitable referencing. 3. Presentation Criteria 1. Is it clearly presented, well organized, and clearly written?( clear presentation, well organized, clearly stated) 2. Is the English satisfactory? (satisfactory english) 3. Is the title appropriate? (Title matches) 4. Does the abstract include the important points of the paper?( abstract contains important information on the article) 5. Are references to related work adequate, up to date and readily available? (links are relevant, relevant, available) 6. Are figures and tables necessary and adequate?( tables and figures are necessary and appropriate) 7. Are the conclusions satisfactory? (conclusion is appropriate) During the review process, the authors were given a one-time opportunity to re-submit the article for review. • Conference submission management system: • The peer review was carried out by the Forum Program Committee, organized according to the order of the General Director of the RSE INP No. 182 dated 22.24.2021. The Program Committee carried out a preliminary selection of articles to be sent to the reviewers. • Number of submissions received: 44 • Number of submissions sent for review: 42 • Number of submissions accepted: 35 • Acceptance Rate (Number of Submissions Accepted / Number of Submissions Received X 100): 79,54 • Average number of reviews per paper: 2 • Total number of reviewers involved: 40 • Any additional info on review process (eg Plagiarism check system): • Review process consisted from few steps: • 1) submission by editorial committee • 2) 2 reviewers received publication (anonymously, author did not know any of reviewer) • 3) after check, authors fixed all mistakes and requirements from reviewers) Checking for plagiarism, showed no repeat or copy of submitted material. • Contact person for queries (Full name, affiliation, institutional email address) Name : Nassurlla Maulen Affiliation: Institute of Nuclear Physics Republic of Kazakhstan Email : nassurlla [email protected] ( additional: [email protected])

Sign in / Sign up

Export Citation Format

Share Document