Citation Patterns Following a Strongly Contradictory Replication Result: Four Case Studies From Psychology

Replication studies that contradict prior findings may facilitate scientific self-correction by triggering a reappraisal of the original studies; however, the research community’s response to replication results has not been studied systematically. One approach for gauging responses to replication results is to examine how they affect citations to original studies. In this study, we explored postreplication citation patterns in the context of four prominent multilaboratory replication attempts published in the field of psychology that strongly contradicted and outweighed prior findings. Generally, we observed a small postreplication decline in the number of favorable citations and a small increase in unfavorable citations. This indicates only modest corrective effects and implies considerable perpetuation of belief in the original findings. Replication results that strongly contradict an original finding do not necessarily nullify its credibility; however, one might at least expect the replication results to be acknowledged and explicitly debated in subsequent literature. By contrast, we found substantial citation bias: The majority of articles citing the original studies neglected to cite relevant replication results. Of those articles that did cite the replication but continued to cite the original study favorably, approximately half offered an explicit defense of the original study. Our findings suggest that even replication results that strongly contradict original findings do not necessarily prompt a corrective response from the research community.

Low-energy sweeteners and body weight: a citation network analysis

ObjectiveReviews on the relationship of low-energy sweeteners (LES) with body weight (BW) have reached widely differing conclusions. To assess possible citation bias, citation analysis was used to quantify the relevant characteristics of cited articles, and explore citation patterns in relation to review conclusions.DesignA systematic search identified reviews published from January 2010 to March 2020. Different characteristics (for example, type of review or research, journal impact factor, conclusions) were extracted from the reviews and cited articles. Logistic regression was used to estimate likelihood of articles with particular characteristics being cited in reviews. A qualitative network analysis linked reviews sub-grouped by conclusions with the types of articles they cited.Main outcome measures(OR; 95% CI) for likelihood that articles with particular characteristics were cited as evidence in reviews.ResultsFrom 33 reviews identified, 183 different articles were cited (including other reviews). Narrative reviews were 62% less likely to be cited than systematic reviews with meta-analysis (OR 0.38; 0.16 to 0.86; p=0.03). Likelihood of being cited was higher for evidence on children than adults (OR 2.27; 1.59 to 3.25; p<0.0001), and with increased journal impact factor (OR 1.15; 1.00 to 1.31; p=0.04). No other factors were statistically significant in the main analysis, and few factors were significant in subgroup analyses. Network analysis showed that reviews concluding a beneficial relationship of LES with BW cited mainly randomised controlled trials, whereas reviews concluding an adverse relationship cited mainly observational studies.ConclusionsOverall reference to the available evidence across reviews appears largely arbitrary, making citation bias likely. Differences in the conclusions of individual reviews map onto different types of evidence cited. Overall, inconsistent and selective use of the available evidence may account for the diversity of conclusions in reviews on LES and BW.Trial registration numberPrior to data analysis, the protocol was registered with the Open Science Framework (https://osf.io/9ghws).

Post-replication citation patterns in psychology: Four case studies

Replication studies that contradict prior findings may facilitate scientific self-correction by triggering a reappraisal of the original studies; however, the research community's response to replication results has not been studied systematically. One approach for gauging responses to replication results is to examine how they impact citations to original studies. In this study, we explored post-replication citation patterns in the context of four prominent multi-laboratory replication attempts published in the field of psychology that strongly contradicted and outweighed prior findings. Generally, we observed a small post-replication decline in the number of favourable citations and a small increase in unfavourable citations. This indicates only modest corrective effects and implies considerable perpetuation of belief in the original findings. Replication results that strongly contradict an original finding do not necessarily nullify its credibility; however, one might at least expect the replication results to be acknowledged and explicitly debated in subsequent literature. By contrast, we found substantial citation bias: the majority of articles citing the original studies neglected to cite relevant replication results. Of those articles that did cite the replication, but continued to cite the original study favourably, approximately half offered an explicit defence of the original study. Our findings suggest that even replication results that strongly contradict original findings do not necessarily prompt a corrective response from the research community.

Citation bias in otolaryngology systematic reviews

Objective: Reproducibility of systemic reviews (SRs) can be hindered by the presence of citation bias. Citation bias may occur when authors of SRs conduct hand-searches of included study reference lists to identify additional studies. Such a practice may lead to exaggerated SR summary effects. The purpose of this paper is to examine the prevalence of hand-searching reference lists in otolaryngology SRs.Methods: The authors searched for systematic reviews published in eight clinical otolaryngology journals using the Cochrane Library and PubMed, with the date parameter of January 1, 2008, to December 31, 2017. Two independent authors worked separately to extract data from each SR for the following elements: whether reference lists were hand-searched, other kinds of supplemental searching, PRISMA adherence, and funding source. Following extraction, the investigators met to review discrepancies and achieve consensus.Results: A total of 539 systemic reviews, 502 from clinical journals and 37 from the Cochrane library, were identified. Of those SRs, 72.4% (390/539) hand-searched reference lists, including 97.3% (36/37) of Cochrane reviews. For 228 (58.5%) of the SRs that hand-searched reference lists, no other supplemental search (e.g., search of trial registries) was conducted.Conclusions: These findings indicate that hand-searching reference lists is a common practice in otolaryngology SRs. Moreover, a majority of studies at risk of citation bias did not attempt to mitigate the bias by conducting additional supplemental searches. The implication is that summary effects in otolaryngology systematic reviews may be biased toward statistically significant findings.

Comment on: The association between early career informal mentorship in academic collaborations and junior author performance

This comment is written in response to the recent article published in Nature Communications (17th November 2020) entitled, ‘The association between early career informal mentorship in academic collaborations and junior author performance’ by AlShebli et al. This short paper provides a theoretical foundation based upon current literature to propose an alternative framework from which to conduct the analysis of the data collected by AlShebli et al. (2020).We believe that AlShebli et al’s (2020) results corroborate, at unique scale, the well-documented gender citation bias. We find that the interpretation of the data, that junior academic careers (both men and women) see greater career benefits when being mentored by senior men, is lacking theoretical foundation and not cognisant of the accepted knowledge on this topic. We are motivated in responding to this article by our concern that should this interpretation go unchallenged, we see a very high risk to the participation and progression of women in STEM academia.

Determinants of citation in the literature on diesel exhaust exposure and lung cancer: a citation analysis

ObjectivesEpidemiological research on the association between diesel exhaust exposure and lung cancer risk has some methodological challenges that give rise to different conclusions and intense debates. This raises the question about the role of selective citation and of citation bias in particular. Our aim was to investigate the occurrence and prevalence of selective citation in this field.DesignCitation analysis.SettingWeb of Science Core Collection.ParticipantsWe identified 96 publications in this network, with 4317 potential citations. For each publication, we extracted characteristics such as study conclusion and funding source. Some of these characteristics are related to the study content: study design, sample size, method of diesel exposure assessment, type of diesel technology under investigation, and whether smoking had been adjusted for.Primary and secondary outcome measuresWhether a citation occurs or not, measured and analysed according to the preregistered protocol. Exploratively we analysed the association between funding source and study conclusion.ResultsMethodological content of a study was clearly related to citation, studies using more sophisticated methods were more likely to be cited. There was some evidence for citation bias: supportive publications had a higher chance of being cited than non-supportive ones, but after adjustment for study quality, this effect decreased substantially (adjusted OR 1.3, 95% CI 1.0 to 1.7). Explorative analyses indicated that three quarters of non-profit funded publications had a supportive study conclusion against only one quarter of the industry-funded publications.ConclusionsThere is evidence for selective citation within this field, but the evidence for citation bias was weak. It seems that factors related to the methodology had more impact on citation than the conclusion of a study. Interestingly, publications that were funded by industry were more skeptical about a causal relationship between diesel exhaust and lung cancer compared to non-profit-funded publications.

Determinants of Citation in Epidemiological Studies on Phthalates: A Citation Analysis

Citing of previous publications is an important factor in knowledge development. Because of the great amount of publications available, only a selection of studies gets cited, for varying reasons. If the selection of citations is associated with study outcome this is called citation bias. We will study determinants of citation in a broader sense, including e.g. study design, journal impact factor or the funding source of the publication. As a case study we assess which factors drive citation in the human literature on phthalates, specifically the metabolite mono(2-ethylhexyl) phthalate (MEHP). A systematic literature search identified all relevant publications on human health effect of MEHP. Data on potential determinants of citation were extracted in duplo. Specialized software was used to create a citation network, including all potential citation pathways. Random effect logistic regression was used to assess whether these determinants influence the likelihood of citation. 112 Publications on MEHP were identified, with 5684 potential citation pathways of which 551 were actual citations. Reporting of a harmful point estimate, journal impact factor, authority of the author, a male corresponding author, research performed in North America and self-citation were positively associated with the likelihood of being cited. In the literature on MEHP, citation is mostly driven by a number of factors that are not related to study outcome. Although the identified determinants do not necessarily give strong indications of bias, it shows selective use of published literature for a variety of reasons.