Citations and journal impact factors: questionable indicators of research quality

‘If there is one thing every bibliometrician agrees, is that you should never use the journal impact factor (JIF) to evaluate research performance for an article or an individual – that is a mortal sin’. Few sentences could define so precisely the uses and misuses of the Journal Impact Factor (JIF) better than Anthony van Raan’s. This manuscript presents a critical overview on the international use, by governments and institutions, of the JIF and/or journal indexing information for individual research quality assessment. Interviews given by Nobel Laureates speaking on this matter are partially illustrated in this work. Furthermore, the authors propose complementary and alternative versions of the journal impact factor, respectively named Complementary (CIF) and Timeless (TIF) Impact Factors, aiming to better assess the average quality of a journal – never of a paper or an author. The idea behind impact factors is not useless, it has just been misused.

Download Full-text

Citation counts and journal impact factors do not capture research quality in the behavioral and brain sciences

10.31234/osf.io/9g5wk ◽

2019 ◽

Author(s):

Michael R Dougherty ◽

Zachary Horne

Keyword(s):

Large Scale ◽

Research Evaluation ◽

Research Quality ◽

Experimental Result ◽

Reproducible Research ◽

Impact Factors ◽

Citation Data ◽

Citation Counts ◽

Reported Data ◽

Journal Impact

Citation data and journal impact factors are important components of faculty dossiers and figure prominently in both promotion decisions and assessments of a researcher's broader societal impact. Although these metrics play a large role in high-stakes decisions, the evidence is mixed regarding whether they are valid proxies for key aspects of research quality. We use data from three large scale studies to assess whether citation counts and impact factors predict four indicators of aspects of research quality: (1) the number of statistical reporting errors in a paper, (2) the evidential value of the reported data, (3) the expected replicability of reported research findings in peer reviewed journals, and (4) the actual replicability of a given experimental result. Both citation counts and impact factors were weak and inconsistent predictors of research quality, so defined, and sometimes negatively related to quality. Our findings impugn the validity of citation data and impact factors as indices of research quality and call into question their usefulness in evaluating scientists and their research. In light of these results, we argue that research evaluation should instead focus on the process of how research is conducted and incentivize behaviors that support open, transparent, and reproducible research.

Download Full-text

Research Quality and Newsworthiness of Published Articles are Partial Predictors of Journal Impact Factors

Evidence Based Library and Information Practice ◽

10.18438/b85022 ◽

2012 ◽

Vol 7 (3) ◽

pp. 90

Author(s):

Jason Martin

Keyword(s):

Impact Factor ◽

Journal Impact Factor ◽

Research Quality ◽

Impact Factors ◽

Validation Set ◽

Literature Service ◽

Journal Impact ◽

Quality Of Research ◽

Mcmaster University

Objective – Determine what characteristics of a journal’s published articles can be used to predict the journal impact factor (JIF). Design – A retrospective cohort study. Setting – The researchers are located at McMaster University, Hamilton, Ontario, Canada. Subjects – The sample consisted of 1,267 clinical research articles from 103 evidence based and clinical journals which were published in 2005 and indexed in the McMaster University Premium LiteratUre Service (PLUS) database and those same journals’ JIF from 2007. Method – The articles were divided 60:40 into a derivation set (760 articles and 99 journals) and a validation set (507 articles and 88 journals). Ten variables which could influence JIF were developed and a multiple linear regression was run on the derivation set and then applied to the validation set. Main Results – The four variables found to be significant were the number of databases which indexed the journal, the number of authors, the quality of research, and the “newsworthiness” of the journal’s published articles. Conclusion – The quality of research and newsworthiness at time of publication of a journal’s articles can predict the journal impact factor with 60% accuracy.

Download Full-text

Research Counts, Not the Journal

10.31219/osf.io/4z39a ◽

2018 ◽

Cited By ~ 3

Author(s):

Miguel Abambres ◽

Tiago Ribeiro ◽

Ana Sousa ◽

Eva Olivia Leontien Lantsoght

Keyword(s):

Impact Factor ◽

Journal Impact Factor ◽

Research Performance ◽

Research Quality ◽

Impact Factors ◽

Critical Overview ◽

Research Quality Assessment ◽

Journal Impact ◽

Better Than

If there is one thing every bibliometrician agrees, is that you should never use the journal impact factor (JIF) to evaluate research performance for an article or an individual-that is a mortal sin'. Few sentences could define so precisely the uses and misuses of the Journal Impact Factor (JIF) better than Anthony van Raan's. This manuscript presents a critical overview on the international use, by governments and institutions, of the JIF and/or journal indexing information for individual research quality assessment. Interviews given by Nobel Laureates speaking on this matter are partially illustrated in this work. Furthermore, the authors propose complementary and alternative versions of the journal impact factor, respectively named Complementary (CIF) and Timeless (TIF) Impact Factors, aiming to better assess the average quality of a journal-never of a paper or an author. The idea behind impact factors is not useless, it has just been misused.

Download Full-text

The integrated interventional radiology match: a bibliometric analysis of matched first-year residents

American Journal of Interventional Radiology ◽

10.25259/ajir_23_2019 ◽

2019 ◽

Vol 3 ◽

pp. 13 ◽

Cited By ~ 1

Author(s):

Vishnu Chandra ◽

Neil Jain ◽

Pratik Shukla ◽

Ethan Wajswol ◽

Sohail Contractor ◽

...

Keyword(s):

Interventional Radiology ◽

Bibliometric Analysis ◽

Statistical Significance ◽

Univariate Analysis ◽

Impact Factors ◽

Index Number ◽

Research Information ◽

H Index ◽

Scopus Database ◽

Journal Impact

Objectives: The integrated interventional radiology (IR) residency has only been established relatively recently as compared to other specialties. Although some preliminary information is available based on survey data five, no comprehensive bibliometric analysis documenting the importance of the quantity and quality of research in applying to an integrated-IR program currently exists. As the first bibliometric analysis of matched IR residents, the data obtained from this study fills a gap in the literature. Materials and Methods: A list of matched residents from the 2018 integrated-IR match were identified by contacting program directors. The Scopus database was used to search for resident research information, including total publications, first-author publications, radiology-related publications, and h-indices. Each matriculating program was categorized into one of five tiers based on the average faculty Hirsch index (h-index). Results: Sixty-three programs and 117 matched residents were identified and reviewed on the Scopus database. For the 2018 cycle, 274 total publications were produced by matched applicants, with a mean of 2.34 ± 0.41 publication per matched applicant. The average h-index for matched applicants was 0.96 ± 0.13. On univariate analysis, the number of radiology-related publications, highest journal impact factor, and h-index were all associated with an increased likelihood of matching into a higher tier program (P < 0.05). Other research variables displayed no statistical significance. All applicants with PhDs matched into tier one programs. Conclusions: Research serves as an important element in successfully matching into an integrated-IR residency. h-index, number of radiology-related manuscripts, and highest journal impact factors are all positively associated with matching into a higher tier program.

Download Full-text

The Socio-economic Context of Journal Impact Factors

Proceedings of the Annual Conference of CAIS / Actes du congrès annuel de l'ACSI ◽

10.29173/cais778 ◽

2013 ◽

Author(s):

Brendan Luyt

Keyword(s):

Perceived Value ◽

Modern Society ◽

Impact Factors ◽

Economic Context ◽

Perte D’Autonomie ◽

Journal Impact

This paper argues that the rise of the JIF is a result of the perceived value of quantification measures in modern society and the restructuring of capitalism. Two key implications of this acceptance are explored: an increase in global academic dependency and a lessening of autonomy in the scientific field.Cet article défend la thèse que la montée du FIRS est le résultat de la valeur perçue des mesures de quantification de la société moderne et de la restructuration du capitalisme. Seront explorées deux conséquences importantes de cette acceptation : une augmentation de la dépendance globale du milieu universitaire et une perte d'autonomie du milieu de la science.

Download Full-text

Dynamics of Journal Impact Factors and Limits to Their Inflation

Journal of Scholarly Publishing ◽

10.3138/jsp.50.1.06 ◽

2018 ◽

Vol 50 (1) ◽

pp. 26-36 ◽

Cited By ~ 5

Author(s):

Igor Fischer ◽

Hans-Jakob Steiger

Keyword(s):

Impact Factors ◽

Journal Impact

Download Full-text

Analysis of the journal impact factor and related bibliometric indicators in education and educational research category

Education for Information ◽

10.3233/efi-200442 ◽

2021 ◽

pp. 1-22

Author(s):

Metin Orbay ◽

Orhan Karamustafaoğlu ◽

Ruben Miranda

Keyword(s):

Impact Factor ◽

Educational Research ◽

Journal Impact Factor ◽

Impact Factors ◽

Bibliometric Indicators ◽

Strongly Correlated ◽

Starting Point ◽

Consistent Assessment ◽

The Impact ◽

Journal Impact

This study analyzes the journal impact factor and related bibliometric indicators in Education and Educational Research (E&ER) category, highlighting the main differences among journal quartiles, using Web of Science (Social Sciences Citation Index, SSCI) as the data source. High impact journals (Q1) publish only slightly more papers than expected, which is different to other areas. The papers published in Q1 journal have greater average citations and lower uncitedness rates compared to other quartiles, although the differences among quartiles are lower than in other areas. The impact factor is only weakly negative correlated (r=-0.184) with the journal self-citation but strongly correlated with the citedness of the median journal paper (r= 0.864). Although this strong correlation exists, the impact factor is still far to be the perfect indicator for expected citations of a paper due to the high skewness of the citations distribution. This skewness was moderately correlated with the citations received by the most cited paper of the journal (r= 0.649) and the number of papers published by the journal (r= 0.484), but no important differences by journal quartiles were observed. In the period 2013–2018, the average journal impact factor in the E&ER has increased largely from 0.908 to 1.638, which is justified by the field growth but also by the increase in international collaboration and the share of papers published in open access. Despite their inherent limitations, the use of impact factors and related indicators is a starting point for introducing the use of bibliometric tools for objective and consistent assessment of researcher.

Download Full-text