Professional and citizen bibliometrics: complementarities and ambivalences in the development and use of indicators—a state-of-the-art report

AbstractBibliometric indicators such as journal impact factors, h-indices, and total citation counts are algorithmic artifacts that can be used in research evaluation and management. These artifacts have no meaning by themselves, but receive their meaning from attributions in institutional practices. We distinguish four main stakeholders in these practices: (1) producers of bibliometric data and indicators; (2) bibliometricians who develop and test indicators; (3) research managers who apply the indicators; and (4) the scientists being evaluated with potentially competing career interests. These different positions may lead to different and sometimes conflicting perspectives on the meaning and value of the indicators. The indicators can thus be considered as boundary objects which are socially constructed in translations among these perspectives. This paper proposes an analytical clarification by listing an informed set of (sometimes unsolved) problems in bibliometrics which can also shed light on the tension between simple but invalid indicators that are widely used (e.g., the h-index) and more sophisticated indicators that are not used or cannot be used in evaluation practices because they are not transparent for users, cannot be calculated, or are difficult to interpret.

Download Full-text

Analysis of the journal impact factor and related bibliometric indicators in education and educational research category

Education for Information ◽

10.3233/efi-200442 ◽

2021 ◽

pp. 1-22

Author(s):

Metin Orbay ◽

Orhan Karamustafaoğlu ◽

Ruben Miranda

Keyword(s):

Impact Factor ◽

Educational Research ◽

Journal Impact Factor ◽

Impact Factors ◽

Bibliometric Indicators ◽

Strongly Correlated ◽

Starting Point ◽

Consistent Assessment ◽

The Impact ◽

Journal Impact

This study analyzes the journal impact factor and related bibliometric indicators in Education and Educational Research (E&ER) category, highlighting the main differences among journal quartiles, using Web of Science (Social Sciences Citation Index, SSCI) as the data source. High impact journals (Q1) publish only slightly more papers than expected, which is different to other areas. The papers published in Q1 journal have greater average citations and lower uncitedness rates compared to other quartiles, although the differences among quartiles are lower than in other areas. The impact factor is only weakly negative correlated (r=-0.184) with the journal self-citation but strongly correlated with the citedness of the median journal paper (r= 0.864). Although this strong correlation exists, the impact factor is still far to be the perfect indicator for expected citations of a paper due to the high skewness of the citations distribution. This skewness was moderately correlated with the citations received by the most cited paper of the journal (r= 0.649) and the number of papers published by the journal (r= 0.484), but no important differences by journal quartiles were observed. In the period 2013–2018, the average journal impact factor in the E&ER has increased largely from 0.908 to 1.638, which is justified by the field growth but also by the increase in international collaboration and the share of papers published in open access. Despite their inherent limitations, the use of impact factors and related indicators is a starting point for introducing the use of bibliometric tools for objective and consistent assessment of researcher.

Download Full-text

Citation counts and journal impact factors do not capture research quality in the behavioral and brain sciences

10.31234/osf.io/9g5wk ◽

2019 ◽

Author(s):

Michael R Dougherty ◽

Zachary Horne

Keyword(s):

Large Scale ◽

Research Evaluation ◽

Research Quality ◽

Experimental Result ◽

Reproducible Research ◽

Impact Factors ◽

Citation Data ◽

Citation Counts ◽

Reported Data ◽

Journal Impact

Citation data and journal impact factors are important components of faculty dossiers and figure prominently in both promotion decisions and assessments of a researcher's broader societal impact. Although these metrics play a large role in high-stakes decisions, the evidence is mixed regarding whether they are valid proxies for key aspects of research quality. We use data from three large scale studies to assess whether citation counts and impact factors predict four indicators of aspects of research quality: (1) the number of statistical reporting errors in a paper, (2) the evidential value of the reported data, (3) the expected replicability of reported research findings in peer reviewed journals, and (4) the actual replicability of a given experimental result. Both citation counts and impact factors were weak and inconsistent predictors of research quality, so defined, and sometimes negatively related to quality. Our findings impugn the validity of citation data and impact factors as indices of research quality and call into question their usefulness in evaluating scientists and their research. In light of these results, we argue that research evaluation should instead focus on the process of how research is conducted and incentivize behaviors that support open, transparent, and reproducible research.

Download Full-text

Letter: A call for a radical change in research evaluation in Spain

El profesional de la información ◽

10.3145/epi.2021.may.09 ◽

2021 ◽

Author(s):

Emilio Delgado-López-Cózar ◽

Ismael Ràfols ◽

Ernest Abadal

Keyword(s):

Research Evaluation ◽

Radical Change ◽

Individual Performance ◽

Scientific Activity ◽

Bibliometric Indicators ◽

Political Leaders ◽

Academic Fields ◽

Evaluation Policies ◽

The Individual ◽

Journal Impact

This letter is a call to the Spanish scientific authorities to abandon current research evaluation policies, which are based on an excessive and indiscriminate use of bibliometric indicators for nearly all areas of scientific activity. This narrow evaluation focus is especially applied to assess the individual performance of researchers. To this end, we first describe the contexts in which the journal impact factor (JIF) and other bibliometric indicators are being used. We then consider the toxic effects of this abuse of indicators. Finally, we outline some significant transformations and initiatives being introduced in various academic fields and regions of the world. These international initiatives offer alternatives to bibliometrics that can improve evaluation processes, and we urge political leaders in Spain to adopt and develop them.

Download Full-text

The integrated interventional radiology match: a bibliometric analysis of matched first-year residents

American Journal of Interventional Radiology ◽

10.25259/ajir_23_2019 ◽

2019 ◽

Vol 3 ◽

pp. 13 ◽

Cited By ~ 1

Author(s):

Vishnu Chandra ◽

Neil Jain ◽

Pratik Shukla ◽

Ethan Wajswol ◽

Sohail Contractor ◽

...

Keyword(s):

Interventional Radiology ◽

Bibliometric Analysis ◽

Statistical Significance ◽

Univariate Analysis ◽

Impact Factors ◽

Index Number ◽

Research Information ◽

H Index ◽

Scopus Database ◽

Journal Impact

Objectives: The integrated interventional radiology (IR) residency has only been established relatively recently as compared to other specialties. Although some preliminary information is available based on survey data five, no comprehensive bibliometric analysis documenting the importance of the quantity and quality of research in applying to an integrated-IR program currently exists. As the first bibliometric analysis of matched IR residents, the data obtained from this study fills a gap in the literature. Materials and Methods: A list of matched residents from the 2018 integrated-IR match were identified by contacting program directors. The Scopus database was used to search for resident research information, including total publications, first-author publications, radiology-related publications, and h-indices. Each matriculating program was categorized into one of five tiers based on the average faculty Hirsch index (h-index). Results: Sixty-three programs and 117 matched residents were identified and reviewed on the Scopus database. For the 2018 cycle, 274 total publications were produced by matched applicants, with a mean of 2.34 ± 0.41 publication per matched applicant. The average h-index for matched applicants was 0.96 ± 0.13. On univariate analysis, the number of radiology-related publications, highest journal impact factor, and h-index were all associated with an increased likelihood of matching into a higher tier program (P < 0.05). Other research variables displayed no statistical significance. All applicants with PhDs matched into tier one programs. Conclusions: Research serves as an important element in successfully matching into an integrated-IR residency. h-index, number of radiology-related manuscripts, and highest journal impact factors are all positively associated with matching into a higher tier program.

Download Full-text

The Socio-economic Context of Journal Impact Factors

Proceedings of the Annual Conference of CAIS / Actes du congrès annuel de l'ACSI ◽

10.29173/cais778 ◽

2013 ◽

Author(s):

Brendan Luyt

Keyword(s):

Perceived Value ◽

Modern Society ◽

Impact Factors ◽

Economic Context ◽

Perte D’Autonomie ◽

Journal Impact

This paper argues that the rise of the JIF is a result of the perceived value of quantification measures in modern society and the restructuring of capitalism. Two key implications of this acceptance are explored: an increase in global academic dependency and a lessening of autonomy in the scientific field.Cet article défend la thèse que la montée du FIRS est le résultat de la valeur perçue des mesures de quantification de la société moderne et de la restructuration du capitalisme. Seront explorées deux conséquences importantes de cette acceptation : une augmentation de la dépendance globale du milieu universitaire et une perte d'autonomie du milieu de la science.

Download Full-text

Dynamics of Journal Impact Factors and Limits to Their Inflation

Journal of Scholarly Publishing ◽

10.3138/jsp.50.1.06 ◽

2018 ◽

Vol 50 (1) ◽

pp. 26-36 ◽

Cited By ~ 5

Author(s):

Igor Fischer ◽

Hans-Jakob Steiger

Keyword(s):

Impact Factors ◽

Journal Impact

Download Full-text

Reactive Balance Control for Legged Robots under Visco-Elastic Contacts

Applied Sciences ◽

10.3390/app11010353 ◽

2020 ◽

Vol 11 (1) ◽

pp. 353

Author(s):

Thomas Flayols ◽

Andrea Del Prete ◽

Majid Khadiv ◽

Nicolas Mansard ◽

Ludovic Righetti

Keyword(s):

Inverse Dynamics ◽

State Of The Art ◽

Balance Control ◽

Contact Stiffness ◽

Poor Performance ◽

Admittance Control ◽

Inverse Dynamics Control ◽

Rigid Contact ◽

Reactive Balance ◽

Shed Light

Contacts between robots and environment are often assumed to be rigid for control purposes. This assumption can lead to poor performance when contacts are soft and/or underdamped. However, the problem of balancing on soft contacts has not received much attention in the literature. This paper presents two novel approaches to control a legged robot balancing on visco-elastic contacts, and compares them to other two state-of-the-art methods. Our simulation results show that performance heavily depends on the contact stiffness and the noises/uncertainties introduced in the simulation. Briefly, the two novel controllers performed best for soft/medium contacts, whereas “inverse-dynamics control under rigid-contact assumptions” was the best one for stiff contacts. Admittance control was instead the most robust, but suffered in terms of performance. These results shed light on this challenging problem, while pointing out interesting directions for future investigation.

Download Full-text