Researchers’ Goals When Assessing Credibility and Impact in Committees and in Their Own Work

In a series of 52 semi-structured interviews with researchers in cell biology, we sought to characterize researchers’ goals when evaluating the credibility (or trustworthiness) and impact of research outputs in two contexts: during researchers' own work (the Discovery context) and when researchers participate in research assessment committees for grant review and hiring and promotion (the Committee context). We have compiled a list of researchers’ goals in these contexts, expressed as desired outcome statements and standardized across the two contexts, which will inform a quantitative survey to validate and prioritize these goals and to identify opportunities for new or improved solutions for research assessment. On the basis of the qualitative data, we examined how these needs intersect in the two contexts. We find that the goals of researchers in the Discovery and Committee context overlap significantly. Both impact and credibility matter in each context. In particular, credibility is the dominant factor in the Discovery context and somewhat less represented but still strongly relevant in the Committee context. Researchers use proxy methods, in particular journal-based proxies, to evaluate all attributes of research outputs and these proxies were reported with similar frequency in both contexts. We also find that researchers seek to understand reproducibility, quality and novelty of research outputs in both contexts, in addition to credibility and impact. While publications remain the dominant unit of research assessment, researchers in our sample also evaluate research data, code and preprints, in both contexts. Our preliminary findings suggest a number of potential opportunities to reduce time, reduce error, or improve the quality of assessment practices, in a manner that avoids journal-based proxies. Amongst these improvements are potential opportunities to (i) provide more reliable signals of credibility, quality, and impact, (ii) apply these signals to publications and preprints, and (iii) improve research assessment guidelines.