Presenting Research Impact: Metrics, Misuse, and Movement

Introduction 

Research metrics are quantitative measurements designed to evaluate research outputs and impacts (Wilsdon, 2015). They consist of different measures and statistical methods for assessing the quality and impact of research. 

Traditionally, research metrics (bibliometrics) focuses on the quantitative analysis of research outputs, most commonly scholarly publications. The most extensively utilized bibliometrics relies on number of citations (citation counts) because citations from other works suggest that the cited work has to a certain extent influenced the citing work. Some examples include Journal Impact Factor and h-index. 

(Read more: Research Metrics@HKUL

Altmetrics are alternative metrics covering measures beyond citation counts, such as social media shares and citations from policies. These indicators base on data gathered automatically by computer programs from online environments such as social media websites and news platforms. 

(Read more: Beyond citations ⁠— Demonstrate your research impact with alternative metrics

[Image curtesy of いらすとや

Attention needed 

Quality 

Citation is not synonymous with “quality”. Metrics often do not distinguish between negative or positive citation – highly cited works may be referenced because they are controversial or even erroneous. 

Altmetric score, a weighted measure of all the attention picked up for a research output, provides little context for the types and reasons of engagement (e.g., positive or negative opinion). It alone is difficult to interpret in terms of broader research impact. 

Discipline 

Among different disciplines, the citation behaviour is also different. For example, papers in Medicine receive a high number of citations, while those in Social Science, Mathematics, or the Humanities may not be that high (Mingers & Leydesdorff, 2015). 

Data source 

The metrics are calculated based on data from the respective sources, which can vary considerably. For example, Scopus covers 27,950 journals (as of March 2023) while Clarivate’s Journal Citation Reports covers 21,522 journals (as of 29 May 2024). 

Manipulation 

Manipulation behaviour, which aims to game the metrics, also occurs. E.g., authors may intentionally submit articles to “predatory journals” or “fake” conferences; Journal editors may ask prospective authors to cite other articles from the journal (Biagioli & Lippman, 2020). 

Some activities included in altmetric scores, e.g., those related to social media, can be prone to being gamed. 

Inherent bias 

Citation counts, and researcher level (h-index) metrics are inherently biased. For example, h-index provides senior researchers with a clear advantage when compared to junior researchers (Aubert Bonn & Bouter, 2023); Reviewers would award higher marks when a paper’s author is famous (Huber et al., 2022). 

[Image curtesy of いらすとや

Presenting impact beyond metrics 

DORA (​San Francisco Declaration on Research Assessment), of which HKU is a signatory, provides recommendations focusing primarily on practices relating to research articles. The key principles include (San Francisco Declaration on Research Assessment, 2012): 

  • The use of journal-based metrics, such as Journal Impact Factors, should be eliminated in funding, appointment, and promotion considerations; 
  • Research should be assessed on its own merits rather than on the journal in which the research is published​; 
  • The opportunities provided by online publication (such as exploring new indicators of significance and impact) should be capitalized. 
[Image curtesy of いらすとや

We have seen the emergence of narrative CVs, which emphasize quality over quantity, and also include narratives about the broader impact (Strinzel et al., 2021). 

(Read more: Communicating research impact in academic CVs

There is also a growing interest in the assessment and demonstration of “broader impacts” brought by research, e.g., benefits for society and culture. In Research Assessment Exercise (RAE) 2026 Framework, research impact assessment should reflect the economical / societal / cultural benefits beyond academia in a scientific way (University Grants Committee, 2023, p. 6). It is crucial to compile an impactful case study which depicts how the piece of research has brought “demonstrable contributions, beneficial effects, valuable changes or advantages” to “the economy, society, culture, public policy or services, health, the environment or quality of life, beyond the academia” (University Grants Committee). 

For example, in the UK Research Excellence Framework 2014 (REF), impact was assessed through case studies, which describe the effects of academic research and are given a score between 1* (“recognised but modest”) and 4* (“outstanding”) (Reichard et al., 2020). High-scoring case studies: 

  • Provided specific and high-magnitude articulations of significance (e.g. specifying “the government’s” or “to the House of Commons”) and reach (e.g. specifying “in England” or “in the US”), instead of focusing on pathway to impact; 
  • Focused more on descriptions of research findings or the quality of research, rather than research outputs and processes; 
  • Were clear, direct, and coherent, often included attributional phrases to establish links between research (cause) and impact (effect), and used fewer ambiguous or uncertain phrases; 
  • Used adjectives more appropriately, as inappropriate use of adjectives might lead to an over-claiming or less factual impression; 
  • Were significantly easier to read and were maintained at interested and educated non-specialist level. 

Conclusion 

As stated in The Metric Tide: Independent Review of the Role of Metrics in Research Assessment and Management (Wilsdon, 2015, p. 139): 

Quantitative evaluation should support – but not supplant – qualitative, expert assessment. 

Extended readings 

References 

Aubert Bonn, N., & Bouter, L. (2023). Research Assessments Should Recognize Responsible Research Practices. Narrative Review of a Lively Debate and Promising Developments. In E. Valdés & J. A. Lecaros (Eds.), Handbook of Bioethical Decisions. Volume II: Scientific Integrity and Institutional Ethics (pp. 441-472). Springer International Publishing. https://doi.org/10.1007/978-3-031-29455-6_27 

Biagioli, M., & Lippman, A. (2020). Introduction: Metrics and the New Ecologies of Academic Misconduct. In Gaming the Metrics: Misconduct and Manipulation in Academic Research (pp. 1-23). The MIT Press. https://doi.org/10.7551/mitpress/11087.003.0001 

Huber, J., Inoua, S., Kerschbamer, R., König-Kersting, C., Palan, S., & Smith, V. L. (2022). Nobel and novice: Author prominence affects peer review. Proceedings of the National Academy of Sciences, 119(41), e2205779119. https://doi.org/10.1073/pnas.2205779119 

Mingers, J., & Leydesdorff, L. (2015). A review of theory and practice in scientometrics. European Journal of Operational Research, 246(1), 1-19. https://doi.org/10.1016/j.ejor.2015.04.002 

Reichard, B., Reed, M. S., Chubb, J., Hall, G., Jowett, L., Peart, A., & Whittle, A. (2020). Writing impact case studies: a comparative study of high-scoring and low-scoring case studies from REF2014. Palgrave Communications, 6(1), 31. https://doi.org/10.1057/s41599-020-0394-7 

San Francisco Declaration on Research Assessment. (2012). https://sfdora.org/about-dora/ 

Strinzel, M., Brown, J., Kaltenbrunner, W., de Rijcke, S., & Hill, M. (2021). Ten ways to improve academic CVs for fairer research assessment. Humanities and Social Sciences Communications, 8(1), 251. https://doi.org/10.1057/s41599-021-00929-0 

University Grants Committee. What is the definition of impact? Retrieved 30 May from https://impact.ugc.edu.hk/faqs 

University Grants Committee. (2023). Research Assessment Exercise (RAE) 2026 Framework. https://www.ugc.edu.hk/doc/eng/ugc/rae/2026/framework.pdf 

Wilsdon, J. (2015). The Metric Tide: Independent Review of the Role of Metrics in Research Assessment and Management. SAGE Publications. https://doi.org/10.4135/9781473978782 

Share