Presenting Research Impact: Metrics, Misuse, and Movement
— by Fanny Liu
Introduction
Research metrics are quantitative measurements designed to evaluate research outputs and impacts (Wilsdon, 2015). They consist of different measures and statistical methods for assessing the quality and impact of research.
Traditionally, research metrics (bibliometrics) focuses on the quantitative analysis of research outputs, most commonly scholarly publications. The most extensively utilized bibliometrics relies on number of citations (citation counts) because citations from other works suggest that the cited work has to a certain extent influenced the citing work. Some examples include Journal Impact Factor and h-index.
(Read more: Research Metrics@HKUL)
Altmetrics are alternative metrics covering measures beyond citation counts, such as social media shares and citations from policies. These indicators base on data gathered automatically by computer programs from online environments such as social media websites and news platforms.
(Read more: Beyond citations — Demonstrate your research impact with alternative metrics)
Attention needed
Quality
Citation is not synonymous with “quality”. Metrics often do not distinguish between negative or positive citation – highly cited works may be referenced because they are controversial or even erroneous.
Altmetric score, a weighted measure of all the attention picked up for a research output, provides little context for the types and reasons of engagement (e.g., positive or negative opinion). It alone is difficult to interpret in terms of broader research impact.
Discipline
Among different disciplines, the citation behaviour is also different. For example, papers in Medicine receive a high number of citations, while those in Social Science, Mathematics, or the Humanities may not be that high (Mingers & Leydesdorff, 2015).
Data source
The metrics are calculated based on data from the respective sources, which can vary considerably. For example, Scopus covers 27,950 journals (as of March 2023) while Clarivate’s Journal Citation Reports covers 21,522 journals (as of 29 May 2024).
Manipulation
Manipulation behaviour, which aims to game the metrics, also occurs. E.g., authors may intentionally submit articles to “predatory journals” or “fake” conferences; Journal editors may ask prospective authors to cite other articles from the journal (Biagioli & Lippman, 2020).
Some activities included in altmetric scores, e.g., those related to social media, can be prone to being gamed.
Inherent bias
Citation counts, and researcher level (h-index) metrics are inherently biased. For example, h-index provides senior researchers with a clear advantage when compared to junior researchers (Aubert Bonn & Bouter, 2023); Reviewers would award higher marks when a paper’s author is famous (Huber et al., 2022).
Presenting impact beyond metrics
We have seen the emergence of narrative CVs, which emphasize quality over quantity, and also include narratives about the broader impact (Strinzel et al., 2021).
(Read more: Communicating research impact in academic CVs)
There is also a growing interest around the world in the assessment and demonstration of “broader impacts” brought by research, e.g., benefits for society and culture. In Research Assessment Exercise (RAE) 2026 Framework, research impact assessment should reflect the economical / societal / cultural benefits beyond academia in a scientific way (University Grants Committee, 2023, p. 6). It is crucial to compile an impactful case study which depicts how the piece of research has brought “demonstrable contributions, beneficial effects, valuable changes or advantages” to “the economy, society, culture, public policy or services, health, the environment or quality of life, beyond the academia” (University Grants Committee).
For example, in the UK Research Excellence Framework 2014 (REF), impact was assessed through case studies, which describe the effects of academic research and are given a score between 1* (“recognised but modest”) and 4* (“outstanding”) (Reichard et al., 2020). High-scoring case studies:
- Provided specific and high-magnitude articulations of significance (e.g. specifying “the government’s” or “to the House of Commons”) and reach (e.g. specifying “in England” or “in the US”), instead of focusing on pathway to impact;
- Focused more on descriptions of research findings or the quality of research, rather than research outputs and processes;
- Were clear, direct, and coherent, often included attributional phrases to establish links between research (cause) and impact (effect), and used fewer ambiguous or uncertain phrases;
- Used adjectives more appropriately, as inappropriate use of adjectives might lead to an over-claiming or less factual impression;
- Were significantly easier to read and were maintained at interested and educated non-specialist level.
Conclusion
As stated in The Metric Tide: Independent Review of the Role of Metrics in Research Assessment and Management (Wilsdon, 2015, p. 139):
Quantitative evaluation should support – but not supplant – qualitative, expert assessment.
Extended readings
- Impact case studies in RAE 2020 Hong Kong: https://impact.ugc.edu.hk/
- Impact case study database for REF 2021 UK: https://results2021.ref.ac.uk/impact
- Impact Studies (Engagement and Impact) at Australian Research Council Data Portal: https://dataportal.arc.gov.au/EI/Web/Impact/ImpactStudies
References
Aubert Bonn, N., & Bouter, L. (2023). Research Assessments Should Recognize Responsible Research Practices. Narrative Review of a Lively Debate and Promising Developments. In E. Valdés & J. A. Lecaros (Eds.), Handbook of Bioethical Decisions. Volume II: Scientific Integrity and Institutional Ethics (pp. 441-472). Springer International Publishing. https://doi.org/10.1007/978-3-031-29455-6_27
Biagioli, M., & Lippman, A. (2020). Introduction: Metrics and the New Ecologies of Academic Misconduct. In Gaming the Metrics: Misconduct and Manipulation in Academic Research (pp. 1-23). The MIT Press. https://doi.org/10.7551/mitpress/11087.003.0001
Huber, J., Inoua, S., Kerschbamer, R., König-Kersting, C., Palan, S., & Smith, V. L. (2022). Nobel and novice: Author prominence affects peer review. Proceedings of the National Academy of Sciences, 119(41), e2205779119. https://doi.org/10.1073/pnas.2205779119
Mingers, J., & Leydesdorff, L. (2015). A review of theory and practice in scientometrics. European Journal of Operational Research, 246(1), 1-19. https://doi.org/10.1016/j.ejor.2015.04.002
Reichard, B., Reed, M. S., Chubb, J., Hall, G., Jowett, L., Peart, A., & Whittle, A. (2020). Writing impact case studies: a comparative study of high-scoring and low-scoring case studies from REF2014. Palgrave Communications, 6(1), 31. https://doi.org/10.1057/s41599-020-0394-7
Strinzel, M., Brown, J., Kaltenbrunner, W., de Rijcke, S., & Hill, M. (2021). Ten ways to improve academic CVs for fairer research assessment. Humanities and Social Sciences Communications, 8(1), 251. https://doi.org/10.1057/s41599-021-00929-0
University Grants Committee. What is the definition of impact? Retrieved 30 May from https://impact.ugc.edu.hk/faqs
University Grants Committee. (2023). Research Assessment Exercise (RAE) 2026 Framework. https://www.ugc.edu.hk/doc/eng/ugc/rae/2026/framework.pdf
Wilsdon, J. (2015). The Metric Tide: Independent Review of the Role of Metrics in Research Assessment and Management. SAGE Publications. https://doi.org/10.4135/9781473978782
