Research Impact beyond the Academia 2: Policy Citations
— by Chloe Ng
In the previous post, we discussed the use of patent citations as alternative metrics to measure research impact beyond academia. In this post, we will focus on policy citations, which try to showcase the impact of research on policy, law and regulation.
Policy Metrics
Policy documents refer to documents prepared by or for policymakers, including policy briefs, working papers, technical reports, and speeches. These documents shape public policies by advocating for changes in laws, regulations, and social programs. However, there is no commonly accepted definition of what constitutes a policy document and what does not.
Policy citation counts the number of policy documents that cite the research output. This helps researchers see how their work has benefited society, such as by suggesting new approaches to social issues, provoking public discussions, and improving policy decisions.
Access Policy Metrics
Within HKU Libraries’ subscriptions, policy metrics can be accessed via Altmetric Explorer, Policy Citation Index on the Web of Science, and SciVal Impact Module. Each platform has a unique scope and coverage, as described below.
Altmetric Explorer tracks policy sources by searching organizational websites directly for citations of research publications. Policy documents are searched for mentions via link searching, identifier analysis, and text mining (Altmetric, n.d.).
The Policy Citation Index on the Web of Science includes data from more than 500 policy sources (Clarivate, n.d.).
Given the differences in data coverage and indexing approaches, the policy citations retrieved may vary for the same article across different platforms. The table below shows the policy citation counts of the five most highly cited* articles by HKU authors in 2020, with data retrieved on 10 March 2025.
(Note: This refers to number of citations received by the articles from other scholarly publications indexed in Scopus.)
These variations highlight that the number of policy citations retrieved depends on the database used. While policy citations alone do not narrate social, economic, and technological impacts, the retrieved numbers and citing policy documents suggest insights into potential social benefits. To achieve a more comprehensive understanding, it is crucial to complement policy metrics with qualitative analyses.
Limitations of Policy Metrics
While policy metrics help understand how science contributes to social good, they should be interpreted cautiously. Study argued that it could be unclear why specific publications were cited in policy documents or whether the cited research did directly influence policy decisions (Newson et al., 2018). The inclusion of research citations in policy documents only indicated that the research was accessible and relevant to the policy content, rather than proving its influence on the policy-making process.
Research can influence policy in multiple ways, yet not all are reflected in policy citations (Newson et al., 2018). Researchers can contribute to policymaking through stakeholder discussions, workshops, commissioned literature reviews, parliamentary inquiries and committees. However, these interactions do not always result in the cited research being formally acknowledged in policy documents.
In addition, the tracking of policy-related citations reveals several biases. Regional biases exist in the current selection of policy-related sites, for example, Altmetric primarily tracking international, English documents (Haunschild & Bornmann, 2017). Citation from local policy documents may be underappreciated. There are also disciplinary differences in policy citation rates. For instance, articles in social science and humanities journals tend to receive more policy citations than those in physical sciences and engineering (Szomszor & Adie, 2022). This imbalance echoes with traditional bibliometrics, where the citation patterns are different across different subjects.
While policy metrics are used to complement traditional, citation-based metrics, we shall note that policy citations alone do not fully capture societal impact. Metrics should be supplemented with qualitative analyses for a more comprehensive understanding of how research contributes to society.
References
Altmetric. (n.d.). Policy documents. https://help.altmetric.com/support/solutions/articles/6000236695-policy-documents
Clarivate. (n.d.). Policy Citation Index. https://webofscience.zendesk.com/hc/en-us/articles/33001278836625-Policy-Citation-Index
Elsevier. (n.d.). Where does the policy data come from?. SciVal Support Center. https://service.elsevier.com/app/answers/detail/a_id/35951/supporthub/scival/
Haunschild, R., & Bornmann, L. (2017). How many scientific papers are mentioned in policy-related documents? An empirical investigation using Web of Science and Altmetric data. Scientometrics, 110(3), 1209–1216. https://doi.org/10.1007/s11192-016-2237-2
Newson, R., Rychetnik, L., King, L., Milat, A., & Bauman, A. (2018). Does citation matter? Research citation in policy documents as an indicator of research impact – an Australian obesity policy case-study. Health Research Policy and Systems, 16(1), 55–55. https://doi.org/10.1186/s12961-018-0326-9
Szomszor, M., & Adie, E. (2022). Overton: A bibliometric database of policy document citations. Quantitative Science Studies, 3(3), 624–650. https://doi.org/10.1162/qss_a_00204
