[Guest Post] Using the Disruptive Index to Benchmark Innovation in AI Research
— by Tsz Chun Brian Lau, Ka Ling Lau
Note: HKU Libraries is committed to fostering the next generation of researchers and advancing research. In our recent collaboration with the Bachelor of Arts and Sciences in Social Data Science (BASc(SDS)) programme, where students tackle real-world challenges in their Final Year Projects, two groups of students explored disruptiveness, i.e., how much a piece of research shifts an existing field of study rather than merely building upon it. Today’s post is by Tsz Chun Brian Lau and Ka Ling Lau. Brian and Ka Ling were both students in the Bachelor of Arts and Sciences in Social Data Science programme at the University of Hong Kong in 2025-2026.
The Disruptive Index (DI) proposed by Wu, Wang and Evans (2019) and further developed by Bornmann et al. (2020) is the indicator that reflects whether later citers cite the focal paper or the papers the focal paper is citing. A paper is “disruptive” if the field moves away from the previous papers and begins to cite the paper directly, and “consolidating” if the future papers continue to cite the previous papers along with it. A high DI may not only mean the paper has a new concept, but also that it has become a new reference point in its own right, compared to the simple count of citations.
Methodology
Based on Google Scholar’s h5-index measure of the quality of AI venues, we focused on the CVPR, NeurIPS, ICLR, and ICML conferences, which are among the most competitive AI conferences in 2020–2024. For each conference, we scraped a list of accepted papers, followed by retrieving reference lists, citations and affiliations of authors, using the OpenAlex API to build a citation network for each paper. Following cleaning and validation, we were able to keep 16,202 papers: 9,620 from CVPR, 3,109 from NeurIPS, 1,895 from ICLR and 1,578 from ICML.
Key Findings
The total output for the four venues almost doubled during the period, from 5,150 papers in 2020 to over 11,150 papers by 2024. In line with the field’s shift towards transformer architectures and the emergence of large language models, mean disruption scores declined in 2021 but have since risen through 2023–2024. We also found a moderate association between institutional QS Computer Science rank and median disruption score (ρ = −0.421, p < 0.001): disruption scores tended to be consistently higher across the top 50 institutions, though ranking alone did not explain much of the variation. When disruption scores were compared against raw citation counts, a small set of papers with relatively few citations but very high disruption came to light; this work is likely to influence the focus of the field before it gets widely recognized, a class of papers that citation-based metrics would not reveal.
Reflection
The most important thing we learned is how to write and code simultaneously, question our assumptions early, and stay willing to revisit our research questions when new problems emerged late in the timeline.
We are grateful to Professor Shihui Feng for guidance throughout. We also extend our gratitude to Dr Man Fung Lo for his invaluable support and to Fanny Liu and Ollie Zhang from HKU Libraries for their exceptional collaboration.
References
Bornmann, L., Devarakonda, S., Tekles, A., et al. (2020). Disruptive papers published in Scientometrics: meaningful results by using an improved variant of the disruption index originally proposed by Wu, Wang, and Evans (2019). Scientometrics, 123, 1149–1155. https://doi.org/10.1007/s11192-020-03406-8
Wu, L., Wang, D., & Evans, J. A. (2019). Large teams develop and small teams disrupt science and technology. Nature, 566(7744), 378–382. https://doi.org/10.1038/s41586-019-0941-9
