Evaluation Study of Academic Search Platforms Based on Generative Artificial Intelligence

Abstract

Abstract: The study explores the integration trend of generative artificial intelligence (GenAI) technology and academic search, and comprehensively evaluates the performance of emerging academic search platforms in practical applications, aiming to provide a new pathway for transformation in the field of information retrieval. Four GenAI-powered academic search platforms, Scopus AI, WoS Research Assistant, SciSpace, and Elicit, are selected to construct the evaluation index system on the dimensions of content generation and retrieval generation. The study reveals their differences in terms of literature coverage, generation accuracy and user experience, and further tests the balance between citation recall and precision rate of GenAI. It is found that the four GenAI academic search platforms are smooth and informative in response, but the coverage and precision of the generated content supported by citations need to be further improved. In comparison, Scopus AI performs relatively better in literature coverage, while WoS Research Assistant stands out in terms of precision rate. SciSpace achieves a more balanced performance in cross-disciplinary adaptability, whereas Elicit shows an overall balanced but slightly weaker performance. Although the fluency of these platforms is high, the problem of “information hallucination” still exists, and the accuracy and reliability of the generated content need to be further improved in the future.

Cui Yuhong, Zhao Jintao, Zhang Huan. Evaluation Study of Academic Search Platforms Based on Generative Artificial Intelligence[J]. Libraly Journal, 2026, 45(5): 27-36.

References

［1］克里斯托夫·曼宁,普拉巴卡夫·拉格万,欣里希·舒策，等.信息检索导论（修订版）[M].北京：人民邮电出版社, 2019： 1.
［2］ Anker M S, Hadzibegovic S, Lena A, et al. The difference in referencing in Web of Science, Scopus, and Google Scholar[J]. ESC Heart Failure, 2019, 6（6）： 1291-1312.
［3］ Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need[C]. Proceedings of the 31st International Conference on Neural Information Processing Systems （NIPS-17）, 2017： 6000-6010.
［4］ Lim W M, Gunasekara A, Pallant J L, et al. Generative AI and the future of education： Ragnar-k or reformation? A paradoxical perspective from management educators[J]. The International Journal of Management Education, 2023, 21（2）： 100-790.
［5］车万翔,窦志成,冯岩松,等.大模型时代的自然语言处理：挑战、机遇与发展[J].中国科学：信息科学,2023, 53（9）： 1645-1687.
［6］ Ai Qingyao, Bai Ting, Cao Zhao, et al. Information retrieval meets large language models： a strategic report from chinese ir community[J]. AI Open, 2023, 4（1）： 80-90.
［7］ Borgeaud S, Mensch A, Hoffmann J, et al. Improving language models by retrieving from trillions of tokens[C]//International Conference on Machine Learning. PMLR, 2022： 2206-2240.
［8］ Pinzolits R. AI in academia： an overview of selected tools and their areas of application[J]. MAP Education and Humanities, 2024, 4（1）：37-50.
［9］ Prillaman M K. Is ChatGPT making scientists hyper-productive? The highs and lows of using AI[J]. Nature, 2024, 627（8002）： 16-17.
[10] Bearman M, Ryan J, Ajjawi R. Discourses of artificial intelligence in higher education： a critical literature review[J]. Higher Education, 2023, 86（2）： 369-385.
[11] 练志闲.人工智能学术搜索工具需要审查和监督[N].中国社会科学报,2023-05-29（3）.
[12] 张玉峰,李敏,晏创业.论知识检索与信息检索[J].中国图书馆学报,2003（5）： 22-25.
[13] 孟凡淇.信息检索模型研究综述[J].信息通信,2013（3）： 76.
[14] 陈志兴.基于语义网的信息检索技术研究定量分析（20022011）[J].图书馆学研究,2012（6）： 19-23； 18.
[15] 赵文娟,刘忠宝,郭慧.语义检索模型中的词元扩展算法研究[J].情报科学,2019, 37（5）： 108-114.

[16] 刘延飞,李超,王忠,等.多智能体深度强化学习及可扩展性研究进展[J/OL].计算机工程与应用,127[2024-10-20].http：//kns.cnki.net/kcms/detail/11.2127.TP.20241015.1742.012.html.

[17] Oyelude A. Artificial intelligence （AI） tools for academic research[J]. Library Hi Tech News, 2024, 41（8）： 18-20.
[18] 林鑫,刘泽妃.ChatGPT生成综述的质量评测与应用策略[J].图书情报工作,2024, 68（18）： 32-40.
[19] 郭亚军,周家华,庞义伟,等.ChatGPT赋能信息检索：原理、测评、场景与进路[J/OL].情报理论与实践,114[2024-10-20].http：//kns.cnki.net/kcms/detail/11.1762.G3.20240729.1533.002.html.
[20] 李蕾,刘钊,王栩彦.用户体验视角下ChatGPT辅助信息检索的可用性研究[J/OL].情报理论与实践,112[20241201].http：//kns.cnki.net/kcms/detail/11.1762.G3.20240903.1130.002.html.
[21] Bascur J P, Verberne S, Van Eck N J, et al. Academic information retrieval using citation clusters： in-depth evaluation based on systematic reviews[J]. Scientometrics, 2023, 128（5）：2895-2921.
[22] Pourreza M, Ensan F. Towards semantic-driven boolean query formalization for biomedical systematic literature reviews[J/OL]. International Journal of Medical Informatics, 110[2022-11-24]. https：//doi.org/10.1016/j.ijmedinf.2022.104928.
[23] 王若佳,李月琳.基于用户体验的健康类搜索引擎可用性评估[J].图书情报工作,2016, 60（7）：92-102.
[24] 裴一蕾,薛万欣,赵宗,等.基于用户体验视角的搜索引擎评价研究[J].情报科学,2013, 31（5）：9497； 1-12.
[25] Aguilera C E, Lopezosa C, Codina L. Scopus AI beta： functional analysis and cases[M]. Barcelona： Universitat Pompeu Fabra, Departament de Comunicació, 2023：248.
[26] Rashkin H, Nikolaev V, Lamm M, et al. Measuring attribution in natural language generation models[J]. Computational Linguistics, 2023, 49（4）：777-840.
[27] Kirici M. New cosine similarity and distance measures for Fermatean fuzzy sets and TOPSIS approach[J]. Knowledge and Information Systems, 2023, 65（2）： 855-868.