Abstract: This study compares and analyzes the content of archival journal articles generated by AI with those written by scholars, delving into the potential of AI technology in academic writing and its relative advantages and limitations compared to human-authored works. The research selected 100 highly cited papers from core journals in the field of archival studies over the past three years. The abstracts, introductions, and conclusions were extracted, and corresponding abstracts were generated using six large language models. A systematic analysis was conducted across multiple dimensions, including semantic similarity, topic modeling, classification detection, ROUGE evaluation, and academic publication detection. The results indicate that the abstracts generated by the six models exhibit a high degree of similarity to those written by scholars, with Tongyi Qianwen particularly excelling in topic refinement, producing content that closely aligns with the professional depth of scholars. In terms of classification detection, the Random Forest (RF) and XGBoost models demonstrated outstanding performance. ROUGE evaluation results show that the quality of abstracts generated by large models has reached or even surpassed traditional algorithm levels, with Wenxin Yiyan 4.0 performing especially well. In academic plagiarism detection tests, both GPT4.0 and Tongyi Qianwen met the standards. Tongyi Qianwen shows an extremely low proportion of suspected AI-generated content in the CNKI AIGC detection. Based on these findings, this study recommends that academic publishing platforms further adapt to the development of new text generation technologies, refine AIGC detection standards, enhance crossplatform collaboration and data sharing, and pay special attention to the detection challenges posed by the Tongyi Qianwen model.

Key words: Large language model, AIGC detection, Academic writing, Text evaluation