基于多模态数据预测公共数字文化服务用户认知负荷

摘要/Abstract

摘要： 本文聚焦公共数字文化多媒体资源的糅杂现状，以优化公共数字文化服务资源建设为目标，研究用户阅读公共数字文化多媒体资源的认知负荷差异及如何应用多模态数据预测用户认知负荷。研究设计脑电眼动协同实验，重构国家公共文化云平台多媒体资源，采集脑电、眼动、量表、访谈等数据，选用经典机器学习算法预测用户认知负荷。结果显示：4 类数据均反馈用户使用多媒体资源时存在认知负荷差异，多模态数据预测二分类认知负荷效果优于单模态数据，多模态主客观协同预测二分类认知负荷效果最佳。研究结论：以用户为中心的公共数字文化服务，应针对理解、记忆、注意3 类服务目标，依据用户认知规律，建设精准化、个性化的多媒体文化资源。

Abstract:

This paper focuses on the complex situation of public digital cultural multimedia resources.Aiming to optimize the construction of public digital cultural service, it studies the differences in cognitiveload when users engage with multimedia resources and explores how multimodal data can predict theload. The research designed an Electroencephalogram & Eye-tracking collaborative experiment toreconstruct the multimedia resources of Chinese National Public Culture Cloud Platform, collected datasuch as EEG, Eye-tracking, NASA_TLX scales, and interviews, and employed classic machine learningalgorithms to predict users’ cognitive load. Findings demonstrate that all four types of data reflect thedifferences existing in cognitive load during used multimedia resource usage. The prediction effect ofbinary-classification cognitive load by multimodal data was better than that of unimodal data, and themultimodal subjective-objective collaborative prediction of binary-classification cognitive load hadthe best effect. The research concludes that user-centered public digital cultural services should alignwith cognitive principles, focusing on understanding, memory, and attention, and leverage precise andpersonalized multimedia cultural resources.

林蕙1, 2 吴丹1, 2（1 武汉大学信息管理学院 2 武汉大学人机交互与用户行为研究中心）. 基于多模态数据预测公共数字文化服务用户认知负荷[J]. 图书馆杂志, 2025, 44(409): 20-32.

Lin Hui1, 2, Wu Dan1, 2 (1 School of Information Management, Wuhan University, 2 Center for Studies of Human-ComputerInteraction and User Behavior, Wuhan University). Predicting User Cognitive Load in Public Digital CulturalServices Based on Multimodal Data[J]. Libraly Journal, 2025, 44(409): 20-32.

参考文献

[ 1 ] Ronchi A M. eCulture：Cultural Content inthe Digital Age[M]. Berlin: Springer BerlinHeidelberg，2009：33–70.

[ 2 ] 孔高敏. 我国基层公共数字文化服务新媒体联动机制研究[J]. 图书馆，2024（7）：47–55.

[ 3 ] Sweller J，van Merrienboer J J G，Paas F G W C.Cognitive Architecture and Instructional Design[J].Educational Psychology Review，1998，10（3）：251–296.

[ 4 ] Mayer R E. Multimedia learning[M]. Amsterdam:Elsevier，2002：85–139.

[ 5 ] 袁曦临. 网络数字阅读行为对阅读脑的改造及其对认知的影响[J]. 图书馆杂志，2016，35（4）：18–26.

[ 6 ] He Y，Guo Z，Wang X，et al. Effects of audiovisualinteractions on working memory task performance—interference or facilitation[J]. Brain Sciences，2022，12（7）：886.

[ 7 ] Ren Y，Zhou Z，Bi J，et al. Effect of attentionalload on audio-visual integration：an ERP study[C]//Proceedings of the 2020 IEEE InternationalConference on Mechatronics and Automation（ICMA）.Beijing，China. IEEE，2020：1443–1448.

[ 8 ] 王玉琴，王咸伟. 媒体组合与学习步调对多媒体学习影响的眼动实验研究[J]. 电化教育研究，2007（11）：61–66.

[ 9 ] 龚少英，段婷，王福兴，等. 装饰图片影响多媒体学习的眼动研究[J]. 心理发展与教育，2014，30（4）：403–410.

[10] 王玉鑫，谢和平，王福兴，等. 多媒体学习的图文整合：空间邻近效应的元分析[J]. 心理发展与教育，2016，32（5）：565–578.

[11] Kosch T，Schmidt A，Thanheiser S，et al.One does not Simply RSVP：Mental Workloadto Select Speed Reading Parameters usingElectroencephalography[C]//Proceedings ofthe 2020 CHI Conference on Human Factors inComputing Systems. Honolulu，HI，USA. ACM，2020：1–13.

[12] Chen C，Lin Y. Effects of different text displaytypes on reading comprehension，sustainedattention and cognitive load in mobile readingcontexts[J]. Interactive Learning Environments，2016，24（3）：553–571.

[13] 熊俊梅，辛亮，高苗苗，等. 视觉和听觉情绪设计对多媒体学习的影响[J]. 心理科学，2018，41（5）：1124–1129.

[14] 黄先蓉，张窈. 数字阅读研究热点与动向：伦理、行为与应用[J]. 出版科学，2020，28（2）：5–16.

[15] 马捷，张光媛，徐晓晨，等. 数字阅读与纸质阅读理解效果及沉浸体验实验研究——以科普知识为例[J]. 图书情报工作，2018，62（16）：35–46.

[16] Cheng K. Reading an augmented reality book：An exploration of learners’ cognitive load，motivation，and attitudes[J]. Australasian Journalof Educational Technology，2017，33（4）：53–69.

[17] Chang C，Warden C A，Liang C，et al. Effects ofdigital game-based learning on achievement，flowand overall cognitive load[J]. Australasian Journal ofEducational Technology，2018，34（4）：155–167.

[18] Hawlitschek A，Joeckel S. Increasing the effectivenessof digital educational games：The effects of a learninginstruction on students’ learning，motivation andcognitive load[J]. Computers in Human Behavior，2017，72：79–86.

[19] Huang C L，Luo Y F，Yang S C，et al. Influenceof Students’ Learning Style，Sense of Presence，and Cognitive Load on Learning Outcomes in anImmersive Virtual Reality Learning Environment[J].Journal of Educational Computing Research，2019，58（3）：596–615.

[20] Minkley N，Xu K M，Krell M. AnalyzingRelationships Between Causal and AssessmentFactors of Cognitive Load：Associations BetweenObjective and Subjective Measures of CognitiveLoad，Stress，Interest，and Self-Concept[J].Frontiers in Education，2021，6：632907.

[21] Mutlu-Bayraktar D，Cosgun V，Altan T. Cognitiveload in multimedia learning environments：Asystematic review[J]. Computers & Education，2019，141：103618.

[22] Chen F，Zhou J，Wang Y，et al. Theoretical Aspectsof Multimodal Cognitive Load Measures[M]. Cham：Springer International Publishing，2016：33–71.

[23] Chen F，Ruiz N，Choi E，et al. Multimodal behaviorand interaction as indicators of cognitive load[J].ACM Trans. Interact. Intell. Syst.，2013，2（4）：22.

[24] Vanneste P，Raes A，Morton J，et al. Towardsmeasuring cognitive load through multimodalphysiological data[J]. Cognition，Technology &Work，2021，23（3）：567–585.

[25] Y L，Y Y，H T，et al. Cognitive Load Predictionfrom Multimodal Physiological Signals usingMultiview Learning[J]. IEEE Journal of Biomedicaland Health Informatics，2023：1–11.

[26] Larmuseau C，Cornelis J，Lancieri L，et al. Multimodal learning analytics to investigatecognitive load during online problem solving[J].British Journal of Educational Technology，2020，51（5）：1548–1562.

[27] Ahmad M I，Robb D A，Keller I，et al. Towards aMultimodal Measure for Physiological Behavioursto Estimate Cognitive Load[C]//Proceedings of the2020HCI International Conference. Copenhagen，Denmark. HCII，2020：3–13.

[28] Gjoreski M，Mahesh B，Koleni T，et al. CognitiveLoad Monitoring With Wearables—Lessons LearnedFrom a Machine Learning Challenge[J]. IEEEAccess，2021，9：103325–103336.

[29] Herbig N，Düwel T，Helali M，et al. InvestigatingMulti-Modal Measures for Cognitive Load Detectionin E-Learning[C]//Proceedings of the 28th ACMConference on User Modeling，Adaptation andPersonalization. Genoa，Italy. ACM，2020：88–97.

[30] Herbig N，Pal S，Vela M，et al. Multi-modalindicators for estimating perceived cognitive loadin post-editing of machine translation[J]. MachineTranslation，2019，33（1）：91–115.

[31] Giannakos M N，Sharma K，Pappas I O，et al.Multimodal data as a means to understand thelearning experience[J]. International Journal ofInformation Management，2019，48：108–119.

[32] Oschlies-Strobel A，Gruss S，Jerg-Bretzke L，et al. Preliminary classification of cognitive loadstates in a human machine interaction scenario[C]//Proceedings of the 2017 International Conference onCompanion Technology （ICCT）.Ulm，Germany.IEEE，2017：1–5.

[33] Canova G. Machine Learning and data fusion ofphysiological signals for assessing a subject’s stresslevel and cognitive load[D]. Turin：Politecnico diTorino，2024.

[34] Hart S G，Staveland L E. Development of NASATLX（Task Load Index）：Results of Empiricaland Theoretical Research[M]. Amsterdam：North-Holland，1988：139–183.

[35] Lahat D，Adali T，Jutten C. Multimodal DataFusion：An Overview of Methods，Challenges，andProspects[J]. Proceedings of the IEEE， 2015， 103（9）：1449–1477.

[36] Miller G A. The magical number seven，plusor minus two：Some limits on our capacity forprocessing information.[J]. Psychological Review，1956，63（2）：81.

[37] Wang G，Tian L，Liu J，et al. Neural mechanismsof cognitive load in Multimedia Learning：a Metaanalysisof EEG frequency Band Modulation[J].Current Psychology，2024，43（37）：29316–29332.

[38] Zammouri A，Moussa A A，Chevallier S. Useof cognitive load measurements to design a newarchitecture of intelligent learning systems[J]. ExpertSystems with Applications，2024，237：121253.

[39] Consalvi L，Ouwehand K，Paas F. Effects ofObserving Urban and Natural Scenes on WorkingMemory Depletion and Restoration：An EEGStudy[J]. Education Sciences，2024，14（11）：1204.

[40] Havugimana F，Moinudin K A，Yeasin M. Deeplearning framework for modeling cognitive loadfrom small and noisy eeg data[J]. IEEE Transactionson Cognitive and Developmental Systems，2023，16（3）：1006–1015.

[41] Liu Y，Yu Y，Ye Z，et al. Fusion of spatial，temporal，and spectral EEG signatures improves multilevelcognitive load prediction[J]. IEEE Transactions onHuman-Machine Systems，2023，53（2）：357–366.

[42] Ruo A，Villani V，Sabattini L. Use of EEG signalsfor mental workload assessment in human-robotcollaboration[C]//Proceedings of the 2022 Human-Friendly Robotics. Delft，The Netherlands. Springer，2022：233–247.

[43] Chikhi S，Matton N，Blanchet S. EEG power spectralmeasures of cognitive workload：A meta-analysis[J].Psychophysiology，2022，59（6）：e14009.

[44] Klimesch W. EEG alpha and theta oscillationsreflect cognitive and memory performance：areview and analysis[J]. Brain Research Reviews，1999，29（2/3）：169–195.

[45] 王国华，田梁浩，聂胜欣，等. 多媒体学习中表征认知负荷的眼动指标研究[J]. 电化教育研究，2023，44（10）：54–62.[46] Zhou Y，Huang S，Xu Z，et al. Cognitive WorkloadRecognition Using EEG Signals and Machine Learning：A Review[J]. IEEE Transactions on Cognitive andDevelopmental Systems，2022，14（3）：799–818.[47] Mutlu-Bayraktar D，Ozel P，Altindis F，et al.Relationship between objective and subjectivecognitive load measurements in multimedia learning[J].Interactive Learning Environments，2023，31（3）：1322–1334.