Research on the Influence of Semantic Manipulation of Multimodal Content of Fake Short Videos on User Trust

Abstract

Abstract:

The dissemination of fake short videos can trigger information risks and thus harm the network ecology. By revealing how different multimodal semantic manipulations in fake short videos influence user trust, this study aims to inhibit the spread of fake short videos from a user-centered perspective. Taking science and technology short videos as a case study, we selected three topics with low user awareness. Guided by information manipulation theory, we applied semantic manipulations—fabrication, concealment, misinterpretation, ambiguity, and incitement—across three modalities: text,audio, and image. A controlled experiment (3*5=15; N=48) was conducted alongside semi-structuredinterviews (N=24) to analyze variations in user-perceived trust. There are significant differences inusers’ perceived trust based on semantic manipulations in different modalities: ① At the text level, usersexhibited higher trust in text fabrication, but lower trust in text concealment, ambiguity, and incitement.② At the audio level, users showed higher trust in audio fabrication and concealment, but lower trust inaudio incitement. ③ At the image level, users perceived more trust in image fabrication but less trust inimage incitement.

Fu Shaoxiong, Song Jinling, Deng Shengli, Zhu Mengdie( College of Information Management, Nanjing Agricultural University, School of Information Management, Wuhan University, School of Information Management, Nanjing University). Research on the Influence of Semantic Manipulation of Multimodal Content of Fake Short Videos on User Trust[J]. Libraly Journal, 2025, 44(406): 108-119.

References

[ 1 ] 中国互联网络信息中心. 第51 次《中国互联网络发展状况统计报告》[EB/OL]. [2023-05-10].https://www.cnnic.net.cn/n4/2023/0303/c88-10757.html.

[ 2 ] Brennen J S， Simon F M， Howard P N， etal. Types， sources， and claims of COVID-19misinformation types， sources， and claims ofCOVID-19 misinformation[D]. Britain：Universityof Oxford， 2020.

[ 3 ] Difonzo N， Robinson N M， Suls J M， et al.Rumors about cancer：Content， sources， coping，transmission， and belief[J]. Journal of HealthCommunication， 2012， 17（9）：1099–1115.

[ 4 ] 罗昕. 新媒体时代“以用户为中心”的信息传播理念[J]. 今传媒， 2008（12）：57–59.

[ 5 ] 范翥峰. 基于神经网络的伪造视频检测技术研究与实现[D]. 成都：电子科技大学， 2022.

[ 6 ] Wang G， Jiang Q， Jin X， et al. MC-LCR：Multi-modal contrastive classification by locallycorrelated representations for effective face forgerydetection[J]. Knowledge-Based Systems， 2022，250：e109114.

[ 7 ] Xue J， Wang Y， Tian Y， et al. Detecting fakenews by exploring the consistency of multimodaldata[J]. Information Processing and Management，2021， 58（5）：102610.

[ 8 ] Wu Y， Zhan P， Zhang Y， et al. Multimodalfusion with co-attention networks for fake newsdetection[C]. Findings of ACL 2021. Thailand，Bangkok， 2021.

[ 9 ] 王昊，龚丽娟，周泽聿，等. 融合语义增强的社交媒体虚假信息检测方法研究[J]. 数据分析与知识发现， 2023， 7（2）：48–60.

[10] 李晓静. 社交媒体用户的信息加工与信任判断——基于眼动追踪的实验研究[J]. 新闻与传播研究， 2017， 24（10）：49–67；127–128.

[11] 邓胜利，付少雄. 社交媒体附加信息对用户信任与分享健康类谣言的影响分析[J]. 情报科学，2018， 36（3）：51–57.

[12] 曹树金，常倞玮. 社交媒体中的突发公共卫生事件信息可信度影响因素研究——以微信为例[J].现代情报， 2020， 40（9）：3–14.

[13] Rosenthal S， Wasenden O C， Gronnevet G A，et al. A tripartite model of trust in Facebook：acceptance of information personalization，privacy concern， and privacy literacy[J]. MediaPsychology， 2019（2）：1–25.

[14] Deng S L， Fu S X， Liu Y， et al. Modelling users’trust in online health rumours：an experimentbasedstudy in China[J]. Information Research：AnInternational Electronic Journal， 2021， 26（1）：e890.

[15] 刘宸玮. 深度伪造音频检测技术的研究与实现[D].广州：广州大学， 2022.

[16] Jin Z W， Cao J， Zhang Y D， et al. Novel visualand statistical image features for microblogs newsverification[J]. IEEE Transactions on Multimedia，2017， 19（3）：598–608.

[17] Pala， Banerjees， Kaur A. What makes rumorrebuttals viral on social media?[C]. 202317th International Conference on UbiquitousInformation Management and Communication（IMCOM）， Seoul， Korea， 2023：1–6.

[18] Singhv K， Ghosh I， Sonagara D. Detecting fakenews stories via multimodal analysis[J]. Journalof the Association for Information Science andTechnology， 2021， 72（1）：3–17.

[19] Zhou Y M， Qing Q C， Qian Z X， et al. Multimodalfake news detection with adaptive unimodalrepresentation aggregation[EB/OL]. [2022-05-27].https://arxiv.org/abs/2205.14304.

[20] Li Y， Wang X. Seeking Health information on social media：A perspective of trust， selfdetermination，and social support[J]. Journal ofOrganizational and End User Computing， 2018，30（1）：1–22.

[21] 潘曙雅，祝宇欢，谢乐滋. 中老年人对虚假短视频的识别判断路径分析[J]. 国际新闻界， 2023，45（2）：127–155.

[22] 段秋婷，张大伟，谢兴政. 社交媒体隐私侵犯情境下的用户信任修复策略效果研究[J]. 图书馆学研究， 2022（3）：75–86；68.

[23] Connolly R， Sanchez O P， Compeau D， etal. Understanding engagement in online healthcommunities：A trust-based perspective[J]. Journalof the Association for Information Systems， 2023，24（2）：345–378.

[24] Mccornack S A. Information manipulationtheory[J]. Communications Monographs， 1992，59（1）：1–16.

[25] 上海辟谣. 断章取义、恶意拼凑，自媒体打着官媒旗号造谣太恶劣[EB/OL]. [2023-0 4 - 2 3 ] . h t t p s : / / p i y a o . j f d a i l y.com/py_7/kUlmJBm8TJp8MpHfQVoa8JpFlcVCrjqfxJR7Mr8tzCl1ItlTd/4ZCgVcuj3vrS7mhYWmp+62bk2hLd.

[26] 叶凤云，常琳，秦琴，等. 基于虚假信息特征的社交媒体大学生用户感知信任研究[J/OL]. 情报理论与实践，1–13[2024-08-12]. http://kns.cnki.net/kcms/detail/11.1762.G3.20240418.1400.002.html.

[27] 张恬. 微博虚假信息产生的传播学解读[J]. 新闻传播， 2013，（11）：172–173.

[28] 莫祖英，王垲烁，赵悦名. 社交媒体用户虚假信息从众传播行为影响模型实证研究[J]. 情报资料工作， 2023， 44（1）：72–81.

[29] 张帅. 社交媒体虚假健康信息特征识别[J]. 图书情报工作， 2021， 65（9）：70–78.

[30] 裘锡圭. 文字学概要（修订本）[M]. 北京：商务印书馆， 2013.

[31] 吴宏浩. 短视频新闻标题制作与传播策略[J]. 新闻世界， 2023， 381（1）：61–64.

[32] Jin X L， Yin M， Zhou Z， et al. The differentialeffects of trusting beliefs on social media users’willingness to adopt and share health knowledge[J].Information Processing and Management， 2021，58（1）：102413.

[33] Vafeiadis M， Xiao A L. Fake news：Howemotions， involvement， need for cognitionand rebuttal evidence（story vs. informational）influence consumer reactions toward a targetedorganization[J]. Public Relations Review， 2021，47（4）：e102088.

[34] 韩宪柱．数字音频技术及其应用[M]. 北京：人民邮电出版社， 2003.

[35] 揭其涛. 内容、情绪及空间：泛媒介时代音频传播的回归与偏向[J]. 媒体融合新观察， 2023，218（2）：26–29.

[36] London J， Li S Y， Sun H S. Seems legit：Aninvestigation of the assessing and sharing ofunverifiable messages on online social networks[J].Information System Research， 2022， 33（3）：978–1001.

[37] Xiao X， Li X， Yao J， et al. Research on VideoFalsity Detection Based on Publisher andPublishing Environment Features[C]//ArtificialIntelligence And Security, ICAIS 2022， PTIII. Berlin, Heidelberg： Springer InternationalPublishing AG， 2022：56–70.

[38] Robson S， Banerjee S， Kaur A， et al. Brand postpopularity on social media： A Systematic LiteratureReview[C]//Proceedings of the InternationalConference on Ubiquitous InformationManagement and Communication, SungkyunkwanUniv， Seoul， IEEE， 2022.

[39] Anotoiadis I， Paltsoglou S， Patoulidis V. Postpopularity and reactions in retail brand pages onFacebook[J]. International Journal of Retail andDistribution Management， 2019， 47（9）：957–973.

[40] Cheung C M K， Lee M K O， Rabjohn N. Theimpact of electronic word-of-mouth the adoption ofonline opinions in online customer communities[J].Internet Research. 2008， 18（3）：229–247.

[41] Wu C， Shaffer D R. Susceptibility to persuasiveappeals as a function of source credibility and priorexperience with the attitude object[J]. Journal ofPersonality and Social Psychology， 1987， 52（4）：677–688.

[42] 钟毅平. 社会认知心理学[M]. 北京：教育科学出版社， 2012：156.

[43] 邓胜利，付少雄. 用户群体特征差异对健康信息搜寻行为的影响研究[J]. 信息资源管理学报，2016，6（4）：5–11.

[44] Pal A， Chua A Y K， Goh H L. How do usersrespond to online rumor rebuttals? [J]. Computersin Human Behavior， 2019，106：e106243.

[45] 王路阳. 知识逻辑VS 传媒逻辑：新媒体语境下专家话语的“曲解化”研究[J]. 社科纵横， 2022，37（4）：133–138.

[46] Song C G， Ning N W， Zhang Y L， et al. A multimodalfake news detection model based on crossmodalattention residual and multichannel convolutionalneural networks[J]. Information Processing andManagement， 2021， 58（1）：e102437.

[47] Vogel S. The power of video in social media[J].Floor Covering Weekly， 2021，70（29）：24.