Please wait a minute...
Journal of ZheJiang University (Engineering Science)  2025, Vol. 59 Issue (4): 661-668    DOI: 10.3785/j.issn.1008-973X.2025.04.001
    
Multimodal emotional feature analysis based on short video resources of traffic incidents
Zhentao DONG1,5(),Kaimin XU1,Qingying WAN2,Xiaofei LIU3,Hao SHEN1,Shuhan LI3,Geqi QI1,4,5,*()
1. School of Traffic and Transportation, Beijing Jiaotong University, Beijing 100044, China
2. School of Electrical Engineering, Beijing Jiaotong University, Beijing 100044, China
3. School of Computer Science and Technology, Beijing Jiaotong University, Beijing 100044, China
4. Key Laboratory of Transport Industry of Big Data Application Technologies for Comprehensive Transport, Beijing Jiaotong University, Beijing 100044, China
5. Key Laboratory of Brain-Machine Intelligence for Information Behavior-Ministry of Education, Shanghai International Studies University, Shanghai 201620, China
Download: HTML     PDF(3695KB) HTML
Export: BibTeX | EndNote (RIS)      

Abstract  

In order to portray the public emotion orientation caused by the public opinion on traffic incidents disseminated in short videos, a physiological feature graph was constructed by the text sentiment analysis and the multimodal physiological signal feature extraction. This work collected 136 highly-liked videos with 38 805 comments on TikTok. Considering all videos as a document set, with each video treated as a document and comments as words, the latent Dirichlet allocation topic model was adopted to obtain the distribution of comments under different topics and the distribution of topics under different videos. Naive Bayes-based SnowNLP was utilized to calculate the sentiment scores of comments and analyze the sentiment tendencies expressed by different opinion topics. Neuroscience experiments were carried out to collect multimodal physiological signals such as EEG, eye movement, ECG, and respiration as well as emotion ratings. Statistical test results show that videos with different sentiment tendencies induce different emotions, and the multimodal physiological features such as the relative spectral power of EEG, blinking frequency, respiration standard deviation, and the very low-frequency power of ECG are specific under different emotions. The emotional semantics embedded in the comments influence public emotion in various ways beyond that evoked by videos.



Key wordstext mining      emotional characteristics      topic model      short video public opinion      electroencephalogram (EEG)     
Received: 26 February 2024      Published: 25 April 2025
CLC:  U 491  
  TP 391  
Fund:  国家自然科学基金资助项目(72101014);脑机协同信息行为教育部重点实验室开放课题项目(2023JYBKFKT009).
Corresponding Authors: Geqi QI     E-mail: ztdong@bjtu.edu.cn;gqqi@bjtu.edu.cn
Cite this article:

Zhentao DONG,Kaimin XU,Qingying WAN,Xiaofei LIU,Hao SHEN,Shuhan LI,Geqi QI. Multimodal emotional feature analysis based on short video resources of traffic incidents. Journal of ZheJiang University (Engineering Science), 2025, 59(4): 661-668.

URL:

https://www.zjujournals.com/eng/10.3785/j.issn.1008-973X.2025.04.001     OR     https://www.zjujournals.com/eng/Y2025/V59/I4/661


基于交通事件短视频资源的多模态情绪特征分析

为了刻画以短视频形式传播的交通事件舆情对公众情绪的导向,通过文本情感分析和多模态生理信号特征提取,构建生理特征图谱. 爬取抖音平台136个高赞视频及38 805条评论,以所有视频为文档集,单个视频为文档,评论为单词,采用隐狄利克雷分布主题模型进行主题挖掘,获得不同主题的评论单词分布和不同视频的主题分布. 使用基于朴素贝叶斯的SnowNLP计算评论单词的情感分数,分析不同舆情主题表达的情感倾向. 开展神经科学实验,采集脑电、眼动、心电和呼吸等多模态生理信号及情绪评分. 统计检验结果表明,不同情感倾向的视频会诱发不同情绪,不同情绪下脑电的相对谱功率、眨眼频率、呼吸标准差和心电极低频功率等多模态生理特征具有特异性,评论文本中蕴含的情感语义会在视频诱发情绪的基础上对公众情绪造成不同方式的影响.


关键词: 文本信息挖掘,  情绪特征,  主题模型,  短视频舆情,  脑电图(EEG) 
Fig.1 Words cloud of traffic topic
Fig.2 Procedures of multimodal neuroscience experiment
Fig.3 ErgoLAB platform for multimodal physiological signal acquisition
Fig.4 Variation of perplexity and coherence with number of topics
Fig.5 Distribution of public opinion topics on traffic incidents
Fig.6 Distribution of sentiment scores for videos under each topic
视频编号视频摘要$ \overline S_{\mathrm{E}} $主题编号
42客机起降禁开遮光板,空乘多次提醒,男子拒不配合0.3291
120北京,市区到机场,无障碍车0.7603
80三轮车逆行被撞得支离破碎,交通事故猛于虎,一次侥幸可能会造成终生不幸0.2942
64如何在北京刷手掌乘地铁?今天我来教你0.3102
639月1日起北京取消二环主路公交专用道0.3314
123疑似为给小蓝车挪地儿,俩男子将小黄车扔到路中,自行车堆成小山0.3285
131北京首批“整车无人”自动驾驶车辆上路测试0.4704
117国庆返程高峰,假期最后一天,北京北四环堵车现状;网友:我是自愿上班的0.3496
56暴雨后的北京丰台站大量乘客滞留,有乘客准备在车站过夜,还有学生无奈退改签4次0.2791
4最高礼仪迎接!C919顺利飞抵北京,商业首航成功,点赞祝贺!0.6921
Tab.1 Detailed information on selected videos for emotional evocation experiment
Fig.7 Summary results of emotional evocation experiment questionnaire
Fig.8 Statistical analysis results of emotional evocation experiment questionnaire
Fig.9 Correlation analysis results of EEG-pleasure ratings
Fig.10 Differences in physiological characteristics of emotion evoked by short videos
Fig.11 Differences in EEG characteristics induced by viewing comments
[1]   MA X, LIU W, ZHOU X, et al Evolution of online public opinion during meteorological disasters[J]. Environmental Hazards, 2020, 19 (4): 375- 397
doi: 10.1080/17477891.2019.1685932
[2]   JIA F, CHEN C C Emotional characteristics and time series analysis of Internet public opinion participants based on emotional feature words[J]. International Journal of Advanced Robotic Systems, 2020, 17 (1): 1- 11
[3]   于群, 英启昊, 曹娜, 等. 基于属性数学理论的电网停电事故网络舆情风险评估[E/OL]. (2024−01−12)[2024−02−20]. https://doi.org/10.13335/j.1000-3673.pst.2023.1923.
[4]   REN S, GONG C, ZHANG C, et al Public opinion communication mechanism of public health emergencies in Weibo: take the COVID-19 epidemic as an example[J]. Frontiers in Public Health, 2023, 11: 1276083
doi: 10.3389/fpubh.2023.1276083
[5]   ALI F, KWAK D, KHAN P, et al Transportation sentiment analysis using word embedding and ontology-based topic modeling[J]. Knowledge-Based Systems, 2019, 174: 27- 42
doi: 10.1016/j.knosys.2019.02.033
[6]   CHEN Y, CHEN H, LIU L, et al. A review on social media data for social transportation [C]// Proceedings of the CICTP 2021 . Xi’an: American Society of Civil Engineers, 2021: 126–136.
[7]   QI B, COSTIN A, JIA M A framework with efficient extraction and analysis of Twitter data for evaluating public opinions on transportation services[J]. Travel Behaviour and Society, 2020, 21: 10- 23
doi: 10.1016/j.tbs.2020.05.005
[8]   汤文蕴, 丁子羿, 马健霄 不同类型重大公共事件下交通管控舆情分析[J]. 浙江大学学报: 工学版, 2022, 56 (11): 2271- 2279
TANG Wenyun, DING Ziyi, MA Jianxiao Public opinion analysis on traffic control under different major public events[J]. Journal of Zhejiang University: Engineering Science, 2022, 56 (11): 2271- 2279
[9]   BAI Z, MA S, LI G A WeChat official account reading quantity prediction model based on text and image feature extraction[J]. IEEE Access, 2022, 10: 28348- 28360
doi: 10.1109/ACCESS.2022.3157715
[10]   LI J, LI Z, MA X, et al Sentiment analysis on online videos by time-sync comments[J]. Entropy, 2023, 25 (7): 1016
doi: 10.3390/e25071016
[11]   LIU J, HU X, SHEN X, et al Electrophysiological representations of multivariate human emotion experience[J]. Cognition and Emotion, 2024, 38 (3): 378- 388
doi: 10.1080/02699931.2023.2297272
[12]   PFEIFFER C, HOLLENSTEIN N, ZHANG C, et al Neural dynamics of sentiment processing during naturalistic sentence reading[J]. NeuroImage, 2020, 218: 116934
doi: 10.1016/j.neuroimage.2020.116934
[13]   赵卿, 张雪英, 陈桂军, 等 基于模态注意力图卷积特征融合的EEG和fNIRS情感识别[J]. 浙江大学学报: 工学版, 2023, 57 (10): 1987- 1997
ZHAO Qing, ZHANG Xueying, CHEN Guijun, et al EEG and fNIRS emotion recognition based on modality attention graph convolution feature fusion[J]. Journal of Zhejiang University: Engineering Science, 2023, 57 (10): 1987- 1997
[14]   郑伟龙, 石振锋, 吕宝粮 用异质迁移学习构建跨被试脑电情感模型[J]. 计算机学报, 2020, 43 (2): 177- 189
ZHENG Weilong, SHI Zhenfeng, LU Baoliang Building cross-subject EEG-based affective models using heterogeneous transfer learning[J]. Chinese Journal of Computers, 2020, 43 (2): 177- 189
doi: 10.11897/SP.J.1016.2020.00177
[15]   BLEI D M, NG A Y, JORDAN M I Latent Dirichlet allocation[J]. The Journal of Machine Learning Research, 2003, 3: 993- 1022
[16]   奇格奇, 张子贤, 卫振林, 等 基于语义挖掘的快递运输货品风险评价研究[J]. 交通运输系统工程与信息, 2021, 21 (4): 248- 255
QI Geqi, ZHANG Zixian, WEI Zhenlin, et al Risk evaluation of express delivery goods based on semantic mining[J]. Journal of Transportation Systems Engineering and Information Technology, 2021, 21 (4): 248- 255
[17]   SYED S, SPRUIT M. Full-text or abstract? Examining topic coherence scores using latent dirichlet allocation [C]// Proceedings of the IEEE International Conference on Data Science and Advanced Analytics . Tokyo: IEEE, 2017: 165–174.
[1] Yixuan LI,Ying LI,Qian XIAO,Lingyue WANG,Ning YIN,Shuo YANG. EEG microstate functional network analysis of different emotional false memories[J]. Journal of ZheJiang University (Engineering Science), 2025, 59(1): 49-61.
[2] Jia-wei LU,Jia-hong ZHENG,Duan-ni LI,Jun XU,Gang XIAO. Short text optimized topic model for service clustering[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(12): 2416-2425.
[3] Hai-dong WU,Zhen-li SI. Intelligent vehicle trajectory tracking control based on linear matrix inequality[J]. Journal of ZheJiang University (Engineering Science), 2020, 54(1): 110-117.
[4] WANG Wei-xing, SUN Shou-qian, LI Chao, TANG Zhi-chuan. Recognition of upper limb motion intention of EEG signal based on convolutional neural network[J]. Journal of ZheJiang University (Engineering Science), 2017, 51(7): 1381-1389.
[5] JING Yao, GUO Bin, WANG Zhu, YU Zhi-wen, ZHOU Xing-she. CrowdReview: personalized product review presentation based on crowd intelligence mining[J]. Journal of ZheJiang University (Engineering Science), 2017, 51(4): 675-681.
[6] TU Ding, CHEN Ling, CHEN Gen cai, WU Yong, WANG Jing chang. Hierarchical online NMF for detecting and tracking topics[J]. Journal of ZheJiang University (Engineering Science), 2016, 50(8): 1618-1626.
[7] SU Jin-song, DONG Huai-lin, CHEN Yi-dong, SHI Xiao-dong, WU Qing-qiang.
Improved statistical machine translation model with topic-based paraphrase
[J]. Journal of ZheJiang University (Engineering Science), 2014, 48(10): 1843-1849.
[8] YANG Bang-hua, HE Mei-yan, LIU Li, LU Wen-yu. EEG classification based on batch incremental SVM in
brain computer interfaces
[J]. Journal of ZheJiang University (Engineering Science), 2013, 47(8): 1431-1436.
[9] ZHU Dan-hua, CHEN Da-jing, CHEN Yu-quan, PAN Min. Enhancement of steady-state visual evoked potentials using
parameter-tuned stochastic resonance
[J]. Journal of ZheJiang University (Engineering Science), 2012, 46(5): 918-922.
[10] QIU Guang, ZHENG Miao, ZHANG Hui, ZHU Jianke, BU Jia-jun, CHEN Chun, HANG Hang. Implicit product feature extraction through regularized topic modeling[J]. Journal of ZheJiang University (Engineering Science), 2011, 45(2): 288-294.