Please wait a minute...
浙江大学学报(工学版)  2026, Vol. 60 Issue (11): 2482-2491    DOI: 10.3785/j.issn.1008-973X.2026.11.017
计算机技术     
基于多视角解耦融合的多模态会话情感识别
张婷婷(),朱小飞*()
重庆理工大学 计算机科学与工程学院,重庆 400054
Multi-view decoupled fusion for multimodal emotion recognition in conversation
Tingting ZHANG(),Xiaofei ZHU*()
College of Computer Science and Engineering, Chongqing University of Technology, Chongqing 400054, China
 全文: PDF(1004 KB)   HTML
摘要:

现有多模态会话情感识别(MERC)方法在上下文建模和多模态融合方面存在局限,为此提出多视角解耦融合(MDF)框架. 设计多视图上下文提取模块,通过构建从近到远的多尺度语境视图,全面捕捉会话中的情感依赖. 结合有监督与无监督学习策略,增强模型的判别能力. 引入多视角模态解耦融合模块,采用Top-K掩码机制有效过滤注意力矩阵中的噪声,通过建模各注意力头的重要性,进一步提升模型表达能力. 在主流数据集上开展的模型性能对比实验结果表明,与先进方法AdaIGN相比,MDF 在上下文提取和模态融合方面优势显著:在IEMOCAP数据集上,MDF的准确率 (Acc) 和加权F1值 (W-F1) 分别提升了2.38和2.31个百分点; 在MELD数据集上,Acc和W-F1分别提升了0.44和0.13个百分点. 实验结果验证了MDF框架的有效性.

关键词: 情感识别;  多模态学习;  多视图上下文提取;  多视角解耦融合;  对比学习    
Abstract:

A multi-view decoupled fusion (MDF) framework was proposed to address the limitations of existing methods for multimodal emotion recognition in conversation (MERC) regarding context modeling and multimodal fusion. A multi-view context extraction module was designed to capture emotional dependencies in conversation through multi-scale contextual views from near to far. Furthermore, the discriminative ability was enhanced by combining supervised and unsupervised learning strategies. In addition, a multi-view modal decoupled fusion module was introduced, in which a Top-K mask mechanism was utilized to filter noise in attention matrices, and the expressive capability was improved by modeling the importance of each attention head. Experimental results on mainstream datasets demonstrated that, compared with the state-of-the-art method AdaIGN, the MDF achieved significant improvements in both context extraction and multimodal fusion. Specifically, on the IEMOCAP dataset, the accuracy (Acc) and weighted F1-score (W-F1) were improved by 2.38 and 2.31 percentage points, respectively; on the MELD dataset, Acc and W-F1 were improved by 0.44 and 0.13 percentage points, respectively. These results validated the effectiveness of the MDF framework.

Key words: emotion recognition    multimodal learning    multi-view context extraction    multi-view decoupled fusion    contrastive learning
收稿日期: 2025-08-30 出版日期: 2026-10-16
CLC:  TP 393  
基金资助: 国家自然科学基金资助项目(62472059);重庆英才计划项目(CSTC2024YCJH-BGZXM0022);重庆市技术创新与应用发展专项重大项目(CSTB2024TIAD-STX0027).
通讯作者: 朱小飞     E-mail: ztt@stu.cqut.edu.cn;zxf@cqut.edu.cn
作者简介: 张婷婷(2000—),女,硕士生,从事自然语言处理研究. orcid.org/0009-0000-9431-0568. E-mail:ztt@stu.cqut.edu.cn
服务  
把本文推荐给朋友
加入引用管理器
E-mail Alert
作者相关文章  
张婷婷
朱小飞

引用本文:

张婷婷,朱小飞. 基于多视角解耦融合的多模态会话情感识别[J]. 浙江大学学报(工学版), 2026, 60(11): 2482-2491.

Tingting ZHANG,Xiaofei ZHU. Multi-view decoupled fusion for multimodal emotion recognition in conversation. Journal of ZheJiang University (Engineering Science), 2026, 60(11): 2482-2491.

链接本文:

https://www.zjujournals.com/eng/CN/10.3785/j.issn.1008-973X.2026.11.017    或    https://www.zjujournals.com/eng/CN/Y2026/V60/I11/2482

图 1  多视角解耦融合框架及核心模块
图 2  多距离视图的构建规则
数据集数据划分总会话数总话语数会话长度
Q1Q3Mean
IEMOCAPTrain+Val1205 81037.05948.42
Test311 62342.55952.35
MELDTrain+Val1 15311 0985.0149.63
Test2802 6105.0139.32
表 1  实验用数据集的信息统计
模型F1%Acc%W-F1%
高兴悲伤中性愤怒兴奋沮丧
DialogueRNN#32.2080.2657.8962.8273.8759.7663.5262.89
MMGCN#45.1477.1664.3668.8274.7161.4066.3666.26
CTNet#51.3079.9065.8067.2078.7058.8068.0067.50
MM-DFN$42.2278.9866.4269.7775.5666.3368.2168.18
SCMM$45.3778.7663.5466.0576.7066.18—67.53
CMCF-SRNet$52.2080.9068.8070.3076.7061.6070.5069.60
MultiDAG$49.6581.4069.5370.3371.6166.9469.1169.08
FrameERC$56.1980.8867.6963.1678.5869.7370.7970.67
AdaIGN$53.0481.4771.2665.8776.3467.7970.4970.74
MDF56.76*83.98*74.84*66.6577.6669.61*72.87*73.05*
表 2  不同模型在IEMOCAP数据集上的性能对比结果
模型F1%Acc%W-F1%
中性惊讶悲伤快乐愤怒
DialogueRNN#76.9747.6920.4150.9245.5260.3157.66
MMGCN#76.3348.1526.7453.0246.0960.4258.31
CTNet#77.4052.7032.5056.0044.6062.0060.50
MM-DFN$77.7650.6922.9354.7847.8262.4959.46
SCMM$——————59.44
CMCF-SRNet$—————62.8062.30
MultiDAG$—————64.4164.00
FrameERC$80.0058.1042.5463.3756.2767.6266.33
AdaIGN$79.7560.5343.7064.5456.1567.6266.79
MDF79.8659.0344.84*64.6057.63*68.06*66.92*
表 3  不同模型在MELD数据集上的性能对比结果
模型IEMOCAPMELD
Acc/%W-F1/%Acc/%W-F1/%
A53.1050.3646.8841.49
V39.9038.9945.7035.52
T69.0369.0367.4166.40
A+T70.8870.8267.7966.54
V+T68.6668.8767.3566.37
A+V64.3163.8649.2043.02
MDF72.8773.0568.0666.92
表 4  不同模态组合输入在2个数据集上的消融实验结果
模型IEMOCAPMELD
Acc/%W-F1/%Acc/%W-F1/%
w/o MCE71.5171.2667.4466.41
w/o S-GRU72.5372.7467.8966.73
w/o D-GRU72.4172.5467.8066.69
w/o S-&D-GRU72.3572.4867.7466.61
w/o MMDF69.7069.5867.3066.51
w/o TM72.7372.6767.9166.79
w/o HW72.5872.6467.9866.74
w/o TM & HW72.2772.4967.8566.66
MDF72.8773.0568.0666.92
表 5  框架核心模块在2个数据集上的消融实验结果
图 3  距离层数在2个数据集上的敏感性分析
图 4  无监督与有监督学习策略参数在2个数据集上的敏感性分析
图 5  Top-K策略在2个数据集上的敏感性分析
图 6  不同情感识别模型在3种会话长度下的性能比较
1 JOSHI A, BHAT A, JAIN A, et al. COGMEN: contextualized GNN based multimodal emotion recognition [C]// Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Seattle: Association for Computational Linguistics, 2022: 4148–4164.
2 ZHENG X, ZHAO G, ZHU L, et al. PERD: personalized emoji recommendation with dynamic user preference [C]// Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval. Madrid: ACM, 2022: 1922–1926.
3 PHANEENDRA BOTTU V S G S, RAGAVAN K Emotion-based music recommendation system integrating facial expression recognition and lyrics sentiment analysis[J]. IEEE Access, 2025, 13: 87740- 87752
doi: 10.1109/ACCESS.2025.3570011
4 FOTOPOULOU E, CASSÀ È L, MAVRAIDI L, et al EduCardia: a modern technology-powered methodology for the assessment and improvement of social and emotional competencies of students in K12 schools[J]. IEEE Access, 2025, 13: 10786- 10804
doi: 10.1109/ACCESS.2025.3529120
5 KHAN N D, ALI KHAN J, LI J, et al Leveraging large language model ChatGPT for enhanced understanding of end-user emotions in social media feedbacks[J]. Expert Systems with Applications, 2025, 261: 125524
doi: 10.1016/j.eswa.2024.125524
6 GIACOMUZZO G, TERRERAN M, JAIN S, et al. DECAF: a discrete-event based collaborative human-robot framework for furniture assembly [C]// Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems. Abu Dhabi: IEEE, 2024: 7085–7091.
7 YE J, WEI Y, WEN X C, et al. Emo-DNA: emotion decoupling and alignment learning for cross-corpus speech emotion recognition [C]// Proceedings of the 31st ACM International Conference on Multimedia. Ottawa: ACM, 2023: 5956–5965.
8 SHI T, HUANG S L. MultiEMO: an attention-based correlation-aware multimodal fusion framework for emotion recognition in conversations [C]// Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Toronto: Association for Computational Linguistics, 2023: 14752–14766.
9 GHOSAL D, MAJUMDER N, PORIA S, et al. DialogueGCN: a graph convolutional neural network for emotion recognition in conversation [C]// Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). Hong Kong: Association for Computational Linguistics, 2019: 154–164.
10 HU J, LIU Y, ZHAO J, et al. MMGCN: multimodal fusion via deep graph convolution network for emotion recognition in conversation [C]// Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). [S.l.]: Association for Computational Linguistics, 2021: 5666–5675.
11 HU D, HOU X, WEI L, et al. MM-DFN: multimodal dynamic fusion network for emotion recognition in conversations [C]// Proceedings of the ICASSP 2022—2022 IEEE International Conference on Acoustics, Speech and Signal Processing. Singapore: IEEE, 2022: 7037–7041.
12 SHEN W, WU S, YANG Y, et al. Directed acyclic graph network for conversational emotion recognition [C]// Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). [S.l.]: Association for Computational Linguistics, 2021: 1551–1560.
13 NGUYEN C T, NGUYEN C B, LE D T, et al. Curriculum learning meets directed acyclic graph for multimodal emotion recognition [C]// Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024). Torino: Association for Computational Linguistics, 2024: 4259–4265.
14 BALTRUŠAITIS T, AHUJA C, MORENCY L P Multimodal machine learning: a survey and taxonomy[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019, 41 (2): 423- 443
doi: 10.1109/TPAMI.2018.2798607
15 LI J, JI D, LI F, et al. HiTrans: a Transformer-based context- and speaker-sensitive model for emotion detection in conversations [C]// Proceedings of the 28th International Conference on Computational Linguistics. Barcelona: Association for Computational Linguistics, 2020: 4190–4200.
16 LIAN Z, LIU B, TAO J CTNet: conversational transformer network for emotion recognition[J]. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021, 29: 985- 1000
doi: 10.1109/TASLP.2021.3049898
17 ZHANG X, LI Y. A cross-modality context fusion and semantic refinement network for emotion recognition in conversation [C]// Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Toronto: Association for Computational Linguistics, 2023: 13099–13110.
18 MA H, WANG J, LIN H, et al A transformer-based model with self-distillation for multimodal emotion recognition in conversations[J]. IEEE Transactions on Multimedia, 2024, 26: 776- 788
doi: 10.1109/TMM.2023.3271019
19 YUN T, LIM H, LEE J, et al. TelME: teacher-leading multimodal fusion network for emotion recognition in conversation [C]// Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). Mexico City: Association for Computational Linguistics, 2024: 82–95.
20 ISHIWATARI T, YASUDA Y, MIYAZAKI T, et al. Relation-aware graph attention networks with relational position encodings for emotion recognition in conversations [C]// Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). [S.l.]: Association for Computational Linguistics, 2020: 7360–7370.
21 TU G, XIE T, LIANG B, et al Adaptive graph learning for multimodal conversational emotion detection[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2024, 38 (17): 19089- 19097
doi: 10.1609/aaai.v38i17.29876
22 LI M, SHI J, BAI L, et al FrameERC: framelet transform based multimodal graph neural networks for emotion recognition in conversation[J]. Pattern Recognition, 2025, 161: 111340
doi: 10.1016/j.patcog.2024.111340
23 LI Z, TANG F, ZHAO M, et al. EmoCaps: emotion capsule based model for conversational emotion recognition [C]// Proceedings of the Findings of the Association for Computational Linguistics. Dublin: Association for Computational Linguistics, 2022: 1610–1618.
24 YANG H, GAO X, WU J, et al. Self-adaptive context and modal-interaction modeling for multimodal emotion recognition [C]// Proceedings of the Findings of the Association for Computational Linguistics. Toronto: Association for Computational Linguistics, 2023: 6267–6281.
25 SCHULLER B, BATLINER A, STEIDL S, et al Recognising realistic emotions and affect in speech: state of the art and lessons learnt from the first challenge[J]. Speech Communication, 2011, 53 (9/10): 1062- 1087
doi: 10.1016/j.specom.2011.01.011
26 LIU Y, OTT M, GOYAL N, et al. RoBERTa: a robustly optimized BERT pretraining approach [EB/OL]. (2019–07–26)[2025–10–22]. https://arxiv.org/pdf/1907.11692.
27 HUANG G, LIU Z, VAN DER MAATEN L, et al. Densely connected convolutional networks [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017: 2261–2269.
28 XIE Y, ZHOU P, KIM S. Decoupled side information fusion for sequential recommendation [C]// Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval. Madrid: ACM, 2022: 1611–1621.
29 BUSSO C, BULUT M, LEE C C, et al IEMOCAP: interactive emotional dyadic motion capture database[J]. Language Resources and Evaluation, 2008, 42 (4): 335- 359
doi: 10.1007/s10579-008-9076-6
30 PORIA S, HAZARIKA D, MAJUMDER N, et al. MELD: a multimodal multi-party dataset for emotion recognition in conversations [C]// Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Florence: Association for Computational Linguistics, 2019: 527–536.
[1] 张雪梅,孙颖,张雪英. 基于无监督图对比学习的语音情感识别[J]. 浙江大学学报(工学版), 2026, 60(4): 782-790.
[2] 王彦乐,张瑞峰,李锵. 融合全局信息和对比学习的图神经网络推荐模型[J]. 浙江大学学报(工学版), 2026, 60(2): 351-359.
[3] 马文煜,夏鸿斌,王晓锋. 面向去噪与用户偏好建模的多模态推荐模型[J]. 浙江大学学报(工学版), 2026, 60(10): 2215-2226.
[4] 谢涛,葛慧丽,陈宁,汪晓锋,李延松,黄晓峰. 知识嵌入增强的对比推荐模型[J]. 浙江大学学报(工学版), 2026, 60(1): 90-98.
[5] 黄文湖,赵邢,谢亮,梁浩然,梁荣华. 基于对比学习的声源定位引导视听分割模型[J]. 浙江大学学报(工学版), 2025, 59(9): 1803-1813.
[6] 孟璇,张雪英,孙颖,周雅茹. 基于电极排列和Transformer的脑电情感识别[J]. 浙江大学学报(工学版), 2025, 59(9): 1872-1880.
[7] 侯越,李前辉,袁鹏,张鑫,王甜甜,郝紫微. 基于对比学习的可扩展交通图像自动标注方法[J]. 浙江大学学报(工学版), 2025, 59(8): 1634-1643.
[8] 李颂元,朱祥维,李玺. 基座模型技术背景下的具身智能体综述[J]. 浙江大学学报(工学版), 2025, 59(2): 213-226.
[9] 陈珂,张文浩. 基于对比学习的零样本对象谣言检测[J]. 浙江大学学报(工学版), 2024, 58(9): 1790-1800.
[10] 刘悦,张雪英,陈桂军,黄丽霞,孙颖. 基于多脑区注意力机制胶囊融合网络的EEG-fNIRS情感识别[J]. 浙江大学学报(工学版), 2024, 58(11): 2247-2257.
[11] 李海烽,张雪英,段淑斐,贾海蓉,Huizhi Liang . 融合生成对抗网络与时间卷积网络的普通话情感识别[J]. 浙江大学学报(工学版), 2023, 57(9): 1865-1875.
[12] 周天琪,杨艳,张继杰,殷少伟,郭增强. 基于无负样本损失和自适应增强的图对比学习[J]. 浙江大学学报(工学版), 2023, 57(2): 259-266.
[13] 赵卿,张雪英,陈桂军,张静. 基于模态注意力图卷积特征融合的EEG和fNIRS情感识别[J]. 浙江大学学报(工学版), 2023, 57(10): 1987-1997.
[14] 孙颖,胡艳香,张雪英,段淑斐. 面向情感语音识别的情感维度PAD预测[J]. 浙江大学学报(工学版), 2019, 53(10): 2041-2048.
[15] 孙凌云, 何博伟, 刘征, 杨智渊. 基于语义细胞的语音情感识别[J]. 浙江大学学报(工学版), 2015, 49(6): 1001-1009.