Please wait a minute...
浙江大学学报(工学版)
浙江大学学报(工学版)  2023, Vol. 57 Issue (10): 1987-1997    DOI: 10.3785/j.issn.1008-973X.2023.10.008
计算机技术、自动化技术     
基于模态注意力图卷积特征融合的EEG和fNIRS情感识别
赵卿(),张雪英*(),陈桂军,张静
太原理工大学 信息与计算机学院，山西 太原 030024
EEG and fNIRS emotion recognition based on modality attention graph convolution feature fusion
Qing ZHAO(),Xue-ying ZHANG*(),Gui-jun CHEN,Jing ZHANG
College of Information and Computer, Taiyuan University of Technology, Taiyuan 030024, China
 全文: PDF(1285 KB)   HTML
摘要：

为了提升情感识别的准确率，从情绪视频引起的脑电（EEG）和功能近红外（fNIRS）数据中提取每个通道的信号之间的联系，并提出基于模态注意力多路图卷积神经网络(MA-MP-GF)的特征融合情感识别方法. 将EEG和fNIRS数据构建为图结构数据，通过多路图卷积分别对每种模态的信号进行特征提取；利用模态注意力图卷积层融合不同模态通道间的连接信息. 模态注意力机制可以赋予不同模态节点不同权重，使得图卷积层能够更加充分提取不同模态节点间连接关系. 对采集的30个被试的4类情感数据进行实验测试，与仅EEG和仅fNIRS单模态识别结果相比，所提出的图卷积融合方法能够获得更高的识别准确率，分别提升了8.06%、22.90%；与当前常用的EEG-fNIRS融合方法相比，所提出的图卷积融合方法的平均识别准确率提升了2.76%~7.36%；图卷积融合方法在加入模态注意力后识别率最高提升了1.68%.
关键词： 图卷积神经网络;  脑电;  功能近红外;  模态注意力;  多模态融合;  情感识别    
Abstract:

A feature fusion emotion recognition method based on modality attention multi-path convolutional neural network was proposed, extracting the connection between the signals of each channel from the electroencephalogram (EEG) and functional near infrared spectroscopy (fNIRS) data induced by emotional video to improve the accuracy of emotion recognition. The EEG and fNIRS data were constructed as graph structure data, and the feature of each mode signal was extracted by multi-path graph convolution. The information of connection between different modal channels was fused by modality attention graph convolution. The modality attention mechanism can give different weights to different modal nodes, thus the graph convolution layer can more fully extract the connection relationship between different modal nodes. Experimental tests were carried out on four types of emotional data collected from 30 subjects. Compared with the results of EEG only and fNIRS only, the recognition accuracy of the proposed graph convolution fusion method was higher, which increased by 8.06% and 22.90% respectively. Compared with the current commonly used EEG and fNIRS fusion method, the average recognition accuracy of the proposed graph convolution fusion method was improved by 2.76%~7.36%. The recognition rate of graph convolution fusion method increased by 1.68% after adding modality attention.
Key words: graph convolution neural network    electroencephalogram    functional near infrared spectroscopy    modality attention    multi-modal fusion    emotion recognition
收稿日期: 2022-11-26 出版日期: 2023-10-18
CLC:  TP 391  
基金资助: 国家自然科学基金资助项目（62271342，62201377）；山西省回国留学人员科研资助项目（HGKY2019025，2022-072）；山西省基础研究计划资助项目（202203021211174）
通讯作者: 张雪英     E-mail: zqmailofficial@163.com;zhangxy@tyut.edu.cn
作者简介: 赵卿（1998—），男，硕士生，从事EEG-fNIRS情感识别研究. orcid.org/0000-0003-3035-9420. E-mail： zqmailofficial@163.com
服务　　
把本文推荐给朋友
加入引用管理器
E-mail Alert
作者相关文章　　
赵卿
张雪英
陈桂军
张静

引用本文:

赵卿,张雪英,陈桂军,张静. 基于模态注意力图卷积特征融合的EEG和fNIRS情感识别[J]. 浙江大学学报(工学版), 2023, 57(10): 1987-1997.

Qing ZHAO,Xue-ying ZHANG,Gui-jun CHEN,Jing ZHANG. EEG and fNIRS emotion recognition based on modality attention graph convolution feature fusion. Journal of ZheJiang University (Engineering Science), 2023, 57(10): 1987-1997.

链接本文:

https://www.zjujournals.com/eng/CN/10.3785/j.issn.1008-973X.2023.10.008    或    https://www.zjujournals.com/eng/CN/Y2023/V57/I10/1987

图 1  EEG-fNIRS数据采集与处理的流程图
图 2  EEG-fNIRS情感数据的采集流程
图 3  双模态融合情感识别框架
图 4  MA-MP-GF-2D-CNN双模态融合情感识别模型
图 5  模态注意力图卷积层
图 6  EEG-fNIRS空间位置映射矩阵
网络类型 输入（滤波器）大小 步长 填充
EEG fNIRS fusion
Conv2D 3*3 2*3 4*4 1 0
ReLU ? ? ? ? ?
Conv2D 3*3 2*3 4*4 1 0
ReLU ? ? ? ? ?
AvePool2D 2*2 2*2 2*2 2 0
Linear 120 60 180 ? ?
ReLU ? ? ? ? ?
Linear 32 20 32 ? ?
Softmax 4 4 4 ? ?
表 1  2D-CNN网络的参数
模态 Acc/% Std/%
EEG(DE) 88.77 6.16
EEG(PSD) 81.56 10.56
HbO 73.93 6.37
HbR 73.41 8.42
EEG(DE)+HbO+HbR 96.83 2.13
EEG(PSD)+HbO+HbR 95.71 2.14
表 2  不同模态平均情感识别结果
图 7  不同被试单模态以及模态融合的情感识别结果
图 8  情感分类的混淆矩阵
图 9  不同特征组合下各融合方法情感识别结果对比
特征组合 Acc(Std)/%
CF SF GF MA-GF MP-GF MA-MP-GF
EEG(DE)+HbO+HbR 91.77(4.31) 92.39(4.27) 94.93(3.12) 95.81(2.78) 96.11(2.50) 96.83(2.13)
EEG(PSD)+HbO+HbR 89.87(3.84) 89.11(4.91) 92.15(3.99) 93.83(3.34) 94.80(2.44) 95.71(2.14)
EEG(DE)+HbO 90.92(4.90) 90.39(5.09) 94.45(3.48) 94.90(3.37) 95.04(3.24) 95.62(3.12)
EEG(DE)+HbR 91.30(4.71) 90.63(4.94) 94.59(3.23) 95.03(3.33) 95.21(2.98) 95.76(2.89)
EEG(PSD)+HbO 87.05(6.03) 85.97(7.14) 90.57(4.99) 91.36(4.52) 92.28(3.70) 92.94(3.48)
EEG(PSD)+HbR 87.96(5.46) 86.24(6.16) 90.63(4.72) 91.82(4.15) 92.88(3.48) 93.60(3.25)
表 3  模态注意力多路图卷积融合模型消融实验结果
图 10  注意力权重分布图
模型 Acc(Std)/%
MA1 MA2 MA3 MA
GF
MP-GF 		94.93(3.12)
96.11(2.50) 		95.52(2.87)
96.36(2.46) 		95.47(2.88)
96.05(2.42) 		95.81(2.78)
96.83(2.13)
表 4  模态注意力机制消融实验结果
模型 Acc(Std)/%
EEG(DE)+HbO+HbR EEG(PSD)+HbO+HbR
SVM[15]
CF[12]
PF[12]		 90.36(5.97)
91.77(4.31)
93.74(3.45) 		81.47(13.29)
89.87(3.84)
91.13(4.02)
TF[12] 92.91(3.22) 90.58(3.91)
LMF[32] 94.07(3.84) 90.69(4.53)
GF(本研究) 94.93(3.12) 92.15(3.99)
MA-GF(本研究) 95.81(2.78) 93.83(3.34)
MP-GF(本研究) 96.11(2.50) 94.80(2.44)
MA-MP-GF(本研究) 96.83(2.13) 95.71(2.14)
表 5  不同融合方法在EEG-fNIRS数据集上的结果
模型 Acc(Std)/%
SVM[15]
CF[12]
PF[12]		 48.08(11.47)
53.24(11.57)
52.30(11.28)
TF[12] 51.03(10.45)
LMF[33] 52.52(11.41)
GF(本研究) 53.82(13.02)
MA-GF(本研究) 54.41(12.73)
MP-GF(本研究) 53.86(13.56)
MA-MP-GF(本研究) 54.44(13.12)
表 6  不同融合方法在EEG-fNIRS数据集的跨被试结果
1 吴朝晖 类脑研究: 为人类构建超级大脑[J]. 浙江大学学报: 工学版, 2020, 54 (3): 425- 426
WU Zhao-hui Cybrain: building superbrain for humans[J]. Journal of Zhejiang University: Engineering Science, 2020, 54 (3): 425- 426
2 PANKSEPP J Neuro-psychoanalysis may enliven the mind brain sciences[J]. Cortex, 2007, 8 (43): 1106- 1107
3 BRUN N C, MOEN A, BORCH K, et al Near-infrared monitoring of cerebral tissue oxygen saturation and blood volume in newborn piglets[J]. American Journal of Physiology-Heart and Circulatory Physiology, 1997, 273 (2): 682- 686
doi: 10.1152/ajpheart.1997.273.2.H682
4 QIU L, ZHONG Y, XIE Q, et al. Multi-modal integration of EEG-fNIRS for characterization of brain activity evoked by preferred music [EB/OL]. [2022-01-31]. https://doi.org/10.3389/fnbot.2022.823435.
5 KOELSTRA S, MUHL C, SOLEYMANI M, et al Deap: a database for emotion analysis, using physiological signals[J]. IEEE Transactions on Affective Computing, 2011, 3 (1): 18- 31
6 ZHENG W L, LU B L Investigating critical frequency bands and channels for EEG-based emotion recognition with deep neural networks[J]. IEEE Transactions on Autonomous Mental Development, 2015, 7 (3): 162- 175
doi: 10.1109/TAMD.2015.2431497
7 SHEN F, DAI G, LIN G, et al EEG-based emotion recognition using 4D convolutional recurrent neural network[J]. Cognitive Neurodynamics, 2020, 14 (6): 815- 828
doi: 10.1007/s11571-020-09634-1
8 CUI F, WANG R, DING W, et al A novel DE-CNN-BiLSTM multi-fusion model for EEG emotion recognition[J]. Mathematics, 2022, 10 (4): 582
doi: 10.3390/math10040582
9 BANDARA D, VELIPASALAR S, BRATT S, et al Building predictive models of emotion with functional near-infrared spectroscopy[J]. International Journal of Human-Computer Studies, 2018, 110: 75- 85
doi: 10.1016/j.ijhcs.2017.10.001
10 HU X, ZHUANG C, WANG F, et al fNIRS evidence for recognizably different positive emotions[J]. Frontiers in Human Neuroscience, 2019, 13: 120
doi: 10.3389/fnhum.2019.00120
11 高宇航, 司娟宁, 何江弘, 等 脑电与功能近红外光谱技术在脑机接口中的应用[J]. 北京生物医学工程, 2022, 41 (3): 318- 325
GAO Yu-hang, SI Juan-ning, HE Jiang-hong, et al Applications of EEG and fNIRS in brain computer interface[J]. Beijing Biomedical Engineering, 2022, 41 (3): 318- 325
doi: 10.3969/j.issn.1002-3208.2022.03.019
12 SUN Z, HUANG Z, DUAN F, et al A novel multimodal approach for hybrid brain–computer interface[J]. IEEE Access, 2020, 8: 89909- 89918
doi: 10.1109/ACCESS.2020.2994226
13 DELIGANI R J, BORGHEAI S B, MCLINDEN J, et al Multimodal fusion of EEG-fNIRS: a mutual information-based hybrid classification framework[J]. Biomedical Optics Express, 2021, 12 (3): 1635- 1650
doi: 10.1364/BOE.413666
14 王恩慧. 基于EEG-fNIRS的情绪识别系统研究[D]. 长春: 吉林大学, 2020: 48-50.
WANG En-hui. Research of emotion recognition system based on EEG-fNIRS [D]. Changchun: Jilin University, 2020: 48-50.
15 SUN Y, AYAZ H, AKANSU A N Multimodal affective state assessment using fNIRS+EEG and spontaneous facial expression[J]. Brain Sciences, 2020, 10 (2): 85- 105
doi: 10.3390/brainsci10020085
16 吴文一. 基于EEG-fNIRS特征融合的多强度负性情绪识别研究[D]. 天津: 天津大学, 2021: 35-47.
WU Wen-yi. Research on multi-intensity negative emotion recognition based on EEG-fNIRS feature fusion [D]. Tianjin: Tianjin University, 2021: 35-47.
17 LI J, LI S, PAN J, et al. Cross-subject EEG emotion recognition with self-organized graph neural network [EB/OL]. [2021-06-09]. https://doi.org/10.3389/fnins.2021.611653.
18 SONG T, ZHENG W, SONG P, et al EEG emotion recognition using dynamical graph convolutional neural networks[J]. IEEE Transactions on Affective Computing, 2018, 11 (3): 532- 541
19 KIPF T N, WELLING M. Semi-supervised classification with graph convolutional networks [EB/OL]. [2016-09-09]. http://doi.org/10.48550/arxiv.1609.02907.
20 DEFFERRARD M, BRESSON X, VANDERGHEYNST P. Convolutional neural networks on graphs with fast localized spectral filtering [EB/OL]. [2016-06-30]. https://doi.org/10.48550/arXiv.1606.09375.
21 HAMILTON W, YING Z, LESKOVEC J. Inductive representation learning on large graphs[EB/OL]. [2017-06-07]. https://doi.org/10.48550/arXiv.1706.02216.
22 VELIKOVI P , CUCURULL G , CASANOVA A , et al. Graph attention networks[EB/OL]. 2017. /arxiv. 1710.10903.
23 张静, 张雪英, 陈桂军, 等 结合3D-CNN和频-空注意力机制的EEG情感识别[J]. 西安电子科技大学学报, 2022, 49 (3): 191- 198
ZHANG Jing, ZHANG Xue-ying, CHEN Gui-jun, et al EEG emotion recognition based on the 3D-CNN and spatial-frequency attention mechanism[J]. Journal of Xidian University, 2022, 49 (3): 191- 198
24 SONG T, ZHENG W, LU C, et al MPED: a multi-modal physiological emotion database for discrete emotion recognition[J]. IEEE Access, 2019, 7: 12177- 12191
doi: 10.1109/ACCESS.2019.2891579
25 SCHAEFER A, NILS F, SANCHEZ X, et al Assessing the effectiveness of a large database of emotion-eliciting films: a new tool for emotion researchers[J]. Cognition and Emotion, 2010, 24 (7): 1153- 1172
doi: 10.1080/02699930903274322
26 BRADLEY M M, LANG P J Measuring emotion: the self-assessment manikin and the semantic differential[J]. Journal of Behavior Therapy and Experimental Psychiatry, 1994, 25 (1): 49- 59
doi: 10.1016/0005-7916(94)90063-9
27 SCHOLKMANN F, KLEISER S, METZ A J, et al A review on continuous wave functional near-infrared spectroscopy and imaging instrumentation and methodology[J]. Neuroimage, 2014, 85: 6- 27
doi: 10.1016/j.neuroimage.2013.05.004
28 ZHANG G, YU M, LIU Y J, et al. SparseDGCNN: recognizing emotion from multichannel EEG signals [EB/OL]. [2021-01-13]. https://doi.org/10.1109/TAFFC.2021.3051332.
29 DUAN R N, ZHU J Y, LU B L. Differential entropy feature for EEG-based emotion classification [C]// 6th International IEEE/EMBS Conference on Neural Engineering(NER). San Diego: IEEE, 2013: 81-84.
30 BAO G, YANG K, TONG L, et al. Linking multi-layer dynamical GCN with style-based recalibration CNN for EEG-based emotion recognition [EB/OL].[2022-02-24]. https://doi.org/10.3389/fnbot.2022.834952.
31 ACHARD S, BULLMORE E Efficiency and cost of economical brain functional networks[J]. PLoS Computational Biology, 2007, 3 (2): 174- 183
32 LIU Z, SHEN Y, LAKSHMINARASIMHAN V B, et al. Efficient low-rank multimodal fusion with modality-specific factors [C]// Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics. Melbourne: ACL, 2018: 2247-2256.
[1] 李海烽,张雪英,段淑斐,贾海蓉,Huizhi Liang . 融合生成对抗网络与时间卷积网络的普通话情感识别[J]. 浙江大学学报(工学版), 2023, 57(9): 1865-1875.
[2] 周传华,操礼春,周家亿,詹凤. 图卷积融合计算时效网络节点重要性评估分析[J]. 浙江大学学报(工学版), 2023, 57(5): 930-938.
[3] 张灵维,周正东,许云飞,王嘉文,吉文韬,宋泽峰. 基于特征融合的语言想象脑电信号分类[J]. 浙江大学学报(工学版), 2023, 57(4): 726-734.
[4] 陈志刚,万永菁,王于蓝,蒋翠玲,陈霞. 基于异构低秩多模态融合网络的后囊膜混浊预测[J]. 浙江大学学报(工学版), 2021, 55(11): 2045-2053.
[5] 刘近贞,叶方方,熊慧. 基于卷积神经网络的多类运动想象脑电信号识别[J]. 浙江大学学报(工学版), 2021, 55(11): 2054-2066.
[6] 闫旭,范晓亮,郑传潘,臧彧,王程,程明,陈龙彪. 基于图卷积神经网络的城市交通态势预测算法[J]. 浙江大学学报(工学版), 2020, 54(6): 1147-1155.
[7] 王雅静,王群,李博闻,刘志文,朴媛媛,遇涛. 基于脑电信号预发作数据段选取的癫痫发作预测[J]. 浙江大学学报(工学版), 2020, 54(11): 2258-2265.
[8] 贾子钰,林友芳,张宏钧,王晶. 基于深度卷积神经网络的睡眠分期模型[J]. 浙江大学学报(工学版), 2020, 54(10): 1899-1905.
[9] 孙颖,胡艳香,张雪英,段淑斐. 面向情感语音识别的情感维度PAD预测[J]. 浙江大学学报(工学版), 2019, 53(10): 2041-2048.
[10] 王卫星, 孙守迁, 李超, 唐智川. 基于卷积神经网络的脑电信号上肢运动意图识别[J]. 浙江大学学报(工学版), 2017, 51(7): 1381-1389.
[11] 童基均, 李琳, 林勤光, 朱丹华. 采用平滑伪Wigner-Ville分布的SSVEP脑机接口系统[J]. 浙江大学学报(工学版), 2017, 51(3): 598-604.
[12] 孙凌云, 何博伟, 刘征, 杨智渊. 基于语义细胞的语音情感识别[J]. 浙江大学学报(工学版), 2015, 49(6): 1001-1009.
[13] 陈实, 郑楷洪, 孙凌云, 李彦. 可穿戴计算设备中振动表情的设计与应用[J]. 浙江大学学报(工学版), 2015, 49(12): 2298-2304.
[14] 杨帮华, 何美燕, 刘丽, 陆文宇. 脑机接口中基于BISVM的EEG分类[J]. J4, 2013, 47(8): 1431-1436.
[15] 朱丹华,陈大竞,陈裕泉,潘敏. 参数调节随机共振增强稳态视觉诱发电位[J]. J4, 2012, 46(5): 918-922.