Please wait a minute...
浙江大学学报(工学版)
Journal of ZheJiang University (Engineering Science)  2023, Vol. 57 Issue (10): 1987-1997    DOI: 10.3785/j.issn.1008-973X.2023.10.008
    
EEG and fNIRS emotion recognition based on modality attention graph convolution feature fusion
Qing ZHAO(),Xue-ying ZHANG*(),Gui-jun CHEN,Jing ZHANG
College of Information and Computer, Taiyuan University of Technology, Taiyuan 030024, China
Download: HTML     PDF(1285KB) HTML
Export: BibTeX | EndNote (RIS)      

Abstract  

A feature fusion emotion recognition method based on modality attention multi-path convolutional neural network was proposed, extracting the connection between the signals of each channel from the electroencephalogram (EEG) and functional near infrared spectroscopy (fNIRS) data induced by emotional video to improve the accuracy of emotion recognition. The EEG and fNIRS data were constructed as graph structure data, and the feature of each mode signal was extracted by multi-path graph convolution. The information of connection between different modal channels was fused by modality attention graph convolution. The modality attention mechanism can give different weights to different modal nodes, thus the graph convolution layer can more fully extract the connection relationship between different modal nodes. Experimental tests were carried out on four types of emotional data collected from 30 subjects. Compared with the results of EEG only and fNIRS only, the recognition accuracy of the proposed graph convolution fusion method was higher, which increased by 8.06% and 22.90% respectively. Compared with the current commonly used EEG and fNIRS fusion method, the average recognition accuracy of the proposed graph convolution fusion method was improved by 2.76%~7.36%. The recognition rate of graph convolution fusion method increased by 1.68% after adding modality attention.

Key words： graph convolution neural network      electroencephalogram      functional near infrared spectroscopy      modality attention      multi-modal fusion      emotion recognition     
Received: 26 November 2022      Published: 18 October 2023
CLC:  TP 391  
Fund:  国家自然科学基金资助项目（62271342，62201377）；山西省回国留学人员科研资助项目（HGKY2019025，2022-072）；山西省基础研究计划资助项目（202203021211174）
Corresponding Authors: Xue-ying ZHANG     E-mail: zqmailofficial@163.com;zhangxy@tyut.edu.cn
Service
E-mail this article
Add to my bookshelf
Add to citation manager
E-mail Alert
RSS
Articles by authors
Qing ZHAO
Xue-ying ZHANG
Gui-jun CHEN
Jing ZHANG
Cite this article:

Qing ZHAO,Xue-ying ZHANG,Gui-jun CHEN,Jing ZHANG. EEG and fNIRS emotion recognition based on modality attention graph convolution feature fusion. Journal of ZheJiang University (Engineering Science), 2023, 57(10): 1987-1997.

URL:

https://www.zjujournals.com/eng/10.3785/j.issn.1008-973X.2023.10.008     OR     https://www.zjujournals.com/eng/Y2023/V57/I10/1987


基于模态注意力图卷积特征融合的EEG和fNIRS情感识别

为了提升情感识别的准确率，从情绪视频引起的脑电（EEG）和功能近红外（fNIRS）数据中提取每个通道的信号之间的联系，并提出基于模态注意力多路图卷积神经网络(MA-MP-GF)的特征融合情感识别方法. 将EEG和fNIRS数据构建为图结构数据，通过多路图卷积分别对每种模态的信号进行特征提取；利用模态注意力图卷积层融合不同模态通道间的连接信息. 模态注意力机制可以赋予不同模态节点不同权重，使得图卷积层能够更加充分提取不同模态节点间连接关系. 对采集的30个被试的4类情感数据进行实验测试，与仅EEG和仅fNIRS单模态识别结果相比，所提出的图卷积融合方法能够获得更高的识别准确率，分别提升了8.06%、22.90%；与当前常用的EEG-fNIRS融合方法相比，所提出的图卷积融合方法的平均识别准确率提升了2.76%~7.36%；图卷积融合方法在加入模态注意力后识别率最高提升了1.68%.


关键词： 图卷积神经网络,  脑电,  功能近红外,  模态注意力,  多模态融合,  情感识别 
Fig.1 Flowchart of EEG-fNIRS data acquisition and processing
Fig.2 Collection process of EEG-fNIRS emotional data
Fig.3 Framework of bimodal fusion emotion recognition
Fig.4 MA-MP-GF-2D-CNN bimodal fusion emotion recognition model
Fig.5 Modality attention graph convolutional layer
Fig.6 EEG-fNIRS spatial position mapping matrix
网络类型 输入（滤波器）大小 步长 填充
EEG fNIRS fusion
Conv2D 3*3 2*3 4*4 1 0
ReLU ? ? ? ? ?
Conv2D 3*3 2*3 4*4 1 0
ReLU ? ? ? ? ?
AvePool2D 2*2 2*2 2*2 2 0
Linear 120 60 180 ? ?
ReLU ? ? ? ? ?
Linear 32 20 32 ? ?
Softmax 4 4 4 ? ?
Tab.1 Parameters of 2D-CNN network
模态 Acc/% Std/%
EEG(DE) 88.77 6.16
EEG(PSD) 81.56 10.56
HbO 73.93 6.37
HbR 73.41 8.42
EEG(DE)+HbO+HbR 96.83 2.13
EEG(PSD)+HbO+HbR 95.71 2.14
Tab.2 Average emotion recognition results of different modalities
Fig.7 Emotional recognition results of different subjects in single mode and mode fusion
Fig.8 Confusion matrix of emotion classification
Fig.9 Comparison of emotion recognition results of different fusion methods under different feature combinations
特征组合 Acc(Std)/%
CF SF GF MA-GF MP-GF MA-MP-GF
EEG(DE)+HbO+HbR 91.77(4.31) 92.39(4.27) 94.93(3.12) 95.81(2.78) 96.11(2.50) 96.83(2.13)
EEG(PSD)+HbO+HbR 89.87(3.84) 89.11(4.91) 92.15(3.99) 93.83(3.34) 94.80(2.44) 95.71(2.14)
EEG(DE)+HbO 90.92(4.90) 90.39(5.09) 94.45(3.48) 94.90(3.37) 95.04(3.24) 95.62(3.12)
EEG(DE)+HbR 91.30(4.71) 90.63(4.94) 94.59(3.23) 95.03(3.33) 95.21(2.98) 95.76(2.89)
EEG(PSD)+HbO 87.05(6.03) 85.97(7.14) 90.57(4.99) 91.36(4.52) 92.28(3.70) 92.94(3.48)
EEG(PSD)+HbR 87.96(5.46) 86.24(6.16) 90.63(4.72) 91.82(4.15) 92.88(3.48) 93.60(3.25)
Tab.3 Experimental results of ablation of modality attention multi-path graph convolution fusion model
Fig.10 Distribution map of attention weight
模型 Acc(Std)/%
MA1 MA2 MA3 MA
GF
MP-GF 		94.93(3.12)
96.11(2.50) 		95.52(2.87)
96.36(2.46) 		95.47(2.88)
96.05(2.42) 		95.81(2.78)
96.83(2.13)
Tab.4 Experimental results of ablation of modality attention mechanism
模型 Acc(Std)/%
EEG(DE)+HbO+HbR EEG(PSD)+HbO+HbR
SVM[15]
CF[12]
PF[12]		 90.36(5.97)
91.77(4.31)
93.74(3.45) 		81.47(13.29)
89.87(3.84)
91.13(4.02)
TF[12] 92.91(3.22) 90.58(3.91)
LMF[32] 94.07(3.84) 90.69(4.53)
GF(本研究) 94.93(3.12) 92.15(3.99)
MA-GF(本研究) 95.81(2.78) 93.83(3.34)
MP-GF(本研究) 96.11(2.50) 94.80(2.44)
MA-MP-GF(本研究) 96.83(2.13) 95.71(2.14)
Tab.5 Results of different fusion methods on EEG-fNIRS dataset
模型 Acc(Std)/%
SVM[15]
CF[12]
PF[12]		 48.08(11.47)
53.24(11.57)
52.30(11.28)
TF[12] 51.03(10.45)
LMF[33] 52.52(11.41)
GF(本研究) 53.82(13.02)
MA-GF(本研究) 54.41(12.73)
MP-GF(本研究) 53.86(13.56)
MA-MP-GF(本研究) 54.44(13.12)
Tab.6 Cross-subject results of different fusion methods on EEG-fNIRS data set
[1]   吴朝晖 类脑研究: 为人类构建超级大脑[J]. 浙江大学学报: 工学版, 2020, 54 (3): 425- 426
WU Zhao-hui Cybrain: building superbrain for humans[J]. Journal of Zhejiang University: Engineering Science, 2020, 54 (3): 425- 426
[2]   PANKSEPP J Neuro-psychoanalysis may enliven the mind brain sciences[J]. Cortex, 2007, 8 (43): 1106- 1107
[3]   BRUN N C, MOEN A, BORCH K, et al Near-infrared monitoring of cerebral tissue oxygen saturation and blood volume in newborn piglets[J]. American Journal of Physiology-Heart and Circulatory Physiology, 1997, 273 (2): 682- 686
doi: 10.1152/ajpheart.1997.273.2.H682
[4]   QIU L, ZHONG Y, XIE Q, et al. Multi-modal integration of EEG-fNIRS for characterization of brain activity evoked by preferred music [EB/OL]. [2022-01-31]. https://doi.org/10.3389/fnbot.2022.823435.
[5]   KOELSTRA S, MUHL C, SOLEYMANI M, et al Deap: a database for emotion analysis, using physiological signals[J]. IEEE Transactions on Affective Computing, 2011, 3 (1): 18- 31
[6]   ZHENG W L, LU B L Investigating critical frequency bands and channels for EEG-based emotion recognition with deep neural networks[J]. IEEE Transactions on Autonomous Mental Development, 2015, 7 (3): 162- 175
doi: 10.1109/TAMD.2015.2431497
[7]   SHEN F, DAI G, LIN G, et al EEG-based emotion recognition using 4D convolutional recurrent neural network[J]. Cognitive Neurodynamics, 2020, 14 (6): 815- 828
doi: 10.1007/s11571-020-09634-1
[8]   CUI F, WANG R, DING W, et al A novel DE-CNN-BiLSTM multi-fusion model for EEG emotion recognition[J]. Mathematics, 2022, 10 (4): 582
doi: 10.3390/math10040582
[9]   BANDARA D, VELIPASALAR S, BRATT S, et al Building predictive models of emotion with functional near-infrared spectroscopy[J]. International Journal of Human-Computer Studies, 2018, 110: 75- 85
doi: 10.1016/j.ijhcs.2017.10.001
[10]   HU X, ZHUANG C, WANG F, et al fNIRS evidence for recognizably different positive emotions[J]. Frontiers in Human Neuroscience, 2019, 13: 120
doi: 10.3389/fnhum.2019.00120
[11]   高宇航, 司娟宁, 何江弘, 等 脑电与功能近红外光谱技术在脑机接口中的应用[J]. 北京生物医学工程, 2022, 41 (3): 318- 325
GAO Yu-hang, SI Juan-ning, HE Jiang-hong, et al Applications of EEG and fNIRS in brain computer interface[J]. Beijing Biomedical Engineering, 2022, 41 (3): 318- 325
doi: 10.3969/j.issn.1002-3208.2022.03.019
[12]   SUN Z, HUANG Z, DUAN F, et al A novel multimodal approach for hybrid brain–computer interface[J]. IEEE Access, 2020, 8: 89909- 89918
doi: 10.1109/ACCESS.2020.2994226
[13]   DELIGANI R J, BORGHEAI S B, MCLINDEN J, et al Multimodal fusion of EEG-fNIRS: a mutual information-based hybrid classification framework[J]. Biomedical Optics Express, 2021, 12 (3): 1635- 1650
doi: 10.1364/BOE.413666
[14]   王恩慧. 基于EEG-fNIRS的情绪识别系统研究[D]. 长春: 吉林大学, 2020: 48-50.
WANG En-hui. Research of emotion recognition system based on EEG-fNIRS [D]. Changchun: Jilin University, 2020: 48-50.
[15]   SUN Y, AYAZ H, AKANSU A N Multimodal affective state assessment using fNIRS+EEG and spontaneous facial expression[J]. Brain Sciences, 2020, 10 (2): 85- 105
doi: 10.3390/brainsci10020085
[16]   吴文一. 基于EEG-fNIRS特征融合的多强度负性情绪识别研究[D]. 天津: 天津大学, 2021: 35-47.
WU Wen-yi. Research on multi-intensity negative emotion recognition based on EEG-fNIRS feature fusion [D]. Tianjin: Tianjin University, 2021: 35-47.
[17]   LI J, LI S, PAN J, et al. Cross-subject EEG emotion recognition with self-organized graph neural network [EB/OL]. [2021-06-09]. https://doi.org/10.3389/fnins.2021.611653.
[18]   SONG T, ZHENG W, SONG P, et al EEG emotion recognition using dynamical graph convolutional neural networks[J]. IEEE Transactions on Affective Computing, 2018, 11 (3): 532- 541
[19]   KIPF T N, WELLING M. Semi-supervised classification with graph convolutional networks [EB/OL]. [2016-09-09]. http://doi.org/10.48550/arxiv.1609.02907.
[20]   DEFFERRARD M, BRESSON X, VANDERGHEYNST P. Convolutional neural networks on graphs with fast localized spectral filtering [EB/OL]. [2016-06-30]. https://doi.org/10.48550/arXiv.1606.09375.
[21]   HAMILTON W, YING Z, LESKOVEC J. Inductive representation learning on large graphs[EB/OL]. [2017-06-07]. https://doi.org/10.48550/arXiv.1706.02216.
[22]   VELIKOVI P , CUCURULL G , CASANOVA A , et al. Graph attention networks[EB/OL]. 2017. /arxiv. 1710.10903.
[23]   张静, 张雪英, 陈桂军, 等 结合3D-CNN和频-空注意力机制的EEG情感识别[J]. 西安电子科技大学学报, 2022, 49 (3): 191- 198
ZHANG Jing, ZHANG Xue-ying, CHEN Gui-jun, et al EEG emotion recognition based on the 3D-CNN and spatial-frequency attention mechanism[J]. Journal of Xidian University, 2022, 49 (3): 191- 198
[24]   SONG T, ZHENG W, LU C, et al MPED: a multi-modal physiological emotion database for discrete emotion recognition[J]. IEEE Access, 2019, 7: 12177- 12191
doi: 10.1109/ACCESS.2019.2891579
[25]   SCHAEFER A, NILS F, SANCHEZ X, et al Assessing the effectiveness of a large database of emotion-eliciting films: a new tool for emotion researchers[J]. Cognition and Emotion, 2010, 24 (7): 1153- 1172
doi: 10.1080/02699930903274322
[26]   BRADLEY M M, LANG P J Measuring emotion: the self-assessment manikin and the semantic differential[J]. Journal of Behavior Therapy and Experimental Psychiatry, 1994, 25 (1): 49- 59
doi: 10.1016/0005-7916(94)90063-9
[27]   SCHOLKMANN F, KLEISER S, METZ A J, et al A review on continuous wave functional near-infrared spectroscopy and imaging instrumentation and methodology[J]. Neuroimage, 2014, 85: 6- 27
doi: 10.1016/j.neuroimage.2013.05.004
[28]   ZHANG G, YU M, LIU Y J, et al. SparseDGCNN: recognizing emotion from multichannel EEG signals [EB/OL]. [2021-01-13]. https://doi.org/10.1109/TAFFC.2021.3051332.
[29]   DUAN R N, ZHU J Y, LU B L. Differential entropy feature for EEG-based emotion classification [C]// 6th International IEEE/EMBS Conference on Neural Engineering(NER). San Diego: IEEE, 2013: 81-84.
[30]   BAO G, YANG K, TONG L, et al. Linking multi-layer dynamical GCN with style-based recalibration CNN for EEG-based emotion recognition [EB/OL].[2022-02-24]. https://doi.org/10.3389/fnbot.2022.834952.
[31]   ACHARD S, BULLMORE E Efficiency and cost of economical brain functional networks[J]. PLoS Computational Biology, 2007, 3 (2): 174- 183
[32]   LIU Z, SHEN Y, LAKSHMINARASIMHAN V B, et al. Efficient low-rank multimodal fusion with modality-specific factors [C]// Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics. Melbourne: ACL, 2018: 2247-2256.
[1] Hai-feng LI,Xue-ying ZHANG,Shu-fei DUAN,Hai-rong JIA,Hui-zhi LIANG. Fusing generative adversarial network and temporal convolutional network for Mandarin emotion recognition[J]. Journal of ZheJiang University (Engineering Science), 2023, 57(9): 1865-1875.
[2] Ling-wei ZHANG,Zheng-dong ZHOU,Yun-fei XU,Jia-wen WANG,Wen-tao JI,Ze-feng SONG. Classification of imagined speech EEG signals based on feature fusion[J]. Journal of ZheJiang University (Engineering Science), 2023, 57(4): 726-734.
[3] Jin-zhen LIU,Fang-fang YE,Hui XIONG. Recognition of multi-class motor imagery EEG signals based on convolutional neural network[J]. Journal of ZheJiang University (Engineering Science), 2021, 55(11): 2054-2066.
[4] Xu YAN,Xiao-liang FAN,Chuan-pan ZHENG,Yu ZANG,Cheng WANG,Ming CHENG,Long-biao CHEN. Urban traffic flow prediction algorithm based on graph convolutional neural networks[J]. Journal of ZheJiang University (Engineering Science), 2020, 54(6): 1147-1155.
[5] Ya-jing WANG,Qun WANG,Bo-wen LI,Zhi-wen LIU,Yuan-yuan PIAO,Tao YU. Seizure prediction based on pre-ictal period selection of EEG signal[J]. Journal of ZheJiang University (Engineering Science), 2020, 54(11): 2258-2265.
[6] Zi-yu JIA,You-fang LIN,Hong-jun ZHANG,Jing WANG. Sleep stage classification model based ondeep convolutional neural network[J]. Journal of ZheJiang University (Engineering Science), 2020, 54(10): 1899-1905.
[7] Ying SUN,Yan-xiang HU,Xue-ying ZHANG,Shu-fei DUAN. Prediction of emotional dimensions PAD for emotional speech recognition[J]. Journal of ZheJiang University (Engineering Science), 2019, 53(10): 2041-2048.
[8] WANG Wei-xing, SUN Shou-qian, LI Chao, TANG Zhi-chuan. Recognition of upper limb motion intention of EEG signal based on convolutional neural network[J]. Journal of ZheJiang University (Engineering Science), 2017, 51(7): 1381-1389.
[9] SUN Ling-yun, HE Bo-wei, LIU Zheng, YANG Zhi-yuan. Speech emotion recognition based on information cell[J]. Journal of ZheJiang University (Engineering Science), 2015, 49(6): 1001-1009.
[10] CHEN Shi, ZHENG Kai Hong, SUN Ling Yun, LI Yan. Design and application of vibration emoticons in wearable computing devices[J]. Journal of ZheJiang University (Engineering Science), 2015, 49(12): 2298-2304.
[11] YANG Bang-hua, HE Mei-yan, LIU Li, LU Wen-yu. EEG classification based on batch incremental SVM in
brain computer interfaces[J]. Journal of ZheJiang University (Engineering Science), 2013, 47(8): 1431-1436.
[12] ZHU Dan-hua, CHEN Da-jing, CHEN Yu-quan, PAN Min. Enhancement of steady-state visual evoked potentials using
parameter-tuned stochastic resonance[J]. Journal of ZheJiang University (Engineering Science), 2012, 46(5): 918-922.
[13] SHU Dan-Hua, TONG Ji-Jun, BO Min, DENG. Electroencephalogram on pain relief of transcutaneous electrical acupoint stimulation[J]. Journal of ZheJiang University (Engineering Science), 2009, 43(12): 2319-2322.