基于多脑区注意力机制胶囊融合网络的EEG-fNIRS情感识别

doi:10.3785/j.issn.1008-973X.2024.11.006

浙江大学学报(工学版)

2024, Vol. 58

Issue (11): 2247-2257 DOI: 10.3785/j.issn.1008-973X.2024.11.006

计算机技术、控制工程

基于多脑区注意力机制胶囊融合网络的EEG-fNIRS情感识别

刘悦(

),张雪英*(

),陈桂军,黄丽霞,孙颖

太原理工大学电子信息与光学工程学院，山西太原 030024

EEG-fNIRS emotion recognition based on multi-brain attention mechanism capsule fusion network

Yue LIU(

),Xueying ZHANG*(

),Guijun CHEN,Lixia HUANG,Ying SUN

College of Electronic Information and Optical Engineering, Taiyuan University of Technology, Taiyuan 030024, China

全文: PDF(1323 KB) HTML

摘要：

为了提高情感识别的准确率，提出多脑区注意力机制和胶囊融合模块的胶囊网络模型(MBA-CF-cCapsNet). 通过情感视频片段诱发采集EEG-fNIRS信号，构建TYUT3.0数据集. 提取EEG和fNIRS的特征，将其映射到矩阵，通过多脑区注意力机制融合EEG和fNIRS的特征，给予不同脑区特征不同的权重，以提取质量更高的初级胶囊. 使用胶囊融合模块，减少进入动态路由机制的胶囊数量，减少模型运行的时间. 利用MBA-CF-cCapsNet模型在TYUT3.0情感数据集上进行实验，与单模态EEG和fNIRS识别结果相比，2种信号结合情感识别的准确率提高了1.53%和14.35%. MBA-CF-cCapsNet模型与原始CapsNet模型相比，平均识别率提高了4.98%，与当前常用的CapsNet情感识别模型相比提高了1%~5%.

关键词： 胶囊网络; EEG; fNIRS; 多脑区注意力机制; 胶囊融合; 情感识别

Abstract:

The multi-brain attention mechanism and capsule fusion module based on CapsNet (MBA-CF-cCapsNet) was proposed in order to improve the accuracy of emotion recognition. EEG-fNIRS signals were evoked by emotional video clips to construct TYUT3.0 dataset, and the features of EEG and fNIRS were extracted and mapped to the matrix. The features of EEG and fNIRS were fused by the multi-brain region attention mechanism, and different weights were given to the features of different brain regions in order to extract higher quality primary capsules. The capsule fusion module was used to reduce the number of capsules entering the dynamic routing mechanism and reduce the running time of the model. The MBA-CF-cCapsNet model was used to conduct experiment on the TYUT3.0 dataset. The accuracy of emotion recognition combined with the two signals increased by 1.53% and 14.35% compared with the results of single-modal EEG and fNIRS. The average recognition rate of the MBA-CF-cCapsNet model increased by 4.98% compared with the original CapsNet model, and was improved by 1%-5% compared with the current commonly used CapsNet emotion recognition model.

Key words: capsule network EEG fNIRS multi-brain attention mechanism capsule fusion emotion recognition

收稿日期: 2023-07-10 出版日期: 2024-10-23

CLC:

TP 393

基金资助: 国家自然科学基金资助项目(62271342，62201377)；山西省回国留学人员科研资助项目(2022-072)；山西省基础研究计划资助项目 (202203021211174).

通讯作者: 张雪英 E-mail: liuyueofficial9935@163.com;zhangxy@tyut.edu.cn

作者简介: 刘悦（1999—），女，硕士生，从事EEG-fNIRS情感识别的研究. orcid.org/0009-0008-7294-3231.E-mail：liuyueofficial9935@163.com

	服务
	把本文推荐给朋友
	加入引用管理器
	E-mail Alert
	作者相关文章
	刘悦
	张雪英
	陈桂军
	黄丽霞
	孙颖

引用本文:

刘悦,张雪英,陈桂军,黄丽霞,孙颖. 基于多脑区注意力机制胶囊融合网络的EEG-fNIRS情感识别[J]. 浙江大学学报(工学版), 2024, 58(11): 2247-2257.

Yue LIU,Xueying ZHANG,Guijun CHEN,Lixia HUANG,Ying SUN. EEG-fNIRS emotion recognition based on multi-brain attention mechanism capsule fusion network. Journal of ZheJiang University (Engineering Science), 2024, 58(11): 2247-2257.

链接本文:

https://www.zjujournals.com/eng/CN/10.3785/j.issn.1008-973X.2024.11.006 或 https://www.zjujournals.com/eng/CN/Y2024/V58/I11/2247

图 1 EEG和fNIRS情感识别框图

图 2 EEG和fNIRS通道分布图

图 3 EEG-fNIRS情感识别的实验范式

图 4 EEG和fNIRS通道映射的矩阵

图 5 MBA-CF-cCapsNet模型

图 6 脑区分布图

图 7 多脑区注意力机制的融合阶段

图 8 多脑区注意力机制的转换阶段

图 9 胶囊融合模块

图 10 动态路由机制

表 1 MBA-CF-cCapsNet模型的实验参数

表 2 不同分段长度下的情感识别结果

表 3 MBA-CF-cCapsNet模型的消融实验

表 4 不同模型的F1分数

图 11 脑区权重分布图

表 5 不同模态数据的情感分类性能

图 12 EEG情感分类混淆矩阵

图 13 fNIRS情感分类混淆矩阵

图 14 EEG-fNIRS情感分类混淆矩阵

表 6 与其他情感识别模型对比

1	吴朝晖类脑研究: 为人类构建超级大脑[J]. 浙江大学学报: 工学版, 2020, 54 (3): 425- 426 WU Zhaohui Cybrain: building superbrain for humans[J]. Journal of Zhejiang University: Engineering Science, 2020, 54 (3): 425- 426
2	QIU L, ZHONG Y, XIE Q, et al Multi-modal integration of EEG-fNIRS for characterization of brain activity evoked by preferred music[J]. Frontiers in Neurorobotics, 2022, 16: 823435 doi: 10.3389/fnbot.2022.823435
3	MAJID R M, JONG H L EEG based emotion recognition from human brain using Hjorth parameters and SVM[J]. International Journal of Bio-Science and Bio-Technology, 2015, 7 (3): 23- 32 doi: 10.14257/ijbsbt.2015.7.3.03
4	LI T Y, FU B L, WU Z X, et al EEG-based emotion recognition using spatial-temporal-connective features via multi-scale CNN[J]. IEEE Access, 2023, 11: 41859- 41867 doi: 10.1109/ACCESS.2023.3270317
5	DU G L, SU J S, ZHANG L L, et al A multi-dimensional graph convolution network for EEG emotion recognition[J]. IEEE Transactions on Instrumentation and Measurement, 2022, 71: 1- 11
6	LI C, HUANG X Y, SONG R C, et al EEG-based seizure prediction via Transformer guided CNN[J]. Measurement, 2022, 203: 111948 doi: 10.1016/j.measurement.2022.111948
7	CHENG J, CHEN M Y, LI C, et al Emotion recognition from multi-channel EEG via deep forest[J]. IEEE Journal of Biomedical and Health Informatics, 2020, 25 (2): 453- 464
8	BANDARA D, VELIPASALAR S, BRATT S, et al Building predictive models of emotion with functional near-infrared spectroscopy[J]. International Journal of Human-Computer Studies, 2018, 110: 75- 85 doi: 10.1016/j.ijhcs.2017.10.001
9	HU X, ZHUANG C, WANG F, et al fNIRS evidence for recognizably different positive emotions[J]. Frontiers in Human Neuroscience, 2019, 13: 120 doi: 10.3389/fnhum.2019.00120
10	SUN Y, AYAZ H, AKANSU AN Multimodal affective state assessment using fNIRS+EEG and spontaneous facial expression[J]. Brain Sciences, 2020, 10 (2): 85- 104 doi: 10.3390/brainsci10020085
11	BECKER H, FLEUREAU J, GUILLOTEL P, et al Emotion recognition based on high-resolution EEG recordings and reconstructed brain sources[J]. IEEE Transactions on Affective Computing, 2020, 11 (2): 244- 257 doi: 10.1109/TAFFC.2017.2768030
12	ZHE S, ZIHAO H, FENG D, et al A novel multimodal approach for hybrid brain–computer interface[J]. IEEE Access, 2020, 8: 89909- 89918 doi: 10.1109/ACCESS.2020.2994226
13	DELIGANI R J, BORGHEAI S B, MCLINDEN J, et al Multimodal fusion of EEG-fNIRS: a mutual information-based hybrid classification framework[J]. Biomedical Optics Express, 2021, 12 (3): 1635- 1650
14	KWAK Y C, SONG W J, KIM S E FGANet: fNIRS-guided attention network for hybrid EEG-fNIRS brain-computer interfaces[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2022, 30: 329- 339 doi: 10.1109/TNSRE.2022.3149899
15	王颖, 高胜轻量型胶囊网络语音情感识别方法[J]. 电子科技大学学报, 2023, 52 (3): 423- 429 WANG Ying, GAO Sheng A speech emotion recognition method based on lightweight capsule network[J]. Journal of University of Electronic Science and Technology of China, 2023, 52 (3): 423- 429 doi: 10.12178/1001-0548.2022086
16	杨巨成, 韩书杰, 毛磊, 等胶囊网络模型综述[J]. 山东大学学报: 工学版, 2019, 49 (6): 1- 10 YANG Jucheng, HAN Shujie, MAO Lei, et al Review of capsule network[J]. Journal of Shandong University: Engineering Science, 2019, 49 (6): 1- 10
17	HINTON G E, OSINDERO S, TEH Y W A fast-learning algorithm for deep belief nets[J]. Neural Computation, 2006, 18 (7): 1527- 1554
18	ZHANG Y, CHENG C, ZHANG Y Multimodal emotion recognition using a hierarchical fusion convolutional neural network[J]. IEEE Access, 2021, 9: 7943- 7951 doi: 10.1109/ACCESS.2021.3049516
19	谌鈫, 陈兰岚, 江润强集成胶囊网络的脑电情绪识别[J]. 计算机工程与应用, 2022, 58 (8): 175- 184 CHEN Qin, CHEN Lanlan, JIANG Runqiang Emotion recognition of EEG based on ensemble CapsNet[J]. Computer Engineering and Applications, 2022, 58 (8): 175- 184 doi: 10.3778/j.issn.1002-8331.2010-0263
20	YU L, DING Y F, CHANG L, et al Multi-channel EEG-based emotion recognition via a multi-level features guided capsule network[J]. Computers in Biology and Medicine, 2020, 123: 103927 doi: 10.1016/j.compbiomed.2020.103927
21	WANG Z H, CHEN C, LI J, et al ST-CapsNet: linking spatial and temporal attention with capsule network for P300 detection improvement[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2023, 31: 991- 1000 doi: 10.1109/TNSRE.2023.3237319
22	LI C, WANG B, ZHANG S L, et al Emotion recognition from EEG based on multi-task learning with capsule network and attention mechanism[J]. Computers in Biology and Medicine, 2022, 143: 105303 doi: 10.1016/j.compbiomed.2022.105303
23	张静, 张雪英, 陈桂军, 等结合3D-CNN和频-空注意力机制的EEG情感识别[J]. 西安电子科技大学学报, 2022, 49 (3): 191- 198 ZHANG Jing, ZHANG Xueying, CHEN Guijun, et al EEG emotion recognition based on the 3D-CNN and spatial-frequency attention mechanism[J]. Journal of Xidian University, 2022, 49 (3): 191- 198
24	GUIDO N, EDGAR L G, ZHENG L, et al Mathematical relations between measures of brain connectivity estimated from electrophysiological recordings for gaussian distributed data[J]. Frontiers in Neuroscience, 2020, 14: 577574 doi: 10.3389/fnins.2020.577574
25	HAO M G, XING T X, JIANG J L, et al Attention mechanisms in computer vision: a survey[J]. Computational Visual Media, 2022, 8 (3): 331- 368 doi: 10.1007/s41095-022-0271-y
26	崔浩阳, 丁偕, 张敬谊基于细胞图卷积的组织病理图像分类研究[J]. 计算机工程与应用, 2020, 56 (24): 223- 228 CUI Haoyang, DING Xie, ZHANG Jingyi Research on classification of histopathological image based on cell graph convolutional network[J]. Computer Engineering and Applications, 2020, 56 (24): 223- 228 doi: 10.3778/j.issn.1002-8331.2009-0364
27	GUANG B, KAI Y, LI T, et al Linking multi-layer dynamical GCN with style-based recalibration CNN for EEG-based emotion recognition[J]. Frontiers in Neurorobotics, 2022, 16: 834952 doi: 10.3389/fnbot.2022.834952
28	HAMILTON W, YING Z, LESKOVEC J. Inductive representation learning on large graphs [J]. Advances in Neural Information Processing Systems, 2017(12): 1024-1034.
29	LI Y, ZHENG W, WANG L, et al From regional to global brain: a novel hierarchical spatial-temporal neural network model for EEG emotion recognition[J]. IEEE Transactions on Affective Computing, 2019, 13 (2): 568- 578
30	MOON S E, JANG S B, LEE J S. Convolutional neural network approach for EEG-based emotion recognition using brain connectivity and its spatial information [C]// IEEE International Conference on Acoustics, Speech and Signal Processing. Calgary: IEEE, 2018: 2556–2560.

[1]	钟博,王鹏飞,王乙乔,王晓玲. 基于深度学习的EEG数据分析技术综述[J]. 浙江大学学报(工学版), 2024, 58(5): 879-890.
[2]	李海烽,张雪英,段淑斐,贾海蓉,Huizhi Liang . 融合生成对抗网络与时间卷积网络的普通话情感识别[J]. 浙江大学学报(工学版), 2023, 57(9): 1865-1875.
[3]	赵卿,张雪英,陈桂军,张静. 基于模态注意力图卷积特征融合的EEG和fNIRS情感识别[J]. 浙江大学学报(工学版), 2023, 57(10): 1987-1997.
[4]	许萌,王丹,李致远,陈远方. IncepA-EEGNet: 融合Inception网络和注意力机制的P300信号检测方法[J]. 浙江大学学报(工学版), 2022, 56(4): 745-753, 782.
[5]	王雅静,王群,李博闻,刘志文,朴媛媛,遇涛. 基于脑电信号预发作数据段选取的癫痫发作预测[J]. 浙江大学学报(工学版), 2020, 54(11): 2258-2265.
[6]	孙颖,胡艳香,张雪英,段淑斐. 面向情感语音识别的情感维度PAD预测[J]. 浙江大学学报(工学版), 2019, 53(10): 2041-2048.
[7]	王卫星, 孙守迁, 李超, 唐智川. 基于卷积神经网络的脑电信号上肢运动意图识别[J]. 浙江大学学报(工学版), 2017, 51(7): 1381-1389.
[8]	童基均, 李琳, 林勤光, 朱丹华. 采用平滑伪Wigner-Ville分布的SSVEP脑机接口系统[J]. 浙江大学学报(工学版), 2017, 51(3): 598-604.
[9]	孙凌云, 何博伟, 刘征, 杨智渊. 基于语义细胞的语音情感识别[J]. 浙江大学学报(工学版), 2015, 49(6): 1001-1009.
[10]	陈实, 郑楷洪, 孙凌云, 李彦. 可穿戴计算设备中振动表情的设计与应用[J]. 浙江大学学报(工学版), 2015, 49(12): 2298-2304.
[11]	谢波陈岭陈根才陈纯. 普通话语音情感识别的特征选择技术[J]. J4, 2007, 41(11): 1816-1822.

Viewed

Full text

Abstract

Cited

Shared

Discussed