|
|
EEG emotion recognition based on electrode arrangement and Transformer |
Xuan MENG( ),Xueying ZHANG*( ),Ying SUN,Yaru ZHOU |
College of Electronic Information Engineering, Taiyuan University of Technology, Taiyuan 030024, China |
|
|
Abstract The RM-STC (Riemannian Manifold Space Transformer CNN) model based on Riemannian manifold space was proposed to explore the true order of information flow represented by Electroencephalogram (EEG) channels and improve emotion recognition performance. Firstly, the spatial covariance matrix features of EEG signals were calculated and mapped to Riemannian manifold space. The Riemannian distance matrix between EEG channels was then computed and subjected to a non-metric multidimensional scale transformation operation to obtain the one-dimensional ranking of the channels. The Pearson correlation coefficient feature matrix was rearranged according to the calculated relative distance order of the channels, allowing the CNN network to better convolve and learn local features. The advantage of modeling long-range dependencies in Transformer networks was utilized to learn global features to supplement the CNN network perspective, and the electrode channel order based on Riemannian manifold space computation was mapped into vector encoding embedded in the position encoding of the Transformer-CNN branch network, adding additional spatial position encoding information to the network. On the DEAP database, the average recognition rates of the valence dimension and arousal dimension by the proposed method reached 90.51% and 90.98%, respectively. The experimental results demonstrated that electrode arrangement based on Riemannian manifold space and effective spatial position encoding could effectively improve the accuracy of emotion recognition.
|
Received: 12 September 2024
Published: 25 August 2025
|
|
Fund: 国家自然科学基金资助项目(62271342). |
Corresponding Authors:
Xueying ZHANG
E-mail: mengxuan202109@163.com;tyzhangxy@163.com
|
基于电极排列和Transformer的脑电情感识别
为了探索脑电通道所表征信息流的真正顺序以提升情感识别效果,提出基于黎曼流形空间的RM-STC模型. 计算脑电信号的空间协方差矩阵特征,将其映射到黎曼流形空间,计算得出脑电通道之间的黎曼距离矩阵;将该距离矩阵进行非度量型多维尺度变换运算获得通道的一维排序;按照计算得出的通道相对远近顺序重新排列皮尔逊相关系数特征矩阵,使CNN网络可以更好地卷积学习局部特征. 利用Transformer网络建模长距离依赖的优势学习全局特征补充CNN网络视角,并将基于黎曼流形空间计算的电极通道顺序映射为向量编码嵌入到Transformer-CNN分支网络的位置编码处,为该网络添加额外的空间位置编码信息. 在DEAP数据库上,本研究所提方法的效价维和唤醒维的平均识别率分别达到90.51%和90.98%,实验结果证明,基于黎曼流形空间的电极排列和有效的空间位置编码可以有效提升情感识别的准确率.
关键词:
脑电信号,
黎曼流形,
电极排列,
Transformer,
情感识别
|
|
[1] |
LI Y, GUO W, WANG Y Emotion recognition with attention mechanism-guided dual-feature multi-path interaction network[J]. Signal, Image and Video Processing, 2024, 18 (1): 617- 626
|
|
|
[2] |
RANA A, JHA S. Emotion based hate speech detection using multimodal learning [EB/OL]. (2022-02-13) [2024-9-17]. https://arxiv.org/abs/2202.06218v1.
|
|
|
[3] |
ZHANG S, ZHAO X, TIAN Q Spontaneous speech emotion recognition using multiscale deep convolutional LSTM[J]. IEEE Transactions on Affective Computing, 2022, 13 (2): 680- 688
doi: 10.1109/TAFFC.2019.2947464
|
|
|
[4] |
ZHANG S, YANG Y, CHEN C, et al Deep learning-based multimodal emotion recognition from audio, visual, and text modalities: a systematic review of recent advancements and future prospects[J]. Expert Systems with Applications, 2024, 237: 121692
doi: 10.1016/j.eswa.2023.121692
|
|
|
[5] |
TOISOUL A, KOSSAIFI J, BULAT A, et al Estimation of continuous valence and arousal levels from faces in naturalistic conditions[J]. Nature Machine Intelligence, 2021, 3 (1): 42- 50
doi: 10.1038/s42256-020-00280-0
|
|
|
[6] |
HAN L, ZHANG X, YIN J EEG emotion recognition based on the TimesNet fusion model[J]. Applied Soft Computing, 2024, 159: 111635
doi: 10.1016/j.asoc.2024.111635
|
|
|
[7] |
HERRMANN C S, STRÜBER D, HELFRICH R F, et al EEG oscillations: from correlation to causality[J]. International Journal of Psychophysiology, 2016, 103: 12- 21
doi: 10.1016/j.ijpsycho.2015.02.003
|
|
|
[8] |
WU X, ZHENG W L, LI Z, et al Investigating EEG-based functional connectivity patterns for multimodal emotion recognition[J]. Journal of Neural Engineering, 2022, 19 (1): 016012
doi: 10.1088/1741-2552/ac49a7
|
|
|
[9] |
WYCZESANY M, CAPOTOSTO P, ZAPPASODI F, et al Hemispheric asymmetries and emotions: evidence from effective connectivity[J]. Neuropsychologia, 2018, 121: 98- 105
doi: 10.1016/j.neuropsychologia.2018.10.007
|
|
|
[10] |
WANG W Brain network features based on theta-gamma cross-frequency coupling connections in EEG for emotion recognition[J]. Neuroscience Letters, 2021, 761: 136106
doi: 10.1016/j.neulet.2021.136106
|
|
|
[11] |
CHENG C, ZHANG Y, LIU L, et al Multi-domain encoding of spatiotemporal dynamics in EEG for emotion recognition[J]. IEEE Journal of Biomedical and Health Informatics, 2023, 27 (3): 1342- 1353
doi: 10.1109/JBHI.2022.3232497
|
|
|
[12] |
MOON S, MOON S E, LEE J S. Resting-state fNIRS classification using connectivity and convolutional neural networks [C]// IEEE International Conference on Systems, Man, and Cybernetics. Prague: IEEE, 2022: 1724–1729.
|
|
|
[13] |
MOON S E, CHEN C J, HSIEH C J, et al Emotional EEG classification using connectivity features and convolutional neural networks[J]. Neural Networks, 2020, 132: 96- 107
doi: 10.1016/j.neunet.2020.08.009
|
|
|
[14] |
CHEN C J, WANG J L A new approach for functional connectivity via alignment of blood oxygen level-dependent signals[J]. Brain Connectivity, 2019, 9 (6): 464- 474
doi: 10.1089/brain.2018.0636
|
|
|
[15] |
DOSE H, MØLLER J S, IVERSEN H K, et al An end-to-end deep learning approach to MI-EEG signal classification for BCIs[J]. Expert Systems with Applications, 2018, 114: 532- 542
doi: 10.1016/j.eswa.2018.08.031
|
|
|
[16] |
GUO J Y, CAI Q, AN J P, et al A Transformer based neural network for emotion recognition and visualizations of crucial EEG channels[J]. Physica A: Statistical Mechanics and Its Applications, 2022, 603: 127700
doi: 10.1016/j.physa.2022.127700
|
|
|
[17] |
ZHENG W, PAN B A spatiotemporal symmetrical transformer structure for EEG emotion recognition[J]. Biomedical Signal Processing and Control, 2024, 87: 105487
doi: 10.1016/j.bspc.2023.105487
|
|
|
[18] |
HU X, CHEN Y, YAN J, et al Masked self-supervised pre-training model for EEG-based emotion recognition[J]. Computational Intelligence, 2024, 40 (3): e12659
doi: 10.1111/coin.12659
|
|
|
[19] |
DEXTER E, ROLLWAGEN-BOLLENS G, BOLLENS S M The trouble with stress: a flexible method for the evaluation of nonmetric multidimensional scaling[J]. Limnology and Oceanography: Methods, 2018, 16 (7): 434- 443
doi: 10.1002/lom3.10257
|
|
|
[20] |
KULKARNI S, PATIL P R. Analysis of DEAP dataset for emotion recognition [C]// International Conference on Intelligent and Smart Computing in Data Analytics: ISCDA 2020. Singapore: Springer Singapore, 2021: 67–76.
|
|
|
[21] |
KOBLER R J, HIRAYAMA J I, ZHAO Q, et al. SPD domain-specific batch normalization to crack interpretable unsupervised domain adaptation in EEG [EB/OL]. (2022-10-12)[2024-09-12]. https://arxiv.org/abs/2206.01323v2.
|
|
|
[22] |
ZHANG G, ETEMAD A Spatio-temporal EEG representation learning on Riemannian manifold and euclidean space[J]. IEEE Transactions on Emerging Topics in Computational Intelligence, 2024, 8 (2): 1469- 1483
doi: 10.1109/TETCI.2023.3332549
|
|
|
[23] |
YIN Z, ZHAO M, WANG Y, et al Recognition of emotions using multimodal physiological signals and an ensemble deep learning model[J]. Computer Methods and Programs in Biomedicine, 2017, 140: 93- 110
doi: 10.1016/j.cmpb.2016.12.005
|
|
|
[24] |
PIHO L, TJAHJADI T A mutual information based adaptive windowing of informative EEG for emotion recognition[J]. IEEE Transactions on Affective Computing, 2020, 11 (4): 722- 735
doi: 10.1109/TAFFC.2018.2840973
|
|
|
[25] |
GARG A, KAPOOR A, BEDI A K, et al. Merged LSTM Model for emotion classification using EEG signals [C]// International Conference on Data Science and Engineering. Patna: IEEE, 2019: 139-143.
|
|
|
|
Viewed |
|
|
|
Full text
|
|
|
|
|
Abstract
|
|
|
|
|
Cited |
|
|
|
|
|
Shared |
|
|
|
|
|
Discussed |
|
|
|
|