Please wait a minute...
浙江大学学报(工学版)
浙江大学学报(工学版)  2024, Vol. 58 Issue (12): 2540-2546    DOI: 10.3785/j.issn.1008-973X.2024.12.013
生物医学工程     
基于注意力机制和深度学习的群体语言想象脑电信号分类
周逸凡1(),张灵维1,2,周正东1,*(),蔡智1,袁梦瑶1,袁晓曦1,杨泽毅1
1. 南京航空航天大学 航空航天结构力学及控制国家重点实验室，江苏 南京 210016
2. 芯原微电子（南京）有限公司，江苏 南京 210000
Classification of group speech imagined EEG signals based on attention mechanism and deep learning
Yifan ZHOU1(),Lingwei ZHANG1,2,Zhengdong ZHOU1,*(),Zhi CAI1,Mengyao YUAN1,Xiaoxi YUAN1,Zeyi YANG1
1. State Key Laboratory of Mechanics and Control for Aerospace Structures, Nanjing University of Aeronautics and Astronautics, Nanjing 210016, China
2. VeriSilicon Holdings (Nanjing) Co. Ltd, Nanjing 210000, China
 全文: PDF(1087 KB)   HTML
摘要：

为了提高群体语言想象脑电信号的分类准确率，提出基于卷积块注意力模块（CBAM）和Inception-V4卷积神经网络的分类方法，其中CBAM被用于关注重要的局部区域，从卷积神经网络（CNN）输出的特征图中提取更加独特的特征，从而提升群体语言想象脑电信号的分类性能. 该方法首先利用短时傅里叶变换将群体语言想象脑电信号转换为时频图，然后使用这些图片对融合了CBAM机制的Inception-V4网络进行训练. 开源数据集上的实验结果表明，所提出的方法使得6类短词的分类准确率达到了52.2%，与基于Inception-V4的分类方法相比，分类准确率提高了4.1个百分点，与基于VGG-16的分类方法相比，分类准确率提高了5.9个百分点. 使用迁移学习也能够大幅缩短训练所需的时间.
关键词： 脑-机接口;  脑电图;  语言想象;  深度学习;  注意力机制    
Abstract:

A classification method based on convolutional block attention module (CBAM) and Inception-V4 convolutional neural network was proposed to improve the classification accuracy of group EEG signals of imagined speech. CBAM was used to emphasize significant localized areas and extract distinctive features from the output feature map of convolutional neural network (CNN), so as to improve the classification performance of group EEG signals of imagined speech. The group EEG signals of imagined speech were converted into time-frequency images by short-time Fourier transform, then the images were used to train the Inception-V4 network incorporating with CBAM. Experiments on an open-accessed dataset showed that the proposed method achieved an accuracy of 52.2% in classifying six types of short words, which was 4.1 percentage points higher than that with Inception-V4 and was 5.9 percentage points higher than that with VGG-16. Furthermore, the training time can be reduced greatly with transfer learning.
Key words: brain-computer interface    electroencephalogram    speech imagery    deep learning    attention mechanism
收稿日期: 2023-10-17 出版日期: 2024-11-25
CLC:  TP 391  
基金资助: 中国航空研究院首批揭榜挂帅项目（F2021109）；上海航天科技创新基金资助项目 (SAST2019-121）；南京航空航天大学科研与实践创新计划资助项目（xcxjh20210104）；江苏高校优势学科建设工程资助项目（PAPD）.
通讯作者: 周正东     E-mail: sz2201046@nuaa.edu.cn;zzd_msc@nuaa.edu.cn
作者简介: 周逸凡（2000—），男，硕士生，从事脑-机接口研究. orcid.org/0009-0004-4981-0132. E-mail：sz2201046@nuaa.edu.cn
服务　　
把本文推荐给朋友
加入引用管理器
E-mail Alert
作者相关文章　　
周逸凡
张灵维
周正东
蔡智
袁梦瑶
袁晓曦
杨泽毅

引用本文:

周逸凡,张灵维,周正东,蔡智,袁梦瑶,袁晓曦,杨泽毅. 基于注意力机制和深度学习的群体语言想象脑电信号分类[J]. 浙江大学学报(工学版), 2024, 58(12): 2540-2546.

Yifan ZHOU,Lingwei ZHANG,Zhengdong ZHOU,Zhi CAI,Mengyao YUAN,Xiaoxi YUAN,Zeyi YANG. Classification of group speech imagined EEG signals based on attention mechanism and deep learning. Journal of ZheJiang University (Engineering Science), 2024, 58(12): 2540-2546.

链接本文:

https://www.zjujournals.com/eng/CN/10.3785/j.issn.1008-973X.2024.12.013    或    https://www.zjujournals.com/eng/CN/Y2024/V58/I12/2540

图 1  脑电信号采集的电极分布位置
图 2  时间-频率图像
图 3  Inception-V4网络框架[16]
图 4  CBAM注意力结构[18]
图 5  融合了CBAM的Inception-V4网络结构图
图 6  基于CBAM和Inception-V4的群体语言想象脑电信号分类方法流程图
图 7  不同窗长下的时频图
图 8  不同窗口长度下群体语言想象脑电信号损失函数的Loss值随训练步数的变化曲线
图 9  Grad-CAM图像
图 10  基于Inception-V4的群体语言想象脑电信号分类混淆矩阵
位置注意力机制
STNECANetCBAM不使用
STEM后20.7%22.4%39.6%48.1%
Avg Pooling前49.3%49.5%52.2%
表 1  不同注意力机制不同插入位置下的分类准确率比较
网络模型分类形式A/%
VMD-RWE+SVM with RBF kernel[22]6分类38.2
Siamese neural network framework[11]6分类44.1
end-to-end Siamese neural network[23]6分类31.4
Deep CNN[24]6分类28.4
VGG-166分类46.3
本研究提出的融合CBAM的Inception-V4模型6分类52.2
表 2  群体语言想象EEG信号分类准确率的比较
编号源域（已训练数据集）目标域（要训练数据集）使用网络tt/hA/%
1无Correto公开数据集Inception-V416848.1
2ImageNet-1000数据集Correto公开数据集Inception-V417147.7
3无Correto公开数据集改Inception-V417252.2
4BCI Competition IV-2b数据集[25]Correto公开数据集改Inception-V46951.8
表 3  迁移学习下信号分类准确率对比
18 WOO S, PARK J, LEE J Y, et al. CBAM: convolutional block attention module [C]// Proceedings of the European Conference on Computer Vision . Munich: Springer Cham, 2018: 3−19.
19 JADERBERG M, SIMONYAN K, ZISSERMAN A, et al. Spatial transformer networks [C]// Proceedings of the 28th International Conference on Neural Information Processing Systems . Montreal: MIT Press, 2015: 2017−2025.
20 WANG Q, WU B, ZHU P, et al. ECA-Net: efficient channel attention for deep convolutional neural networks [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . Seattle: IEEE, 2020: 11534−11542.
21 SELVARAJU R R, COGSWELL M, DAS A, et al. Grad-cam: visual explanations from deep networks via gradient-based localization [C]// Proceedings of the IEEE International Conference on Computer Vision . Venice: IEEE, 2017: 618−626.
22 BISWAS S, SINHA R. Lateralization of brain during EEG based covert speech classification [C]// 2018 15th IEEE India Council International Conference . Coimbatore: IEEE, 2018: 1−5.
23 LEE D Y, LEE M, LEE S W. Classification of imagined speech using Siamese neural network [C]// 2020 IEEE International Conference on Systems, Man, and Cybernetics . Toronto: IEEE, 2020: 2979−2984.
24 COONEY C, KORIK A, FOLLI R, et al Evaluation of hyperparameter optimization in machine and deep learning methods for decoding imagined speech EEG[J]. Sensors, 2020, 20 (16): 4629
doi: 10.3390/s20164629
25 DAGDEVIR E, TOKMAKCI M Determination of effective signal processing stages for brain computer interface on BCI competition IV data set 2b: a review study[J]. IETE Journal of Research, 2023, 69 (6): 3144- 3155
doi: 10.1080/03772063.2021.1914204
1 NGUYEN C H, KARAVAS G K, ARTEMIADIS P Inferring imagined speech using EEG signals: a new approach using Riemannian manifold features[J]. Journal of Neural Engineering, 2017, 15 (1): 016002
2 ROYER A S, DOUD A J, ROSE M L EEG control of a virtual helicopter in 3-dimensional space using intelligent control strategies[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2010, 18 (6): 581- 589
doi: 10.1109/TNSRE.2010.2077654
3 BALAM V P, CHINARA S Statistical channel selection method for detecting drowsiness through single-channel EEG-based BCI system[J]. IEEE Transactions on Instrumentation and Measurement, 2021, 70 (7): 1- 9
4 KHARE S K, BAJAJ V, SINHA G R Adaptive tunable Q wavelet transform-based emotion identification[J]. IEEE Transactions on Instrumentation and Measurement, 2020, 69 (12): 9609- 9617
doi: 10.1109/TIM.2020.3006611
5 KAMBLE K S, SENGUPTA J Ensemble machine learning-based affective computing for emotion recognition using dual-decomposed EEG signals[J]. IEEE Sensors Journal, 2021, 22 (3): 2496- 2507
6 WIERZGAŁA P, ZAPAŁA D, WOJCIK G M Most popular signal processing methods in motor-imagery BCI: a review and meta-analysis[J]. Frontiers in Neuroinformatics, 2018, 12 (11): 78
7 LEE S H, LEE M, LEE S W. EEG representations of spatial and temporal features in imagined speech and overt speech [C]// Pattern Recognition: 5th Asian Conference . Auckland: Springer International Publishing, 2020: 387−400.
8 刘艳鹏, 龚安民, 丁鹏, 等 基于言语想象的脑机交互关键技术[J]. 生物医学工程学杂志, 2022, 39 (3): 596- 611
LIU Yanpeng, GONG Anmin, DING Peng, et al Key technology of brain-computer interaction based on speech imagery[J]. Journal of Biomedical Engineering, 2022, 39 (3): 596- 611
doi: 10.7507/1001-5515.202107018
9 VAN DEN BERG B, VAN DONKELAAR S, ALIMARDANI M. Inner speech classification using EEG signals: a deep learning approach [C]// 2021 IEEE 2nd International Conference on Human-Machine Systems . Magdeburg: IEEE, 2021: 1−4.
10 RUSNAC A L, GRIGORE O CNN architectures and feature extraction methods for EEG imaginary speech recognition[J]. Sensors, 2022, 22 (13): 4679- 4698
doi: 10.3390/s22134679
11 LEE D Y, LEE M, LEE S W Decoding imagined speech based on deep metric learning for intuitive BCI communication[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2021, 29 (7): 1363- 1374
12 CORETTO G A P, GAREIS I E, RUFINER H L. Open access database of EEG signals recorded during imagined speech [C]// 12th International Symposium on Medical Information Processing and Analysis . Munich: SPIE, 2017: 1016002.
13 CAO Y, OOSTENVELD R, ALDAY P M, et al Are alpha and beta oscillations spatially dissociated over the cortex in context-driven spoken-word production?[J]. Psychophysiology, 2022, 59 (6): e13999
doi: 10.1111/psyp.13999
14 WEI X, SUN T, ZHENG L, et al. Diagnosis of loose core fault in saturable reactor of thyristor valve based on vibration signal time-frequency analysis and CNN [C]// 5th International Conference on Energy Systems and Electrical Power . Changsha: Journal of Physics: Conference Series, 2023, 2584(1): 012079.
15 PODDER P, KHAN T Z, KHAN M H, et al Comparative performance analysis of hamming, hanning and blackman window[J]. International Journal of Computer Applications, 2014, 96 (18): 1- 7
doi: 10.5120/16891-6927
16 SZEGEDY C, IOFFE S, VANHOUCKE V, et al. Inception-V4, inception-resnet and the impact of residual connections on learning [C]// Proceedings of the AAAI Conference on Artificial Intelligence . San Francisco: AAAI, 2017, 31(1): 4278−4284.
[1] 李凡,杨杰,冯志成,陈智超,付云骁. 基于图像识别的弓网接触点检测方法[J]. 浙江大学学报(工学版), 2024, 58(9): 1801-1810.
[2] 肖力,曹志刚,卢浩冉,黄志坚,蔡袁强. 基于深度学习和梯度优化的弹性超材料设计[J]. 浙江大学学报(工学版), 2024, 58(9): 1892-1901.
[3] 李灿林,王新玥,马利庄,邵志文,张文娇. 融合注意力机制和结构线提取的图像卡通化[J]. 浙江大学学报(工学版), 2024, 58(8): 1728-1737.
[4] 李忠良,陈麒,石琳,杨朝,邹先明. 时间感知组合的动态知识图谱补全[J]. 浙江大学学报(工学版), 2024, 58(8): 1738-1747.
[5] 吴书晗,王丹,陈远方,贾子钰,张越棋,许萌. 融合注意力的滤波器组双视图图卷积运动想象脑电分类[J]. 浙江大学学报(工学版), 2024, 58(7): 1326-1335.
[6] 李林睿,王东升,范红杰. 基于法条知识的事理型类案检索方法[J]. 浙江大学学报(工学版), 2024, 58(7): 1357-1365.
[7] 马现伟,范朝辉,聂为之,李东,朱逸群. 对失效传感器具备鲁棒性的故障诊断方法[J]. 浙江大学学报(工学版), 2024, 58(7): 1488-1497.
[8] 杨军,张琛. 基于边界点估计与稀疏卷积神经网络的三维点云语义分割[J]. 浙江大学学报(工学版), 2024, 58(6): 1121-1132.
[9] 李运堂,李恒杰,张坤,王斌锐,关山越,陈源. 基于新型编码解码网络的复杂输电线识别[J]. 浙江大学学报(工学版), 2024, 58(6): 1133-1141.
[10] 宋娟,贺龙喜,龙会平. 基于深度学习的隧道衬砌多病害检测算法[J]. 浙江大学学报(工学版), 2024, 58(6): 1161-1173.
[11] 钟博,王鹏飞,王乙乔,王晓玲. 基于深度学习的EEG数据分析技术综述[J]. 浙江大学学报(工学版), 2024, 58(5): 879-890.
[12] 邢志伟,朱书杰,李彪. 基于改进图卷积神经网络的航空行李特征感知[J]. 浙江大学学报(工学版), 2024, 58(5): 941-950.
[13] 刘毅,陈一丹,高琳,洪姣. 基于多尺度特征融合的轻量化道路提取模型[J]. 浙江大学学报(工学版), 2024, 58(5): 951-959.
[14] 魏翠婷,赵唯坚,孙博超,刘芸怡. 基于改进Mask R-CNN与双目视觉的智能配筋检测[J]. 浙江大学学报(工学版), 2024, 58(5): 1009-1019.
[15] 宦海,盛宇,顾晨曦. 基于遥感图像道路提取的全局指导多特征融合网络[J]. 浙江大学学报(工学版), 2024, 58(4): 696-707.