Please wait a minute...
浙江大学学报(工学版)
浙江大学学报(工学版)  2024, Vol. 58 Issue (7): 1326-1335    DOI: 10.3785/j.issn.1008-973X.2024.07.002
计算机与控制工程     
融合注意力的滤波器组双视图图卷积运动想象脑电分类
吴书晗1(),王丹1,*(),陈远方2,贾子钰3,张越棋1,许萌1
1. 北京工业大学 信息学部，北京 100124
2. 北京机械设备研究所，北京 100854
3. 中国科学院自动化研究所脑网络组研究中心，北京 100190
Attention-fused filter bank dual-view graph convolution motor imagery EEG classification
Shuhan WU1(),Dan WANG1,*(),Yuanfang CHEN2,Ziyu JIA3,Yueqi ZHANG1,Meng XU1
1. Faculty of Information Technology, Beijing University of Technology, Beijing 100124, China
2. Beijing Institute of Machinery and Equipment, Beijing 100854, China
3. Brainnetome Center, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China
 全文: PDF(2102 KB)   HTML
摘要：

在运动想象任务中，传统卷积神经网络难以准确表达大脑多区域协同神经活动；图卷积网络（GCN）能够在图数据中考虑节点（脑区）间的连接和关系，适于表示不同脑区的协同任务，为此提出融合注意力的滤波器组双视图GCN（AFB-DVGCN）. 由滤波器组构建双分支网络，提取不同频段的时域和空域信息；采用双视图图卷积空间特征提取方法实现信息互补；利用有效通道注意力机制增强特征和捕捉不同特征图的交互信息，以提高分类准确率. 在公开数据集BCI Competition IV-2a和OpenBMI上的验证结果表明，AFB-DVGCN的分类性能良好，其分类准确率显著高于对比网络的分类准确率.
关键词： 脑机接口;  运动想象;  深度学习;  图卷积网络;  注意力机制    
Abstract:

In motor imagery tasks, the brain often involves simultaneous activation of multiple regions, and traditional convolutional neural networks struggle to accurately represent the coordinated neural activity across these regions. Graph convolutional network GCN is suitable for representing the collaborative tasks of different brain regions by considering the connections and relationships between nodes (brain regions) in graph data. Attention-fused filter bank dual-view GCN（AFB-DVGCN）was proposed. A dual-branch network was constructed using filter banks to extract temporal and spatial information from different frequency bands. Information complementarity was achieved by a convolutional spatial feature extraction method for dual-view graphs. In order to improve the classification accuracy, the effective channel attention mechanism was utilized to enhance features and capture the interaction information between different feature maps. Validation results in the publicly available datasets BCI Competition IV-2a and OpenBMI show that AFB-DVGCN has achieved good classification performance, and the classification accuracy is significantly higher than that of the comparison networks.
Key words: brain-computer interface    motor imagery    deep learning    graph convolutional network    attention mechanism
收稿日期: 2023-08-20 出版日期: 2024-07-01
CLC:  TN 911.7  
基金资助: 国家自然科学基金资助项目（12275295）；中国博士后科学基金面上项目（2023M740171）.
通讯作者: 王丹     E-mail: wush@emails.bjut.edu.cn;wangdan@bjut.edu.cn
作者简介: 吴书晗（1999—），男，硕士生，从事脑机接口、运动想象研究. orcid.org/0009-0005-7709-6893. E-mail：wush@emails.bjut.edu.cn
服务　　
把本文推荐给朋友
加入引用管理器
E-mail Alert
作者相关文章　　
吴书晗
王丹
陈远方
贾子钰
张越棋
许萌

引用本文:

吴书晗,王丹,陈远方,贾子钰,张越棋,许萌. 融合注意力的滤波器组双视图图卷积运动想象脑电分类[J]. 浙江大学学报(工学版), 2024, 58(7): 1326-1335.

Shuhan WU,Dan WANG,Yuanfang CHEN,Ziyu JIA,Yueqi ZHANG,Meng XU. Attention-fused filter bank dual-view graph convolution motor imagery EEG classification. Journal of ZheJiang University (Engineering Science), 2024, 58(7): 1326-1335.

链接本文:

https://www.zjujournals.com/eng/CN/10.3785/j.issn.1008-973X.2024.07.002    或    https://www.zjujournals.com/eng/CN/Y2024/V58/I7/1326

图 1  融合注意力的滤波器组双视图图卷积网络的整体架构图
模块网络层F1核大小层可训练参数输出尺寸激活函数备注
1输入———$ \text{(}\text{1}\text{,}{{N}}_{\text{e}}\text{,}{{F}}_{\text{t}}\text{)} $——
DVGCN——$ {{F}}_{\text{t}}\text{+}\text{2?}{{N}}_{\text{e}}\text{?}{{N}}_{\text{e}} $$ \text{(1}\text{,}{{N}}_{\text{e}}\text{,}{{F}}_{\text{t}}\text{)} $——
二维卷积50$ \text{(1,}\text{25}\text{)} $$ {{F}}_{\text{1}}\text{?}{25}\text{+}{{F}}_{\text{1}} $$ ({{F}}_{\text{1}},{N}_{{\mathrm{e}}},{F}_{{\mathrm{t}}}-24) $—$ c\text{=1} $
2合并———$ ({2\cdot F}_{1},{N}_{{\mathrm{e}}},{{F}}_{\text{t}}-24) $——
二维卷积50($ {N}_{{\mathrm{e}}} $,1)$ \text{2?}{{F}}_{\text{1}}\text{?}{{F}}_{\text{1}}\text{?}{{N}}_{\text{e}} $$ ({F}_{1},1,{{F}}_{\text{t}}-24) $—$c\text{=1} $
批归一化——$ 2{\cdot F}_{1} $$ ({F}_{1},1,{{F}}_{\text{t}}-24) $平方激活—
平均池化—$ \left(\mathrm{1,75}\right) $—$ ({F}_{1},1,({{F}}_{\text{t}}-99)//15+1) $——
3丢弃层—————$ {p}\text{=0.5} $
ECA——$ 3 $$ ({{F}}_{\text{1}},1,({{F}}_{\text{t}}-99)//15+1) $——
分类压平———$ ({{F}}_{\text{1}}\cdot (({{F}}_{\text{t}}-99)//15+1\left)\right) $——
全连接层——$ {{F}}_{\text{1}}\text{?}\text{((}{{F}}_{\text{t}}-\text{99)//15+1)}\text{?}{N}_{{\mathrm{c}}}+{N}_{{\mathrm{c}}} $$ {N}_{{\mathrm{c}}} $—c=0.5
表 1  融合注意力的滤波器组双视图图卷积网络架构
图 2  有效通道注意力的结构
分类方法BCI Competition IV-2aOpenBMI
acc/%kapstdacc/%kapstd
ShallowConvNet72.8010.6370.10868.9070.3780.153
EEGNet66.1270.5480.15169.6390.3930.154
MSFBCNN73.6880.6490.11569.7130.3940.166
FBCNet71.1030.6150.14665.9630.3170.144
Sinc-ShallowNet73.3410.6450.13068.0280.3610.156
FB-Sinc-ShallowNet73.1100.6410.12668.3700.3670.156
AFB-DVGCN77.3920.6990.10372.6570.4530.142
表 2  跨会话实验对比分类结果
图 3  所提网络在2个公开数据集上的混淆矩阵
图 4  融合注意力的滤波器组双视图图卷积网络在数据集BCI Competition IV-2a上的可视化结果
图 5  融合注意力的滤波器组双视图图卷积网络在数据集OpenBMI上的可视化结果
变体网络acc/%
BCI Competition IV-2aOpenBMI
无FB多分支74.30668.315
无距离脑视图74.57668.935
无功能脑视图74.82669.611
无ECA76.27371.889
AFB-DVGCN 77.392* 72.657*
表 3  消融实验结果
图 6  数据集BCI Competition IV-2a中被试4的左手试次的地形图
图 7  数据集OpenBMI中被试33的左手试次的地形图
图 8  2个公开数据集在不同频段下的距离邻接矩阵可视化
图 9  数据集BCI Competition IV-2a的功能邻接矩阵可视化
图 10  数据集OpenBMI的功能邻接矩阵可视化
Nfbacc/%
BCI Competition IV-2aOpenBMI
174.30668.315
2 77.392 72.658
473.07168.704
表 4  不同滤波器数量下融合注意力的滤波器组双视图图卷积网络的分类准确率
注意力机制BCI Competition IV-2aOpenBMI
acc/%kapstdNTPacc/%kapstdNTP
Triplet Attention[30]72.9550.6550.156129 14070.0650.4010.167112 702
NAM[31]74.6530.6620.108128 94069.5830.3921.660112 502
SimAM[32]72.7620.6370.116128 84067.3430.3470.142112 402
ECA[9] 77.392*0.6990.103128 843 72.657*0.4530.142112 405
表 5  不同注意力机制在2个公开数据集上的性能对比
1 刘近贞, 叶方方, 熊慧 基于卷积神经网络的多类运动想象脑电信号识别[J]. 浙江大学学报: 工学版, 2021, 55 (11): 2054- 2066
LIU Jinzhen, YE Fangfang, XIONG Hui Recognition of multi-class motor imagery EEG signals based on convolutional neural network[J]. Journal of Zhejiang University: Engineering Science, 2021, 55 (11): 2054- 2066
2 ALTAHERI H, MUHAMMAD G, ALSULAIMAN M, et al Deep learning techniques for classification of electroencephalogram (EEG) motor imagery (MI) signals: a review[J]. Neural Computing and Applications, 2023, 35: 14681- 14722
doi: 10.1007/s00521-021-06352-5
3 ANG K K, CHIN Z Y, ZHANG H, et al. Filter bank common spatial pattern (FBCSP) in brain-computer interface [C]// 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence) . Hong Kong: IEEE, 2008: 2390–2397.
4 SALAMI A, ANDREU-PEREZ J, GILLMEISTER H EEG-ITNet: an explainable inception temporal convolutional network for motor imagery classification[J]. IEEE Access, 2022, 10: 36672- 36685
doi: 10.1109/ACCESS.2022.3161489
5 SCHIRRMEISTER R T, SPRINGENBERG J T, FIEDERER L D J, et al Deep learning with convolutional neural networks for EEG decoding and visualization[J]. Human Brain Mapping, 2017, 38 (11): 5391- 5420
doi: 10.1002/hbm.23730
6 LAWHERN V J, SOLON A J, WAYTOWICH N R, et al EEGNet: a compact convolutional neural network for EEG-based brain-computer interfaces[J]. Journal of Neural Engineering, 2018, 15 (5): 056013
doi: 10.1088/1741-2552/aace8c
7 JIA Z, LIN Y, WANG J, et al. GraphSleepNet: adaptive spatial-temporal graph convolutional networks for sleep stage classification [C]// Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence . [S. l.]: ACM, 2020: 1324–1330.
8 李珍琦, 王晶, 贾子钰, 等 融合注意力的多维特征图卷积运动想象分类[J]. 计算机科学与探索, 2022, 16 (9): 2050- 2060
LI Zhenqi, WANG Jing, JIA Ziyu, et al Attention-based multi-dimensional feature graph convolutional network for motor imagery classification[J]. Journal of Frontiers of Computer Science and Technology, 2022, 16 (9): 2050- 2060
doi: 10.3778/j.issn.1673-9418.2103079
9 WANG Q, WU B, ZHU P, et al. ECA-Net: efficient channel attention for deep convolutional neural networks [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . Seattle: IEEE, 2020: 11534–11542.
10 徐冰冰, 岑科廷, 黄俊杰, 等 图卷积神经网络综述[J]. 计算机学报, 2020, 43 (5): 755- 780
TU Bingbing, CEN Keting, HUANG Junjie, et al A survey on graph convolutional neural network[J]. Chinese Journal of Computers, 2020, 43 (5): 755- 780
doi: 10.11897/SP.J.1016.2020.00755
11 ZHANG D, CHEN K, JIAN D, et al Motor imagery classification via temporal attention cues of graph embedded EEG signals[J]. IEEE Journal of Biomedical and Health Informatics, 2020, 24 (9): 2570- 2579
doi: 10.1109/JBHI.2020.2967128
12 SUN B, ZHANG H, WU Z, et al Adaptive spatiotemporal graph convolutional networks for motor imagery classification[J]. IEEE Signal Processing Letters, 2021, 28: 219- 223
doi: 10.1109/LSP.2021.3049683
13 HOU Y, JIA S, LUN X, et al Deep feature mining via the attention-based bidirectional long short term memory graph convolutional neural network for human motor imagery recognition[J]. Frontiers in Bioengineering and Biotechnology, 2022, 9: 706229
doi: 10.3389/fbioe.2021.706229
14 HOU Y, JIA S, LUN X, et al. GCNs-Net: a graph convolutional neural network approach for decoding time-resolved EEG motor imagery signals [J]. IEEE Transactions on Neural Networks and Learning Systems , 2022: 1–12.
15 MA W, WANG C, SUN X, et al MBGA-Net: a multi-branch graph adaptive network for individualized motor imagery EEG classification[J]. Computer Methods and Programs in Biomedicine, 2023, 240: 107641
doi: 10.1016/j.cmpb.2023.107641
16 BAHDANAU D, CHO K H, BENGIO Y. Neural machine translation by jointly learning to align and translate [C]// 3rd International Conference on Learning Representations 2015 . San Diego: [s. n.], 2015.
17 LI D, XU J, WANG J, et al A multi-scale fusion convolutional neural network based on attention mechanism for the visualization analysis of EEG signals decoding[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2020, 28 (12): 2615- 2626
doi: 10.1109/TNSRE.2020.3037326
18 LI Y, GUO L, LIU Y, et al A temporal-spectral-based squeeze-and-excitation feature fusion network for motor imagery EEG decoding[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2021, 29: 1534- 1545
doi: 10.1109/TNSRE.2021.3099908
19 YU Z, CHEN W, ZHANG T Motor imagery EEG classification algorithm based on improved lightweight feature fusion network[J]. Biomedical Signal Processing and Control, 2022, 75: 103618
doi: 10.1016/j.bspc.2022.103618
20 DEFFERRARD M, BRESSON X, VANDERGHEYNST P. Convolutional neural networks on graphs with fast localized spectral filtering [C]// Proceedings of the 30th Information Processing Systems . Barcelona: [s. n.], 2016: 3844–3852.
21 TANGERMANN M, MÜLLER K R, AERTSEN A, et al Review of the BCI competition IV[J]. Frontiers in Neuroscience, 2012, 6: 55
22 LEE M H, KWON O Y, KIM Y J, et al EEG dataset and OpenBMI toolbox for three BCI paradigms: an investigation into BCI illiteracy[J]. GigaScience, 2019, 8 (5): giz002
doi: 10.1093/gigascience/giz002
23 LIU K, YANG M, YU Z, et al FBMSNet: a filter-bank multi-scale convolutional neural network for EEG-based motor imagery decoding[J]. IEEE Transactions on Biomedical Engineering, 2022, 70 (2): 436- 445
24 KINGMA D P, BA J. Adam: a method for stochastic optimization [EB/OL]. (2017-01-30)[2023-08-20]. https://arxiv.org/pdf/1412.6980.pdf.
25 WU H, NIU Y, LI F, et al A parallel multiscale filter bank convolutional neural networks for motor imagery EEG classification[J]. Frontiers in Neuroscience, 2019, 13: 1275
doi: 10.3389/fnins.2019.01275
26 MANE R, ROBINSON N, VINOD A P, et al. A multi-view CNN with novel variance layer for motor imagery brain computer interface [C]// 2020 42nd Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) . Montreal: IEEE, 2020.
27 BORRA D, FANTOZZI S, MAGOSSO E Interpretable and lightweight convolutional neural network for EEG decoding: application to movement execution and imagination[J]. Neural Networks, 2020, 129: 55- 74
doi: 10.1016/j.neunet.2020.05.032
28 CHEN J, YI W, WANG D. Filter bank Sinc-ShallowNet with EMD-based mixed noise adding data augmentation for motor imagery classification [C]// 2021 43rd Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) . Mexico: IEEE, 2021: 5837–5841.
29 VAN DER MAATEN L, HINTON G Visualizing data using t-SNE[J]. Journal of Machine Learning Research, 2008, 9: 2579- 2605
30 MISRA D, NALAMADA T, ARASANIPALAI A U, et al. Rotate to attend: convolutional triplet attention module [C]// Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision . Waikoloa: IEEE, 2021: 3139–3148.
31 LIU Y, SHAO Z, TENG Y, et al. NAM: normalization-based attention module [EB/OL]. (2021-11-24)[2023-08-20]. https://arxiv.org/pdf/2111.12419.pdf.
[1] 杨军,张琛. 基于边界点估计与稀疏卷积神经网络的三维点云语义分割[J]. 浙江大学学报(工学版), 2024, 58(6): 1121-1132.
[2] 李运堂,李恒杰,张坤,王斌锐,关山越,陈源. 基于新型编码解码网络的复杂输电线识别[J]. 浙江大学学报(工学版), 2024, 58(6): 1133-1141.
[3] 宋娟,贺龙喜,龙会平. 基于深度学习的隧道衬砌多病害检测算法[J]. 浙江大学学报(工学版), 2024, 58(6): 1161-1173.
[4] 邢志伟,朱书杰,李彪. 基于改进图卷积神经网络的航空行李特征感知[J]. 浙江大学学报(工学版), 2024, 58(5): 941-950.
[5] 刘毅,陈一丹,高琳,洪姣. 基于多尺度特征融合的轻量化道路提取模型[J]. 浙江大学学报(工学版), 2024, 58(5): 951-959.
[6] 魏翠婷,赵唯坚,孙博超,刘芸怡. 基于改进Mask R-CNN与双目视觉的智能配筋检测[J]. 浙江大学学报(工学版), 2024, 58(5): 1009-1019.
[7] 钟博,王鹏飞,王乙乔,王晓玲. 基于深度学习的EEG数据分析技术综述[J]. 浙江大学学报(工学版), 2024, 58(5): 879-890.
[8] 何勇禧,韩虎,孔博. 基于多依赖图和知识融合的方面级情感分析模型[J]. 浙江大学学报(工学版), 2024, 58(4): 737-747.
[9] 宦海,盛宇,顾晨曦. 基于遥感图像道路提取的全局指导多特征融合网络[J]. 浙江大学学报(工学版), 2024, 58(4): 696-707.
[10] 罗向龙,王亚飞,王彦博,王立新. 基于双向门控式宽度学习系统的监测数据结构变形预测[J]. 浙江大学学报(工学版), 2024, 58(4): 729-736.
[11] 宋明俊,严文,邓益昭,张俊然,涂海燕. 轻量化机器人抓取位姿实时检测算法[J]. 浙江大学学报(工学版), 2024, 58(3): 599-610.
[12] 钱庆杰,余军合,战洪飞,王瑞,胡健. 基于DL-BiGRU多特征融合的注塑件尺寸预测方法[J]. 浙江大学学报(工学版), 2024, 58(3): 646-654.
[13] 姚鑫骅,于涛,封森文,马梓健,栾丛丛,沈洪垚. 基于图神经网络的零件机加工特征识别方法[J]. 浙江大学学报(工学版), 2024, 58(2): 349-359.
[14] 秦思怡,盖绍彦,达飞鹏. 混合采样下多级特征聚合的视频目标检测算法[J]. 浙江大学学报(工学版), 2024, 58(1): 10-19.
[15] 申自浩,唐雨雨,王辉,刘沛骞,刘琨. 基于聚类和深度学习的车联网轨迹隐私保护机制[J]. 浙江大学学报(工学版), 2024, 58(1): 20-28.