Attention-fused filter bank dual-view graph convolution motor imagery EEG classification

doi:10.3785/j.issn.1008-973X.2024.07.002

Journal of ZheJiang University (Engineering Science)

2024, Vol. 58

Issue (7): 1326-1335 DOI: 10.3785/j.issn.1008-973X.2024.07.002

Attention-fused filter bank dual-view graph convolution motor imagery EEG classification

Shuhan WU1(

),Dan WANG1,*(

),Yuanfang CHEN2,Ziyu JIA3,Yueqi ZHANG1,Meng XU1

1. Faculty of Information Technology, Beijing University of Technology, Beijing 100124, China
2. Beijing Institute of Machinery and Equipment, Beijing 100854, China
3. Brainnetome Center, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China

Download:

HTML

PDF(2102KB) HTML
Export: BibTeX | EndNote (RIS)

Abstract

In motor imagery tasks, the brain often involves simultaneous activation of multiple regions, and traditional convolutional neural networks struggle to accurately represent the coordinated neural activity across these regions. Graph convolutional network GCN is suitable for representing the collaborative tasks of different brain regions by considering the connections and relationships between nodes (brain regions) in graph data. Attention-fused filter bank dual-view GCN（AFB-DVGCN）was proposed. A dual-branch network was constructed using filter banks to extract temporal and spatial information from different frequency bands. Information complementarity was achieved by a convolutional spatial feature extraction method for dual-view graphs. In order to improve the classification accuracy, the effective channel attention mechanism was utilized to enhance features and capture the interaction information between different feature maps. Validation results in the publicly available datasets BCI Competition IV-2a and OpenBMI show that AFB-DVGCN has achieved good classification performance, and the classification accuracy is significantly higher than that of the comparison networks.

Key words： brain-computer interface motor imagery deep learning graph convolutional network attention mechanism

Received: 20 August 2023 Published: 01 July 2024

CLC:	TN 911.7
	R 318

Fund: 国家自然科学基金资助项目（12275295）；中国博士后科学基金面上项目（2023M740171）.

Corresponding Authors: Dan WANG E-mail: wush@emails.bjut.edu.cn;wangdan@bjut.edu.cn

	Service
	E-mail this article
	Add to my bookshelf
	Add to citation manager
	E-mail Alert
	RSS
	Articles by authors
	Shuhan WU
	Dan WANG
	Yuanfang CHEN
	Ziyu JIA
	Yueqi ZHANG
	Meng XU

Cite this article:

Shuhan WU,Dan WANG,Yuanfang CHEN,Ziyu JIA,Yueqi ZHANG,Meng XU. Attention-fused filter bank dual-view graph convolution motor imagery EEG classification. Journal of ZheJiang University (Engineering Science), 2024, 58(7): 1326-1335.

URL:

https://www.zjujournals.com/eng/10.3785/j.issn.1008-973X.2024.07.002 OR https://www.zjujournals.com/eng/Y2024/V58/I7/1326

融合注意力的滤波器组双视图图卷积运动想象脑电分类

在运动想象任务中，传统卷积神经网络难以准确表达大脑多区域协同神经活动；图卷积网络（GCN）能够在图数据中考虑节点（脑区）间的连接和关系，适于表示不同脑区的协同任务，为此提出融合注意力的滤波器组双视图GCN（AFB-DVGCN）. 由滤波器组构建双分支网络，提取不同频段的时域和空域信息；采用双视图图卷积空间特征提取方法实现信息互补；利用有效通道注意力机制增强特征和捕捉不同特征图的交互信息，以提高分类准确率. 在公开数据集BCI Competition IV-2a和OpenBMI上的验证结果表明，AFB-DVGCN的分类性能良好，其分类准确率显著高于对比网络的分类准确率.

关键词： 脑机接口, 运动想象, 深度学习, 图卷积网络, 注意力机制

Fig.1 Overall architecture of attention-fused filter bank dual-view graph convolutional network

Tab.1 Architecture of attention-fused filter bank dual-view graph convolutional network

Fig.2 Structure of effective channel attention

Tab.2 Hold-out comparative classification results

Fig.3 Confusion matrices on two publicly available datasets for proposed network

Fig.4 Visualization results of attention-fused filter bank dual-view graph convolutional network on BCI Competition IV-2a dataset

Fig.5 Visualization results of attention-fused filter bank dual-view graph convolutional network on OpenBMI dataset

Tab.3 Results of ablation experiments

Fig.6 Topographical distribution of power during left-hand trials for subject 4 in BCI Competition IV-2a dataset

Fig.7 Topographical distribution of power during left-hand trials for subject 33 in OpenBMI dataset

Fig.8 Distance adjacency matrices visualization for two publicly available datasets in different frequency bands

Fig.9 Functional adjacency matrices visualization for BCI Competition IV-2a dataset

Fig.10 Functional adjacency matrices visualization for OpenBMI dataset

Tab.4 Classification accuracy of attention-fused filter bank dual-view graph convolutional network with different numbers of filters

Tab.5 Performance comparison of different attention mechanisms in two publicly available datasets


[1]	刘近贞, 叶方方, 熊慧基于卷积神经网络的多类运动想象脑电信号识别[J]. 浙江大学学报: 工学版, 2021, 55 (11): 2054- 2066 LIU Jinzhen, YE Fangfang, XIONG Hui Recognition of multi-class motor imagery EEG signals based on convolutional neural network[J]. Journal of Zhejiang University: Engineering Science, 2021, 55 (11): 2054- 2066

[2]	ALTAHERI H, MUHAMMAD G, ALSULAIMAN M, et al Deep learning techniques for classification of electroencephalogram (EEG) motor imagery (MI) signals: a review[J]. Neural Computing and Applications, 2023, 35: 14681- 14722 doi: 10.1007/s00521-021-06352-5

[3]	ANG K K, CHIN Z Y, ZHANG H, et al. Filter bank common spatial pattern (FBCSP) in brain-computer interface [C]// 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence) . Hong Kong: IEEE, 2008: 2390–2397.

[4]	SALAMI A, ANDREU-PEREZ J, GILLMEISTER H EEG-ITNet: an explainable inception temporal convolutional network for motor imagery classification[J]. IEEE Access, 2022, 10: 36672- 36685 doi: 10.1109/ACCESS.2022.3161489

[5]	SCHIRRMEISTER R T, SPRINGENBERG J T, FIEDERER L D J, et al Deep learning with convolutional neural networks for EEG decoding and visualization[J]. Human Brain Mapping, 2017, 38 (11): 5391- 5420 doi: 10.1002/hbm.23730

[6]	LAWHERN V J, SOLON A J, WAYTOWICH N R, et al EEGNet: a compact convolutional neural network for EEG-based brain-computer interfaces[J]. Journal of Neural Engineering, 2018, 15 (5): 056013 doi: 10.1088/1741-2552/aace8c

[7]	JIA Z, LIN Y, WANG J, et al. GraphSleepNet: adaptive spatial-temporal graph convolutional networks for sleep stage classification [C]// Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence . [S. l.]: ACM, 2020: 1324–1330.

[8]	李珍琦, 王晶, 贾子钰, 等融合注意力的多维特征图卷积运动想象分类[J]. 计算机科学与探索, 2022, 16 (9): 2050- 2060 LI Zhenqi, WANG Jing, JIA Ziyu, et al Attention-based multi-dimensional feature graph convolutional network for motor imagery classification[J]. Journal of Frontiers of Computer Science and Technology, 2022, 16 (9): 2050- 2060 doi: 10.3778/j.issn.1673-9418.2103079

[9]	WANG Q, WU B, ZHU P, et al. ECA-Net: efficient channel attention for deep convolutional neural networks [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . Seattle: IEEE, 2020: 11534–11542.

[10]	徐冰冰, 岑科廷, 黄俊杰, 等图卷积神经网络综述[J]. 计算机学报, 2020, 43 (5): 755- 780 TU Bingbing, CEN Keting, HUANG Junjie, et al A survey on graph convolutional neural network[J]. Chinese Journal of Computers, 2020, 43 (5): 755- 780 doi: 10.11897/SP.J.1016.2020.00755

[11]	ZHANG D, CHEN K, JIAN D, et al Motor imagery classification via temporal attention cues of graph embedded EEG signals[J]. IEEE Journal of Biomedical and Health Informatics, 2020, 24 (9): 2570- 2579 doi: 10.1109/JBHI.2020.2967128

[12]	SUN B, ZHANG H, WU Z, et al Adaptive spatiotemporal graph convolutional networks for motor imagery classification[J]. IEEE Signal Processing Letters, 2021, 28: 219- 223 doi: 10.1109/LSP.2021.3049683

[13]	HOU Y, JIA S, LUN X, et al Deep feature mining via the attention-based bidirectional long short term memory graph convolutional neural network for human motor imagery recognition[J]. Frontiers in Bioengineering and Biotechnology, 2022, 9: 706229 doi: 10.3389/fbioe.2021.706229

[14]	HOU Y, JIA S, LUN X, et al. GCNs-Net: a graph convolutional neural network approach for decoding time-resolved EEG motor imagery signals [J]. IEEE Transactions on Neural Networks and Learning Systems , 2022: 1–12.

[15]	MA W, WANG C, SUN X, et al MBGA-Net: a multi-branch graph adaptive network for individualized motor imagery EEG classification[J]. Computer Methods and Programs in Biomedicine, 2023, 240: 107641 doi: 10.1016/j.cmpb.2023.107641

[16]	BAHDANAU D, CHO K H, BENGIO Y. Neural machine translation by jointly learning to align and translate [C]// 3rd International Conference on Learning Representations 2015 . San Diego: [s. n.], 2015.

[17]	LI D, XU J, WANG J, et al A multi-scale fusion convolutional neural network based on attention mechanism for the visualization analysis of EEG signals decoding[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2020, 28 (12): 2615- 2626 doi: 10.1109/TNSRE.2020.3037326

[18]	LI Y, GUO L, LIU Y, et al A temporal-spectral-based squeeze-and-excitation feature fusion network for motor imagery EEG decoding[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2021, 29: 1534- 1545 doi: 10.1109/TNSRE.2021.3099908

[19]	YU Z, CHEN W, ZHANG T Motor imagery EEG classification algorithm based on improved lightweight feature fusion network[J]. Biomedical Signal Processing and Control, 2022, 75: 103618 doi: 10.1016/j.bspc.2022.103618

[20]	DEFFERRARD M, BRESSON X, VANDERGHEYNST P. Convolutional neural networks on graphs with fast localized spectral filtering [C]// Proceedings of the 30th Information Processing Systems . Barcelona: [s. n.], 2016: 3844–3852.

[21]	TANGERMANN M, MÜLLER K R, AERTSEN A, et al Review of the BCI competition IV[J]. Frontiers in Neuroscience, 2012, 6: 55

[22]	LEE M H, KWON O Y, KIM Y J, et al EEG dataset and OpenBMI toolbox for three BCI paradigms: an investigation into BCI illiteracy[J]. GigaScience, 2019, 8 (5): giz002 doi: 10.1093/gigascience/giz002

[23]	LIU K, YANG M, YU Z, et al FBMSNet: a filter-bank multi-scale convolutional neural network for EEG-based motor imagery decoding[J]. IEEE Transactions on Biomedical Engineering, 2022, 70 (2): 436- 445

[24]	KINGMA D P, BA J. Adam: a method for stochastic optimization [EB/OL]. (2017-01-30)[2023-08-20]. https://arxiv.org/pdf/1412.6980.pdf.

[25]	WU H, NIU Y, LI F, et al A parallel multiscale filter bank convolutional neural networks for motor imagery EEG classification[J]. Frontiers in Neuroscience, 2019, 13: 1275 doi: 10.3389/fnins.2019.01275

[26]	MANE R, ROBINSON N, VINOD A P, et al. A multi-view CNN with novel variance layer for motor imagery brain computer interface [C]// 2020 42nd Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) . Montreal: IEEE, 2020.

[27]	BORRA D, FANTOZZI S, MAGOSSO E Interpretable and lightweight convolutional neural network for EEG decoding: application to movement execution and imagination[J]. Neural Networks, 2020, 129: 55- 74 doi: 10.1016/j.neunet.2020.05.032

[28]	CHEN J, YI W, WANG D. Filter bank Sinc-ShallowNet with EMD-based mixed noise adding data augmentation for motor imagery classification [C]// 2021 43rd Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) . Mexico: IEEE, 2021: 5837–5841.

[29]	VAN DER MAATEN L, HINTON G Visualizing data using t-SNE[J]. Journal of Machine Learning Research, 2008, 9: 2579- 2605

[30]	MISRA D, NALAMADA T, ARASANIPALAI A U, et al. Rotate to attend: convolutional triplet attention module [C]// Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision . Waikoloa: IEEE, 2021: 3139–3148.

[31]	LIU Y, SHAO Z, TENG Y, et al. NAM: normalization-based attention module [EB/OL]. (2021-11-24)[2023-08-20]. https://arxiv.org/pdf/2111.12419.pdf.

[1]	Jun YANG,Chen ZHANG. Semantic segmentation of 3D point cloud based on boundary point estimation and sparse convolution neural network[J]. Journal of ZheJiang University (Engineering Science), 2024, 58(6): 1121-1132.

[2]	Yuntang LI,Hengjie LI,Kun ZHANG,Binrui WANG,Shanyue GUAN,Yuan CHEN. Recognition of complex power lines based on novel encoder-decoder network[J]. Journal of ZheJiang University (Engineering Science), 2024, 58(6): 1133-1141.

[3]	Juan SONG,Longxi HE,Huiping LONG. Deep learning-based algorithm for multi defect detection in tunnel lining[J]. Journal of ZheJiang University (Engineering Science), 2024, 58(6): 1161-1173.

[4]	Bo ZHONG,Pengfei WANG,Yiqiao WANG,Xiaoling WANG. Survey of deep learning based EEG data analysis technology[J]. Journal of ZheJiang University (Engineering Science), 2024, 58(5): 879-890.

[5]	Zhiwei XING,Shujie ZHU,Biao LI. Airline baggage feature perception based on improved graph convolutional neural network[J]. Journal of ZheJiang University (Engineering Science), 2024, 58(5): 941-950.

[6]	Yi LIU,Yidan CHEN,Lin GAO,Jiao HONG. Lightweight road extraction model based on multi-scale feature fusion[J]. Journal of ZheJiang University (Engineering Science), 2024, 58(5): 951-959.

[7]	Cuiting WEI,Weijian ZHAO,Bochao SUN,Yunyi LIU. Intelligent rebar inspection based on improved Mask R-CNN and stereo vision[J]. Journal of ZheJiang University (Engineering Science), 2024, 58(5): 1009-1019.

[8]	Hai HUAN,Yu SHENG,Chenxi GU. Global guidance multi-feature fusion network based on remote sensing image road extraction[J]. Journal of ZheJiang University (Engineering Science), 2024, 58(4): 696-707.

[9]	Xianglong LUO,Yafei WANG,Yanbo WANG,Lixin WANG. Structural deformation prediction of monitoring data based on bi-directional gate board learning system[J]. Journal of ZheJiang University (Engineering Science), 2024, 58(4): 729-736.

[10]	Mingjun SONG,Wen YAN,Yizhao DENG,Junran ZHANG,Haiyan TU. Light-weight algorithm for real-time robotic grasp detection[J]. Journal of ZheJiang University (Engineering Science), 2024, 58(3): 599-610.

[11]	Qingjie QIAN,Junhe YU,Hongfei ZHAN,Rui WANG,Jian HU. Dimension prediction method of injection molded parts based on multi-feature fusion of DL-BiGRU[J]. Journal of ZheJiang University (Engineering Science), 2024, 58(3): 646-654.

[12]	Xinhua YAO,Tao YU,Senwen FENG,Zijian MA,Congcong LUAN,Hongyao SHEN. Recognition method of parts machining features based on graph neural network[J]. Journal of ZheJiang University (Engineering Science), 2024, 58(2): 349-359.

[13]	Siyi QIN,Shaoyan GAI,Feipeng DA. Video object detection algorithm based on multi-level feature aggregation under mixed sampler[J]. Journal of ZheJiang University (Engineering Science), 2024, 58(1): 10-19.

[14]	Zihao SHEN,Yuyu TANG,Hui WANG,Peiqian LIU,Kun LIU. Clustering and deep learning based trajectory privacy protection mechanism for Internet of vehicles[J]. Journal of ZheJiang University (Engineering Science), 2024, 58(1): 20-28.

[15]	Xuefei SUN,Ruifeng ZHANG,Xin GUAN,Qiang LI. Lightweight and efficient human pose estimation with enhanced priori skeleton structure[J]. Journal of ZheJiang University (Engineering Science), 2024, 58(1): 50-60.

Viewed

Full text

Abstract

Cited

Shared

Discussed