Please wait a minute...
浙江大学学报(工学版)
Journal of ZheJiang University (Engineering Science)  2021, Vol. 55 Issue (11): 2054-2066    DOI: 10.3785/j.issn.1008-973X.2021.11.005
    
Recognition of multi-class motor imagery EEG signals based on convolutional neural network
Jin-zhen LIU1,2(),Fang-fang YE1,2,Hui XIONG1,2,*()
1. School of Control Science and Engineering, TIANGONG University, Tianjin 300387, China
2. Tianjin Key Laboratory of Advanced Technology of Electrical Engineering and Energy, TIANGONG University, Tianjin 300387, China
Download: HTML     PDF(1117KB) HTML
Export: BibTeX | EndNote (RIS)      

Abstract  

An automatic classification method of motor imagery (MI) signals based on deep learning was proposed, aiming at the traditional multi-class MI recognition method of electroencephalogram (EEG) signals that requires cumbersome preprocessing and feature extraction. In terms of sample representation, the multi-channel EEG signals were converted into a one-dimensional sequence signal for processing, which increased the number of samples while neglecting the influence of spatial information related to the channel position. And according to the characteristics of the input signal, the multi-layer one-dimensional convolutional neural network was used to learn time-frequency information in EEG signals under different motor imagery states, and automatically complete the feature extraction and classification. The proposed method was compared with a variety of methods on the public dataset and the classification of the actual collected data was completed. The proposed method was used to do end-to-end learning of EEG signals without prior knowledge. Experimental results show that the method can obtain higher multi-classification accuracy and reduce the impact of individual differences on classification. The proposed method is conducive to the development of MI-based brain-computer interface systems.

Key words： motor imagery      electroencephalogram signal      convolutional neural network      individual difference      brain-computer interface     
Received: 28 December 2020      Published: 05 November 2021
CLC:  TP 29  
Fund:  国家自然科学基金资助项目（61871288）；天津市自然科学基金资助项目（18JCYBJC90400，18JCQNJC84000）
Corresponding Authors: Hui XIONG     E-mail: liujinzhen@tiangong.edu.cn;xionghui@tiangong.edu.cn
Service
E-mail this article
Add to my bookshelf
Add to citation manager
E-mail Alert
RSS
Articles by authors
Jin-zhen LIU
Fang-fang YE
Hui XIONG
Cite this article:

Jin-zhen LIU,Fang-fang YE,Hui XIONG. Recognition of multi-class motor imagery EEG signals based on convolutional neural network. Journal of ZheJiang University (Engineering Science), 2021, 55(11): 2054-2066.

URL:

https://www.zjujournals.com/eng/10.3785/j.issn.1008-973X.2021.11.005     OR     https://www.zjujournals.com/eng/Y2021/V55/I11/2054


基于卷积神经网络的多类运动想象脑电信号识别

针对传统多类运动想象（MI）脑电信号的识别方法须进行繁琐的预处理以及特征提取问题，提出基于深度学习的MI信号自动分类方法.在样本表示方面，提出将多通道脑电（EEG）信号转化为一维序列信号处理，在增加样本数量的同时又能够忽略与通道位置相关的空间信息的影响；根据输入信号的特点，采用多层一维卷积神经网络学习不同运动想象状态时脑电信号中的时频信息，自动完成特征提取和分类工作. 将所提出的方法在公共数据集上与多种方法进行比较，并完成对实际采集数据集的分类. 利用所提方法在不需要先验知识的条件下，对脑电信号进行端到端的学习. 结果表明该方法可以获得更高的多分类准确率以及降低个体差异对分类的影响. 所提出的方法有利于促进基于MI 的脑机接口系统的开发.


关键词： 运动想象,  脑电信号,  卷积神经网络,  个体差异,  脑机接口 
Fig.1 EEG data collection and processing flow
层数 类型 输出特征尺寸 核尺寸
0 输入 750×1 ?
1 卷积 750×32 9
2 池化 375×32 2
3 卷积 375×32 9
4 池化 187×32 2
5 卷积 187×32 9
6 池化 93×32 2
8 全连接 64 ?
9 全连接层 4 ?
Tab.1 Details of CNN structure
Fig.2 1D-CNN model for MI EEG data classification
Fig.3 Paradigm of BCI Competition IV 2a
Fig.4 Accuracy curve during CNN training under different input forms
对比方法 输入设计 分类器 网络特点
DFFN[30] CSP CNN 考虑相邻层和跨层特征之间的相关性，减少卷积运算过程中的信息丢失
HC-CNN[31] 相同标签样本进行分割后交换重组，然后按照频率带划分，同一频率带的不同片段进行交换重组 CNN 对不同频率带的脑电信号设计不同的卷积核尺度，利用一维卷积核在时间和空间维度进行卷积操作
PSO-CNN[32] 滤波后的信号输入到CSP滤波器，同时计算不同频率带的PSD，引入粒子群优化算法进行特征优化 CNN 只有1层卷积层，紧跟着2层全连接层
CNN-LSTM[33] FBCSP CNN+LSTM 混合网络提取多模态特征
DWT-CNN[34] 滑动窗口进行数据增强，计算最优频率带的PSD CNN 结构与VGGNet相似，3 × 3卷积核分解为4 × 1或者2 × 1
SS-MEMDBF[35] 多元经验模式分解(MEMD) 黎曼几何 ?
RBM-SVM[36] 分别提取时-频-空域特征，利用参数t分布随机邻居嵌入进行参数降维 SVM ?
ETRCNN[37] 脑电图地形表征能量 CNN 网络能够从结合了时频特征和脑功能连通性的输入特征中学习到多模态信息
C2CM[38] FBCSP CNN 分别在时间和空间维度进行卷积，但须针对不同的对象更改参数
3D-CNN[39] 根据电极的3D分布将脑电信号映射为2D图像表示 CNN 构造3路不同卷积核尺寸的卷积网络，最后将3路网络获取的特征融合
FBCSP[40] FBCSP SVM ?
Tab.2 Introduction of methods and models used in comparative references
被试者 A/%
本研究方法 DFFN[30] HC-CNN[31] PSO-CNN[32] CNN-LSTM[33] SS-MEMDBF[35] RBM-SVM[36] ETRCNN[37]
1 99.14 85.40 90.07 93.30 98.82 91.49 86.61 85.88
2 99.49 69.30 80.28 84.59 98.64 60.56 61.26 75.41
3 99.68 90.29 97.08 91.68 96.92 94.16 87.27 91.32
4 99.01 71.07 89.66 84.55 96.50 76.72 75.20 83.45
5 99.61 65.41 97.04 86.54 92.75 58.52 64.55 72.11
6 98.86 69.45 87.04 76.92 91.84 68.52 65.91 91.72
7 98.58 88.18 92.14 94.03 95.07 78.67 83.78 85.71
8 99.82 86.46 98.51 93.20 95.25 97.01 89.91 91.32
9 98.98 93.54 92.31 92.24 99.23 93.85 92.08 93.23
平均值 99.24 79.90 91.57 85.56 96.13 79.94 78.51 85.57
均方差 0.3998 10.25 5.41 5.46 2.486 14.13 11.29 7.08
Tab.3 Comparison of classification accuracy between proposed method and comparative references
被试者 Kappa值
所提方法 CNN-LSTM[33] RBM-SVM[36] ETRCNN[37] C2CM[38] 3D-CNN[39] FBCSP[40]
1 0.9885 0.8500 0.8214 0.8420 0.8750 0.6990 0.6800
2 0.9933 0.5400 0.4838 0.6630 0.6528 0.4590 0.4200
3 0.9957 0.8700 0.7696 0.8770 0.9028 0.7880 0.7500
4 0.9863 0.7800 0.6664 0.7610 0.6667 0.5940 0.4800
5 0.9948 0.7700 0.5024 0.5710 0.6250 0.6470 0.4000
6 0.9848 0.6600 0.5301 0.8910 0.4549 0.5380 0.2700
7 0.9810 0.9500 0.7837 0.8090 0.8959 0.6530 0.7700
8 0.9976 0.8300 0.8655 0.8900 0.8333 0.7020 0.7600
9 0.9864 0.9000 0.8942 0.9070 0.7951 0.7130 0.6100
平均值 0.9898 0.7944 0.7019 0.8012 0.7446 0.6437 0.5711
均方差 0.0053 0.1197 0.1518 0.1097 0.1446 0.0942 0.1737
Tab.4 Comparison of classification Kappa value between proposed method and comparative references
Fig.5 Classification confusion matrix results of subjects 1 and 7
想象类别 被试者2 被试者8
PR RE F-score PR RE F-score
0 1.00 1.00 1.00 1.00 1.00 1.00
1 1.00 1.00 1.00 1.00 1.00 1.00
2 1.00 0.99 1.00 1.00 1.00 1.00
3 1.00 1.00 1.00 1.00 1.00 1.00
Tab.5 Classification precision, recall and F-score results of subjects 2 and 8
运动想象类别 AUC
被试者9 被试者3
0 1.00 0.99
1 1.00 1.00
2 1.00 1.00
3 1.00 1.00
Tab.6 Classification AUC results of subjects 3 and 9
Fig.6 Paradigm of actual MI experiment
Fig.7 Scene figure of MI experiment
Fig.8 Classification accuracy of different subjects under different input lengths
时间长度 $\overline {A}$/% MSE
1 s 83.29 1.7882
2 s 88.90 0.8894
3 s（无下采样） 89.87 1.6286
3 s（下采样） 89.10 1.6587
Tab.7 Classification accuracy statistics table of actual collected dataset
[1]   GRAIMANN B, ALLISON B, PFURTSCHELLER G. Brain-computer interface: a gentle introduction [M]. The frontiers collection. Berlin, Heidelberg: Springer, 2009: 1-27.
[2]   ADAMS M, BEN-SALEM S, ISLAM Z, et al. Towards an SSVEP-BCI controlled smart home [C]// 2019 IEEE International Conference on Systems Man and Cybernetics (SMC). Bari: IEEE, 2019: 2737-2742.
[3]   CHEN L, WANG Z P, HE F, et al. An online hybrid brain-computer interface combining multiple physiological signals for webpage browse [C]// 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society. Milan: IEEE, 2015: 1152-1155.
[4]   WONG C M, TANG Q, WAN F, et al. A multi-channel SSVEP-based BCI for computer games with analogue control [C]// Proceedings of the 2015 IEEE International Conference on Computational Intelligence and Virtual Environments for Measurement Systems and Application (CIVEMSA). Shenzhen: IEEE, 2015: 1-6.
[5]   PFURTSCHELLER G, LOPES DA SILVA F H Event-related EEG/MEG synchronization and desynchronization: basic principles[J]. Clinical Neurophysiology, 1999, 110 (11): 1842- 1857
doi: 10.1016/S1388-2457(99)00141-8
[6]   巫嘉陵, 高忠科 脑机接口技术及其在神经科学中的应用[J]. 中国现代神经疾病杂志, 2021, 21 (1): 3- 8
WU Jia-ling, GAO Zhong-ke Brain-computer interface technology and its applications in neuroscience[J]. Chinese Journal of Modern Neurological Diseases, 2021, 21 (1): 3- 8
doi: 10.3969/j.issn.1672-6731.2021.01.002
[7]   BAIG M Z, ASLAML N, SHUM H P H Filtering techniques for channel selection in motor imagery EEG applications: a survey[J]. Artificial Intelligence Review, 2020, 53 (3): 1207- 1232
[8]   霍首君, 郝琰, 石慧宇, 等 基于深度卷积网络的运动想象脑电信号模式识别[J]. 计算机应用, 2020, 41 (4): 1042- 1048
HUO Shou-jun, HAO Yan, SHI Hui-yu, et al Pattern recognition of motor imagery EEG based on deep convolutional networks[J]. Computer Applications, 2020, 41 (4): 1042- 1048
[9]   CHATTERJEE R, BANDYOPADHYAY T, SANYAL D K, et al. Comparative analysis of feature extraction techniques in motor imagery EEG signal classification [C]// Proceedings of First International Conference on Smart System, Innovations and Computing. Smart Innovation, Systems and Technologies. Jaipur: Springer, 2018, 79: 73-83.
[10]   WANG Y J, GAO S K, GAO X R, et al. Common spatial pattern method for channel selection in motor imagery based brain-computer interface [C]// International Conference of the Engineering in Medicine and Biology Society. Shanghai: IEEE, 2005: 5392-5395.
[11]   ANG K K, CHINY C, ZHANG H H, et al. Filter bank common spatial pattern (FBCSP) in brain-computer interface [C]// Proceedings of the IEEE International Joint Conference on Neural Networks. Hong Kong: IEEE, 2008: 2391–2398.
[12]   施锦河, 沈继忠, 王攀 四类运动想象脑电信号特征提取与分类算法[J]. 浙江大学学报:工学版, 2012, 46 (2): 338- 344
SHI Jin-he, SHEN Ji-zhong, WANG Pan Feature extraction and classification of four-class motor imagery EEG data[J]. Journal of Zhejiang University: Engineering Science, 2012, 46 (2): 338- 344
doi: 10.3785/j.issn.1008-973X.2012.02.025
[13]   TYAGI A, NEHRA V. Time frequency analysis of non-stationary motor imagery EEG signal [C]// 2017 International Conference on Computing and Communication Technologies for Smart Nation (IC3TSN). Gurgaon: IEEE, 2017: 44-50.
[14]   LIU J, CHENG Y, ZHANG W. Deep learning EEG response representation for brain computer interface [C]// 34th Chinese Control Conference. Hangzhou: IEEE, 2015: 3518-3523.
[15]   KUMAR S, SHARMA A, MAMUN K, et al. A deep learning approach for motor imagery EEG signal classification [C]// 2016 3rd Asia-Pacific World Congress on Computer Science and Engineering (APWC on CSE). Nadi: IEEE, 2016: 34-39.
[16]   LEE H K, CHOI Y. A convolutional neural networks scheme for classification of motor imagery EEG based on wavelet time-frequency image [C]// 2018 International Conference on Information Networking (ICOIN). Chiang Mai: IEEE, 2018: 906-909.
[17]   ZHU X Y, LI P Y, LI C B, et al Separated channel convolutional neural network to realize the training free motor imagery BCI systems[J]. Biomedical Signal Processing and Control, 2019, 49: 396- 403
doi: 10.1016/j.bspc.2018.12.027
[18]   YANG T, PHUA K S, YU J H, et al. Image-based motor imagery EEG classification using convolutional neural network [C]// 2019 IEEE EMBS International Conference on Biomedical and Health Informatics (BHI). Chicago: IEEE, 2019: 1-4.
[19]   LI Y, ZHANG X R, LEI M Y, et al A channel-projection mixed-scale convolutional neural network for motor imagery EEG decoding[J]. IEEE Transaction on Neural Systems and Rehabilitation Engineering, 2019, 27 (6): 1170- 1180
doi: 10.1109/TNSRE.2019.2915621
[20]   WU H, NIU Y, LI F, et al A parallel multiscale filter bank convolutional neural networks for motor imagery EEG classification[J]. Frontiers in Neuroscience, 2019, 26 (13): 1275
[21]   ROY S, MCCREADIE K, PRASAD G. Can a single model deep learning approach enhance classification accuracy of an EEG-based brain-computer interface? [C]// 2019 IEEE International Conference on Systems, Man and Cybernetics (SMC). Bari: IEEE, 2019: 1317-1321.
[22]   TANG X B, ZHAO J C, FU W L, et al. A novel classification algorithm for MI-EEG based on deep learning [C]// 2019 IEEE 8th Joint International Information Technology and Artificial Intelligence Conference (ITAIC). Chongqing: IEEE, 2019: 606-611.
[23]   SHEN Y R, LU H T, JIA J. Classification of motor imagery EEG signal with deep learning models [C]// International Conference on Intelligent Science and Big Data Engineering. Dalian: Springer, 2017: 181-190.
[24]   LASHGARI E, LIANG D, MAOZ U Data augmentation for deep-learning-based electroencephalography[J]. Journal of Neuroscience Methods, 2020, 346: 108885
doi: 10.1016/j.jneumeth.2020.108885
[25]   BASHIVAN P, RISH I, YEASIN M, et al. Learning representations from EEG with deep recurrent-convolutional neural networks [EB/OL]. (2015-11-19) [2020-12-28]. https://arxiv.org/abs/1511.06448v3.
[26]   CHANG L, DENG X M, ZHOU M Q, et al Convolutional neural networks in image understanding[J]. Acta Automatica Sinica, 2016, 42 (9): 1300- 1312
[27]   DING Y, ROBINSON N, ZENG Q H, et al. TSception: a deep learning framework for emotion detection using EEG [EB/OL]. (2020-04-02) [2020-12-28]. https://arxiv.org/abs/2004.02965.
[28]   王卫星, 孙守迁, 李超, 等 基于卷积神经网络的脑电信号上肢运动意图识别[J]. 浙江大学学报:工学版, 2017, 51 (7): 1381- 1389
WANG Wei-xing, SUN Shou-qian, LI Chao, et al Recognition of upper limb motion intention of EEG signal based on convolutional neural network[J]. Journal of Zhejiang University: Engineering Science, 2017, 51 (7): 1381- 1389
[29]   TANGERMANN M, MÜLLER K R, AERTSEN A, et al Review of the BCI competition IV[J]. Frontiers Neuroscience, 2012, 6: 55
[30]   LI D L, WANG J H, XU J C, et al Densely feature fusion based on convolutional neural networks for motor imagery EEG classification[J]. IEEE Access, 2019, 7: 132720- 132730
doi: 10.1109/ACCESS.2019.2941867
[31]   DAI G, ZHOU J, HUANG J, et al HS-CNN: a CNN with hybrid convolutional scale for EEG motor imagery classification[J]. Journal of Neural Engineering, 2020, 17 (1): 016025
doi: 10.1088/1741-2552/ab405f
[32]   MAJIDOV I, WHANGBO T Efficient classification of motor imagery electroencephalography signal using deep learning methods[J]. Sensors, 2019, 19 (7): 1736- 1749
doi: 10.3390/s19071736
[33]   ZHANG R, QUN Z, DOU L, et al A novel hybrid deep learning scheme for four-class motor imagery classification[J]. Journal of Neural Engineering, 2019, 16 (6): 1- 11
[34]   MA X G, WANG D S, LIU D H, et al DWT and CNN based multi-class motor imagery electroencephalographic signal recognition[J]. Journal of Neural Engineering, 2020, 17 (1): 016073
doi: 10.1088/1741-2552/ab6f15
[35]   GAUR P, PACHORI R B, WANG H, et al A multi-class EEG-based BCI classification using multivariate empirical mode decomposition based filtering and Riemannian Geometry[J]. Expert Systems with Applications, 2018, 95: 201- 211
doi: 10.1016/j.eswa.2017.11.007
[36]   XU J, ZHENG H, WANG J, et al Recognition of EEG signal motor imagery intention based on deep multi-view feature learning[J]. Sensors, 2020, 20 (12): 3496
doi: 10.3390/s20123496
[37]   XU M, YAO J, ZHANG Z, et al Learning EEG topographical representation for classification via convolutional neural network[J]. Pattern Recognition, 2020, 105: 107390
doi: 10.1016/j.patcog.2020.107390
[38]   SAKHAVI S, GUAN C, YAN S Learning temporal information for brain-computer interface using convolutional neural networks[J]. IEEE Transaction on Neural Networks and Learning Systems, 2018, 29 (11): 5619- 5629
doi: 10.1109/TNNLS.2018.2789927
[39]   ZHAO X, ZHANG H, ZHU G, et al A multi-branch 3D convolutional neural network for EEG-based motor imagery classification[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2019, 27 (10): 2164- 2177
doi: 10.1109/TNSRE.2019.2938295
[40]   ANG K K, CHIN Z Y, WANG C, et al Filter bank common spatial pattern algorithm on BCI Competition IV datasets 2a and 2b[J]. Frontiers in Neuroscience, 2012, 6: 39
[1] Li HE,Shan-min PANG. Face reconstruction from voice based on age-supervised learning and face prior information[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(5): 1006-1016.
[2] Yun-hao WANG,Ming-hui SUN,Yi XIN,Bo-xuan ZHANG. Robot tactile recognition system based on piezoelectric film sensor[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(4): 702-710.
[3] Pei-zhi WEN,Jun-mou CHEN,Yan-nan XIAO,Ya-yuan WEN,Wen-ming HUANG. Underwater image enhancement algorithm based on GAN and multi-level wavelet CNN[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(2): 213-224.
[4] Song REN,Qian-wen ZHU,Xin-yue TU,Chao DENG,Xiao-shu WANG. Lining disease identification of highway tunnel based on deep learning[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(1): 92-99.
[5] Ying-jie ZHENG,Song-rong WU,Ruo-yu WEI,Zhen-wei TU,Jin LIAO,Dong LIU. Metro location point matching and false alarm elimination based on FCM algorithm of target image[J]. Journal of ZheJiang University (Engineering Science), 2021, 55(3): 586-593.
[6] Qiao-hong CHEN,YI CHEN,Wen-shu Li,Yu-bo JIA. Clothing image classification based on multi-scale SE-Xception[J]. Journal of ZheJiang University (Engineering Science), 2020, 54(9): 1727-1735.
[7] Deng-wen ZHOU,Jin-yue TIAN,Lu-yao MA,Xiu-xiu SUN. Lightweight image semantic segmentation based on multi-level feature cascaded network[J]. Journal of ZheJiang University (Engineering Science), 2020, 54(8): 1516-1524.
[8] Yan-nan ZHANG,Xiao-hong HUANG,Yan MA,Qun CONG. Method with recording text classification based on deep learning[J]. Journal of ZheJiang University (Engineering Science), 2020, 54(7): 1264-1271.
[9] Hai-jin WANG,Zong-yu YIN,Zhen-zheng KE,Ying-jie GUO,Hui-yue DONG. Wear monitoring of helical milling tool based on one-dimensional convolutional neural network[J]. Journal of ZheJiang University (Engineering Science), 2020, 54(5): 931-939.
[10] Ping YANG,Dan WANG,Zi-jian KAGN,Tong LI,Li-hua FU,Yue-ren YU. Prediction model of paroxysmal atrial fibrillation based on pattern recognition and ensemble CNN-LSTM[J]. Journal of ZheJiang University (Engineering Science), 2020, 54(5): 1039-1048.
[11] Lie-jun JIN,Jian-ming ZHAN,Jun-hua CHEN,Tao WANG. Drill pipe fault diagnosis method based on one-dimensional convolutional neural network[J]. Journal of ZheJiang University (Engineering Science), 2020, 54(3): 467-474.
[12] Xiao-feng FU,Li NIU,Zhuo-qun HU,Jian-jun LI,Qing WU. Deep micro-expression spotting network training based on concept of transition frame[J]. Journal of ZheJiang University (Engineering Science), 2020, 54(11): 2128-2137.
[13] Hong-guang LI,Ying GUO,Ping SUI,Zi-sen QI. Frequency hopping modulation recognition of convolutional neural network based on time-frequency characteristics[J]. Journal of ZheJiang University (Engineering Science), 2020, 54(10): 1945-1954.
[14] Gang YE,Yi-bo LI,Zhu-xi MA,Jie CHENG. End-to-end aluminum strip surface defects detection and recognition method based on ViBe[J]. Journal of ZheJiang University (Engineering Science), 2020, 54(10): 1906-1914.
[15] Zi-yu JIA,You-fang LIN,Hong-jun ZHANG,Jing WANG. Sleep stage classification model based ondeep convolutional neural network[J]. Journal of ZheJiang University (Engineering Science), 2020, 54(10): 1899-1905.