Please wait a minute...
浙江大学学报(工学版)
浙江大学学报(工学版)  2021, Vol. 55 Issue (11): 2054-2066    DOI: 10.3785/j.issn.1008-973X.2021.11.005
生物医学工程     
基于卷积神经网络的多类运动想象脑电信号识别
刘近贞1,2(),叶方方1,2,熊慧1,2,*()
1. 天津工业大学 控制科学与工程学院，天津 300387
2. 天津工业大学 电工电能新技术天津市重点实验室，天津 300387
Recognition of multi-class motor imagery EEG signals based on convolutional neural network
Jin-zhen LIU1,2(),Fang-fang YE1,2,Hui XIONG1,2,*()
1. School of Control Science and Engineering, TIANGONG University, Tianjin 300387, China
2. Tianjin Key Laboratory of Advanced Technology of Electrical Engineering and Energy, TIANGONG University, Tianjin 300387, China
 全文: PDF(1117 KB)   HTML
摘要：

针对传统多类运动想象（MI）脑电信号的识别方法须进行繁琐的预处理以及特征提取问题，提出基于深度学习的MI信号自动分类方法.在样本表示方面，提出将多通道脑电（EEG）信号转化为一维序列信号处理，在增加样本数量的同时又能够忽略与通道位置相关的空间信息的影响；根据输入信号的特点，采用多层一维卷积神经网络学习不同运动想象状态时脑电信号中的时频信息，自动完成特征提取和分类工作. 将所提出的方法在公共数据集上与多种方法进行比较，并完成对实际采集数据集的分类. 利用所提方法在不需要先验知识的条件下，对脑电信号进行端到端的学习. 结果表明该方法可以获得更高的多分类准确率以及降低个体差异对分类的影响. 所提出的方法有利于促进基于MI 的脑机接口系统的开发.
关键词： 运动想象;  脑电信号;  卷积神经网络;  个体差异;  脑机接口    
Abstract:

An automatic classification method of motor imagery (MI) signals based on deep learning was proposed, aiming at the traditional multi-class MI recognition method of electroencephalogram (EEG) signals that requires cumbersome preprocessing and feature extraction. In terms of sample representation, the multi-channel EEG signals were converted into a one-dimensional sequence signal for processing, which increased the number of samples while neglecting the influence of spatial information related to the channel position. And according to the characteristics of the input signal, the multi-layer one-dimensional convolutional neural network was used to learn time-frequency information in EEG signals under different motor imagery states, and automatically complete the feature extraction and classification. The proposed method was compared with a variety of methods on the public dataset and the classification of the actual collected data was completed. The proposed method was used to do end-to-end learning of EEG signals without prior knowledge. Experimental results show that the method can obtain higher multi-classification accuracy and reduce the impact of individual differences on classification. The proposed method is conducive to the development of MI-based brain-computer interface systems.
Key words: motor imagery    electroencephalogram signal    convolutional neural network    individual difference    brain-computer interface
收稿日期: 2020-12-28 出版日期: 2021-11-05
CLC:  TP 29  
基金资助: 国家自然科学基金资助项目（61871288）；天津市自然科学基金资助项目（18JCYBJC90400，18JCQNJC84000）
通讯作者: 熊慧     E-mail: liujinzhen@tiangong.edu.cn;xionghui@tiangong.edu.cn
作者简介: 刘近贞（1985—），女，博士，从事生物医学信息检测和处理研究. orcid.org/0000-0001-8367-4999. E-mail: liujinzhen@tiangong.edu.cn
服务　　
把本文推荐给朋友
加入引用管理器
E-mail Alert
作者相关文章　　
刘近贞
叶方方
熊慧

引用本文:

刘近贞,叶方方,熊慧. 基于卷积神经网络的多类运动想象脑电信号识别[J]. 浙江大学学报(工学版), 2021, 55(11): 2054-2066.

Jin-zhen LIU,Fang-fang YE,Hui XIONG. Recognition of multi-class motor imagery EEG signals based on convolutional neural network. Journal of ZheJiang University (Engineering Science), 2021, 55(11): 2054-2066.

链接本文:

https://www.zjujournals.com/eng/CN/10.3785/j.issn.1008-973X.2021.11.005    或    https://www.zjujournals.com/eng/CN/Y2021/V55/I11/2054

图 1  EEG数据的采集和处理流程
层数 类型 输出特征尺寸 核尺寸
0 输入 750×1 ?
1 卷积 750×32 9
2 池化 375×32 2
3 卷积 375×32 9
4 池化 187×32 2
5 卷积 187×32 9
6 池化 93×32 2
8 全连接 64 ?
9 全连接层 4 ?
表 1  CNN结构的详细信息
图 2  用于运动想象脑电信号分类的一维卷积神经网络模型
图 3  BCI Competition IV 2a数据集的实验范式
图 4  不同输入形式下CNN训练过程中的准确率曲线
对比方法 输入设计 分类器 网络特点
DFFN[30] CSP CNN 考虑相邻层和跨层特征之间的相关性，减少卷积运算过程中的信息丢失
HC-CNN[31] 相同标签样本进行分割后交换重组，然后按照频率带划分，同一频率带的不同片段进行交换重组 CNN 对不同频率带的脑电信号设计不同的卷积核尺度，利用一维卷积核在时间和空间维度进行卷积操作
PSO-CNN[32] 滤波后的信号输入到CSP滤波器，同时计算不同频率带的PSD，引入粒子群优化算法进行特征优化 CNN 只有1层卷积层，紧跟着2层全连接层
CNN-LSTM[33] FBCSP CNN+LSTM 混合网络提取多模态特征
DWT-CNN[34] 滑动窗口进行数据增强，计算最优频率带的PSD CNN 结构与VGGNet相似，3 × 3卷积核分解为4 × 1或者2 × 1
SS-MEMDBF[35] 多元经验模式分解(MEMD) 黎曼几何 ?
RBM-SVM[36] 分别提取时-频-空域特征，利用参数t分布随机邻居嵌入进行参数降维 SVM ?
ETRCNN[37] 脑电图地形表征能量 CNN 网络能够从结合了时频特征和脑功能连通性的输入特征中学习到多模态信息
C2CM[38] FBCSP CNN 分别在时间和空间维度进行卷积，但须针对不同的对象更改参数
3D-CNN[39] 根据电极的3D分布将脑电信号映射为2D图像表示 CNN 构造3路不同卷积核尺寸的卷积网络，最后将3路网络获取的特征融合
FBCSP[40] FBCSP SVM ?
表 2  对比文献使用方法和模型的介绍
被试者 A/%
本研究方法 DFFN[30] HC-CNN[31] PSO-CNN[32] CNN-LSTM[33] SS-MEMDBF[35] RBM-SVM[36] ETRCNN[37]
1 99.14 85.40 90.07 93.30 98.82 91.49 86.61 85.88
2 99.49 69.30 80.28 84.59 98.64 60.56 61.26 75.41
3 99.68 90.29 97.08 91.68 96.92 94.16 87.27 91.32
4 99.01 71.07 89.66 84.55 96.50 76.72 75.20 83.45
5 99.61 65.41 97.04 86.54 92.75 58.52 64.55 72.11
6 98.86 69.45 87.04 76.92 91.84 68.52 65.91 91.72
7 98.58 88.18 92.14 94.03 95.07 78.67 83.78 85.71
8 99.82 86.46 98.51 93.20 95.25 97.01 89.91 91.32
9 98.98 93.54 92.31 92.24 99.23 93.85 92.08 93.23
平均值 99.24 79.90 91.57 85.56 96.13 79.94 78.51 85.57
均方差 0.3998 10.25 5.41 5.46 2.486 14.13 11.29 7.08
表 3  所提方法与对比文献的分类准确率对比
被试者 Kappa值
所提方法 CNN-LSTM[33] RBM-SVM[36] ETRCNN[37] C2CM[38] 3D-CNN[39] FBCSP[40]
1 0.9885 0.8500 0.8214 0.8420 0.8750 0.6990 0.6800
2 0.9933 0.5400 0.4838 0.6630 0.6528 0.4590 0.4200
3 0.9957 0.8700 0.7696 0.8770 0.9028 0.7880 0.7500
4 0.9863 0.7800 0.6664 0.7610 0.6667 0.5940 0.4800
5 0.9948 0.7700 0.5024 0.5710 0.6250 0.6470 0.4000
6 0.9848 0.6600 0.5301 0.8910 0.4549 0.5380 0.2700
7 0.9810 0.9500 0.7837 0.8090 0.8959 0.6530 0.7700
8 0.9976 0.8300 0.8655 0.8900 0.8333 0.7020 0.7600
9 0.9864 0.9000 0.8942 0.9070 0.7951 0.7130 0.6100
平均值 0.9898 0.7944 0.7019 0.8012 0.7446 0.6437 0.5711
均方差 0.0053 0.1197 0.1518 0.1097 0.1446 0.0942 0.1737
表 4  所提方法与对比文献分类Kappa值对比
图 5  被试者1和7分类的混淆矩阵结果
想象类别 被试者2 被试者8
PR RE F-score PR RE F-score
0 1.00 1.00 1.00 1.00 1.00 1.00
1 1.00 1.00 1.00 1.00 1.00 1.00
2 1.00 0.99 1.00 1.00 1.00 1.00
3 1.00 1.00 1.00 1.00 1.00 1.00
表 5  被试者2和8分类的精度、召回率和F-score结果
运动想象类别 AUC
被试者9 被试者3
0 1.00 0.99
1 1.00 1.00
2 1.00 1.00
3 1.00 1.00
表 6  被试者3和9分类的AUC结果
图 6  实际运动想象实验范式
图 7  运动想象实验现场图
图 8  不同被试者在不同输入长度下分类的准确率
时间长度 $\overline {A}$/% MSE
1 s 83.29 1.7882
2 s 88.90 0.8894
3 s（无下采样） 89.87 1.6286
3 s（下采样） 89.10 1.6587
表 7  实际采集数据集的分类准确率统计表
1 GRAIMANN B, ALLISON B, PFURTSCHELLER G. Brain-computer interface: a gentle introduction [M]. The frontiers collection. Berlin, Heidelberg: Springer, 2009: 1-27.
2 ADAMS M, BEN-SALEM S, ISLAM Z, et al. Towards an SSVEP-BCI controlled smart home [C]// 2019 IEEE International Conference on Systems Man and Cybernetics (SMC). Bari: IEEE, 2019: 2737-2742.
3 CHEN L, WANG Z P, HE F, et al. An online hybrid brain-computer interface combining multiple physiological signals for webpage browse [C]// 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society. Milan: IEEE, 2015: 1152-1155.
4 WONG C M, TANG Q, WAN F, et al. A multi-channel SSVEP-based BCI for computer games with analogue control [C]// Proceedings of the 2015 IEEE International Conference on Computational Intelligence and Virtual Environments for Measurement Systems and Application (CIVEMSA). Shenzhen: IEEE, 2015: 1-6.
5 PFURTSCHELLER G, LOPES DA SILVA F H Event-related EEG/MEG synchronization and desynchronization: basic principles[J]. Clinical Neurophysiology, 1999, 110 (11): 1842- 1857
doi: 10.1016/S1388-2457(99)00141-8
6 巫嘉陵, 高忠科 脑机接口技术及其在神经科学中的应用[J]. 中国现代神经疾病杂志, 2021, 21 (1): 3- 8
WU Jia-ling, GAO Zhong-ke Brain-computer interface technology and its applications in neuroscience[J]. Chinese Journal of Modern Neurological Diseases, 2021, 21 (1): 3- 8
doi: 10.3969/j.issn.1672-6731.2021.01.002
7 BAIG M Z, ASLAML N, SHUM H P H Filtering techniques for channel selection in motor imagery EEG applications: a survey[J]. Artificial Intelligence Review, 2020, 53 (3): 1207- 1232
8 霍首君, 郝琰, 石慧宇, 等 基于深度卷积网络的运动想象脑电信号模式识别[J]. 计算机应用, 2020, 41 (4): 1042- 1048
HUO Shou-jun, HAO Yan, SHI Hui-yu, et al Pattern recognition of motor imagery EEG based on deep convolutional networks[J]. Computer Applications, 2020, 41 (4): 1042- 1048
9 CHATTERJEE R, BANDYOPADHYAY T, SANYAL D K, et al. Comparative analysis of feature extraction techniques in motor imagery EEG signal classification [C]// Proceedings of First International Conference on Smart System, Innovations and Computing. Smart Innovation, Systems and Technologies. Jaipur: Springer, 2018, 79: 73-83.
10 WANG Y J, GAO S K, GAO X R, et al. Common spatial pattern method for channel selection in motor imagery based brain-computer interface [C]// International Conference of the Engineering in Medicine and Biology Society. Shanghai: IEEE, 2005: 5392-5395.
11 ANG K K, CHINY C, ZHANG H H, et al. Filter bank common spatial pattern (FBCSP) in brain-computer interface [C]// Proceedings of the IEEE International Joint Conference on Neural Networks. Hong Kong: IEEE, 2008: 2391–2398.
12 施锦河, 沈继忠, 王攀 四类运动想象脑电信号特征提取与分类算法[J]. 浙江大学学报:工学版, 2012, 46 (2): 338- 344
SHI Jin-he, SHEN Ji-zhong, WANG Pan Feature extraction and classification of four-class motor imagery EEG data[J]. Journal of Zhejiang University: Engineering Science, 2012, 46 (2): 338- 344
doi: 10.3785/j.issn.1008-973X.2012.02.025
13 TYAGI A, NEHRA V. Time frequency analysis of non-stationary motor imagery EEG signal [C]// 2017 International Conference on Computing and Communication Technologies for Smart Nation (IC3TSN). Gurgaon: IEEE, 2017: 44-50.
14 LIU J, CHENG Y, ZHANG W. Deep learning EEG response representation for brain computer interface [C]// 34th Chinese Control Conference. Hangzhou: IEEE, 2015: 3518-3523.
15 KUMAR S, SHARMA A, MAMUN K, et al. A deep learning approach for motor imagery EEG signal classification [C]// 2016 3rd Asia-Pacific World Congress on Computer Science and Engineering (APWC on CSE). Nadi: IEEE, 2016: 34-39.
16 LEE H K, CHOI Y. A convolutional neural networks scheme for classification of motor imagery EEG based on wavelet time-frequency image [C]// 2018 International Conference on Information Networking (ICOIN). Chiang Mai: IEEE, 2018: 906-909.
17 ZHU X Y, LI P Y, LI C B, et al Separated channel convolutional neural network to realize the training free motor imagery BCI systems[J]. Biomedical Signal Processing and Control, 2019, 49: 396- 403
doi: 10.1016/j.bspc.2018.12.027
18 YANG T, PHUA K S, YU J H, et al. Image-based motor imagery EEG classification using convolutional neural network [C]// 2019 IEEE EMBS International Conference on Biomedical and Health Informatics (BHI). Chicago: IEEE, 2019: 1-4.
19 LI Y, ZHANG X R, LEI M Y, et al A channel-projection mixed-scale convolutional neural network for motor imagery EEG decoding[J]. IEEE Transaction on Neural Systems and Rehabilitation Engineering, 2019, 27 (6): 1170- 1180
doi: 10.1109/TNSRE.2019.2915621
20 WU H, NIU Y, LI F, et al A parallel multiscale filter bank convolutional neural networks for motor imagery EEG classification[J]. Frontiers in Neuroscience, 2019, 26 (13): 1275
21 ROY S, MCCREADIE K, PRASAD G. Can a single model deep learning approach enhance classification accuracy of an EEG-based brain-computer interface? [C]// 2019 IEEE International Conference on Systems, Man and Cybernetics (SMC). Bari: IEEE, 2019: 1317-1321.
22 TANG X B, ZHAO J C, FU W L, et al. A novel classification algorithm for MI-EEG based on deep learning [C]// 2019 IEEE 8th Joint International Information Technology and Artificial Intelligence Conference (ITAIC). Chongqing: IEEE, 2019: 606-611.
23 SHEN Y R, LU H T, JIA J. Classification of motor imagery EEG signal with deep learning models [C]// International Conference on Intelligent Science and Big Data Engineering. Dalian: Springer, 2017: 181-190.
24 LASHGARI E, LIANG D, MAOZ U Data augmentation for deep-learning-based electroencephalography[J]. Journal of Neuroscience Methods, 2020, 346: 108885
doi: 10.1016/j.jneumeth.2020.108885
25 BASHIVAN P, RISH I, YEASIN M, et al. Learning representations from EEG with deep recurrent-convolutional neural networks [EB/OL]. (2015-11-19) [2020-12-28]. https://arxiv.org/abs/1511.06448v3.
26 CHANG L, DENG X M, ZHOU M Q, et al Convolutional neural networks in image understanding[J]. Acta Automatica Sinica, 2016, 42 (9): 1300- 1312
27 DING Y, ROBINSON N, ZENG Q H, et al. TSception: a deep learning framework for emotion detection using EEG [EB/OL]. (2020-04-02) [2020-12-28]. https://arxiv.org/abs/2004.02965.
28 王卫星, 孙守迁, 李超, 等 基于卷积神经网络的脑电信号上肢运动意图识别[J]. 浙江大学学报:工学版, 2017, 51 (7): 1381- 1389
WANG Wei-xing, SUN Shou-qian, LI Chao, et al Recognition of upper limb motion intention of EEG signal based on convolutional neural network[J]. Journal of Zhejiang University: Engineering Science, 2017, 51 (7): 1381- 1389
29 TANGERMANN M, MÜLLER K R, AERTSEN A, et al Review of the BCI competition IV[J]. Frontiers Neuroscience, 2012, 6: 55
30 LI D L, WANG J H, XU J C, et al Densely feature fusion based on convolutional neural networks for motor imagery EEG classification[J]. IEEE Access, 2019, 7: 132720- 132730
doi: 10.1109/ACCESS.2019.2941867
31 DAI G, ZHOU J, HUANG J, et al HS-CNN: a CNN with hybrid convolutional scale for EEG motor imagery classification[J]. Journal of Neural Engineering, 2020, 17 (1): 016025
doi: 10.1088/1741-2552/ab405f
32 MAJIDOV I, WHANGBO T Efficient classification of motor imagery electroencephalography signal using deep learning methods[J]. Sensors, 2019, 19 (7): 1736- 1749
doi: 10.3390/s19071736
33 ZHANG R, QUN Z, DOU L, et al A novel hybrid deep learning scheme for four-class motor imagery classification[J]. Journal of Neural Engineering, 2019, 16 (6): 1- 11
34 MA X G, WANG D S, LIU D H, et al DWT and CNN based multi-class motor imagery electroencephalographic signal recognition[J]. Journal of Neural Engineering, 2020, 17 (1): 016073
doi: 10.1088/1741-2552/ab6f15
35 GAUR P, PACHORI R B, WANG H, et al A multi-class EEG-based BCI classification using multivariate empirical mode decomposition based filtering and Riemannian Geometry[J]. Expert Systems with Applications, 2018, 95: 201- 211
doi: 10.1016/j.eswa.2017.11.007
36 XU J, ZHENG H, WANG J, et al Recognition of EEG signal motor imagery intention based on deep multi-view feature learning[J]. Sensors, 2020, 20 (12): 3496
doi: 10.3390/s20123496
37 XU M, YAO J, ZHANG Z, et al Learning EEG topographical representation for classification via convolutional neural network[J]. Pattern Recognition, 2020, 105: 107390
doi: 10.1016/j.patcog.2020.107390
38 SAKHAVI S, GUAN C, YAN S Learning temporal information for brain-computer interface using convolutional neural networks[J]. IEEE Transaction on Neural Networks and Learning Systems, 2018, 29 (11): 5619- 5629
doi: 10.1109/TNNLS.2018.2789927
39 ZHAO X, ZHANG H, ZHU G, et al A multi-branch 3D convolutional neural network for EEG-based motor imagery classification[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2019, 27 (10): 2164- 2177
doi: 10.1109/TNSRE.2019.2938295
40 ANG K K, CHIN Z Y, WANG C, et al Filter bank common spatial pattern algorithm on BCI Competition IV datasets 2a and 2b[J]. Frontiers in Neuroscience, 2012, 6: 39
[1] 何立,庞善民. 结合年龄监督和人脸先验的语音-人脸图像重建[J]. 浙江大学学报(工学版), 2022, 56(5): 1006-1016.
[2] 王云灏,孙铭会,辛毅,张博宣. 基于压电薄膜传感器的机器人触觉识别系统[J]. 浙江大学学报(工学版), 2022, 56(4): 702-710.
[3] 温佩芝,陈君谋,肖雁南,温雅媛,黄文明. 基于生成式对抗网络和多级小波包卷积网络的水下图像增强算法[J]. 浙江大学学报(工学版), 2022, 56(2): 213-224.
[4] 任松,朱倩雯,涂歆玥,邓超,王小书. 基于深度学习的公路隧道衬砌病害识别方法[J]. 浙江大学学报(工学版), 2022, 56(1): 92-99.
[5] 郑英杰,吴松荣,韦若禹,涂振威,廖进,刘东. 基于目标图像FCM算法的地铁定位点匹配及误报排除方法[J]. 浙江大学学报(工学版), 2021, 55(3): 586-593.
[6] 陈巧红,陈翊,李文书,贾宇波. 多尺度SE-Xception服装图像分类[J]. 浙江大学学报(工学版), 2020, 54(9): 1727-1735.
[7] 周登文,田金月,马路遥,孙秀秀. 基于多级特征并联的轻量级图像语义分割[J]. 浙江大学学报(工学版), 2020, 54(8): 1516-1524.
[8] 张彦楠,黄小红,马严,丛群. 基于深度学习的录音文本分类方法[J]. 浙江大学学报(工学版), 2020, 54(7): 1264-1271.
[9] 闫旭,范晓亮,郑传潘,臧彧,王程,程明,陈龙彪. 基于图卷积神经网络的城市交通态势预测算法[J]. 浙江大学学报(工学版), 2020, 54(6): 1147-1155.
[10] 沈宗礼,余建波. 基于迁移学习与深度森林的晶圆图缺陷识别[J]. 浙江大学学报(工学版), 2020, 54(6): 1228-1239.
[11] 汪海晋,尹宗宇,柯臻铮,郭英杰,董辉跃. 基于一维卷积神经网络的螺旋铣刀具磨损监测[J]. 浙江大学学报(工学版), 2020, 54(5): 931-939.
[12] 杨萍,王丹,康子健,李童,付利华,余悦任. 基于模式识别和集成CNN-LSTM的阵发性房颤预测模型[J]. 浙江大学学报(工学版), 2020, 54(5): 1039-1048.
[13] 金列俊,詹建明,陈俊华,王涛. 基于一维卷积神经网络的钻杆故障诊断[J]. 浙江大学学报(工学版), 2020, 54(3): 467-474.
[14] 付晓峰,牛力,胡卓群,李建军,吴卿. 基于过渡帧概念训练的微表情检测深度网络[J]. 浙江大学学报(工学版), 2020, 54(11): 2128-2137.
[15] 贾子钰,林友芳,张宏钧,王晶. 基于深度卷积神经网络的睡眠分期模型[J]. 浙江大学学报(工学版), 2020, 54(10): 1899-1905.