Please wait a minute...
浙江大学学报(工学版)
浙江大学学报(工学版)  2019, Vol. 53 Issue (10): 1994-2002    DOI: 10.3785/j.issn.1008-973X.2019.10.017
自动化技术、计算机技术     
基于滤波器裁剪的卷积神经网络加速算法
李浩(),赵文杰*(),韩波
浙江大学 航空航天学院，浙江 杭州 310027
Convolutional neural network acceleration algorithm based on filters pruning
Hao LI(),Wen-jie ZHAO*(),Bo HAN
College of Aeronautics and Astronautics, Zhejiang University, Hangzhou 310027, China
 全文: PDF(2004 KB)   HTML
摘要：

针对卷积神经网络（CNN）模型的压缩和加速问题，提出基于滤波器裁剪的新型卷积神经网络模型加速算法. 通过计算卷积层中滤波器的标准差值衡量该滤波器的重要程度，裁剪对神经网络准确率影响较小的滤波器及对应的特征图，可以有效地降低计算成本. 与裁剪权重不同，该算法不会导致网络稀疏连接，不需要应用特殊的稀疏矩阵计算库. 基于CIFAR-10数据集的实验结果表明，该滤波器裁剪算法能够对VGG-16和ResNet-110模型加速30%以上，通过微调继承的预训练参数可以使结果接近或达到原始模型的精度.
关键词： 深度学习;  卷积神经网络（CNN）;  模型压缩;  滤波器;  特征图    
Abstract:

A new model acceleration algorithm of convolutional neural network (CNN) was proposed based on filters pruning in order to promote the compression and acceleration of the CNN model. The computational cost could be effectively reduced by calculating the standard deviation of filters in the convolutional layer to measure its importance and pruning filters with less influence on the accuracy of the neural network and its corresponding feature map. The algorithm did not cause the network to be sparsely connected unlike the method of pruning weight value, so there was no need of the support of special sparse convolution libraries. The experimental results based on the CIFAR-10 dataset show that the filters pruning algorithm can accelerate the VGG-16 and ResNet-110 models by more than 30%. Results can be close to or reach the accuracy of the original model by fine-tuning the inherited pre-training parameters.
Key words: deep learning    convolutional neural network (CNN)    model compress    filter    feature map
收稿日期: 2018-12-05 出版日期: 2019-09-30
CLC:  TP 183  
通讯作者: 赵文杰     E-mail: lhmzl2012@163.com;zhaowenjie8@zju.edu.cn
作者简介: 李浩（1993—），男，硕士生，从事计算机视觉研究. orcid.org/0000-0001-6731-1856. E-mail： lhmzl2012@163.com
服务　　
把本文推荐给朋友
加入引用管理器
E-mail Alert
作者相关文章　　
李浩
赵文杰
韩波

引用本文:

李浩,赵文杰,韩波. 基于滤波器裁剪的卷积神经网络加速算法[J]. 浙江大学学报(工学版), 2019, 53(10): 1994-2002.

Hao LI,Wen-jie ZHAO,Bo HAN. Convolutional neural network acceleration algorithm based on filters pruning. Journal of ZheJiang University (Engineering Science), 2019, 53(10): 1994-2002.

链接本文:

http://www.zjujournals.com/eng/CN/10.3785/j.issn.1008-973X.2019.10.017    或    http://www.zjujournals.com/eng/CN/Y2019/V53/I10/1994

图 1  裁剪滤波器致使其对应特征图和下一层滤波器的移除
图 2  ResNet模型的裁剪
图 3  利用传统方法提取图像特征
图 4  ImageNet数据集上预训练模型VGG-16的卷积层可视化特征图
图 5  CIFAR-10数据集上的预训练模型VGG-16的裁剪
卷积层 Fb/106 Fa/106 ΔF/%
conv 1_1 1.8 0.9 50
conv 1_2 38 19 50
conv 2_1 19 14.25 25
conv 2_2 38 28.5 25
conv 3_1 19 14.25 25
conv 3_2 38 28.5 25
conv 3_3 38 38 0
conv 4_1 19 9.5 50
conv 4_2 38 9.5 75
conv 4_3 38 9.5 75
conv 5_1 9.4 2.35 75
conv 5_2 9.4 2.35 75
conv 5_3 9.4 2.35 75
FC6 0.26 0.13 50
FC7 5.1×10?3 5.1×10?3 0
总量 315 179 43.2
表 1  裁剪CIFAR-10数据集上的预训练模型VGG-16
模型 y/% Fw/106 ΔF/% M/106 ΔM/%
VGG-16 93.33 313 ? 15 ?
VGG-16-A 93.47 179 43.2 5.4 64
ResNet-56 93.11 127 ? 0.85 ?
ResNet-
56-A 		93.07 112 11.8 0.77 9.4
ResNet-
56-B 		93.03 92.3 27.3 0.72 15.3
ResNet-110 93.62 256 ? 1.73 ?
ResNet-110-A 93.67 213 16.8 1.68 2.9
ResNet-110-B 93.48 155 39.5 1.14 34.1
ResNet-34 73.29 3 640 ? 21.6 ?
ResNet-
34-A 		72.61 3 080 15.5 19.9 7.9
ResNet-
34-B 		72.31 2 760 24.2 19.3 10.6
表 2  裁剪、重训练多个模型及其结果
图 6  ResNet-56/110模型各卷积层的裁剪敏感度
图 7  ResNet-34模型各卷积层的裁剪敏感度
%
裁剪算法 微调 随机初值训练 多次迭代训练
m=20 m=40 m=60 m=20 m=40 m=60 m=20 m=40 m=60
标准差 72.37 64.85 53.36 72.41 64.93 53.31 72.65 65.16 54.45
APoZ 71.14 62.82 52.63 71.11 63.12 52.67 71.39 63.31 53.44
L1范数 72.28 64.77 53.35 72.33 64.76 53.39 72.47 64.81 53.56
Random 71.45 63.34 51.92 71.49 63.40 52.01 71.67 63.82 52.76
Mean Activation 71.62 63.42 51.39 71.61 63.59 51.41 71.83 63.71 52.13
表 3  不同裁剪方法的准确率对比
1 KRIZHEVSKY A, SUTSKEVER I, HINTON G E. ImageNet classification with deep convolutional neural networks [C] // International Conference on Neural Information Processing Systems. Lake Tahoe: Curran Associates Inc., 2012.
2 GRAVES A, SCHMIDHUBER J Framewise phoneme classification with bidirectional LSTM and other neural network architectures[J]. Neural Netw, 2005, 18 (5): 602- 610
3 SZEGEDY C, VANHOUCKE V, IOFFE S, et al. Rethinking the inception architecture for computer vision [C] // International Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 2818-2826.
4 DENIL M, SHAKIBI B, DINH L, et al. Predicting parameters in deep learning [C] // Advances in Neural Information Processing Systems. Lake Tahoe: MIT, 2013: 2148-2156.
5 SRINIVAS S, BABU R V. Data-free parameter pruning for deep neural networks [EB/OL]. [2018-09-06]. http://arxiv.org/abs/1507.06149.
6 HAN S, POOL J, TRAN J, et al. Learning both weights and connections for efficient neural network [C] // Advances in Neural Information Processing Systems. Montreal: MIT, 2015: 1135-1143.
7 MARIET Z, SRA S. Diversity networks: neural network compression using determinantal point processes [EB/OL]. [2018-05-13]. http://arxiv.org/abs/1511.05077.
8 HAN S, MAO H, DALLY W J. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding [EB/OL]. [2018-08-09]. http://arxiv.org/abs/1510.00149.
9 SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition [EB/OL]. [2018-07-22]. http://arxiv.org/abs/1409.1556.
10 IANDOLA F N, HAN S, MOSKEWICZ M W, et al. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and < 0.5 mb model size [EB/OL]. [2018-07-14]. http://arxiv.org/abs/1602.07360.
11 HAN S, LIU X, MAO H, et al EIE: efficient inference engine on compressed deep neural network[J]. ACM Sigarch Computer Architecture News, 2016, 44 (3): 243- 254
doi: 10.1145/3007787.3001163
12 MATHIEU M, HENAFF M, LECUN Y. Fast training of convolutional networks through FFTs [EB/OL]. [2018-09-03]. http://arxiv.org/abs/1312.5851.
13 RASTEGARI M, ORDONEZ V, REDMON J, et al. XNOR-Net: ImageNet classification using binary convolutional neural networks [C] // European Conference on Computer Vision. Cham: Springer, 2016: 525-542.
14 WEN W, WU C, WANG Y, et al. Learning structured sparsity in deep neural networks [C] // Advances in Neural Information Processing Systems. Barcelona: MIT, 2016: 2074-2082.
15 LI H, KADAV A, DURDANOVIC I, et al. Pruning filters for efficient convents [EB/OL]. [2018-09-11]. http://arxiv.org/abs/1608.08710.
16 MITTAL D, BHARDWAJ S, KHAPRA M M, et al. Recovering from random pruning: on the plasticity of deep convolutional neural networks [EB/OL]. [2018-09-12]. http://arxiv.org/abs/1801.10447.
17 ZHU M, GUPTA S. To prune, or not to prune: exploring the efficacy of pruning for model compression [EB/OL]. [2018-06-23]. http://arxiv.org/abs/1710.01878.
18 HE K, ZHANG X, REN S, et al. Deep residual learning for image recognition [C] // Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 770-778.
19 ZAGORUYKO S. 92.45% on CIFAR-10 in Torch[EB/OL].[2018-07-30]. http://torch.ch/blog/2015/07/30/cifar.html.
20 IOFFE S, SZEGEDY C. Batch normalization: accelerating deep network training by reducing internal covariate shift [EB/OL]. [2018-07-16]. http://arxiv.org/abs/1502.03167.
21 HU H, PENG R, TAI Y W, et al. Network trimming: a data-driven neuron pruning approach towards efficient deep architectures [EB/OL]. [2018-07-19]. http://arxiv.org/abs/1607.03250.
[1] 许佳辉,王敬昌,陈岭,吴勇. 基于图神经网络的地表水水质预测模型[J]. 浙江大学学报(工学版), 2021, 55(4): 601-607.
[2] 王虹力,郭斌,刘思聪,刘佳琪,仵允港,於志文. 边端融合的终端情境自适应深度感知模型[J]. 浙江大学学报(工学版), 2021, 55(4): 626-638.
[3] 张腾,蒋鑫龙,陈益强,陈前,米涛免,陈彪. 基于腕部姿态的帕金森病用药后开-关期检测[J]. 浙江大学学报(工学版), 2021, 55(4): 639-647.
[4] 徐利锋,黄海帆,丁维龙,范玉雷. 基于改进DenseNet的水果小目标检测[J]. 浙江大学学报(工学版), 2021, 55(2): 377-385.
[5] 许豪灿,李基拓,陆国栋. 由LeNet-5从单张着装图像重建三维人体[J]. 浙江大学学报(工学版), 2021, 55(1): 153-161.
[6] 黄毅鹏,胡冀苏,钱旭升,周志勇,赵文露,马麒,沈钧康,戴亚康. SE-Mask-RCNN：多参数MRI前列腺癌分割方法[J]. 浙江大学学报(工学版), 2021, 55(1): 203-212.
[7] 陈巧红,陈翊,李文书,贾宇波. 多尺度SE-Xception服装图像分类[J]. 浙江大学学报(工学版), 2020, 54(9): 1727-1735.
[8] 郑浦,白宏阳,李伟,郭宏伟. 复杂背景下的小目标检测算法[J]. 浙江大学学报(工学版), 2020, 54(9): 1777-1784.
[9] 周登文,田金月,马路遥,孙秀秀. 基于多级特征并联的轻量级图像语义分割[J]. 浙江大学学报(工学版), 2020, 54(8): 1516-1524.
[10] 张彦楠,黄小红,马严,丛群. 基于深度学习的录音文本分类方法[J]. 浙江大学学报(工学版), 2020, 54(7): 1264-1271.
[11] 明涛,王丹,郭继昌,李锵. 基于多尺度通道重校准的乳腺癌病理图像分类[J]. 浙江大学学报(工学版), 2020, 54(7): 1289-1297.
[12] 闫旭,范晓亮,郑传潘,臧彧,王程,程明,陈龙彪. 基于图卷积神经网络的城市交通态势预测算法[J]. 浙江大学学报(工学版), 2020, 54(6): 1147-1155.
[13] 杨萍,王丹,康子健,李童,付利华,余悦任. 基于模式识别和集成CNN-LSTM的阵发性房颤预测模型[J]. 浙江大学学报(工学版), 2020, 54(5): 1039-1048.
[14] 汪周飞,袁伟娜. 基于深度学习的多载波系统信道估计与检测[J]. 浙江大学学报(工学版), 2020, 54(4): 732-738.
[15] 杨冰,莫文博,姚金良. 融合局部特征与深度学习的三维掌纹识别[J]. 浙江大学学报(工学版), 2020, 54(3): 540-545.