|
|
|
| Combination pruning method based on reinforcement learning and 3σ criterion |
Shao-ming XU( ),Yu LI*(),Qing-long YUAN |
| School of Information Science and Engineering, East China University of Science and Technology, Shanghai 200237, China |
|
|
|
Abstract In order to resolve the problem that deep neural network with complex structure and redundant parameters could not be deployed to the resource constrained embedded system, an efficient combination pruning method based on reinforcement learning and 3σ criterion was proposed, which was inspired by the effect of sparsity rate on performance. Firstly, an optimal global sparsity rate was determined according to the influence of sparsity rate on accuracy, which could achieve a good balance between sparsity rate and accuracy. Secondly, under the guidance of optimal global sparsity rate, the reinforcement learning method was used to search the optimal pruning rate of each convolutional layer automatically, and the unimportant weights were cut off on the basis of the pruning rate. Then, the weight pruning threshold of each fully connected layer was determined by 3σ criterion, and for each fully connected layer, the weight which below the threshold would be pruned. Finally, the accuracy of model recognition was restored by retraining. Experimental results showed that the proposed pruning method could compress the parameters of VGG16, ResNet56 and ResNet50 network by 83.33%, 70.1% and 80.9% respectively, and the model’s recognition accuracy could be reduced by 1.55%, 1.98% and 1.86% respectively.
|
|
Received: 12 March 2022
Published: 31 March 2023
|
|
|
|
Corresponding Authors:
Yu LI
E-mail: X18912726309@163.com;liyu@ecust.edu.cn
|
基于强化学习和3σ准则的组合剪枝方法
针对结构复杂、参数冗余的深度神经网络无法部署到资源受限的嵌入式系统的问题，受稀疏率对性能影响的启示，提出基于强化学习和3σ准则的组合剪枝方法. 根据稀疏率对准确率的影响，确定最佳全局稀疏率，使稀疏率和精度达到较好平衡. 在最佳全局稀疏率的指导下，利用强化学习方法自动搜索每层卷积层的最佳剪枝率，根据剪枝率剪去不重要的权重. 通过3σ准则确定全连接层每层的权重剪枝阈值，对全连接层进行权重剪枝. 通过再训练来恢复模型识别的精度. 实验结果表明，所提剪枝方法可以将网络VGG16、ResNet56和ResNet50的参数，分别压缩83.33%、70.1%和80.9%，模型的识别准确率分别降低1.55%、1.98%和1.86%.
关键词：
深度神经网络,
模型压缩,
稀疏率,
强化学习,
组合剪枝
|
|
| [21] |
CHOLLET F. Xception: deep learning with depthwise separable convolutions [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017: 1251-1258.
|
|
|
| [22] |
HOWARD A G, ZHU M, CHEN B, et al. MobileNets: efficient convolutional neural networks for mobile vision applications [EB/OL]. (2017-04-17) [2022-12-10]. https: arxiv.org/pdf/1704.04861. pdf.
|
|
|
| [23] |
HE Y H, LIN J, LIU Z J, et al. AMC: AutoML for model compression and acceleration on mobile devices [C]// Proceedings of the European Conference on Computer Vision. [S. l.]: Springer, 2018: 815-832.
|
|
|
| [24] |
ASHOK A, RHINEHART N, BEAINY F, et al. N2N learning: network to network compression via policy gradient reinforcement learning [EB/OL]. (2017-12-17)[2022-12-10]. https://arxiv.org/pdf/1709.06030v1.pdf.
|
|
|
| [25] |
LIU Z C, MU H Y, ZHANG X Y, et al. MetaPruning: meta learning for automatic neural network channel pruning [C]// Proceedings of the IEEE/CVF International Conference on Computer Vision. Seoul: IEEE, 2019: 3296-3305.
|
|
|
| [26] |
LILLICRAP T P, HUNT J J, PRITZEL A, et al. Continuous control with deep reinforcement learning [EB/OL]. (2019-07-15)[2022-12-12]. https://arxiv.org/pdf/1509.02971.pdf.
|
|
|
| [27] |
LIN M B, JI R R, ZHANG Y X. Channel pruning via automatic structure search [EB/OL]. (2020-06-29)[2022-12-16]. https://arxiv.org/pdf/2001.08565.pdf.
|
|
|
| [28] |
HUANG Z Z, SHAO W Q, WANG X J. Rethinking the pruning criteria for convolutional neural network [C]// Conference and Workshop on Neural Information Processing Systems. Montreal: MIT Press, 2021, 34: 16305-16318.
|
|
|
| [29] |
盛骤, 谢式千, 潘承毅. 概率论与数理统计[M]. 北京高等教育出版社, 2008: 112-114.
|
|
|
| [30] |
LIN S H, JI R R, YAN C Q, et al. Towards optimal structured CNN pruning via generative adversarial learning [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2019: 2790-2799.
|
|
|
| [31] |
LI H, KADAV A, DURDANOVIC I, et al. Pruning filters for efficient convnets [EB/OL]. (2017-03-10)[2022-12-20]. https://arxiv.org/pdf/1608.08710.pdf.
|
|
|
| [32] |
LUO J H, WU J X. An entropy-based pruning method for CNN compression [EB/OL]. (2017-06-19)[2022-12-20]. https: arxiv.org/pdf/1706.05791.pdf.
|
|
|
| [33] |
HU H Y, PENG R, TAI Y W, et al. Network trimming: a data-driven neuron pruning approach towards efficient deep architectures [EB/OL]. (2016-07-12)[2022-12-20]. https://arxiv.org/pdf/1607.03250.pdf.
|
|
|
| [1] |
KRIZHEVSKY A, SUTSKEVER I, HINTON G E. ImageNet classification with deep convolutional neural networks [C]// Proceedings of the 25th International Conference on Neural Information Processing Systems. Lake Tahoe: Curran Associates Inc, 2012: 1097-1105.
|
|
|
| [2] |
ZHANG L B, HUANG S L, LIU W. Intra-class part swapping for fine-grained image classification [C]// Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. Waikoloa: IEEE, 2021: 3209-3218.
|
|
|
| [3] |
REN S K, HE K M, GIRSHICK R, et al. Object detection networks on convolutional feature maps [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2017, 39(7): 1476-1481.
|
|
|
| [4] |
REDMON J, DIVVALA S, GIRSHICK R, et al. You only look once: unified, real-time object detection [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 779-788.
|
|
|
| [5] |
BLAIVAS M, ARNTFIELD R, WHITE M Creation and testing of a deep learning algorithm to automatically identify and label vessels, nerves, tendons, and bones on cross-sectional point-of-care ultrasound scans for peripheral intravenous catheter placement by novices[J]. Journal of Ultrasound in Medicine, 2020, 39 (9): 1721- 1727
doi: 10.1002/jum.15270
|
|
|
| [6] |
LONG J, SHELHAMER E, DARRELL T. Fully convolutional networks for semantic segmentation [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Boston: IEEE, 2015: 3431-3440.
|
|
|
| [7] |
TANZI L, PIAZZOLLA P, PORPIGLIA F, et al Real-time deep learning semantic segmentation during intra-operative surgery for 3D augmented reality assistance[J]. International Journal of Computer Assisted Radiology and Surgery, 2021, 16: 1435- 1445
doi: 10.1007/s11548-021-02432-y
|
|
|
| [8] |
张哲晗, 方薇, 杜丽丽, 等 基于编码-解码卷积神经网络的遥感图像语义分割[J]. 光学学报, 2020, 40 (3): 46- 55
ZHANG Zhe-han, FANG Wei, DU Li-li, et al Semantic segmentation of remote sensing image based on coding-decoding convolutional neural network[J]. Acta Optica Sinica, 2020, 40 (3): 46- 55
|
|
|
| [9] |
吕永发. 基于深度学习的手机表面缺陷检测算法[D]. 郑州: 郑州大学, 2020.
LV Yong-fa. Mobile phone surface detect detection algorithm based on deep learning [D]. Zhengzhou: Zhengzhou University, 2020.
|
|
|
| [10] |
JIANG Y, WANG W, ZHAO C. A machine vision-based realtime anomaly detection method for industrial products using deep learning [C]// 2019 Chinese Automation Congress. Hangzhou: IEEE, 2019: 4842-4847.
|
|
|
| [11] |
GONG R H, LIU X L, JIANG S H, et al. Differentiable soft quantization: bridging full-precision and low-bit neural networks [C]// Proceedings of the IEEE/CVF International Conference on Computer Vision. Seoul: IEEE, 2019: 4852-4861.
|
|
|
| [12] |
COURBARIAUX M, HUBARA I, SOUDRY D, et al. Binarized neural networks: training neural networks with weights and activations constrained to +1 or −1 [EB/OL]. (2016-03-17)[2022-12-10]. https:arxiv.org/pdf/1602.02830.pdf.
|
|
|
| [13] |
LI F F, LIU B, WANG X X. Ternary weight networks [EB/OL]. (2022-11-20)[2022-12-10]. https://arxiv.org/pdf/1605.04711.pdf.
|
|
|
| [14] |
KALMAN D A singularly valuable decomposition: the SVD of a matrix[J]. The College Mathematics Journal, 1996, 27 (1): 2- 23
doi: 10.1080/07468342.1996.11973744
|
|
|
| [15] |
KIM Y D, PARK E, YOO S, et al. Compression of deep convolutional neural networks for fast and low power mobile applications [EB/OL]. (2016-12-24) [2022-12-10]. https://arxiv.org/pdf/1511.06530.pdf.
|
|
|
| [16] |
LATHAUWER L D Decompositions of a higher-order tensor in block terms[J]. SIAM Journal on Matrix Analysis and Applications, 2008, 30 (3): 1022- 1032
doi: 10.1137/060661685
|
|
|
| [17] |
ZAGORUYKO S, KOMODAKIS N. Paying more attention to attention: improving the performance of convolutional neural networks via attention transfer [EB/OL]. (2017-12-12)[2022-12-10]. https://arxiv.org/pdf/1612.03928.pdf.
|
|
|
| [18] |
ZHANG L F, SONG J B, GAO A, et al. Be your own teacher: improve the performance of convolutional neural networks via self distillation [C]// Proceedings of the IEEE/CVF International Conference on Computer Vision. Seoul: IEEE, 2019: 3713-3722.
|
|
|
| [19] |
IANDOLA F N, HAN S, MOSKEWICZ M W, et al. SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and<0.5MB model size [EB/OL]. (2016-11-04)[2022-12-10]. https:arxiv.org/pdf/1602.07360.pdf.
|
|
|
| [20] |
ZHANG X Y, ZHOU X Y, LIN M X, et al. ShuffleNet: an extremely efficient convolutional neural network for mobile devices [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 6848-6856.
|
|
|
|
Viewed |
|
|
|
Full text
|
|
|
|
|
Abstract
|
|
|
|
|
Cited |
|
|
|
|
| |
Shared |
|
|
|
|
| |
Discussed |
|
|
|
|
