Please wait a minute...
浙江大学学报(工学版)
Journal of ZheJiang University (Engineering Science)  2022, Vol. 56 Issue (11): 2145-2155    DOI: 10.3785/j.issn.1008-973X.2022.11.005
    
Optimization method of CNC milling parameters based on deep reinforcement learning
Qi-lin DENG1(),Juan LU2,Yong-hui CHEN1,Jian FENG1,Xiao-ping LIAO1,3,Jun-yan MA1,3,*()
1. College of Mechanical Engineering, Guangxi University, Nanning 530004, China
2. Department of Mechanical and Marine Engineering, Beibu Gulf University, Qinzhou 535011, China
3. Guangxi Key Laboratory of Manufacturing Systems and Advanced Manufacturing Technology, Guangxi University, Nanning 530004, China
Download: HTML     PDF(1928KB) HTML
Export: BibTeX | EndNote (RIS)      

Abstract  

A deep reinforcement learning-based optimization method for CNC milling machining parameters was proposed to improve the machine tool effectiveness and the machining efficiency in CNC machining, and the applicability of deep reinforcement learning to machining parameters optimization problems was explored. The combined cutting force and material removal rate were selected as the optimization objectives of effectiveness and efficiency. The optimization function of combined cutting force and milling parameters were constructed using genetic algorithm optimization back propagation neural network (GA-BPNN) and the optimization function of material removal rate was established using empirical formulas. The competing network architecture (Dueling DQN) algorithm was applied to obtain Pareto frontier for combined cutting force and material removal rate multi-objective optimization and the decision solution was selected from Pareto frontier by combining the superior-inferior solution distance method and the entropy value method. The effectiveness of the Dueling DQN algorithm for machining parameter optimization was verified based on milling tests on 45 steel. Compared with the empirically selected machining parameters, the machining solution obtained by Dueling DQN optimization resulted in 8.29% reduction of combined cutting force and 4.95% improvement of machining efficiency, which provided guidance for the multi-objective optimization method of machining parameters and the selection of machining parameters.

Key words： milling      processing parameter      back propagation neural network      deep reinforcement learning      multi-objective optimization     
Received: 04 December 2021      Published: 02 December 2022
CLC:  TH 16  
Fund:  国家自然科学基金资助项目(51665005，52165062)；广西自然科学基金资助项目（2020JJD160004，2019JJB160048，2018GXNSFAA138158）；广西高校中青年教师基础能力提升资助项目（2020KY10014）
Corresponding Authors: Jun-yan MA     E-mail: 602096993@qq.com;191159191@qq.com
Service
E-mail this article
Add to my bookshelf
Add to citation manager
E-mail Alert
RSS
Articles by authors
Qi-lin DENG
Juan LU
Yong-hui CHEN
Jian FENG
Xiao-ping LIAO
Jun-yan MA
Cite this article:

Qi-lin DENG,Juan LU,Yong-hui CHEN,Jian FENG,Xiao-ping LIAO,Jun-yan MA. Optimization method of CNC milling parameters based on deep reinforcement learning. Journal of ZheJiang University (Engineering Science), 2022, 56(11): 2145-2155.

URL:

https://www.zjujournals.com/eng/10.3785/j.issn.1008-973X.2022.11.005     OR     https://www.zjujournals.com/eng/Y2022/V56/I11/2145


基于深度强化学习的数控铣削加工参数优化方法

为了提高数控加工中的机床效能和加工效率，探究深度强化学习在加工参数优化问题中的适用性，提出一种基于深度强化学习的数控铣削加工参数优化方法. 选取切削力合力和材料除去率作为效能和效率的优化目标，利用遗传算法优化反向传播神经网络（GA-BPNN）构建切削力合力和铣削参数的优化函数，并采用经验公式建立材料除去率的优化函数. 应用竞争网络架构（Dueling DQN）算法获得切削力合力和材料除去率多目标优化的Pareto前沿，并结合优劣解距离法和熵值法从Pareto前沿中选择决策解. 基于45钢的铣削试验，验证了Dueling DQN算法用于加工参数优化的有效性，相比经验选取加工参数，通过Dueling DQN优化得到的加工方案使切削力合力降低了8.29%，加工效率提高了4.95%，为加工参数的多目标优化方法和加工参数的选择提供了指导.


关键词： 铣削加工,  加工参数,  反向传播神经网络,  深度强化学习,  多目标优化 
Fig.1 Framework of machining parameters (spindle speed、feed rate、cutting width、cutting depth) optimization
Fig.2 Milling test platform
水平 加工参数
n /( ${\rm{r}} \cdot {\rm{mi} }{ {\rm{n} }^{ - 1} }$) f /( ${{\rm{mm}}} \cdot {r^{ - 1} }$) ae /mm ap /mm
1 1 500 0.08 2.00 0.20
2 1 900 0.10 4.00 0.40
3 2 300 0.12 6.00 0.60
Tab.1 Experimental factors and their levels
序号 n/
( ${\rm{r}} \cdot {\rm{mi}}{{\rm{n}}^{ - 1} }$) 		f/
( ${{\rm{mm}}} \cdot {{\rm{r}}^{ - 1} }$) 		ae/
mm 		ap/
mm 		Fc/
N 		R/
( ${ {{\rm{mm}}} ^3} \cdot {\min ^{ - 1} }$)
1 1 500 0.08 2 0.2 17.241 48.0
2 1 500 0.08 4 0.4 33.117 192.0
3 1 500 0.08 6 0.6 44.120 432.0
4 1 500 0.10 2 0.6 44.246 180.0
5 1 500 0.10 4 0.2 23.873 120.0
6 1 500 0.10 6 0.4 33.256 360.0
7 1 500 0.12 2 0.4 35.638 144.0
8 1 500 0.12 4 0.6 53.547 432.0
9 1 500 0.12 6 0.2 25.398 216.0
10 1 900 0.08 2 0.6 38.787 182.4
11 1 900 0.08 4 0.2 21.276 121.6
12 1 900 0.08 6 0.4 27.223 364.8
13 1 900 0.10 2 0.4 29.856 152.0
14 1 900 0.10 4 0.6 49.820 456.0
15 1 900 0.10 6 0.2 19.837 228.0
16 1 900 0.12 2 0.2 22.726 91.2
17 1 900 0.12 4 0.4 37.849 364.8
18 1 900 0.12 6 0.6 45.732 820.8
19 2 300 0.08 2 0.4 42.285 147.2
20 2 300 0.08 4 0.6 65.958 441.6
21 2 300 0.08 6 0.2 32.286 220.8
22 2 300 0.10 2 0.2 26.537 92.00
23 2 300 0.10 4 0.4 52.899 368.0
24 2 300 0.10 6 0.6 70.342 828.0
25 2 300 0.12 2 0.6 62.847 331.2
26 2 300 0.12 4 0.2 44.824 220.8
27 2 300 0.12 6 0.4 50.243 662.4
Tab.2 Combined cutting force and material  removal  rates for 27 sets of Taguchi test datas
组数 n/( ${\rm{r}} \cdot {\rm{mi}}{{\rm{n}}^{ - 1} }$) f/( ${{\rm{mm}}} \cdot {{\rm{r}}^{ - 1} }$) ae/ ${{\rm{mm}}}$ ap / ${{\rm{mm}}}$ $ F_{\rm{c}}^\& $/N
1 1 500 0.12 5 0.60 56.809
2 1 800 0.10 3 0.60 61.264
3 2 000 0.08 3 0.20 27.541
4 1 600 0.10 5 0.20 22.172
5 2 000 0.09 4 0.20 31.305
6 1 600 0.08 3 0.21 15.546
7 1 400 0.10 4 0.21 14.013
8 1 800 0.08 3 0.21 18.663
Tab.3 Test set sample data
Fig.3 Comparison of predicted and measured values of combined cutting force
模型 MSE MAPE/% R2
GA-BPNN 12.417 8.854 0.932
SVR 33.903 16.129 0.884
GBRT 44.887 21.249 0.847
Tab.4 Predictors of three models
Fig.4 Dueling DQN process for optimizing four machining parameters
组数 n/
(r·min?1) 		f/
(mm·r?1) 		ae/
mm 		ap/
mm 		Fc/
N 		R/
(mm3·min?1) 		Ci
1 1 700.000 0.103 6.000 0.411 30.120 431.797 0.591
2 1 725.455 0.100 6.000 0.411 29.975 427.337 0.585
3 1 830.909 0.098 6.000 0.404 30.568 434.142 0.581
4 1 801.818 0.101 5.964 0.415 31.512 449.089 0.580
5 1 674.545 0.107 6.000 0.367 27.372 394.839 0.576
Tab.5 Process parameter combination decision results
序号 Fc&/N Fc*/N $ l_{{\rm{n}}} ^{} $/% $ l_{{\rm{n}}} ^\& $/%
1 28.033 — 6.929 —
2 29.194 — 3.074 —
3 28.667 30.120 4.824 4.802
4 28.753 — 4.538 —
5 28.721 — 4.645 —
Tab.6 Comparison of optimized value of combined cutting force with measured value
Fig.5 Pareto front solution results for each algorithm
方法 N MID RAS U/s
Dueling DQN 173 26.014 0.013 1 063
DDPG 92 39.161 0.025 1 052
DQN 123 29.589 0.018 1 065
NSGA-II 140 28.713 0.018 501
Tab.7 Comparison of the optimization performance of different algorithms for multi-objective optimization problems with milling machining parameters
方法 n/(r·min?1) f/(mm·r?1) ae/mm ap/mm
Dueling DQN 1 700.000 0.103 6.000 0.411
DDPG 1 808.154 0.116 5.972 0.349
DQN 1 809.091 0.098 6.000 0.404
NSGA-II 1 720.582 0.119 5.984 0.357
方法 Fc&/N Fc*/N R/(mm3·min?1)
Dueling DQN 29.194 30.120 431.797
DDPG 32.847 33.463 437.158
DQN 33.211 34.327 429.753
NSGA-II 33.117 33.728 437.403
Tab.8 Comparison of combined cutting forces and material removal rate results for each method optimization
方法 n/(r·min?1) f/(mm·r?1) ae/mm ap/mm
Dueling DQN 1 700 0.103 6 0.411
经验铣削参数 1 900 0.080 6 0.450
方法 Fc&/N R/(mm3·min?1) $ G_{\rm{c}}^{} $/% $ G_{\rm{R}}^{} $/%
Dueling DQN 29.194 431.797 8.29 4.95
经验铣削参数 31.614 410.400
Tab.9 Comparison of Dueling DQN optimization results with empirical results
[1]   SAHU N K, ANDHARE A B Multi-objective optimization for improving machinability of Ti-6Al-4V using RSM and advanced algorithms[J]. Journal of Computational Design and Engineering, 2019, 6 (1): 1- 12
doi: 10.1016/j.jcde.2018.04.004
[2]   SHIHAB S K, GATTMAH J, KADHIM H M Experimental investigation of surface integrity and multi-objective optimization of end milling for hybrid Al7075 matrix composites[J]. Silicon, 2020, 13 (5): 1403- 1419
[3]   XIE H B, WANG Z J Study of cutting forces using FE, ANOVA, and BPNN in elliptical vibration cutting of titanium alloy Ti-6Al-4V[J]. The International Journal of Advanced Manufacturing Technology, 2019, 105 (12): 5105- 5120
doi: 10.1007/s00170-019-04537-w
[4]   TIEN D H, DUC Q T, VAN T N, et al Online monitoring and multi-objective optimization of technological parameters in high-speed milling process[J]. The International Journal of Advanced Manufacturing Technology, 2021, 112 (9-10): 2461- 2483
doi: 10.1007/s00170-020-06444-x
[5]   李建斌, 武颖莹, 李鹏宇, 等 基于局部线性嵌入和支持向量机回归的TBM施工参数预测[J]. 浙江大学学报: 工学版, 2021, 55 (8): 1426- 1435
LI Jian-bin, WU Ying-ying, LI Peng-yu, et al TBM tunneling parameters prediction based on locally linear embedding and support vector regression[J]. Journal of Zhejiang University: Engineering Science, 2021, 55 (8): 1426- 1435
[6]   陈超逸, 鲁娟, 陈楷, 等 车削表面粗糙度解析模型与DDQN-SVR预测模型研究[J]. 机械工程学报, 2021, 57 (13): 262- 272
CHEN Chao-yi, LU Juan, CHEN Kai, et al Research on analytical model and DDQN-SVR prediction model of turning surface roughness[J]. Journal of Mechanical Engineering, 2021, 57 (13): 262- 272
doi: 10.3901/JME.2021.13.262
[7]   巩超光, 胡天亮, 叶瑛歆 基于数字孪生的铣削参数动态多目标优化策略[J]. 计算机集成制造系统, 2021, 27 (2): 478- 486
GONG Chao-guang, HU Tian-liang, YE Ying-xin Dynamic multi-objective optimization strategy of milling parameters based on digital twin[J]. Computer Integrated Manufacturing Systems, 2021, 27 (2): 478- 486
doi: 10.13196/j.cims.2021.02.015
[8]   CHENG Y N, YANG J L, QIN C, et al Tool design and cutting parameter optimization for side milling blisk[J]. The International Journal of Advanced Manufacturing Technology, 2019, 100 (9-12): 2495- 2508
doi: 10.1007/s00170-018-2846-4
[9]   GHOSH T, WANG Y, MARTINSEN K, et al A surrogate-assisted optimization approach for multi-response end milling of aluminum alloy AA3105[J]. The International Journal of Advanced Manufacturing Technology, 2020, 111 (9-10): 2419- 2439
doi: 10.1007/s00170-020-06209-6
[10]   HE K, TANG R, JIN M Pareto fronts of machining parameters for trade-off among energy consumption, cutting force and processing time[J]. International Journal of Production Economics, 2017, 185: 113- 127
doi: 10.1016/j.ijpe.2016.12.012
[11]   OSORIOPINZON J C, ABOLGHASEM S, MARANON A, et al Cutting parameter optimization of Al-6063-O using numerical simulations and particle swarm optimization[J]. The International Journal of Advanced Manufacturing Technology, 2020, 111 (9-10): 2507- 2532
doi: 10.1007/s00170-020-06200-1
[12]   Van H P Application of singularity vibration for minimum energy consumption in high-speed milling[J]. International Journal of Modern Physics B, 2021, 35: 2140008
doi: 10.1142/S0217979221400087
[13]   LI B, TIAN X T, ZHANG M Modeling and multi-objective optimization of cutting parameters in the high-speed milling using RSM and improved TLBO algorithm[J]. The International Journal of Advanced Manufacturing Technology, 2020, 111 (7-8): 2323- 2335
doi: 10.1007/s00170-020-06284-9
[14]   翁剑, 庄可佳, 浦栋麟, 等 基于机器学习和多目标算法的钛合金插铣优化[J]. 中国机械工程, 2021, 32 (7): 771- 777
WENG Jian, ZHUANG Ke-jia, PU Dong-lin, et al Plunge milling of tianium alloys based on machine learning and multi-objective optimization[J]. China Mechanical Engineering, 2021, 32 (7): 771- 777
doi: 10.3969/j.issn.1004-132X.2021.07.002
[15]   RUST J Structural estimation of markov decision processes[J]. Handbook of Econometrics, 1994, 3081- 3143
[16]   LI K W, ZHANG T, WANG R Deep reinforcement learning for multi-objective optimization[J]. IEEE Transactions on Cybernetics, 2021, 51 (6): 3103- 3114
doi: 10.1109/TCYB.2020.2977661
[17]   施群, 吕雷, 谢家骏 可变环境下仿人机器人智能姿态控制[J]. 机械工程学报, 2020, 56 (3): 64- 72
SHI Qun, LV Lei, XIE Jia-jun Intelligent posture control of humanoid robot in variable environment[J]. Journal of Mechanical Engineering, 2020, 56 (3): 64- 72
doi: 10.3901/JME.2020.03.064
[18]   LAN S, PANDA R, ZHU Q, et al. FFNet: video fast-forwarding via reinforcement learning [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. USA: Salt Lake City, 2018: 6771-6780.
[19]   MNIH V, KAVUKCUOGLU K, SILVER D, et al Human level control through deep reinforcement learning[J]. Nature, 2015, 518 (7540): 529- 533
doi: 10.1038/nature14236
[20]   WANG Z Y, SCHAUL T, HESSEL M, et al. Dueling network architectures for deep reinforcement learning [C]// Proceedings of the 33rd International Conference on Machine Learning. USA: New York, 2016, 46: 1995-2003.
[21]   SUN G L, AYEPAHMENSAH D, XU R, et al End-to-end CNN-based dueling deep Q-Network for autonomous cell activation in Cloud-RANs[J]. Journal of Network and Computer Applications, 2020, 169: 102757
doi: 10.1016/j.jnca.2020.102757
[22]   BAN T W An autonomous transmission scheme using dueling DQN for D2D communication networks[J]. IEEE Transactions on Vehicular Technology, 2020, 69 (45): 16348- 16352
[23]   ZHANG X W, EHMANN K F, YU T B, et al Cutting forces in micro-end-milling processes[J]. International Journal of Machine Tools and Manufacture, 2016, 107: 21- 40
[24]   HAN F J, LI L, CAI W, et al Parameters optimization considering the trade-off between cutting power and R based on linear decreasing particle swarm algorithm in milling[J]. Journal of Cleaner Production, 2020, 262: 121388
doi: 10.1016/j.jclepro.2020.121388
[25]   MOREIRA L C, LI W D, LU X, et al Energy-efficient machining process analysis and optimization based on BS EN24T alloy steel as case studies[J]. Robotics and Computer-Integrated Manufacturing, 2019, 58: 1- 12
doi: 10.1016/j.rcim.2019.01.011
[26]   SOEPANGKAT B, NORCAHYO R, PRAMUJATI B, et al Multi-objective optimization in face milling process with cryogenic cooling using grey fuzzy analysis and BPNN-GA methods[J]. Engineering Computations, 2020, 36 (5): 1542- 1565
[27]   MNIH V, KAVUKCUOGLU K, SILVER D, et al Playing atari with deep reinforcement learning[J]. Computer Science, 2013, 1- 9
[28]   XU L H, HUANG C Z, LI C W, et al Estimation of tool wear and optimization of cutting parameters based on novel ANFIS-PSO method toward intelligent machining[J]. Journal of Intelligent Manufacturing, 2020, 32 (1): 77- 90
[29]   SUN G L, AYEPAH M D, BUDKEVICH A, et al Autonomous cell activation for energy saving in cloud-RANs based on dueling deep q-network[J]. Knowledge-Based Systems, 2020, 192: 105347
[30]   KUMAR R, BILGA P S, SINGH S Multi-objective optimization using different methods of assigning weights to energy consumption responses, surface roughness and material removal rate during rough turning operation[J]. Journal of Cleaner Production, 2017, 164: 45- 57
doi: 10.1016/j.jclepro.2017.06.077
[31]   SEN B, MIA M, MANDAL U K, et al Multi-objective optimization for MQL-assisted end milling operation: an intelligent hybrid strategy combining GEP and NTOPSIS[J]. Neural Computing and Applications, 2019, 31 (12): 8693- 8717
doi: 10.1007/s00521-019-04450-z
[32]   BEHNAMIAN J, ZANDIEH M, GHOMI S A multi-phase covering pareto-optimal front method to multi-objective parallel machine scheduling[J]. International Journal of Production Research, 2010, 48 (17-18): 4949- 4976
[33]   SUN G L, XIONG K, BOATENG G O, et al Resource slicing and customization in RAN with dueling deep Q-network[J]. Journal of Network and Computer Applications, 2020, 157 (3): 102573
[1] Xia HUA,Xin-qing WANG,Ting RUI,Fa-ming SHAO,Dong WANG. Vision-driven end-to-end maneuvering object tracking of UAV[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(7): 1464-1472.
[2] Zhi-min LIU,Bao-Lin YE,Yao-dong ZHU,Qing YAO,Wei-min WU. Traffic signal control method based on deep reinforcement learning[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(6): 1249-1256.
[3] Wan-liang WANG,Ya-wen JIN,Jia-cheng CHEN,Guo-qing LI,Ming-zhi HU,Jian-hang DONG. Multi-objective particle swarm optimization algorithm with multi-role and multi-strategy[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(3): 531-541.
[4] Jun-heng XU,Xiao-jun YANG,Bing LI. Design of wing mechanism with variable camber based on cross-spring flexural pivots[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(3): 444-451, 509.
[5] Jun-jie CHEN,Hong-jun LI,Zhang-hua CAO. Performance-aware resource allocation algorithm for core network control plane[J]. Journal of ZheJiang University (Engineering Science), 2021, 55(9): 1782-1787.
[6] Yi-fan MA,Fan-yu ZHAO,Xin WANG,Zhong-he JIN. Satellite earth observation task planning method based on improved pointer networks[J]. Journal of ZheJiang University (Engineering Science), 2021, 55(2): 395-401.
[7] Xiao-zhu LI,Wei-qing WANG. Bi-level robust game optimal scheduling of regional comprehensive energy system[J]. Journal of ZheJiang University (Engineering Science), 2021, 55(1): 177-188.
[8] Kai-jun LOU,Feng YU,Tang-dai XIA,Jian MA. Stability analysis of diaphragm wall retained structure in clay[J]. Journal of ZheJiang University (Engineering Science), 2020, 54(9): 1697-1705.
[9] Hai-jin WANG,Zong-yu YIN,Zhen-zheng KE,Ying-jie GUO,Hui-yue DONG. Wear monitoring of helical milling tool based on one-dimensional convolutional neural network[J]. Journal of ZheJiang University (Engineering Science), 2020, 54(5): 931-939.
[10] Xiang-fei MENG,Ren-guang WANG,Yuan-li XU. Torque distribution strategy of pure electric driving mode for dual planetary vehicle[J]. Journal of ZheJiang University (Engineering Science), 2020, 54(11): 2214-2223.
[11] Hua HUANG,Wen-qiang DENG,Yuan LI,Run-lan GUO. Mass matching design of machine tool parts based on spatial dynamics optimization[J]. Journal of ZheJiang University (Engineering Science), 2020, 54(10): 2009-2017.
[12] Jia-shuang FAN,Sui-huai YU,Jian-jie CHU,Hui WANG,Chen CHEN,Wen-zhe CUN,Tian CHEN,Jia-yan GUO. Optimal decision-making method of design scheme in cloud service mode[J]. Journal of ZheJiang University (Engineering Science), 2020, 54(1): 143-151.
[13] Zhi-lin SUN,Zhen-yu CHEN,Zheng-zhi DENG,Yu-yu DAI,Dan XU. Relation between sediment mass flux and volume runoff under natural condition of Lancang River[J]. Journal of ZheJiang University (Engineering Science), 2019, 53(5): 932-939.
[14] ZHANG De-sheng, LIU An, CHEN Jian, ZHAO Rui-jie, SHI Wei-dong. Multi-objective optimization of horizontal axis tidal current turbine using particle swarm optimization[J]. Journal of ZheJiang University (Engineering Science), 2018, 52(12): 2349-2355.
[15] YU Yang, XIA Chun-he, HU Xiao-yun. Defense scheme generation method using mixed path attack graph[J]. Journal of ZheJiang University (Engineering Science), 2017, 51(9): 1745-1759.