Please wait a minute...
浙江大学学报(工学版)
浙江大学学报(工学版)  2022, Vol. 56 Issue (11): 2145-2155    DOI: 10.3785/j.issn.1008-973X.2022.11.005
机械与能源工程     
基于深度强化学习的数控铣削加工参数优化方法
邓齐林1(),鲁娟2,陈勇辉1,冯健1,廖小平1,3,马俊燕1,3,*()
1. 广西大学 机械工程学院，广西 南宁 530004
2. 北部湾大学 机械与船舶海洋工程学院，广西 钦州 535011
3. 广西大学 制造系统与先进制造技术重点实验室，广西 南宁 530004
Optimization method of CNC milling parameters based on deep reinforcement learning
Qi-lin DENG1(),Juan LU2,Yong-hui CHEN1,Jian FENG1,Xiao-ping LIAO1,3,Jun-yan MA1,3,*()
1. College of Mechanical Engineering, Guangxi University, Nanning 530004, China
2. Department of Mechanical and Marine Engineering, Beibu Gulf University, Qinzhou 535011, China
3. Guangxi Key Laboratory of Manufacturing Systems and Advanced Manufacturing Technology, Guangxi University, Nanning 530004, China
 全文: PDF(1928 KB)   HTML
摘要：

为了提高数控加工中的机床效能和加工效率，探究深度强化学习在加工参数优化问题中的适用性，提出一种基于深度强化学习的数控铣削加工参数优化方法. 选取切削力合力和材料除去率作为效能和效率的优化目标，利用遗传算法优化反向传播神经网络（GA-BPNN）构建切削力合力和铣削参数的优化函数，并采用经验公式建立材料除去率的优化函数. 应用竞争网络架构（Dueling DQN）算法获得切削力合力和材料除去率多目标优化的Pareto前沿，并结合优劣解距离法和熵值法从Pareto前沿中选择决策解. 基于45钢的铣削试验，验证了Dueling DQN算法用于加工参数优化的有效性，相比经验选取加工参数，通过Dueling DQN优化得到的加工方案使切削力合力降低了8.29%，加工效率提高了4.95%，为加工参数的多目标优化方法和加工参数的选择提供了指导.
关键词： 铣削加工;  加工参数;  反向传播神经网络;  深度强化学习;  多目标优化    
Abstract:

A deep reinforcement learning-based optimization method for CNC milling machining parameters was proposed to improve the machine tool effectiveness and the machining efficiency in CNC machining, and the applicability of deep reinforcement learning to machining parameters optimization problems was explored. The combined cutting force and material removal rate were selected as the optimization objectives of effectiveness and efficiency. The optimization function of combined cutting force and milling parameters were constructed using genetic algorithm optimization back propagation neural network (GA-BPNN) and the optimization function of material removal rate was established using empirical formulas. The competing network architecture (Dueling DQN) algorithm was applied to obtain Pareto frontier for combined cutting force and material removal rate multi-objective optimization and the decision solution was selected from Pareto frontier by combining the superior-inferior solution distance method and the entropy value method. The effectiveness of the Dueling DQN algorithm for machining parameter optimization was verified based on milling tests on 45 steel. Compared with the empirically selected machining parameters, the machining solution obtained by Dueling DQN optimization resulted in 8.29% reduction of combined cutting force and 4.95% improvement of machining efficiency, which provided guidance for the multi-objective optimization method of machining parameters and the selection of machining parameters.
Key words: milling    processing parameter    back propagation neural network    deep reinforcement learning    multi-objective optimization
收稿日期: 2021-12-04 出版日期: 2022-12-02
CLC:  TH 16  
基金资助: 国家自然科学基金资助项目(51665005，52165062)；广西自然科学基金资助项目（2020JJD160004，2019JJB160048，2018GXNSFAA138158）；广西高校中青年教师基础能力提升资助项目（2020KY10014）
通讯作者: 马俊燕     E-mail: 602096993@qq.com;191159191@qq.com
作者简介: 邓齐林（1996—），男，硕士生，从事智能制造研究. orcid.org/0000-0003-4255-4418. E-mail： 602096993@qq.com
服务　　
把本文推荐给朋友
加入引用管理器
E-mail Alert
作者相关文章　　
邓齐林
鲁娟
陈勇辉
冯健
廖小平
马俊燕

引用本文:

邓齐林,鲁娟,陈勇辉,冯健,廖小平,马俊燕. 基于深度强化学习的数控铣削加工参数优化方法[J]. 浙江大学学报(工学版), 2022, 56(11): 2145-2155.

Qi-lin DENG,Juan LU,Yong-hui CHEN,Jian FENG,Xiao-ping LIAO,Jun-yan MA. Optimization method of CNC milling parameters based on deep reinforcement learning. Journal of ZheJiang University (Engineering Science), 2022, 56(11): 2145-2155.

链接本文:

https://www.zjujournals.com/eng/CN/10.3785/j.issn.1008-973X.2022.11.005    或    https://www.zjujournals.com/eng/CN/Y2022/V56/I11/2145

图 1  工艺参数（主轴转速、进给量、切削宽度和深度）优化求解框架
图 2  铣削试验平台
水平 加工参数
n /( ${\rm{r}} \cdot {\rm{mi} }{ {\rm{n} }^{ - 1} }$) f /( ${{\rm{mm}}} \cdot {r^{ - 1} }$) ae /mm ap /mm
1 1 500 0.08 2.00 0.20
2 1 900 0.10 4.00 0.40
3 2 300 0.12 6.00 0.60
表 1  试验因素及其水平
序号 n/
( ${\rm{r}} \cdot {\rm{mi}}{{\rm{n}}^{ - 1} }$) 		f/
( ${{\rm{mm}}} \cdot {{\rm{r}}^{ - 1} }$) 		ae/
mm 		ap/
mm 		Fc/
N 		R/
( ${ {{\rm{mm}}} ^3} \cdot {\min ^{ - 1} }$)
1 1 500 0.08 2 0.2 17.241 48.0
2 1 500 0.08 4 0.4 33.117 192.0
3 1 500 0.08 6 0.6 44.120 432.0
4 1 500 0.10 2 0.6 44.246 180.0
5 1 500 0.10 4 0.2 23.873 120.0
6 1 500 0.10 6 0.4 33.256 360.0
7 1 500 0.12 2 0.4 35.638 144.0
8 1 500 0.12 4 0.6 53.547 432.0
9 1 500 0.12 6 0.2 25.398 216.0
10 1 900 0.08 2 0.6 38.787 182.4
11 1 900 0.08 4 0.2 21.276 121.6
12 1 900 0.08 6 0.4 27.223 364.8
13 1 900 0.10 2 0.4 29.856 152.0
14 1 900 0.10 4 0.6 49.820 456.0
15 1 900 0.10 6 0.2 19.837 228.0
16 1 900 0.12 2 0.2 22.726 91.2
17 1 900 0.12 4 0.4 37.849 364.8
18 1 900 0.12 6 0.6 45.732 820.8
19 2 300 0.08 2 0.4 42.285 147.2
20 2 300 0.08 4 0.6 65.958 441.6
21 2 300 0.08 6 0.2 32.286 220.8
22 2 300 0.10 2 0.2 26.537 92.00
23 2 300 0.10 4 0.4 52.899 368.0
24 2 300 0.10 6 0.6 70.342 828.0
25 2 300 0.12 2 0.6 62.847 331.2
26 2 300 0.12 4 0.2 44.824 220.8
27 2 300 0.12 6 0.4 50.243 662.4
表 2  27组Taguchi试验数据的切削力合力和材料去除率
组数 n/( ${\rm{r}} \cdot {\rm{mi}}{{\rm{n}}^{ - 1} }$) f/( ${{\rm{mm}}} \cdot {{\rm{r}}^{ - 1} }$) ae/ ${{\rm{mm}}}$ ap / ${{\rm{mm}}}$ $ F_{\rm{c}}^\& $/N
1 1 500 0.12 5 0.60 56.809
2 1 800 0.10 3 0.60 61.264
3 2 000 0.08 3 0.20 27.541
4 1 600 0.10 5 0.20 22.172
5 2 000 0.09 4 0.20 31.305
6 1 600 0.08 3 0.21 15.546
7 1 400 0.10 4 0.21 14.013
8 1 800 0.08 3 0.21 18.663
表 3  测试集样本数据
图 3  切削力合力预测值与测量值比较
模型 MSE MAPE/% R2
GA-BPNN 12.417 8.854 0.932
SVR 33.903 16.129 0.884
GBRT 44.887 21.249 0.847
表 4  三个模型的预测指标
图 4  Dueling DQN优化4个工艺参数的过程
组数 n/
(r·min?1) 		f/
(mm·r?1) 		ae/
mm 		ap/
mm 		Fc/
N 		R/
(mm3·min?1) 		Ci
1 1 700.000 0.103 6.000 0.411 30.120 431.797 0.591
2 1 725.455 0.100 6.000 0.411 29.975 427.337 0.585
3 1 830.909 0.098 6.000 0.404 30.568 434.142 0.581
4 1 801.818 0.101 5.964 0.415 31.512 449.089 0.580
5 1 674.545 0.107 6.000 0.367 27.372 394.839 0.576
表 5  工艺参数组合决策结果
序号 Fc&/N Fc*/N $ l_{{\rm{n}}} ^{} $/% $ l_{{\rm{n}}} ^\& $/%
1 28.033 — 6.929 —
2 29.194 — 3.074 —
3 28.667 30.120 4.824 4.802
4 28.753 — 4.538 —
5 28.721 — 4.645 —
表 6  切削力合力优化值与测量值的比较结果
图 5  各算法的帕累托前沿解结果
方法 N MID RAS U/s
Dueling DQN 173 26.014 0.013 1 063
DDPG 92 39.161 0.025 1 052
DQN 123 29.589 0.018 1 065
NSGA-II 140 28.713 0.018 501
表 7  不同算法在铣削工艺参数多目标优化问题上的优化性能比较
方法 n/(r·min?1) f/(mm·r?1) ae/mm ap/mm
Dueling DQN 1 700.000 0.103 6.000 0.411
DDPG 1 808.154 0.116 5.972 0.349
DQN 1 809.091 0.098 6.000 0.404
NSGA-II 1 720.582 0.119 5.984 0.357
方法 Fc&/N Fc*/N R/(mm3·min?1)
Dueling DQN 29.194 30.120 431.797
DDPG 32.847 33.463 437.158
DQN 33.211 34.327 429.753
NSGA-II 33.117 33.728 437.403
表 8  各方法优化下的切削力合力和材料去除率结果对比
方法 n/(r·min?1) f/(mm·r?1) ae/mm ap/mm
Dueling DQN 1 700 0.103 6 0.411
经验铣削参数 1 900 0.080 6 0.450
方法 Fc&/N R/(mm3·min?1) $ G_{\rm{c}}^{} $/% $ G_{\rm{R}}^{} $/%
Dueling DQN 29.194 431.797 8.29 4.95
经验铣削参数 31.614 410.400
表 9  Dueling DQN优化结果与经验结果对比
1 SAHU N K, ANDHARE A B Multi-objective optimization for improving machinability of Ti-6Al-4V using RSM and advanced algorithms[J]. Journal of Computational Design and Engineering, 2019, 6 (1): 1- 12
doi: 10.1016/j.jcde.2018.04.004
2 SHIHAB S K, GATTMAH J, KADHIM H M Experimental investigation of surface integrity and multi-objective optimization of end milling for hybrid Al7075 matrix composites[J]. Silicon, 2020, 13 (5): 1403- 1419
3 XIE H B, WANG Z J Study of cutting forces using FE, ANOVA, and BPNN in elliptical vibration cutting of titanium alloy Ti-6Al-4V[J]. The International Journal of Advanced Manufacturing Technology, 2019, 105 (12): 5105- 5120
doi: 10.1007/s00170-019-04537-w
4 TIEN D H, DUC Q T, VAN T N, et al Online monitoring and multi-objective optimization of technological parameters in high-speed milling process[J]. The International Journal of Advanced Manufacturing Technology, 2021, 112 (9-10): 2461- 2483
doi: 10.1007/s00170-020-06444-x
5 李建斌, 武颖莹, 李鹏宇, 等 基于局部线性嵌入和支持向量机回归的TBM施工参数预测[J]. 浙江大学学报: 工学版, 2021, 55 (8): 1426- 1435
LI Jian-bin, WU Ying-ying, LI Peng-yu, et al TBM tunneling parameters prediction based on locally linear embedding and support vector regression[J]. Journal of Zhejiang University: Engineering Science, 2021, 55 (8): 1426- 1435
6 陈超逸, 鲁娟, 陈楷, 等 车削表面粗糙度解析模型与DDQN-SVR预测模型研究[J]. 机械工程学报, 2021, 57 (13): 262- 272
CHEN Chao-yi, LU Juan, CHEN Kai, et al Research on analytical model and DDQN-SVR prediction model of turning surface roughness[J]. Journal of Mechanical Engineering, 2021, 57 (13): 262- 272
doi: 10.3901/JME.2021.13.262
7 巩超光, 胡天亮, 叶瑛歆 基于数字孪生的铣削参数动态多目标优化策略[J]. 计算机集成制造系统, 2021, 27 (2): 478- 486
GONG Chao-guang, HU Tian-liang, YE Ying-xin Dynamic multi-objective optimization strategy of milling parameters based on digital twin[J]. Computer Integrated Manufacturing Systems, 2021, 27 (2): 478- 486
doi: 10.13196/j.cims.2021.02.015
8 CHENG Y N, YANG J L, QIN C, et al Tool design and cutting parameter optimization for side milling blisk[J]. The International Journal of Advanced Manufacturing Technology, 2019, 100 (9-12): 2495- 2508
doi: 10.1007/s00170-018-2846-4
9 GHOSH T, WANG Y, MARTINSEN K, et al A surrogate-assisted optimization approach for multi-response end milling of aluminum alloy AA3105[J]. The International Journal of Advanced Manufacturing Technology, 2020, 111 (9-10): 2419- 2439
doi: 10.1007/s00170-020-06209-6
10 HE K, TANG R, JIN M Pareto fronts of machining parameters for trade-off among energy consumption, cutting force and processing time[J]. International Journal of Production Economics, 2017, 185: 113- 127
doi: 10.1016/j.ijpe.2016.12.012
11 OSORIOPINZON J C, ABOLGHASEM S, MARANON A, et al Cutting parameter optimization of Al-6063-O using numerical simulations and particle swarm optimization[J]. The International Journal of Advanced Manufacturing Technology, 2020, 111 (9-10): 2507- 2532
doi: 10.1007/s00170-020-06200-1
12 Van H P Application of singularity vibration for minimum energy consumption in high-speed milling[J]. International Journal of Modern Physics B, 2021, 35: 2140008
doi: 10.1142/S0217979221400087
13 LI B, TIAN X T, ZHANG M Modeling and multi-objective optimization of cutting parameters in the high-speed milling using RSM and improved TLBO algorithm[J]. The International Journal of Advanced Manufacturing Technology, 2020, 111 (7-8): 2323- 2335
doi: 10.1007/s00170-020-06284-9
14 翁剑, 庄可佳, 浦栋麟, 等 基于机器学习和多目标算法的钛合金插铣优化[J]. 中国机械工程, 2021, 32 (7): 771- 777
WENG Jian, ZHUANG Ke-jia, PU Dong-lin, et al Plunge milling of tianium alloys based on machine learning and multi-objective optimization[J]. China Mechanical Engineering, 2021, 32 (7): 771- 777
doi: 10.3969/j.issn.1004-132X.2021.07.002
15 RUST J Structural estimation of markov decision processes[J]. Handbook of Econometrics, 1994, 3081- 3143
16 LI K W, ZHANG T, WANG R Deep reinforcement learning for multi-objective optimization[J]. IEEE Transactions on Cybernetics, 2021, 51 (6): 3103- 3114
doi: 10.1109/TCYB.2020.2977661
17 施群, 吕雷, 谢家骏 可变环境下仿人机器人智能姿态控制[J]. 机械工程学报, 2020, 56 (3): 64- 72
SHI Qun, LV Lei, XIE Jia-jun Intelligent posture control of humanoid robot in variable environment[J]. Journal of Mechanical Engineering, 2020, 56 (3): 64- 72
doi: 10.3901/JME.2020.03.064
18 LAN S, PANDA R, ZHU Q, et al. FFNet: video fast-forwarding via reinforcement learning [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. USA: Salt Lake City, 2018: 6771-6780.
19 MNIH V, KAVUKCUOGLU K, SILVER D, et al Human level control through deep reinforcement learning[J]. Nature, 2015, 518 (7540): 529- 533
doi: 10.1038/nature14236
20 WANG Z Y, SCHAUL T, HESSEL M, et al. Dueling network architectures for deep reinforcement learning [C]// Proceedings of the 33rd International Conference on Machine Learning. USA: New York, 2016, 46: 1995-2003.
21 SUN G L, AYEPAHMENSAH D, XU R, et al End-to-end CNN-based dueling deep Q-Network for autonomous cell activation in Cloud-RANs[J]. Journal of Network and Computer Applications, 2020, 169: 102757
doi: 10.1016/j.jnca.2020.102757
22 BAN T W An autonomous transmission scheme using dueling DQN for D2D communication networks[J]. IEEE Transactions on Vehicular Technology, 2020, 69 (45): 16348- 16352
23 ZHANG X W, EHMANN K F, YU T B, et al Cutting forces in micro-end-milling processes[J]. International Journal of Machine Tools and Manufacture, 2016, 107: 21- 40
24 HAN F J, LI L, CAI W, et al Parameters optimization considering the trade-off between cutting power and R based on linear decreasing particle swarm algorithm in milling[J]. Journal of Cleaner Production, 2020, 262: 121388
doi: 10.1016/j.jclepro.2020.121388
25 MOREIRA L C, LI W D, LU X, et al Energy-efficient machining process analysis and optimization based on BS EN24T alloy steel as case studies[J]. Robotics and Computer-Integrated Manufacturing, 2019, 58: 1- 12
doi: 10.1016/j.rcim.2019.01.011
26 SOEPANGKAT B, NORCAHYO R, PRAMUJATI B, et al Multi-objective optimization in face milling process with cryogenic cooling using grey fuzzy analysis and BPNN-GA methods[J]. Engineering Computations, 2020, 36 (5): 1542- 1565
27 MNIH V, KAVUKCUOGLU K, SILVER D, et al Playing atari with deep reinforcement learning[J]. Computer Science, 2013, 1- 9
28 XU L H, HUANG C Z, LI C W, et al Estimation of tool wear and optimization of cutting parameters based on novel ANFIS-PSO method toward intelligent machining[J]. Journal of Intelligent Manufacturing, 2020, 32 (1): 77- 90
29 SUN G L, AYEPAH M D, BUDKEVICH A, et al Autonomous cell activation for energy saving in cloud-RANs based on dueling deep q-network[J]. Knowledge-Based Systems, 2020, 192: 105347
30 KUMAR R, BILGA P S, SINGH S Multi-objective optimization using different methods of assigning weights to energy consumption responses, surface roughness and material removal rate during rough turning operation[J]. Journal of Cleaner Production, 2017, 164: 45- 57
doi: 10.1016/j.jclepro.2017.06.077
31 SEN B, MIA M, MANDAL U K, et al Multi-objective optimization for MQL-assisted end milling operation: an intelligent hybrid strategy combining GEP and NTOPSIS[J]. Neural Computing and Applications, 2019, 31 (12): 8693- 8717
doi: 10.1007/s00521-019-04450-z
32 BEHNAMIAN J, ZANDIEH M, GHOMI S A multi-phase covering pareto-optimal front method to multi-objective parallel machine scheduling[J]. International Journal of Production Research, 2010, 48 (17-18): 4949- 4976
33 SUN G L, XIONG K, BOATENG G O, et al Resource slicing and customization in RAN with dueling deep Q-network[J]. Journal of Network and Computer Applications, 2020, 157 (3): 102573
[1] 华夏,王新晴,芮挺,邵发明,王东. 视觉感知的无人机端到端目标跟踪控制技术[J]. 浙江大学学报(工学版), 2022, 56(7): 1464-1472.
[2] 刘智敏,叶宝林,朱耀东,姚青,吴维敏. 基于深度强化学习的交通信号控制方法[J]. 浙江大学学报(工学版), 2022, 56(6): 1249-1256.
[3] 王万良,金雅文,陈嘉诚,李国庆,胡明志,董建杭. 多角色多策略多目标粒子群优化算法[J]. 浙江大学学报(工学版), 2022, 56(3): 531-541.
[4] 徐钧恒,杨晓钧,李兵. 基于交叉簧片式铰链的变弯度机翼机构设计[J]. 浙江大学学报(工学版), 2022, 56(3): 444-451, 509.
[5] 陈俊杰,李洪均,曹张华. 性能感知的核心网控制面资源分配算法[J]. 浙江大学学报(工学版), 2021, 55(9): 1782-1787.
[6] 马一凡,赵凡宇,王鑫,金仲和. 基于改进指针网络的卫星对地观测任务规划方法[J]. 浙江大学学报(工学版), 2021, 55(2): 395-401.
[7] 李笑竹,王维庆. 区域综合能源系统两阶段鲁棒博弈优化调度[J]. 浙江大学学报(工学版), 2021, 55(1): 177-188.
[8] 楼恺俊,俞峰,夏唐代,马健. 黏土中地下连续墙支护结构的稳定性分析[J]. 浙江大学学报(工学版), 2020, 54(9): 1697-1705.
[9] 黄华,邓文强,李源,郭润兰. 基于空间动力学优化的机床结构件质量匹配设计[J]. 浙江大学学报(工学版), 2020, 54(10): 2009-2017.
[10] 孙志林,陈震宇,邓争志,戴俣俣,许丹. 澜沧江自然条件下输沙质量通量与体积径流量的关系[J]. 浙江大学学报(工学版), 2019, 53(5): 932-939.
[11] 童水光,赵航,刘会琴,童哲铭,余跃,唐宁,吴伟杰,李进富,从飞云,张昊,王寅华,郝国帅. 中开多级离心泵效率优化计算方法[J]. 浙江大学学报(工学版), 2019, 53(5): 988-996.
[12] 毕晓君, 王朝. 基于超平面投影的高维多目标进化算法[J]. 浙江大学学报(工学版), 2018, 52(7): 1284-1293.
[13] 张德胜, 刘安, 陈健, 赵睿杰, 施卫东. 采用粒子群算法的水平轴潮流能水轮机翼型多目标优化[J]. 浙江大学学报(工学版), 2018, 52(12): 2349-2355.
[14] 余洋, 夏春和, 胡潇云. 采用混和路径攻击图的防御方案生成方法[J]. 浙江大学学报(工学版), 2017, 51(9): 1745-1759.
[15] 李建丽, 丁丁, 李涛. 基于二次聚类的多目标混合云任务调度算法[J]. 浙江大学学报(工学版), 2017, 51(6): 1233-1241.