Please wait a minute...
浙江大学学报(工学版)
浙江大学学报(工学版)  2023, Vol. 57 Issue (3): 466-476    DOI: 10.3785/j.issn.1008-973X.2023.03.004
计算机与控制工程     
基于深度学习三维成型的钢板表面缺陷检测
兰欢1(),余建波1,2,*()
1. 同济大学 机械与能源工程学院，上海 201804
2. 上海市大型构件智能制造机器人技术协同创新中心, 上海 201620
Steel surface defect detection based on deep learning 3D reconstruction
Huan LAN1(),Jian-bo YU1,2,*()
1. School of Mechanical Engineering, Tongji University, Shanghai 201804, China
2. Shanghai Collaborative Innovation Center of Intelligent Manufacturing Robot Technology for Large Components, Shanghai 201620, China
 全文: PDF(5141 KB)   HTML
摘要：

为了解决二维检测方法难以检测带有深度信息的缺陷问题，提出全新的三维重建网络. 提出基于多尺度特征增强的级联式三维重建网络(MFE-CasMVSNet)，并与点云数据处理技术结合，用于钢板表面缺陷检测. 为了提高三维重建的精度，提出位置导向的特征增强模块(PFEM)和多尺度特征自适应融合模块(MFAFM)，对特征进行有效提取并减少信息丢失. 提出基于曲率稀疏化的密度聚类方法(CS-DBSCAN)，用于精确识别不同部位的缺陷. 引入三维检测框，实现对缺陷的定位与检测可视化. 实验结果表明，相较于图像几何的重建方法，MFE-CasMVSNet能够更加精确、快速地实现钢板表面的三维重建. 相较于二维检测，三维缺陷检测能够精确获取缺陷的三维形状信息，实现对钢板表面缺陷的多维度检测.
关键词： 表面缺陷检测;  深度学习;  三维重建;  点云分割;  缺陷定位    
Abstract:

A new 3D reconstruction network was proposed in order to resolve the difficulty of 2D detection method to detect defects with depth information. CasMVSNet with multiscale feature enhancement (MFE-CasMVSNet) was combined with the technology of point cloud processing for steel plate surface defect detection. In order to improve the accuracy of 3D reconstruction, a position-oriented feature enhancement module (PFEM) and a multiscale feature adaptive fusion module (MFAFM) were proposed to effectively extract features and reduce information loss. A density clustering method, curvature-sparse-guided density-based spatial clustering of applications with noise (CS-DBSCAN), was proposed for accurately extracting defects in different parts, and the 3D detection box was introduced to locate and visualize defects. Experimental results show that compared with the reconstruction method based on images, MFE-CasMVSNet can realize the 3D reconstruction of steel plate surface more accurately and quickly. Compared with 2D detection, 3D visual defect detection can accurately obtain the 3D shape information of defects and realize the multi-dimensional detection of steel plate surface defects.
Key words: surface defect detection    deep learning    3D reconstruction    point cloud segmentation    defect location
收稿日期: 2022-03-27 出版日期: 2023-03-31
CLC:  TH 164  
基金资助: 国家自然科学基金资助项目 (92167107，71777173)；中央高校基本业务经费资助项目
通讯作者: 余建波     E-mail: 13120521883@163.com;jbyu@tongji.edu.cn
作者简介: 兰欢（1998—），男，硕士生，从事三维重建、缺陷检测研究. orcid.org /0000-0001-5657-5581. E-mail： 13120521883@163.com
服务　　
把本文推荐给朋友
加入引用管理器
E-mail Alert
作者相关文章　　
兰欢
余建波

引用本文:

兰欢,余建波. 基于深度学习三维成型的钢板表面缺陷检测[J]. 浙江大学学报(工学版), 2023, 57(3): 466-476.

Huan LAN,Jian-bo YU. Steel surface defect detection based on deep learning 3D reconstruction. Journal of ZheJiang University (Engineering Science), 2023, 57(3): 466-476.

链接本文:

https://www.zjujournals.com/eng/CN/10.3785/j.issn.1008-973X.2023.03.004    或    https://www.zjujournals.com/eng/CN/Y2023/V57/I3/466

图 1  本研究所提网络的钢板表面缺陷三维检测流程
图 2  多尺度特征增强的级联式三维重建网络结构
图 3  位置引导的特征增强模块结构
图 4  多尺度特征自适应融合模块结构
图 5  平面扫描法
图 6  深度图细化
图 7  带有深度信息的钢板常见表面缺陷
图 8  待检测缺陷钢板
缺陷类型 ld wd hd
割痕缺陷1 40.0 3.0 2.0
割痕缺陷2 33.0 3.0 1.2
焊点缺陷 9.1 6.0 1.3
钻痕缺陷 7.0 7.0 2.5
结疤缺陷 4.4 4.1 0.7
表 1  钢板缺陷的尺寸
方法 Acc/mm Com/mm OR/mm GPU /MB tR/s
COLMAP 0.400 0.664 0.532 — —
Tola 0.342 1.190 0.766 — —
Gipuma 0.283 0.873 0.578 — —
MVSNet 0.456 0.646 0.551 10823 1.210
RMVSNet 0.383 0.452 0.417 7577 1.280
CasMVSNet 0.325 0.385 0.355 5360 0.496
本研究 0.317 0.376 0.347 6272 0.532
表 2  不同方法在DTU数据集上重建效果的比较
添加模块 Acc /mm Com /mm OR /mm GPU /MB tR/s
— 0.325 0.385 0.355 5374 0.494
PFEM 0.315 0.381 0.348 5613 0.512
MFAFM 0.320 0.382 0.351 5529 0.509
PFEM+MFAFM 0.317 0.376 0.347 6272 0.532
表 3  添加不同模块的CasMVSNet消融实验结果
图 9  钢板的部分图像集和重建效果
图 10  钢板缺陷的重建效果
图 11  不同方法的钢板重建效果对比
方法 tr /s 方法 tr /s
COLMAP 579 RMVSNet 76
VisualSFM 263 CasMVSNet 43
OpenMVG+MVS 345 本研究 49
表 4  不同方法的重建时间比较
图 12  基于点云的三维缺陷检测流程
图 13  不同点云处理方法的缺陷聚类效果比较
图 14  基于三维检测框的缺陷检测可视化
缺陷类型 ${b_1}/{\rm{mm}}$ ${b_2}/{\rm{mm}}$ ${{\varepsilon }}/{\text{% }}$
割痕缺陷1 长 40.0 33.88 15.3
宽 3.0 3.13 4.3
深 2.0 1.85 7.5
割痕缺陷2 长 33.0 27.13 17.8
宽 3.0 3.17 5.7
深 1.2 1.10 8.3
焊点缺陷 长 9.1 9.59 5.4
宽 6.0 5.74 4.3
高 1.3 1.39 7.0
钻痕缺陷 长 7.0 6.69 4.4
宽 7.0 6.68 4.6
高 2.5 2.26 9.6
结疤缺陷 长 4.4 4.64 5.5
宽 4.1 4.30 4.9
高 0.7 0.76 8.6
表 5  多尺度特征增强的级联式三维重建网络对钢板表面缺陷的三维检测精度
检测步骤 $t/s$
割痕 焊痕 钻痕 结疤
滤波，降采样 1.78 1.98 1.46 1.34
点云平面分割 0.54 0.59 0.72 0.84
点云缺陷聚类 0.66 0.53 0.65 0.49
三维可视化 0.45 0.17 0.21 0.11
合计 3.43 3.27 3.04 2.78
表 6  基于所提点云数据处理流程的钢板表面缺陷检测各步骤耗时
图 15  基于多尺度特征增强的级联式三维重建网络的钢板表面缺陷三维检测结果
图 16  基于RetinaNet的钢板缺陷误检结果
1 程训, 余建波 基于机器视觉的加工刀具磨损监测方法[J]. 浙江大学学报: 工学版, 2021, 55 (5): 896- 904
CHEN Xun, YU Jian-bo Monitoring method for machining tool wear based on machine vision[J]. Journal of Zhejiang University: Engineering Science, 2021, 55 (5): 896- 904
2 CHU M F, GONG R F, GAO S, et al Steel surface defects recognition based on multi-type statistical features and enhanced twin support vector machine[J]. Chemometrics and Intelligent Laboratory Systems, 2017, 171: 140- 150
doi: 10.1016/j.chemolab.2017.10.020
3 王典洪, 甘胜丰, 张伟民, 等 基于监督双限制连接 Isomap算法的带钢表面缺陷图像分类方法[J]. 自动化学报, 2014, 40 (5): 883- 891
WANG Dian-hong, GAN Sheng-feng, ZHANG Wei-min, et al Strip surface defect image classification based on double-limited and supervised-connect Isomap algorithm[J]. Acta Automatica Sinica, 2014, 40 (5): 883- 891
4 WANG H Y, ZHANG J W, TIAN Y, et al A Simple guidance template-based defect detection method for strip steel surfaces[J]. IEEE Transactions on Industrial Informatics, 2019, 15 (5): 2798- 2809
doi: 10.1109/TII.2018.2887145
5 KOYUNCU I, ÇETIN O, KATIRCIOĞLU F, et al. Edge dedection application with FPGA based Sobel operator [C]// Signal Processing and Communications Applications Conference. Malatya: IEEE, 2015: 1829–1832.
6 HE Y, SONG K C, MENG Q G, et al An end-to-end steel surface defect detection approach via fusing multiple hierarchical features[J]. IEEE Transactions on Instrumentation and Measurement, 2020, 69 (4): 1493- 1504
doi: 10.1109/TIM.2019.2915404
7 LIU Y, XU K, XU J W Periodic surface defect detection in steel plates based on deep learning[J]. Applied Sciences-Basel, 2019, 9 (15): 3127
doi: 10.3390/app9153127
8 金侠挺, 王耀南, 张辉, 等 基于贝叶斯 CNN和注意力网络的钢轨表面缺陷检测系统[J]. 自动化学报, 2019, 45 (12): 2312- 2327
JIN Xia-ting, WANG Yao-nan, ZHANG Hui DeepRail: automatic visual detection system for railway surface defect using Bayesian CNN and attention network[J]. Acta Automatica Sinica, 2019, 45 (12): 2312- 2327
9 LIU J Q, ZHANG Z J, LIN Z Y, et al Characterization method of surface crack based on laser thermography[J]. IEEE Access, 2021, 9: 76395- 76402
doi: 10.1109/ACCESS.2021.3081435
10 GUNATILAKE A, PIYATHILAKA L, TRAN A, et al Stereo vision combined with laser profiling for mapping of pipeline internal defects[J]. IEEE Sensors Journal, 2021, 21 (10): 11926- 11934
doi: 10.1109/JSEN.2020.3040396
11 孙彬, 王建华, 赫东锋, 等 基于激光测量的航发叶片表面几何缺陷识别技术[J]. 自动化学报, 2020, 46 (3): 594- 599
SUN Bin, WANG Jian-hua, HE Dong-feng, et al Identification of aero-engine blade surface geometric defects with laser measurement[J]. Acta Automatica Sinica, 2020, 46 (3): 594- 599
12 ZHAO C, ZHU H, WANG X. Steel plate surface defect recognition method based on depth information [C]// Data Driven Control and Learning Systems Conference. Dali: IEEE, 2019: 322-327.
13 EIGEN D, PUHRSCH C, FERGUS R. Depth map prediction from a single image using a multi-scale deep network [C]// Conference on Neural Information Processing Systems. Montreal: [s. n.], 2014: 2366-2374.
14 YAO Y, LUO Z, LI S, et al. MVSNet: depth inference for unstructured multi-view stereo [C]// European Conference on Computer Vision. Munich: Springer, 2018: 785-801.
15 IM S, JEON H G, LIN S, et al. DPSNet: end-to-end deep plane sweep stereo [EB/OL]. [2022-03-04]. https://arxiv.org/pdf/1905.00538.pdf.
16 YU Z, GAO S. Fast-MVSNet: sparse-to-dense multi-view stereo with learned propagation and gauss-newton refinement [C]// Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020: 1949-1958.
17 CHEN R, HAN S, XU J, et al. Point-based multi-view stereo network [C]// International Conference on Computer Vision. Seoul: IEEE, 2019: 1538-1547.
18 GU X, FAN Z, ZHU S, et al. Cascade cost volume for high-resolution multi-view stereo and stereo matching [C]// Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020: 2492-2501.
19 LIN T Y, DOLLAR P, GIRSHICK R, et al. Feature pyramid networks for object detection [C]// Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017: 2117-2125.
20 HOU Q B, ZHOU D Q, FENG J S. Coordinate attention for efficient mobile network design [C]// Conference on Computer Vision and Pattern Recognition. Nashille: IEEE, 2021: 13708-13717.
21 HU J, SHEN L, SUN G, et al Squeeze-and-excitation networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 42 (8): 2011- 2023
doi: 10.1109/TPAMI.2019.2913372
22 ESTER M, KRIEGEL H P, SANDER J, et al. A density-based algorithm for discovering clusters in large spatial databases with noise [C]// International Conference on Knowledge Discovery and Data Mining. Portland: [s. n.], 1996: 226-231.
23 GOOTTSCHALK S, LIN M C, MANOCHA D. OBBTree: a hierarchical structure for rapid interference detection [C]// Conference on Computer Graphics and Interactive Techniques. New Orleans: ACM, 1996, 30: 171-180.
24 AANAES H, JENSEN R R, VOGIATZIS G, et al Large-scale data for multiple-view stereopsis[J]. International Journal of Computer Vision, 2016, 120 (2): 153- 168
doi: 10.1007/s11263-016-0902-9
25 SCHONBERGER J L, FRAHM J M. Structure-from-motion revisited [C]// Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2016: 4104-4113.
26 TOLA E, STRECHA C, FUA P Efficient large-scale multi-view stereo for ultra high-resolution image sets[J]. Machine Vision and Applications, 2012, 23: 903- 920
[1] 曾耀,高法钦. 基于改进YOLOv5的电子元件表面缺陷检测算法[J]. 浙江大学学报(工学版), 2023, 57(3): 455-465.
[2] 曾菊香,王平辉,丁益东,兰林,蔡林熹,管晓宏. 面向节点分类的图神经网络节点嵌入增强模型[J]. 浙江大学学报(工学版), 2023, 57(2): 219-225.
[3] 鲁建厦,包秦,汤洪涛,邵益平,赵文彬. 无设备人体追踪系统的择优标签方法[J]. 浙江大学学报(工学版), 2023, 57(2): 415-425.
[4] 马骏驰,迪骁鑫,段宗涛,唐蕾. 程序表示学习综述[J]. 浙江大学学报(工学版), 2023, 57(1): 155-169.
[5] 叶晨,战洪飞,林颖俊,余军合,王瑞,钟武昌. 基于推理-情境感知激活模型的设计知识推荐[J]. 浙江大学学报(工学版), 2023, 57(1): 32-46.
[6] 刘近贞,陈飞,熊慧. 多尺度残差网络模型的开放式电阻抗成像算法[J]. 浙江大学学报(工学版), 2022, 56(9): 1789-1795.
[7] 王万良,王铁军,陈嘉诚,尤文波. 融合多尺度和多头注意力的医疗图像分割方法[J]. 浙江大学学报(工学版), 2022, 56(9): 1796-1805.
[8] 郝琨,王阔,王贝贝. 基于改进Mobilenet-YOLOv3的轻量级水下生物检测算法[J]. 浙江大学学报(工学版), 2022, 56(8): 1622-1632.
[9] 夏杰锋,唐武勤,杨强. 光伏航拍红外图像的热斑自动检测方法[J]. 浙江大学学报(工学版), 2022, 56(8): 1640-1647.
[10] 赵永胜,李瑞祥,牛娜娜,赵志勇. 数字孪生驱动的机身形状控制方法[J]. 浙江大学学报(工学版), 2022, 56(7): 1457-1463.
[11] 何立,庞善民. 结合年龄监督和人脸先验的语音-人脸图像重建[J]. 浙江大学学报(工学版), 2022, 56(5): 1006-1016.
[12] 张雪芹,李天任. 基于Cycle-GAN和改进DPN网络的乳腺癌病理图像分类[J]. 浙江大学学报(工学版), 2022, 56(4): 727-735.
[13] 褚晶辉,史李栋,井佩光,吕卫. 适用于目标检测的上下文感知知识蒸馏网络[J]. 浙江大学学报(工学版), 2022, 56(3): 503-509.
[14] 程若然,赵晓丽,周浩军,叶翰辰. 基于深度学习的中文字体风格转换研究综述[J]. 浙江大学学报(工学版), 2022, 56(3): 510-519, 530.
[15] 袁天乐,袁巨龙,朱勇建,郑翰辰. 基于改进YOLOv5的推力球轴承表面缺陷检测算法[J]. 浙江大学学报(工学版), 2022, 56(12): 2349-2357.