Steel surface defect detection based on deep learning 3D reconstruction
Huan LAN1(),Jian-bo YU1,2,*()
1. School of Mechanical Engineering, Tongji University, Shanghai 201804, China 2. Shanghai Collaborative Innovation Center of Intelligent Manufacturing Robot Technology for Large Components, Shanghai 201620, China
A new 3D reconstruction network was proposed in order to resolve the difficulty of 2D detection method to detect defects with depth information. CasMVSNet with multiscale feature enhancement (MFE-CasMVSNet) was combined with the technology of point cloud processing for steel plate surface defect detection. In order to improve the accuracy of 3D reconstruction, a position-oriented feature enhancement module (PFEM) and a multiscale feature adaptive fusion module (MFAFM) were proposed to effectively extract features and reduce information loss. A density clustering method, curvature-sparse-guided density-based spatial clustering of applications with noise (CS-DBSCAN), was proposed for accurately extracting defects in different parts, and the 3D detection box was introduced to locate and visualize defects. Experimental results show that compared with the reconstruction method based on images, MFE-CasMVSNet can realize the 3D reconstruction of steel plate surface more accurately and quickly. Compared with 2D detection, 3D visual defect detection can accurately obtain the 3D shape information of defects and realize the multi-dimensional detection of steel plate surface defects.
Huan LAN,Jian-bo YU. Steel surface defect detection based on deep learning 3D reconstruction. Journal of ZheJiang University (Engineering Science), 2023, 57(3): 466-476.
Fig.1Steel plate surface three-dimensional defect detection process by proposed net
Fig.2Structure of CasMVSNet with multiscale feature enhancement
Fig.3Structure of position-guided feature enhancement module
Fig.4Structure of multiscale feature adaptive fusion module
Fig.5Illustration of planar scanning method
Fig.6Depth map refinement
Fig.7Common surface defects of steel plate with depth information
Fig.8Defective steel plate to be tested
缺陷类型
ld
wd
hd
割痕缺陷1
40.0
3.0
2.0
割痕缺陷2
33.0
3.0
1.2
焊点缺陷
9.1
6.0
1.3
钻痕缺陷
7.0
7.0
2.5
结疤缺陷
4.4
4.1
0.7
Tab.1Defect size of steel plate mm
方法
Acc/mm
Com/mm
OR/mm
GPU /MB
tR/s
COLMAP
0.400
0.664
0.532
—
—
Tola
0.342
1.190
0.766
—
—
Gipuma
0.283
0.873
0.578
—
—
MVSNet
0.456
0.646
0.551
10823
1.210
RMVSNet
0.383
0.452
0.417
7577
1.280
CasMVSNet
0.325
0.385
0.355
5360
0.496
本研究
0.317
0.376
0.347
6272
0.532
Tab.2Comparison of reconstruction effects with different methods on DTU
添加模块
Acc /mm
Com /mm
OR /mm
GPU /MB
tR/s
—
0.325
0.385
0.355
5374
0.494
PFEM
0.315
0.381
0.348
5613
0.512
MFAFM
0.320
0.382
0.351
5529
0.509
PFEM+MFAFM
0.317
0.376
0.347
6272
0.532
Tab.3Ablation experimental results of CasMVSNet with different modules added
Fig.9Part of photo collection and reconstruction effect of steel plate
Fig.10Reconstruction effect of steel plate defect
Fig.11Comparison of steel plate reconstruction with different methods
方法
tr /s
方法
tr /s
COLMAP
579
RMVSNet
76
VisualSFM
263
CasMVSNet
43
OpenMVG+MVS
345
本研究
49
Tab.4Comparison of reconstruction time among different methods
Fig.12Process of three-dimensional defect detection based on point cloud
Fig.13Comparison of defect clustering effects of different point cloud processing methods
Fig.14Visualization of defect detection based on three-dimensional detection box
缺陷类型
${b_1}/{\rm{mm}}$
${b_2}/{\rm{mm}}$
${{\varepsilon }}/{\text{% }}$
割痕缺陷1
长
40.0
33.88
15.3
宽
3.0
3.13
4.3
深
2.0
1.85
7.5
割痕缺陷2
长
33.0
27.13
17.8
宽
3.0
3.17
5.7
深
1.2
1.10
8.3
焊点缺陷
长
9.1
9.59
5.4
宽
6.0
5.74
4.3
高
1.3
1.39
7.0
钻痕缺陷
长
7.0
6.69
4.4
宽
7.0
6.68
4.6
高
2.5
2.26
9.6
结疤缺陷
长
4.4
4.64
5.5
宽
4.1
4.30
4.9
高
0.7
0.76
8.6
Tab.5Three-dimensional detection accuracy for steel plate surface defect by CasMVSNet with multiscale feature enhancement
检测步骤
$t/s$
割痕
焊痕
钻痕
结疤
滤波，降采样
1.78
1.98
1.46
1.34
点云平面分割
0.54
0.59
0.72
0.84
点云缺陷聚类
0.66
0.53
0.65
0.49
三维可视化
0.45
0.17
0.21
0.11
合计
3.43
3.27
3.04
2.78
Tab.6Time consuming of each step of steel plate surface defect detection based on proposed point cloud processing process
Fig.15Three-dimensional detection results of steel plate surface defects based on CasMVSNet with multiscale feature enhancement
Fig.16False detection results of steel plate defects based on RetinaNet
[1]
程训, 余建波 基于机器视觉的加工刀具磨损监测方法[J]. 浙江大学学报: 工学版, 2021, 55 (5): 896- 904 CHEN Xun, YU Jian-bo Monitoring method for machining tool wear based on machine vision[J]. Journal of Zhejiang University: Engineering Science, 2021, 55 (5): 896- 904
[2]
CHU M F, GONG R F, GAO S, et al Steel surface defects recognition based on multi-type statistical features and enhanced twin support vector machine[J]. Chemometrics and Intelligent Laboratory Systems, 2017, 171: 140- 150
doi: 10.1016/j.chemolab.2017.10.020
[3]
王典洪, 甘胜丰, 张伟民, 等 基于监督双限制连接 Isomap算法的带钢表面缺陷图像分类方法[J]. 自动化学报, 2014, 40 (5): 883- 891 WANG Dian-hong, GAN Sheng-feng, ZHANG Wei-min, et al Strip surface defect image classification based on double-limited and supervised-connect Isomap algorithm[J]. Acta Automatica Sinica, 2014, 40 (5): 883- 891
[4]
WANG H Y, ZHANG J W, TIAN Y, et al A Simple guidance template-based defect detection method for strip steel surfaces[J]. IEEE Transactions on Industrial Informatics, 2019, 15 (5): 2798- 2809
doi: 10.1109/TII.2018.2887145
[5]
KOYUNCU I, ÇETIN O, KATIRCIOĞLU F, et al. Edge dedection application with FPGA based Sobel operator [C]// Signal Processing and Communications Applications Conference. Malatya: IEEE, 2015: 1829–1832.
[6]
HE Y, SONG K C, MENG Q G, et al An end-to-end steel surface defect detection approach via fusing multiple hierarchical features[J]. IEEE Transactions on Instrumentation and Measurement, 2020, 69 (4): 1493- 1504
doi: 10.1109/TIM.2019.2915404
[7]
LIU Y, XU K, XU J W Periodic surface defect detection in steel plates based on deep learning[J]. Applied Sciences-Basel, 2019, 9 (15): 3127
doi: 10.3390/app9153127
[8]
金侠挺, 王耀南, 张辉, 等 基于贝叶斯 CNN和注意力网络的钢轨表面缺陷检测系统[J]. 自动化学报, 2019, 45 (12): 2312- 2327 JIN Xia-ting, WANG Yao-nan, ZHANG Hui DeepRail: automatic visual detection system for railway surface defect using Bayesian CNN and attention network[J]. Acta Automatica Sinica, 2019, 45 (12): 2312- 2327
[9]
LIU J Q, ZHANG Z J, LIN Z Y, et al Characterization method of surface crack based on laser thermography[J]. IEEE Access, 2021, 9: 76395- 76402
doi: 10.1109/ACCESS.2021.3081435
[10]
GUNATILAKE A, PIYATHILAKA L, TRAN A, et al Stereo vision combined with laser profiling for mapping of pipeline internal defects[J]. IEEE Sensors Journal, 2021, 21 (10): 11926- 11934
doi: 10.1109/JSEN.2020.3040396
[11]
孙彬, 王建华, 赫东锋, 等 基于激光测量的航发叶片表面几何缺陷识别技术[J]. 自动化学报, 2020, 46 (3): 594- 599 SUN Bin, WANG Jian-hua, HE Dong-feng, et al Identification of aero-engine blade surface geometric defects with laser measurement[J]. Acta Automatica Sinica, 2020, 46 (3): 594- 599
[12]
ZHAO C, ZHU H, WANG X. Steel plate surface defect recognition method based on depth information [C]// Data Driven Control and Learning Systems Conference. Dali: IEEE, 2019: 322-327.
[13]
EIGEN D, PUHRSCH C, FERGUS R. Depth map prediction from a single image using a multi-scale deep network [C]// Conference on Neural Information Processing Systems. Montreal: [s. n.], 2014: 2366-2374.
[14]
YAO Y, LUO Z, LI S, et al. MVSNet: depth inference for unstructured multi-view stereo [C]// European Conference on Computer Vision. Munich: Springer, 2018: 785-801.
[15]
IM S, JEON H G, LIN S, et al. DPSNet: end-to-end deep plane sweep stereo [EB/OL]. [2022-03-04]. https://arxiv.org/pdf/1905.00538.pdf.
[16]
YU Z, GAO S. Fast-MVSNet: sparse-to-dense multi-view stereo with learned propagation and gauss-newton refinement [C]// Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020: 1949-1958.
[17]
CHEN R, HAN S, XU J, et al. Point-based multi-view stereo network [C]// International Conference on Computer Vision. Seoul: IEEE, 2019: 1538-1547.
[18]
GU X, FAN Z, ZHU S, et al. Cascade cost volume for high-resolution multi-view stereo and stereo matching [C]// Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020: 2492-2501.
[19]
LIN T Y, DOLLAR P, GIRSHICK R, et al. Feature pyramid networks for object detection [C]// Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017: 2117-2125.
[20]
HOU Q B, ZHOU D Q, FENG J S. Coordinate attention for efficient mobile network design [C]// Conference on Computer Vision and Pattern Recognition. Nashille: IEEE, 2021: 13708-13717.
[21]
HU J, SHEN L, SUN G, et al Squeeze-and-excitation networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 42 (8): 2011- 2023
doi: 10.1109/TPAMI.2019.2913372
[22]
ESTER M, KRIEGEL H P, SANDER J, et al. A density-based algorithm for discovering clusters in large spatial databases with noise [C]// International Conference on Knowledge Discovery and Data Mining. Portland: [s. n.], 1996: 226-231.
[23]
GOOTTSCHALK S, LIN M C, MANOCHA D. OBBTree: a hierarchical structure for rapid interference detection [C]// Conference on Computer Graphics and Interactive Techniques. New Orleans: ACM, 1996, 30: 171-180.
[24]
AANAES H, JENSEN R R, VOGIATZIS G, et al Large-scale data for multiple-view stereopsis[J]. International Journal of Computer Vision, 2016, 120 (2): 153- 168
doi: 10.1007/s11263-016-0902-9
[25]
SCHONBERGER J L, FRAHM J M. Structure-from-motion revisited [C]// Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2016: 4104-4113.
[26]
TOLA E, STRECHA C, FUA P Efficient large-scale multi-view stereo for ultra high-resolution image sets[J]. Machine Vision and Applications, 2012, 23: 903- 920
Jun-chi MA,Xiao-xin DI,Zong-tao DUAN,Lei TANG. Survey on program representation learning[J]. Journal of ZheJiang University (Engineering Science), 2023, 57(1): 155-169.
