Please wait a minute...
浙江大学学报(工学版)
Journal of ZheJiang University (Engineering Science)  2023, Vol. 57 Issue (3): 466-476    DOI: 10.3785/j.issn.1008-973X.2023.03.004
    
Steel surface defect detection based on deep learning 3D reconstruction
Huan LAN1(),Jian-bo YU1,2,*()
1. School of Mechanical Engineering, Tongji University, Shanghai 201804, China
2. Shanghai Collaborative Innovation Center of Intelligent Manufacturing Robot Technology for Large Components, Shanghai 201620, China
Download: HTML     PDF(5141KB) HTML
Export: BibTeX | EndNote (RIS)      

Abstract  

A new 3D reconstruction network was proposed in order to resolve the difficulty of 2D detection method to detect defects with depth information. CasMVSNet with multiscale feature enhancement (MFE-CasMVSNet) was combined with the technology of point cloud processing for steel plate surface defect detection. In order to improve the accuracy of 3D reconstruction, a position-oriented feature enhancement module (PFEM) and a multiscale feature adaptive fusion module (MFAFM) were proposed to effectively extract features and reduce information loss. A density clustering method, curvature-sparse-guided density-based spatial clustering of applications with noise (CS-DBSCAN), was proposed for accurately extracting defects in different parts, and the 3D detection box was introduced to locate and visualize defects. Experimental results show that compared with the reconstruction method based on images, MFE-CasMVSNet can realize the 3D reconstruction of steel plate surface more accurately and quickly. Compared with 2D detection, 3D visual defect detection can accurately obtain the 3D shape information of defects and realize the multi-dimensional detection of steel plate surface defects.

Key words： surface defect detection      deep learning      3D reconstruction      point cloud segmentation      defect location     
Received: 27 March 2022      Published: 31 March 2023
CLC:  TH 164  
Fund:  国家自然科学基金资助项目 (92167107，71777173)；中央高校基本业务经费资助项目
Corresponding Authors: Jian-bo YU     E-mail: 13120521883@163.com;jbyu@tongji.edu.cn
Service
E-mail this article
Add to my bookshelf
Add to citation manager
E-mail Alert
RSS
Articles by authors
Huan LAN
Jian-bo YU
Cite this article:

Huan LAN,Jian-bo YU. Steel surface defect detection based on deep learning 3D reconstruction. Journal of ZheJiang University (Engineering Science), 2023, 57(3): 466-476.

URL:

https://www.zjujournals.com/eng/10.3785/j.issn.1008-973X.2023.03.004     OR     https://www.zjujournals.com/eng/Y2023/V57/I3/466


基于深度学习三维成型的钢板表面缺陷检测

为了解决二维检测方法难以检测带有深度信息的缺陷问题，提出全新的三维重建网络. 提出基于多尺度特征增强的级联式三维重建网络(MFE-CasMVSNet)，并与点云数据处理技术结合，用于钢板表面缺陷检测. 为了提高三维重建的精度，提出位置导向的特征增强模块(PFEM)和多尺度特征自适应融合模块(MFAFM)，对特征进行有效提取并减少信息丢失. 提出基于曲率稀疏化的密度聚类方法(CS-DBSCAN)，用于精确识别不同部位的缺陷. 引入三维检测框，实现对缺陷的定位与检测可视化. 实验结果表明，相较于图像几何的重建方法，MFE-CasMVSNet能够更加精确、快速地实现钢板表面的三维重建. 相较于二维检测，三维缺陷检测能够精确获取缺陷的三维形状信息，实现对钢板表面缺陷的多维度检测.


关键词： 表面缺陷检测,  深度学习,  三维重建,  点云分割,  缺陷定位 
Fig.1 Steel plate surface three-dimensional defect detection process by proposed net
Fig.2 Structure of CasMVSNet with multiscale feature enhancement
Fig.3 Structure of position-guided feature enhancement module
Fig.4 Structure of multiscale feature adaptive fusion module
Fig.5 Illustration of planar scanning method
Fig.6 Depth map refinement
Fig.7 Common surface defects of steel plate with depth information
Fig.8 Defective steel plate to be tested
缺陷类型 ld wd hd
割痕缺陷1 40.0 3.0 2.0
割痕缺陷2 33.0 3.0 1.2
焊点缺陷 9.1 6.0 1.3
钻痕缺陷 7.0 7.0 2.5
结疤缺陷 4.4 4.1 0.7
Tab.1 Defect size of steel plate mm
方法 Acc/mm Com/mm OR/mm GPU /MB tR/s
COLMAP 0.400 0.664 0.532 — —
Tola 0.342 1.190 0.766 — —
Gipuma 0.283 0.873 0.578 — —
MVSNet 0.456 0.646 0.551 10823 1.210
RMVSNet 0.383 0.452 0.417 7577 1.280
CasMVSNet 0.325 0.385 0.355 5360 0.496
本研究 0.317 0.376 0.347 6272 0.532
Tab.2 Comparison of reconstruction effects with different methods on DTU
添加模块 Acc /mm Com /mm OR /mm GPU /MB tR/s
— 0.325 0.385 0.355 5374 0.494
PFEM 0.315 0.381 0.348 5613 0.512
MFAFM 0.320 0.382 0.351 5529 0.509
PFEM+MFAFM 0.317 0.376 0.347 6272 0.532
Tab.3 Ablation experimental results of CasMVSNet with different modules added
Fig.9 Part of photo collection and reconstruction effect of steel plate
Fig.10 Reconstruction effect of steel plate defect
Fig.11 Comparison of steel plate reconstruction with different methods
方法 tr /s 方法 tr /s
COLMAP 579 RMVSNet 76
VisualSFM 263 CasMVSNet 43
OpenMVG+MVS 345 本研究 49
Tab.4 Comparison of reconstruction time among different methods
Fig.12 Process of three-dimensional defect detection based on point cloud
Fig.13 Comparison of defect clustering effects of different point cloud processing methods
Fig.14 Visualization of defect detection based on three-dimensional detection box
缺陷类型 ${b_1}/{\rm{mm}}$ ${b_2}/{\rm{mm}}$ ${{\varepsilon }}/{\text{% }}$
割痕缺陷1 长 40.0 33.88 15.3
宽 3.0 3.13 4.3
深 2.0 1.85 7.5
割痕缺陷2 长 33.0 27.13 17.8
宽 3.0 3.17 5.7
深 1.2 1.10 8.3
焊点缺陷 长 9.1 9.59 5.4
宽 6.0 5.74 4.3
高 1.3 1.39 7.0
钻痕缺陷 长 7.0 6.69 4.4
宽 7.0 6.68 4.6
高 2.5 2.26 9.6
结疤缺陷 长 4.4 4.64 5.5
宽 4.1 4.30 4.9
高 0.7 0.76 8.6
Tab.5 Three-dimensional detection accuracy for steel plate surface defect by CasMVSNet with multiscale feature enhancement
检测步骤 $t/s$
割痕 焊痕 钻痕 结疤
滤波，降采样 1.78 1.98 1.46 1.34
点云平面分割 0.54 0.59 0.72 0.84
点云缺陷聚类 0.66 0.53 0.65 0.49
三维可视化 0.45 0.17 0.21 0.11
合计 3.43 3.27 3.04 2.78
Tab.6 Time consuming of each step of steel plate surface defect detection based on proposed point cloud processing process
Fig.15 Three-dimensional detection results of steel plate surface defects based on CasMVSNet with multiscale feature enhancement
Fig.16 False detection results of steel plate defects based on RetinaNet
[1]   程训, 余建波 基于机器视觉的加工刀具磨损监测方法[J]. 浙江大学学报: 工学版, 2021, 55 (5): 896- 904
CHEN Xun, YU Jian-bo Monitoring method for machining tool wear based on machine vision[J]. Journal of Zhejiang University: Engineering Science, 2021, 55 (5): 896- 904
[2]   CHU M F, GONG R F, GAO S, et al Steel surface defects recognition based on multi-type statistical features and enhanced twin support vector machine[J]. Chemometrics and Intelligent Laboratory Systems, 2017, 171: 140- 150
doi: 10.1016/j.chemolab.2017.10.020
[3]   王典洪, 甘胜丰, 张伟民, 等 基于监督双限制连接 Isomap算法的带钢表面缺陷图像分类方法[J]. 自动化学报, 2014, 40 (5): 883- 891
WANG Dian-hong, GAN Sheng-feng, ZHANG Wei-min, et al Strip surface defect image classification based on double-limited and supervised-connect Isomap algorithm[J]. Acta Automatica Sinica, 2014, 40 (5): 883- 891
[4]   WANG H Y, ZHANG J W, TIAN Y, et al A Simple guidance template-based defect detection method for strip steel surfaces[J]. IEEE Transactions on Industrial Informatics, 2019, 15 (5): 2798- 2809
doi: 10.1109/TII.2018.2887145
[5]   KOYUNCU I, ÇETIN O, KATIRCIOĞLU F, et al. Edge dedection application with FPGA based Sobel operator [C]// Signal Processing and Communications Applications Conference. Malatya: IEEE, 2015: 1829–1832.
[6]   HE Y, SONG K C, MENG Q G, et al An end-to-end steel surface defect detection approach via fusing multiple hierarchical features[J]. IEEE Transactions on Instrumentation and Measurement, 2020, 69 (4): 1493- 1504
doi: 10.1109/TIM.2019.2915404
[7]   LIU Y, XU K, XU J W Periodic surface defect detection in steel plates based on deep learning[J]. Applied Sciences-Basel, 2019, 9 (15): 3127
doi: 10.3390/app9153127
[8]   金侠挺, 王耀南, 张辉, 等 基于贝叶斯 CNN和注意力网络的钢轨表面缺陷检测系统[J]. 自动化学报, 2019, 45 (12): 2312- 2327
JIN Xia-ting, WANG Yao-nan, ZHANG Hui DeepRail: automatic visual detection system for railway surface defect using Bayesian CNN and attention network[J]. Acta Automatica Sinica, 2019, 45 (12): 2312- 2327
[9]   LIU J Q, ZHANG Z J, LIN Z Y, et al Characterization method of surface crack based on laser thermography[J]. IEEE Access, 2021, 9: 76395- 76402
doi: 10.1109/ACCESS.2021.3081435
[10]   GUNATILAKE A, PIYATHILAKA L, TRAN A, et al Stereo vision combined with laser profiling for mapping of pipeline internal defects[J]. IEEE Sensors Journal, 2021, 21 (10): 11926- 11934
doi: 10.1109/JSEN.2020.3040396
[11]   孙彬, 王建华, 赫东锋, 等 基于激光测量的航发叶片表面几何缺陷识别技术[J]. 自动化学报, 2020, 46 (3): 594- 599
SUN Bin, WANG Jian-hua, HE Dong-feng, et al Identification of aero-engine blade surface geometric defects with laser measurement[J]. Acta Automatica Sinica, 2020, 46 (3): 594- 599
[12]   ZHAO C, ZHU H, WANG X. Steel plate surface defect recognition method based on depth information [C]// Data Driven Control and Learning Systems Conference. Dali: IEEE, 2019: 322-327.
[13]   EIGEN D, PUHRSCH C, FERGUS R. Depth map prediction from a single image using a multi-scale deep network [C]// Conference on Neural Information Processing Systems. Montreal: [s. n.], 2014: 2366-2374.
[14]   YAO Y, LUO Z, LI S, et al. MVSNet: depth inference for unstructured multi-view stereo [C]// European Conference on Computer Vision. Munich: Springer, 2018: 785-801.
[15]   IM S, JEON H G, LIN S, et al. DPSNet: end-to-end deep plane sweep stereo [EB/OL]. [2022-03-04]. https://arxiv.org/pdf/1905.00538.pdf.
[16]   YU Z, GAO S. Fast-MVSNet: sparse-to-dense multi-view stereo with learned propagation and gauss-newton refinement [C]// Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020: 1949-1958.
[17]   CHEN R, HAN S, XU J, et al. Point-based multi-view stereo network [C]// International Conference on Computer Vision. Seoul: IEEE, 2019: 1538-1547.
[18]   GU X, FAN Z, ZHU S, et al. Cascade cost volume for high-resolution multi-view stereo and stereo matching [C]// Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020: 2492-2501.
[19]   LIN T Y, DOLLAR P, GIRSHICK R, et al. Feature pyramid networks for object detection [C]// Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017: 2117-2125.
[20]   HOU Q B, ZHOU D Q, FENG J S. Coordinate attention for efficient mobile network design [C]// Conference on Computer Vision and Pattern Recognition. Nashille: IEEE, 2021: 13708-13717.
[21]   HU J, SHEN L, SUN G, et al Squeeze-and-excitation networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 42 (8): 2011- 2023
doi: 10.1109/TPAMI.2019.2913372
[22]   ESTER M, KRIEGEL H P, SANDER J, et al. A density-based algorithm for discovering clusters in large spatial databases with noise [C]// International Conference on Knowledge Discovery and Data Mining. Portland: [s. n.], 1996: 226-231.
[23]   GOOTTSCHALK S, LIN M C, MANOCHA D. OBBTree: a hierarchical structure for rapid interference detection [C]// Conference on Computer Graphics and Interactive Techniques. New Orleans: ACM, 1996, 30: 171-180.
[24]   AANAES H, JENSEN R R, VOGIATZIS G, et al Large-scale data for multiple-view stereopsis[J]. International Journal of Computer Vision, 2016, 120 (2): 153- 168
doi: 10.1007/s11263-016-0902-9
[25]   SCHONBERGER J L, FRAHM J M. Structure-from-motion revisited [C]// Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2016: 4104-4113.
[26]   TOLA E, STRECHA C, FUA P Efficient large-scale multi-view stereo for ultra high-resolution image sets[J]. Machine Vision and Applications, 2012, 23: 903- 920
[1] Yao ZENG,Fa-qin GAO. Surface defect detection algorithm of electronic components based on improved YOLOv5[J]. Journal of ZheJiang University (Engineering Science), 2023, 57(3): 455-465.
[2] Ju-xiang ZENG,Ping-hui WANG,Yi-dong DING,Lin LAN,Lin-xi CAI,Xiao-hong GUAN. Graph neural network based node embedding enhancement model for node classification[J]. Journal of ZheJiang University (Engineering Science), 2023, 57(2): 219-225.
[3] Jian-sha LU,Qin BAO,Hong-tao TANG,Yi-ping SHAO,Wen-bin ZHAO. Optimal tag selection method for device-free human tracking system[J]. Journal of ZheJiang University (Engineering Science), 2023, 57(2): 415-425.
[4] Jun-chi MA,Xiao-xin DI,Zong-tao DUAN,Lei TANG. Survey on program representation learning[J]. Journal of ZheJiang University (Engineering Science), 2023, 57(1): 155-169.
[5] Chen YE,Hong-fei ZHAN,Ying-jun LIN,Jun-he YU,Rui WANG,Wu-chang ZHONG. Design knowledge recommendation based on inference-context-aware activation model[J]. Journal of ZheJiang University (Engineering Science), 2023, 57(1): 32-46.
[6] Jin-zhen LIU,Fei CHEN,Hui XIONG. Open electrical impedance imaging algorithm based on multi-scale residual network model[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(9): 1789-1795.
[7] Wan-liang WANG,Tie-jun WANG,Jia-cheng CHEN,Wen-bo YOU. Medical image segmentation method combining multi-scale and multi-head attention[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(9): 1796-1805.
[8] Kun HAO,Kuo WANG,Bei-bei WANG. Lightweight underwater biological detection algorithm based on improved Mobilenet-YOLOv3[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(8): 1622-1632.
[9] Yong-sheng ZHAO,Rui-xiang LI,Na-na NIU,Zhi-yong ZHAO. Shape control method of fuselage driven by digital twin[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(7): 1457-1463.
[10] Li HE,Shan-min PANG. Face reconstruction from voice based on age-supervised learning and face prior information[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(5): 1006-1016.
[11] Xue-qin ZHANG,Tian-ren LI. Breast cancer pathological image classification based on Cycle-GAN and improved DPN network[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(4): 727-735.
[12] Jing-hui CHU,Li-dong SHI,Pei-guang JING,Wei LV. Context-aware knowledge distillation network for object detection[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(3): 503-509.
[13] Ruo-ran CHENG,Xiao-li ZHAO,Hao-jun ZHOU,Han-chen YE. Review of Chinese font style transfer research based on deep learning[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(3): 510-519, 530.
[14] Tian-le YUAN,Ju-long YUAN,Yong-jian ZHU,Han-chen ZHENG. Surface defect detection algorithm of thrust ball bearing based on improved YOLOv5[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(12): 2349-2357.
[15] Fan ZHONG,Zheng-yao BAI. 3D point cloud super-resolution with dynamic residual graph convolutional networks[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(11): 2251-2259.