Please wait a minute...
浙江大学学报(工学版)
Journal of ZheJiang University (Engineering Science)  2023, Vol. 57 Issue (3): 477-485    DOI: 10.3785/j.issn.1008-973X.2023.03.005
    
Surface defect identification of cross scene strip based on unsupervised domain adaptation
Kun LIU(),Xiao-song YANG
College of Artificial Intelligence, Hebei University of Technology, Tianjin 300131, China
Download: HTML     PDF(1277KB) HTML
Export: BibTeX | EndNote (RIS)      

Abstract  

In view of the poor generalization performance of the deep learning model at surface defect identification of cross scene strip, an end-to-end multi-level aligned domain adaptation neural network (MADA) was proposed, which could achieve pixel-level illumination distribution alignment and feature-level texture distribution alignment, respectively. The source and target domain data were projected into the illumination subspace by MADA to achieve the pixel-level illumination distribution alignment, through the non-reference pixel-level illumination distribution alignment module and the illumination loss function. The adversarial learning of texture feature extractor and feature-level domain discriminator were used to achieve the texture distribution alignment of the source and target domain. The experiment achieved an F1 measure of 98% in Handan strip surface defect dataset and 86.6% in Severstal strip surface defect dataset. Experimental results showed that the proposed method has better generalization performance than other domain adaptation methods.

Key words： strip surface defect identification      domain adaptation      cross scene      generalization      illumination      texture     
Received: 10 March 2022      Published: 31 March 2023
CLC:  TP 391  
Fund:  国家自然科学基金资助项目（62173124）；河北省自然科学基金资助项目（F2019202305）
Service
E-mail this article
Add to my bookshelf
Add to citation manager
E-mail Alert
RSS
Articles by authors
Kun LIU
Xiao-song YANG
Cite this article:

Kun LIU,Xiao-song YANG. Surface defect identification of cross scene strip based on unsupervised domain adaptation. Journal of ZheJiang University (Engineering Science), 2023, 57(3): 477-485.

URL:

https://www.zjujournals.com/eng/10.3785/j.issn.1008-973X.2023.03.005     OR     https://www.zjujournals.com/eng/Y2023/V57/I3/477


基于无监督域适应的跨场景带钢表面缺陷识别

深度学习模型面对跨场景的带钢表面缺陷识别时存在泛化性能差的问题，为此提出端到端的多级对齐域适应神经网络模型(MADA)，实现源域与目标域数据的像素级光照分布对齐与特征级纹理分布对齐. MADA通过无参考像素级光照分布对齐模块和光照校正损失函数，将源域与目标域数据投影到光照子空间，实现源域与目标域的像素级光照分布对齐. 利用纹理特征提取器和特征级域鉴别器的对抗学习，实现源域和目标域数据的纹理分布对齐. 实验在邯郸钢铁集团带钢表面缺陷数据集的F1指数达到98%，在谢维尔钢铁集团带钢表面缺陷数据集上的F1指数达到86.6%. 实验结果表明，与其他域适应方法相比，所提方法具有更好的泛化性能.


关键词： 带钢表面缺陷识别,  域适应,  跨场景,  泛化,  光照,  纹理 
Fig.1 Structure of multi-level alignment domain adaptation neural network
Fig.2 Strip surface defect datasets with different lighting scenarios
方法 HSDD_N1?HSDD_N2 HSDD_N1?HSDD_N3 HSDD_N2?HSDD_N1 HSDD_N2?HSDD_N3 HSDD_N3?HSDD_N1 HSDD_N3?HSDD_N2 $ \bar F $
P R F P R F P R F P R F P R F P R F
ResNet50 53 43 47 31 36 35 76 50 49 62 60 68 39 39 39 82 72 69 51
DAN 89 86 86 62 45 40 87 84 85 91 88 88 79 66 68 96 96 96 77
JAN 85 85 85 62 61 61 90 90 90 88 87 87 74 74 74 90 90 90 81
DANN 94 94 94 91 91 91 92 92 92 94 94 94 92 92 92 92 92 92 92
ASAN 92 92 92 94 94 94 91 91 91 94 94 95 90 90 90 92 93 93 92
GVB 95 95 95 96 96 96 95 95 95 96 96 97 91 91 90 93 93 93 94
MADA 98 99 99 99 99 99 98 98 98 98 98 98 97 97 97 98 98 98 98
Tab.1 Comparison of evaluation indicators with different methods on HSDD %
方法 SSDD_1?SSDD_2 SSDD_1?SSDD_3 SSDD_2?SSDD_1 SSDD_2?SSDD_3 SSDD_3?SSDD_1 SSDD_3?SSDD_2 $ \bar F $
P R F P R F P R F P R F P R F P R F
ResNet50 78 78 78 60 61 59 80 72 73 79 79 79 67 67 66 82 77 77 72
DAN 84 84 84 67 66 66 78 78 78 85 85 85 69 69 69 83 84 84 77
JAN 84 83 83 65 65 65 84 84 84 84 84 84 70 70 70 84 84 84 78
DANN 82 82 82 72 73 72 88 86 86 88 88 88 74 72 73 85 85 85 81
ASAN 88 87 88 75 73 74 89 89 86 91 91 91 75 75 75 85 85 85 83
GVB 88 88 87 76 75 76 90 90 90 91 91 91 79 77 78 86 86 86 84
MADA 90 92 90 80 80 79 91 91 91 92 92 92 82 82 82 85 85 85 86
Tab.2 Comparison of evaluation indicators with different methods on SSDD %
Fig.3 Loss and accuracy of target domain for different methods
Fig.4 Feature visualization results extracted by different methods for target domain
Fig.5 Heat map of features extracted by different methods for target domain
${L_{{\rm{exp}}} }$ ${L_{{\rm{spa}}} }$ ${L_{ {\rm{tuA} } } }$ P/% R/% F/%
— — — 90.04 86.07 85.11
√ — — 94.30 92.93 92.35
— √ — 90.83 88.76 88.08
— — √ 91.29 88.27 86.75
√ √ — 97.36 97.03 97.20
√ — √ 96.27 95.50 95.56
— √ √ 90.92 88.10 86.99
√ √ √ 98.36 98.03 98.20
Tab.3 Ablation experiment of light loss function
[1]   HU L, DUAN F, DING K, et al Research on surface detects on line detection system for steel plate using computer vision[J]. Iron and Steel, 2005, 40 (2): 59- 61
[2]   WANG J, LI Q, GAN J, et al Surface defect detection via entity sparsity pursuit with intrinsic priors[J]. IEEE Transactions on Industrial Informatics, 2020, 16 (1): 141- 150
doi: 10.1109/TII.2019.2917522
[3]   SAMSUDIAN S S, AROF H, HARUN S W, et al Steel surface defect classification using multi-resolution empirical mode decomposition and LBP[J]. Measurement Science and Technology, 2020, 32 (1): 015601
[4]   MENTOURI Z, MOUSSAOUI A, BOUDJEHEM D, et al Steel strip surface defect identification using multiresolution binarized image features[J]. Journal of Failure Analysis and Prevention, 2020, 20: 1917- 1927
doi: 10.1007/s11668-020-01012-7
[5]   ZAGHDOUDI R, SERIDI H, BOUDIAF A, et al. Binary Gabor pattern (BGP) descriptor and principal component analysis (PCA) for steel surface defects classification [C]// 2020 International Conference on Advanced Aspects of Software Engineering.Constantine: IEEE, 2020: 1–7.
[6]   KNITTER-PIĄTKOWSKA A, DOBRZYCKI A Application of wavelet transform to damage identification in the steel structure elements[J]. Applied Sciences, 2020, 10 (22): 8198
[7]   MENTOURI Z, DOGHMANE H, GHERFI K, et al. Tool combination for the description of steel surface image and defect classification [C]// The 2nd International Conference on Embedded Systems and Artificial Intelligence. Fez:[s.n.], 2021: 1-13.
[8]   SARDA K, ACERNESE A, NOLÈ V, et al A multi-step anomaly detection strategy based on robust distances for the steel industry[J]. IEEE Access, 2021, 9: 53827
doi: 10.1109/ACCESS.2021.3070659
[9]   TSAI D M, CHEN M C, LI W C, et al A fast regularity measure for surface defect detection[J]. Machine Vision and Applications, 2012, 23: 869- 886
doi: 10.1007/s00138-011-0403-3
[10]   BOUDANI F Z, NACEREDDINE N, LAICHE N. Content-based image retrieval for surface defects of hot rolled steel strip using wavelet-based LBP [C]// International Workshop on Artificial Intelligence and Pattern Recognition. [S.l.]: Springer, 2021: 404–413.
[11]   徐科, 宋敏, 杨朝霖, 等 隐马尔可夫树模型在带钢表面缺陷在线检测中的应用[J]. 机械工程学报, 2013, 49 (22): 34- 40
XU Ke, SONG Min, YANG Chao-lin, et al Application of hidden Markov tree model to on-line detection of surface defects for steel strips[J]. Journal of Mechanical Engineering, 2013, 49 (22): 34- 40
doi: 10.3901/JME.2013.22.034
[12]   TAO X, ZHANG D, MA W, et al Automatic metallic surface defect detection and recognition with convolutional neural networks[J]. Applied Sciences, 2018, 8 (9): 1575
[13]   HE Y, SONG K, MENG Q, et al An end-to-end steel surface defect detection approach via fusing multiple hierarchical features[J]. IEEE Transactions on Instrumentation and Measurement, 2019, 69 (4): 1493- 1504
[14]   DONG H, SONG K, HE Y, et al PGA-net: pyramid feature fusion and global context attention network for automated surface defect detection[J]. IEEE Transactions on Industrial Informatics, 2019, 16 (12): 7448- 7458
[15]   CHEN F C, JAHANSHAHI M R NB-CNN: deep learning-based crack detection using convolutional neural network and naïve bayes data fusion[J]. IEEE Transactions on Industrial Electronics, 2018, 65 (5): 4392- 4400
doi: 10.1109/TIE.2017.2764844
[16]   彭大芹, 刘恒, 许国良, 等 基于双向特征融合卷积神经网络的液晶面板缺陷检测算法[J]. 广东通信技术, 2019, 39 (4): 66- 73
PENG Da-qin, LIU Heng, XU Guo-liang, et al Defect detection algorithm of liquid crystal panel based on bidirectional feature fusion convolution neural network[J]. Guangdong Communication Technology, 2019, 39 (4): 66- 73
[17]   ZHANG Y, QIU Z, YAO T, et al. Fully convolutional adaptation networks for semantic segmentation [C]// IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 6810–6818.
[18]   LIU D, ZHANG D, SONG Y, et al. Unsupervised instance segmentation in microscopy images via panoptic domain adaptation and task re-weighting [C]// IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020: 4242–4251.
[19]   ZHENG Y, HUANG D, LIU S, et al. Cross-domain object detection through coarse-to-fine feature adaptation [C]// IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020: 13763–13772.
[20]   SINDAGI V A, SRIVASTAVA S Domain adaptation for automatic OLED panel defect detection using adaptive support vector data description[J]. International Journal of Computer Vision, 2017, 122: 193- 211
doi: 10.1007/s11263-016-0953-y
[21]   GOETZ A, DURMAZ A R, MÜLLER M, et al Addressing materials’ microstructure diversity using transfer learning[J]. NPJ Computational Materials, 2022, 8: 27
doi: 10.1038/s41524-021-00695-2
[22]   FAN R, WANG H, BOCUS M J, et al. We learn better road pothole detection: from attention aggregation to adversarial domain adaptation [C]// European Conference on Computer Vision. [S.l.]: Springer, 2020: 285–300.
[23]   GUO C, LI C, GUO J, et al. Zero-reference deep curve estimation for low-light image enhancement [C]// IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020: 1777–1786.
[24]   HE K, ZHANG X, REN S, et al. Deep residual learning for image recognition [C]// IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 770–778.
[25]   LONG M, CAO Y, WANG J, et al. Learning transferable features with deep adaptation networks[C]// Proceedings of the 32th International Conference on Machine Learning. Lille:[s.n.], 2015: 97-105.
[26]   LONG M, ZHU H, WANG J, et al. Deep transfer learning with joint adaptation networks [C]// Proceedings of the 34th International Conference on Machine Learning. Sydney:[s.n.], 2017: 2208–2217.
[27]   GANIN Y, LEMPITSKY V. Unsupervised domain adaptation by backpropagation [C]// Proceedings of the 32nd International Conference on Machine Learning. Lille:[s.n.], 2015: 1180–1189.
[28]   RAAB C, VATH P, MEIER P, et al. Bridging adversarial and statistical domain transfer via spectral adaptation networks [C]// Proceedings of the Asian Conference on Computer Vision.[S.l.]: Spring, 2020: 457–473.
[29]   CUI S, WANG S, ZHUO J, et al. Gradually vanishing bridge for adversarial domain adaptation [C]// IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020: 12452–12461.
[30]   VAN DER MAATEN L, HINTON G Visualizing data using t-SNE[J]. Journal of Machine Learning Research, 2008, 9 (11): 2579- 2605
[1] Lan WU,Han WANG,Bin-quan LI,Chong-yang LI,Fan-shi KONG. Multi-source unsupervised domain adaption method based on self-supervised task[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(4): 754-763.
[2] Jiong-hui SONG,Qi LI,Jing WANG,Zhi-hai XU,Hua-jun FENG,Yue-ting CHEN. Continuous digital zooming algorithm of dual-focal camera based on texture restoration[J]. Journal of ZheJiang University (Engineering Science), 2021, 55(8): 1510-1517.
[3] You ZHAN,Qiang LI,Xiao-tian MA,Chen-ping WANG,Yan-jun QIU. Macro and micro texture based prediction of pavement surface friction[J]. Journal of ZheJiang University (Engineering Science), 2021, 55(4): 684-694.
[4] Zhi-gang CHEN,Yong-jing WAN,Yu-lan WANG,Cui-ling JIANG,Xia CHEN. Prediction of posterior capsular opacification based on heterogeneous low-rank multimodal fusion network[J]. Journal of ZheJiang University (Engineering Science), 2021, 55(11): 2045-2053.
[5] Tai-heng ZHANG,Biao MEI,Lei QIAO,Hao-jie YANG,Wei-dong ZHU. Detection method for composite hole guided by texture boundary[J]. Journal of ZheJiang University (Engineering Science), 2020, 54(12): 2294-2300.
[6] Xiao-min LU,Hao-wen YAN,Lu KANG,Fang WU. Point cluster selection based on weighted network Voronoi diagram[J]. Journal of ZheJiang University (Engineering Science), 2019, 53(3): 571-578.
[7] Hao SUI,Gao-feng QIN,Xiang-bo CUI,Xin-jiang LU. Robust fuzzy T-S modeling method based on minimizing mean and variance of modeling error[J]. Journal of ZheJiang University (Engineering Science), 2019, 53(2): 382-387.
[8] CHEN De, HAN Sen, SU Qian, HAN Xiao. Evaluation indicator of surface texture of asphalt pavement based on skid-resistance and noise reduction performance[J]. Journal of ZheJiang University (Engineering Science), 2017, 51(5): 896-903.
[9] DONG Li yan, ZHU Qi, LI Yong li. Model combination algorithm based on consensus maximization[J]. Journal of ZheJiang University (Engineering Science), 2017, 51(2): 416-421.
[10] ZHANG Jian ting,ZHANG Li min. Image segmentation using novel adaptive robust bilateral filter[J]. Journal of ZheJiang University (Engineering Science), 2016, 50(9): 1703-1710.
[11] DONG Wei, SHEN Hui liang. Sparse acquisition and reconstruction of bidirectional texture functions[J]. Journal of ZheJiang University (Engineering Science), 2016, 50(12): 2364-2370.
[12] FENG Zhi lin, ZHOU Jia nan, CHEN Wei jie, YIN Jian wei. Mixed state MRF detection algorithm for ink jet printing motion texture[J]. Journal of ZheJiang University (Engineering Science), 2015, 49(9): 1642-1650.
[13] MA Li-gang, ZHANG Le-ping, DENG Jing-song, WANG Ya-jie, WANG Ke. Land use classification using ZY1-“02C” remote sensing images[J]. Journal of ZheJiang University (Engineering Science), 2013, 47(8): 1508-1516.
[14] LI Di, CHEN Xiang-jian, XU Zhi-jun, BAI Yue. Type-II fuzzy neural networks with self-organizing recurrent intervals
for image stabilization of the airborne camera[J]. Journal of ZheJiang University (Engineering Science), 2012, 46(8): 1540-1545.
[15] DAI Yuan-ming, WEI Wei, LIN Yi-ning. An improved Mean-shift tracking algorithm based on
color and texture feature[J]. Journal of ZheJiang University (Engineering Science), 2012, 46(2): 212-217.