Please wait a minute...
浙江大学学报(工学版)
Journal of ZheJiang University (Engineering Science)  2023, Vol. 57 Issue (10): 1998-2010    DOI: 10.3785/j.issn.1008-973X.2023.10.009
    
Image data enhancement method based on improved generative adversarial network
Yan ZHAN(),Die HU,Hong-tao TANG,Jian-sha LU,Jian TAN,Chang-rui LIU
College of Mechanical Engineering, Zhejiang University of Technology, Hangzhou 310023, China
Download: HTML     PDF(5142KB) HTML
Export: BibTeX | EndNote (RIS)      

Abstract  

An image data enhancement method based on data distribution fitting, generative adversarial neural network and image super-resolution reconstruction was proposed to improve the accuracy of machine learning model. The maximum likelihood estimation and sampling algorithm were used to generate two-dimensional noise conforming to the original data distribution for counter-training. The problem of random noise input in traditional image generation models was overcome. Layer by layer training method was used to generate high-resolution images to correct the shortcomings of difficult mapping to high-resolution images. The effectiveness of the proposed method was verified by taking the gray image data enhancement of bearing roller surface. The results showed that the image quality generated by the proposed method was superior. Compared with the image generated by the traditional method, the image peak signal-to-noise ratio was increased by 13.07%, the structural similarity was increased by 32.40%, Fréchet inception distance was reduced by 37.58%, and the average accuracy of the model after data enhancement was increased by 7.89%.

Key words： image data enhancement      distribution fitting      sampling algorithm      generative adversarial neural network      image super-resolution reconstruction     
Received: 13 December 2022      Published: 18 October 2023
CLC:  TP 391.4  
Fund:  浙江省科技计划资助项目[重点研发（尖兵）项目]（2023C01063）
Service
E-mail this article
Add to my bookshelf
Add to citation manager
E-mail Alert
RSS
Articles by authors
Yan ZHAN
Die HU
Hong-tao TANG
Jian-sha LU
Jian TAN
Chang-rui LIU
Cite this article:

Yan ZHAN,Die HU,Hong-tao TANG,Jian-sha LU,Jian TAN,Chang-rui LIU. Image data enhancement method based on improved generative adversarial network. Journal of ZheJiang University (Engineering Science), 2023, 57(10): 1998-2010.

URL:

https://www.zjujournals.com/eng/10.3785/j.issn.1008-973X.2023.10.009     OR     https://www.zjujournals.com/eng/Y2023/V57/I10/1998


基于改进生成对抗网络的图像数据增强方法

为了提高机器学习模型的精确度，提出基于数据分布拟合、生成式对抗神经网络和图像超分辨率重建的图像数据增强方法. 该方法将最大似然估计和采样算法生成的符合原始数据分布的二维噪声用于对抗训练，克服了在生成模型中传统图像噪声输入随意的问题；采用逐层训练方式生成高分辨率图像，改进高分辨率图像映射困难、参数冗余的缺点. 以轴承滚子表面灰度图像数据增强为例，验证所提方法的有效性. 研究结果表明，所提方法生成的图像质量更优，相比传统方法生成的图像峰值信噪比提高13.07%，结构相似性提高32.40%，弗雷歇初始距离降低37.58%，且数据增强后的模型平均精确度提升7.89%.


关键词： 图像数据增强,  分布拟合,  采样算法,  生成式对抗网络,  图像超分辨率重建 
Fig.1 Image data enhancement method based on traditional GAN
Fig.2 Flowchart of image data enhancement method based on generative adversarial neural network super-resolution reconstruction
Fig.3 Flowchart of generating random noise
Fig.4 Flowchart of Metropolis-Hastings sampling algorithm
Fig.5 Architecture diagram of WGAN-GP network
Fig.6 Overall structure diagram of WGAN-GP+SR model
Fig.7 Image data and gray histogram after down-sampling
拟合分布名称 估计参数 SSE
α β ε
卡方分布 2.848 45.687 2.848 0.004 551
伽马分布 ?20.172 8.656 9.378 0.004 411
高斯分布 65.852 31.368 0 0.004 014
T分布 66.366 17.124 2.734 0.002 338
拉普拉斯分布 68.000 19.693 0 0.002 049
韦布尔分布 69.000 20.150 0.926 0.002 051
柯西分布 69.070 11.861 0 0.002 038
Tab.1 Gray histogram fitting results of bearing roller end face
Fig.8 Gray histogram fitting effect of bearing roller end face
Fig.9 Optimal distribution fitting effect of gray histogram of bearing roller surface
图像名称 估计参数 SSE
α β ε
倒角面 8.305 0.475 2.000 0.046284
侧面（峰1） 8.360 4.882 0 0.007293
侧面（峰2） 141.441 21.260 0 0.004029
侧面（峰3） 254.950 1.754 0 0.023585
Tab.2 Fitting results of gray histogram of bearing roller chamfering and side surface
Fig.10 One-dimensional random noise distribution of end face generated by Box-Muller transform
Fig.11 Sampling results of MH sampling algorithm
Fig.12 Comparison between distribution of sampled data and distribution of raw data
Fig.13 Comparison diagram of output results of each model
Fig.14 Change curve of CGAN loss function value
Fig.15 Change curve of WGAN-GP loss function value
Fig.16 Change curve of SRCNN loss function value
模型 PSNR/dB SSIM FID
侧面 倒角面 端面 侧面 倒角面 端面 侧面 倒角面 端面
CGAN 48.043 3 54.327 9 52.459 2 0.184 2 0.288 9 0.248 0 472.899 4 343.297 9 372.268 5
WGAN-GP 57.523 0 62.487 9 53.688 9 0.634 2 0.831 7 0.396 4 363.993 2 166.245 8 207.444 2
WGAN-GP+SR 63.543 6 68.508 5 64.347 9 0.807 0 0.941 1 0.717 6 224.5837 132.936 6 102.986 2
Tab.3 Quality evaluation table of image generated by each model
Fig.17 Different standard deviation noise and corresponding image generation
噪声标准差 $\sigma $ PSNR/dB SSIM FID
0.168 6 56.580 9 0.599 2 312.831 4
0.215 5 56.375 6 0.371 3 406.542 9
0.784 0 55.263 4 0.165 7 430.592 1
Tab.4 Quality evaluation table of image generated by each model
Fig.18 Training process FID curve of bearing roller data set with different sampling methods
Fig.19 Comparison of WGAN-GP+SR and WGAN-GP images
Fig.20 Map diagram of Yolov5 target detection model training process
Fig.21 Comparison of bearing roller defect detection
[1]   康守强, 胡明武, 王玉静, 等 基于特征迁移学习的变工况下滚动轴承故障诊断方法[J]. 中国电机工程学报, 2019, 39 (3): 764- 772
KANG Shou-qiang, HU Ming-wu, WANG Yu-jing, et al Fault diagnosis method of a rolling bearing under variable working conditions based on feature transfer learning[J]. Proceedings of the CSEE, 2019, 39 (3): 764- 772
doi: 10.13334/J.0258-8013.PCSEE.180130
[2]   肖雄, 肖宇雄, 张勇军, 等 基于二维灰度图的数据增强方法在电机轴承故障诊断的应用研究[J]. 中国电机工程学报, 2021, 41 (2): 738- 749
XIAO Xiong, XIAO Yu-xiong, ZHANG Yong-jun, et al Research on the application of the data augmentation method based on 2D gray pixel images in the fault diagnosis of motor bearing[J]. Proceedings of the CSEE, 2021, 41 (2): 738- 749
doi: 10.13334/j.0258-8013.pcsee.200834
[3]   PAN S J, YANG Q A survey on transfer learning[J]. IEEE Transactions on Knowledge and Data Engineering, 2009, 22 (10): 1345- 1359
[4]   SALAKHUTDINOV R, LAROCHELLE H. Efficient learning of deep Boltzmann machines [C]// Proceedings of the 13th International Conference on Artificial Intelligence and Statistics. Sardinia: JMLR, 2010: 693-700.
[5]   BENGIO Y, LAUFER E, ALAIN G, et al. Deep generative stochastic networks trainable by backprop [C]// International Conference on Machine Learning. Beijing: PMLR, 2014: 226-234.
[6]   KINGMA D P, WELLING M. Auto-encoding variational bayes [EB/OL]. [2013-12-20]. https://arxiv.org/abs/1312.6114.
[7]   VANDEN OORD A, KALCHBRENNER N, ESPEHOLT L, et al. Conditional image generation with pixel-cnn decoders [C]// 30th Conference on Neural Information Processing Systems. Barcelona: CA, 2016: 4797-4805.
[8]   VAN OORD A, KALCHBRENNER N, KAVUKCUOGLU K. Pixel recurrent neural networks [C]// International Conference on Machine Learning. New York: JMLR, 2016: 1747-1756.
[9]   GOODFELLOW I, POUGET-ABADIE J, MIRZA M, et al Generative adversarial networks[J]. Communications of the ACM, 2020, 63 (11): 139- 144
doi: 10.1145/3422622
[10]   MIRZA M, OSINDERO S. Conditional generative adversarial nets [EB/OL]. [2014-11-06]. https://arxiv.org/abs/1411.1784.
[11]   RADFORD A, METZ L, CHINTALA S. Unsupervised representation learning with deep convolutional generative adversarial networks [EB/OL]. [2015-11-19]. https://arxiv.org/abs/1511.06434.
[12]   ARJOVSKY M, CHINTALA S, BOTTOU L. Wasserstein generative adversarial networks [C]// International Conference on Machine Learning. Sydney: PMLR, 2017: 214-223.
[13]   GULRAJANI I, AHMED F, ARJOVSKY M, et al. Improved training of wasserstein gans [EB/OL]. [2017-03-31]. https://arxiv.org/abs/1704.00028.
[14]   BERTHELOT D, SCHUMM T, METZ L. Boundary equilibrium generative adversarial networks [EB/OL]. [2017-03-21]. https://arxiv.org/abs/1703.10717.
[15]   KARRAS T, AILA T, LAINE S, et al. Progressive growing of gans for improved quality, stability, and variation [C]// International Conference on Learning Representations. Vancouver: JMLR, 2018: 26.
[16]   KARRAS T, LAINE S, AILA T. A style-based generator architecture for generative adversarial networks [C]// CVF Conference on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2019: 4396-4405.
[17]   KARRAS T, LAINE S, AITTALA M, et al. Analyzing and improving the image quality of style GAN [C]// CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020: 8107-8116.
[18]   KARRAS T, AITTALA M, LAINE S, et al Alias-free generative adversarial networks[J]. Advances in Neural Information Processing Systems, 2021, 34: 852- 863
[19]   DONG C, LOY C C, HE K, et al Image super-resolution using deep convolutional networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015, 38 (2): 295- 307
[20]   DEON A F, MENYAEV Y A Twister generator of random normal numbers by box-muller model[J]. International Journal of Trends in Computer Science, 2020, 16 (1): 1- 13
[21]   何新林, 戚宗锋, 李建勋 基于隐变量后验生成对抗网络的不平衡学习[J]. 上海交通大学学报, 2021, 55 (5): 557- 565
HE Xin-lin, QI Zong-feng, LI Jian-xun Unbalanced learning of generative adversarial network based on latent posterior[J]. Journal of Shanghai Jiaotong University, 2021, 55 (5): 557- 565
doi: 10.16183/j.cnki.jsjtu.2019.264
[22]   ARJOVSKY M, BOTTOU L. Towards principled methods for training generative adversarial networks [EB/OL]. [2017-01-17]. https://arxiv.org/abs/1701.04862.
[1] Xin-dong LV,Jiao LI,Zhen-nan DENG,Hao FENG,Xin-tong CUI,Hong-xia DENG. Structured image super-resolution network based on improved Transformer[J]. Journal of ZheJiang University (Engineering Science), 2023, 57(5): 865-874.