Please wait a minute...
Journal of ZheJiang University (Engineering Science)  2022, Vol. 56 Issue (2): 213-224    DOI: 10.3785/j.issn.1008-973X.2022.02.001
    
Underwater image enhancement algorithm based on GAN and multi-level wavelet CNN
Pei-zhi WEN1(),Jun-mou CHEN1,Yan-nan XIAO1,Ya-yuan WEN2,Wen-ming HUANG1
1. School of Computer Science and Information Security, Guilin University of Electronic Technology, Guilin 541004, China
2. College of Electronic Engineering, Guangxi Normal University, Guilin 541004, China
Download: HTML     PDF(1464KB) HTML
Export: BibTeX | EndNote (RIS)      

Abstract  

An underwater image enhancement algorithm was proposed based on generative adversarial networks (GAN) and improved convolutional neural networks (CNN) in order to solve the problems of haze blurring and color distortion of underwater image. Generative adversarial network was used to synthesize underwater images to effectively expand the paired underwater data set. The underwater image was decomposed by multi-scale wavelet transform without losing the feature resolution. Then, combined with CNN, the compact learning method was used to extract features from multi-scale images, and skip connection was used to prevent gradient dispersion. Finally, the fog blur effect of the underwater image was resolved. In order to improve the color correction ability of the model and overcome the problem of color distortion of underwater images, the correlation between different channels of color images was learned by using the style cost function. Experimental results show that, in subjective visual and objective indicators, the proposed algorithm is superior to the contrast algorithm in comprehensive performance and robustness.



Key wordsimage processing      underwater image enhancement      multi-level wavelet transform      convolutional neural networks      generative adversarial networks     
Received: 29 March 2021      Published: 03 March 2022
CLC:  TP 391  
Cite this article:

Pei-zhi WEN,Jun-mou CHEN,Yan-nan XIAO,Ya-yuan WEN,Wen-ming HUANG. Underwater image enhancement algorithm based on GAN and multi-level wavelet CNN. Journal of ZheJiang University (Engineering Science), 2022, 56(2): 213-224.

URL:

https://www.zjujournals.com/eng/10.3785/j.issn.1008-973X.2022.02.001     OR     https://www.zjujournals.com/eng/Y2022/V56/I2/213


基于生成式对抗网络和多级小波包卷积网络的水下图像增强算法

为了解决水下图像的雾模糊和偏色问题,针对水下图像成像模型提出基于生成式对抗网络(GAN)和改进卷积神经网络(CNN)的水下图像增强算法. 利用生成式对抗网络合成水下图像,以对配对式水下图像数据集进行有效扩充. 利用多级小波变换,以不丢失特征分辨率的方式对水下图像进行多尺度分解,然后结合卷积神经网络利用紧凑式学习方式对多尺度图像进行特征提取,并利用跳跃连接以防止梯度弥散,克服水下图像的雾模糊效应. 利用风格代价函数学习彩色图像各通道间的相关性,提高模型的色彩校正能力,克服水下图像色彩失真的问题. 实验结果表明,相较对比算法,在主观视觉和客观指标上,本研究所提算法拥有更优秀的综合性能及鲁棒性.


关键词: 图像处理,  水下图像增强,  多级小波变换,  卷积神经网络,  生成式对抗网络 
Fig.1 Underwater image generation model
Fig.2 Parameter estimator of underwater image generation model
Fig.3 Conv block of parameter estimator
Fig.4 Underwater image enhancement model
Fig.5 Comparison of visual enhancement effect between proposed algorithm and other algorithms in URPC dataset
Fig.6 Comparison of visual enhancement effect between proposed algorithm and other algorithms in EUVP dataset
Fig.7 Comparison of detail of experimental image between proposed algorithm and UWGAN algorithm
算法 UIQM UCIQE NIQE
原始图像 2.787 0.432 6.973
IBLA 2.612 0.558 7.220
UDCP 2.073 0.523 6.924
ULAP 2.394 0.524 6.717
RGHS 2.752 0.575 7.164
Sea-thru 3.022 0.533 6.844
UWGAN 2.981 0.490 6.225
FunieGAN 3.054 0.494 10.300
FunieGAN-up 2.915 0.479 7.744
本研究算法 3.341 0.525 5.866
本研究算法-无色彩校正 3.021 0.497 5.988
Tab.1 Comparison of indicators between proposed algorithm and other algorithms in URPC dataset
算法 SSIM PSNR UIQM UCIQE NIQE
原始图像 0.814 29.320 2.549 0.550 8.192
IBLA 0.464 15.040 1.045 0.698 11.930
UDCP 0.734 19.430 2.028 0.588 7.958
ULAP 0.694 23.190 2.213 0.577 8.073
RGHS 0.778 26.660 2.337 0.598 8.084
Sea-thru 0.772 21.380 2.593 0.591 8.123
UWGAN 0.880 29.020 3.345 0.554 7.294
FunieGAN 0.796 26.300 3.236 0.559 11.820
FunieGAN-up 0.768 26.200 3.029 0.557 8.259
本研究算法 0.932 30.610 3.196 0.570 7.212
本研究算法-无色彩校正 0.851 28.910 3.003 0.553 7.558
Tab.2 Comparison of indicators between proposed algorithm and other algorithms in EUVP dataset
算法 t/s 算法 t/s
IBLA 5.595 UWGAN 0.010
UDCP 2.411 FunieGAN 0.017
ULAP 0.380 FunieGAN-up 0.017
RGHS 1.005 本研究算法 0.025
Sea-thru 2.910 本研究算法-无色彩校正 0.025
Tab.3 Average processing time per image of algorithms
[1]   STANKIEWICZ P, TAN Y T, KOBILAROV M Adaptive s-ampling with an autonomous underwater vehicle in static marine environments[J]. Journal of Field Robotics, 2021, 38 (4): 572- 597
doi: 10.1002/rob.22005
[2]   RUMSON A. Development of autonomous subsea pipeline inspection capabilities[C]// Global Oceans 2020: Singapore–US Gulf Coast. Biloxi: IEEE, 2020: 1-6.
[3]   CEJKA J, BRUNO F, SKARLATOS D, et al Detecting square markers in underwater environments[J]. Remote Sensing, 2019, 11 (4): 23
[4]   ANCUTI C, ANCUTI C O, HABER T, et al. Enhancing underwater images and videos by fusion[C]// 2012 IEEE Conference on Computer Vision and Pattern Recognition. Providence: IEEE, 2012: 81-88.
[5]   HUANG D, WANG Y, SONG W, et al. Shallow-water image enhancement using relative global histogram stretching based on adaptive parameter acquisition[C]// International Conference on Multimedia Modeling. Bangkok: Springer, 2018: 453-465.
[6]   DREWS P, NASCIMENTO E, MORAES F, et al. Transmission estimation in underwater single images[C]// Proceedings of the IEEE International Conference on Computer Vision Workshops. Sydney: IEEE, 2013: 825-830.
[7]   PENG Y, COSMAN P C Underwater image restoration based on image blurriness and light absorption[J]. IEEE Transactions on Image Processing, 2017, 26 (4): 1579- 1594
doi: 10.1109/TIP.2017.2663846
[8]   SONG W, WANG Y, HUANG D, et al. A rapid scene depth estimation model based on underwater light attenuation prior for underwater image restoration[C]// Pacific Rim Conference on Multimedia. Hefei: Springer, 2018: 678-688.
[9]   LI J, SKINNER K A, EUSTICE R M, et al WaterGAN: uns-upervised generative network to enable real-time color correction of monocular underwater images[J]. IEEE Robotics and Automation Letters, 2018, 3 (1): 387- 394
[10]   LI C Y, GUO J C, GUO C L Emerging from water: underwater image color correction based on weakly supervised color transfer[J]. IEEE Signal Processing Letters, 2018, 25 (3): 323- 327
doi: 10.1109/LSP.2018.2792050
[11]   FABBRI C, ISLAM M J, SATTAR J. Enhancing underwater imagery using generative adversarial networks[C]// 2018 IEEE International Conference on Robotics and Automation. Brisbane: IEEE, 2018: 7159-7165.
[12]   WANG N, ZHOU Y, HAN F, et al. UWGAN: underwater GAN for real-world underwater color restoration and dehazing [EB/OL]. (2019-12-21). https://arxiv.org/ftp/arxiv/papers/1912/1912.10269.pdf.
[13]   LIU P, ZHANG H, ZHANG K, et al. Multi-level wavelet-CNN for image restoration[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. Salt Lake City: IEEE, 2018: 773-782.
[14]   JAFFE J S Computer modeling and the design of optimal underwater imaging systems[J]. IEEE Journal of Oceanic Engineering, 1990, 15 (2): 101- 111
doi: 10.1109/48.50695
[15]   AKKAYNAK D, TREIBITZ T. A revised underwater image formation model[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 6723-6732.
[16]   AKKAYNAK D, TREIBITZ T, SHLESINGER T, et al. What is the space of attenuation coefficients in underwater computer vision?[C]// Proceedings of the IEEE Conference on Com-puter Vision and Pattern Recognition. Honolulu: IEEE, 2017: 4931-4940.
[17]   GALDRAN A, PARDO D, PICON A, et al Automatic red channel underwater image restoration[J]. Journal of Visual Communication and Image Representation, 2015, 26: 132- 145
doi: 10.1016/j.jvcir.2014.11.006
[18]   ZHAO X W, JIN T, QU S Deriving inherent optical properties from background color and underwater image enhancement[J]. Ocean Engineering, 2015, 94: 163- 172
doi: 10.1016/j.oceaneng.2014.11.036
[19]   AKKAYNAK D, TREIBITZ T. Sea-thru: a method for removing water from underwater images[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Reco-gnition. Long Beach: IEEE, 2019: 1682-1691.
[20]   GODARD C, MAC AODHA O, FIRMAN M, et al. Digging into self-supervised monocular depth estimation[C]// Proceedings of the IEEE/CVF International Conference on Computer Vision. Seoul: IEEE, 2019: 3828-3838.
[21]   WEI K, FU Y, YANG J, et al. A physics-based noise formation model for extreme low-light raw denoising [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020: 2758-2767.
[22]   RADFORD A, METZ L, CHINTALA S. Unsupervised representation learning with deep convolutional generative adversarial networks [EB/OL]. (2015-11-19). https://arxiv.org/pdf/1511.06434.pdf.
[23]   RONNEBERGER O, FISCHER P, BROX T. U-net: convolutional networks for biomedical image segmentation[C]// International Conference on Medical Image Computing and Computer-assisted Intervention. Munich: Springer, 2015: 234-241.
[24]   ODENA A, DUMOULIN V, OLAH C. Deconvolution and checkerboard artifacts [EB/OL]. [2021-03-01]. https://distill.pub/2016/deconv-checkerboard/.
[25]   GATYS L A, ECKER A S, BETHGE M. A neural algorithm of artistic style [EB/OL]. (2015-08-26). https://arxiv.org/pdf/1508.06576.pdf.
[26]   ISLAM M J, XIA Y, SATTAR J Fast underwater image enh-ancement for improved visual perception[J]. IEEE Robotics and Automation Letters, 2020, 5 (2): 3227- 3234
doi: 10.1109/LRA.2020.2974710
[27]   SILBERMAN N, FERGUS R. Indoor scene segmentation using a structured light sensor[C]// 2011 IEEE International Conference on Computer Vision Workshops. Barcelona: IEEE, 2011: 601-608.
[28]   SILBERMAN N, HOIEM D, KOHLI P, et al. Indoor segme-ntation and support inference from rgbd images[C]// European Conference on Computer Vision. Florence: Springer, 2012: 746-760.
[29]   URPC竞赛项目: 水下目标检测 [DS/OL]. [2019-8-8]. http://www.cnurpc.org/a/xwjrz/2019/0808/129.html.
[30]   YANG M, SOWMYA A An underwater color image quality evaluation metric[J]. IEEE Transactions on Image Processing, 2015, 24 (12): 6062- 6071
doi: 10.1109/TIP.2015.2491020
[31]   PANETTA K, GAO C, AGAIAN S Human-visual-system-inspired underwater image quality measures[J]. IEEE Journal of Oceanic Engineering, 2015, 41 (3): 541- 551
[32]   MITTAL A, SOUNDARARAJAN R, BOVIK A C Making a “completely blind” image quality analyzer[J]. IEEE Signal Processing Letters, 2013, 20 (3): 209- 212
doi: 10.1109/LSP.2012.2227726
[1] Tong CHEN,Jian-feng GUO,Xin-zhong HAN,Xue-li XIE,Jian-xiang XI. Visible and infrared image matching method based on generative adversarial model[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(1): 63-74.
[2] Zhu-ye XU,Xiao-qiang ZHAO,Hong-mei JIANG. 3D model fitting method based on point distribution model[J]. Journal of ZheJiang University (Engineering Science), 2021, 55(12): 2373-2381.
[3] Yan-nan ZHANG,Xiao-hong HUANG,Yan MA,Qun CONG. Method with recording text classification based on deep learning[J]. Journal of ZheJiang University (Engineering Science), 2020, 54(7): 1264-1271.
[4] Ying LI,Fang CHENG,Zhi-lin ZHAO. Machining precision online measurement of large span pin hole using structured light[J]. Journal of ZheJiang University (Engineering Science), 2020, 54(3): 557-565.
[5] Wan-liang WANG,Xiao-han YANG,Yan-wei ZHAO,Nan GAO,Chuang LV,Zhao-juan ZHANG. Image enhancement algorithm with convolutional auto-encoder network[J]. Journal of ZheJiang University (Engineering Science), 2019, 53(9): 1728-1740.
[6] ZHOU Hao, LI Ning, LI Yuan, ZHAO Meng-hao, CEN Ke-fa. Experimental study on ethanol spray combustion characteristics under oxy-fuel conditions[J]. Journal of ZheJiang University (Engineering Science), 2018, 52(9): 1821-1827.
[7] ZHANG Cheng-zhi, FENG Hua-jun, XU Zhi-hai, LI Qi, CHEN Yue-ting. Piecewise noise variance estimation of images based on wavelet transform[J]. Journal of ZheJiang University (Engineering Science), 2018, 52(9): 1804-1810.
[8] GUO Bao-zhen, ZUO Wan-li, WANG Ying. Double CNN sentence classification model with attention mechanism of word embeddings[J]. Journal of ZheJiang University (Engineering Science), 2018, 52(9): 1729-1737.
[9] ZHOU Jia-li, CHEN Yi-jun, WU Min. Image acquisition and preprocessing method based on FPGA monitor[J]. Journal of ZheJiang University (Engineering Science), 2018, 52(2): 398-405.
[10] HUANG Song, YI Ben-shun. Underwater image restoration algorithm based on adaptive transmission ratio[J]. Journal of ZheJiang University (Engineering Science), 2018, 52(1): 166-173.
[11] ZHOU Hao, MA Wei chen, YANG Yu, CHEN Jian zhong. Investigation of flame characteristics of low NOx swirl burner[J]. Journal of ZheJiang University (Engineering Science), 2016, 50(4): 698-703.
[12] WU Yi,YIN Jun, ZHU Li, YUAN Yong ming. Two dimensional Arimoto gray entropy image thresholding based on bee colony optimization or decomposition[J]. Journal of ZheJiang University (Engineering Science), 2015, 49(9): 1625-1633.
[13] WANG Yuan-yuan, GUO Yan-en, SHI Guo-quan, WEI Jun-xia, XIA Shun-ren. Algorithm for seabed terrain boundary extraction based on acoustic images[J]. Journal of ZheJiang University (Engineering Science), 2015, 49(2): 376-383.
[14] WANG Han, XIA Xin-xing, YU Chao, ZHONG Qing, GAO Qian-jue, LI Hai-feng, LIU Xu. Brightness uniformity correction method of light field integral three-dimensional display[J]. Journal of ZheJiang University (Engineering Science), 2015, 49(1): 1-5.
[15] WANG Han, XIA Xin-xing, YU Chao, ZHONG Qing, GAO Qian-jue, LI Hai-feng, LIU Xu. integral three-dimensional display[J]. Journal of ZheJiang University (Engineering Science), 2014, 48(8): 1-6.