| Image Processing Algorithms |
|
|
|
|
| Prior-based single image rain and haze removal |
| LIANG Chuping1,2, FENG Yidan1,2, XIE Haoran3, WEI Mingqiang1,2, YAN Xuefeng1 |
1.School of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, Nanjing 210016,China
2.MIIT Key Laboratory of Pattern Analysis and Machine Intelligence, Nanjing 210016, China
3.Department of Computer and Decision Sciences, Lingnan University, HongKong 999077, China |
|
|
|
Abstract Single image rain and haze removal is an important task in computer vision. Most state-of-the-art methods focus on the features of rain while ignoring the disturbance from haze and rain accumulation.To solve this problem,this paper proposes a novel strategy of rain and haze removal from a single image and develops a three-stage framework.First,only the low-frequency of the input image is dehazed based on dark channel prior,where a depth map is used as guidance of transmission to improve the result; Then the streaks in high-frequency can be learned through a residual network; At last,local details are refined by a conditional generative adversarial network (cGAN).A high-quality synthetic dataset and real photo data are used in testing.Experiment results show that this framework has better performance on rainy image restoration than baseline,by comparing the value of PSNR and SSIM with ground-truth images. Our method is robust in restoring clean scene from images with various haze density or rain distribution.
|
|
Received: 23 September 2020
Published: 20 May 2021
|
|
|
|
Cite this article:
LIANG Chuping, FENG Yidan, XIE Haoran, WEI Mingqiang, YAN Xuefeng. Prior-based single image rain and haze removal. Journal of Zhejiang University (Science Edition), 2021, 48(3): 270-281.
URL:
https://www.zjujournals.com/sci/EN/Y2021/V48/I3/270
|
基于先验知识的单幅图像雨雾去除方法
单幅图像雨雾去除是计算机视觉研究的热点之一，现有的去雨方法往往需根据雨的特性恢复图像，忽视了场景中的雾以及场景深处雨累积的影响。提出了一种新型单幅图像去雨雾方法，设计了一种基于先验知识的三阶段单幅图像去雨雾框架。首先，将深度图作为图像透射率的引导，利用暗通道先验知识对输入图像的低频部分去雾，然后，采用残差网络学习高频雨痕特征，最后，引入条件生成对抗网络（conditional generative adversarial network，cGAN），对图像局部细节进行精细化修复，cGAN对图像的空间信息和纹理细节更为敏感，能有效恢复去雨雾过程中丢失或未能填补的细节。分别用高质量的合成雨雾数据集和真实数据集进行实验测试，结果表明，本文方法在峰值信噪比和结构相似度上均较基准方法优，能将雾浓度各异和雨分布变化情况下的图像恢复至细节丰富的干净场景图。
关键词：
单幅图像去雨雾,
暗通道先验,
残差网络,
条件生成对抗网络
|
|
1 HE K，SUN J，TANG X，et al. Single image haze removal using dark channel prior［J］.IEEE Transactions on Pattern Analysis & Machine Intelligence，2011，33（12）：2341-2353. DOI：10.1109/tpami.2010.168
2 GARG K，NAYAR S K.Vision and rain［J］.International Journal of Computer Vision，2007，75（1）：3-27. DOI：10.1007/s11263-006-0028-6
3 HU X，FU C W，ZHU L，et al.Depth-attentional features for single-image rain removal［C］//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Long Beach：IEEE，2019：8022-8031. DOI：10.1109/cvpr.2019.00821
4 MCCARTNEY E J.Optics of the Atmosphere：Scattering by Molecules and Particles［M］.New York：John Wiley and Sons，Inc，1976.
5 NARASIMHAN S G，NAYAR S K. Vision and the atmosphere［J］. International Journal of Computer Vision，2002，48（3）：233-254.
6 孙小明，孙俊喜，赵立荣，等. 暗原色先验单幅图像去雾改进算法［J］. 中国图像图形学报，2014，19（3）：381-385. SUN X M，SUN J X，ZHAO L R，et al. Improved algorithm for single image haze removing using dark channel prior［J］. Journal of Image and Graphics，2014，19（3）：381-385.
7 TAN R T.Visibility in bad weather from a single image［C］//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Alaska：IEEE，2008：1-8. DOI：10.1109/cvpr.2008.4587643
8 ZHU Q S，MAI J M，SHAO L. A fast single image haze removal algorithm using color attenuation prior［J］. IEEE Transactions on Image Processing，2015，24（11）：3522-3533. DOI：10.1109/tip.2015. 2446191
9 TANG K T，YANG J C，WANG J. Investigating haze-relevant features in a learning framework for image dehazing［C］//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Columbus：IEEE，2014：2995-3000. DOI：10.1109/cvpr.2014.383
10 CAI B L，XU X M，JIA K，et al. DehazeNet：An end-to-end system for single image haze removal［J］. IEEE Transactions on Image Processing，2016，25（11）：5187-5198. DOI：10.1109/tip.2016.2598681
11 REN W Q，LIU S，ZHANG H，et al. Single image dehazing via multi-scale convolutional neural networks［C］//Proceedings of the European Conference on Computer Vision. Switzerland：Springer，Cham，2016：154-169. DOI：10.1007/978-3-319-46475-6_10
12 LI B Y，PENG X L，WANG Z Y，et al. AOD-Net：All-in-one dehazing network［C］//Proceedings of the IEEE International Conference on Computer Vision.Venice：IEEE，2017：4770-4778. DOI：10.1109/ICCV.2017.511
13 REN W Q，MA L，ZHANG J W，et al. Gated fusion network for single image dehazing［C］//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Salt Lake City：IEEE，2018：3253-3261. DOI：10.1109/cvpr.2018.00343
14 KANG L W，LIN C W，FU Y H. Automatic single-image-based rain streaks removal via image decomposition［J］.IEEE Transactions on Image Processing，2011，21（4）：1742-1755. DOI：10.1109/TIP.2011.2179057
15 LUO Y，XU Y，JI H.Removing rain from a single image via discriminative sparse coding［C］//Proceedings of the IEEE International Conference on Computer Vision. Santiago：IEEE，2015：3397-3405. DOI：10.1109/iccv.2015.388
16 LI Y，TAN R T，GUO X，et al. Rain streak removal using layer priors［C］//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Salt Lake City：IEEE，2016：2736-2744. DOI：10.1109/cvpr.2016.299
17 ZHU L，FU C W，LISCHINSKI D，et al. Joint bi-layer optimization for single-image rain streak removal［C］//Proceedings of the IEEE International Conference on Computer Vision. Hawaii：IEEE，2017：2526-2534. DOI：10.1109/iccv.2017.276
18 ZHANG H，PATEL V M. Convolutional sparse and low-rank coding-based rain streak removal［C］//Proceedings of the IEEE Winter Conference on Applications of Computer Vision. Santa Rosa：IEEE，2017：1259-1267. DOI：10.1109/wacv. 2017.145
19 FU X Y，HUANG J B，ZENG D L，et al. Removing rain from single images via a deep detail network ［C］// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Honolulu：IEEE，2017：3855-3863. DOI：10.1109/cvpr.2017.186
20 ZHANG H，SINDAGI V，PATEL V M. Image de-raining using a conditional generative adversarial network［J］.IEEE Transactions on Circuits and Systems for Video Technology，2020，30（11）：3943-3956. DOI：10.1109/TCSVT.2019.2920407
21 YANG W H，TAN R T，FENG J S，et al. Joint rain detection and removal via iterative region dependent multi-task learning［EB/OL］. （2016-09-23）. https：//arxiv.org/abs/1609.07769v1
22 LI X，WU J L，LIN Z C，et al. Recurrent squeeze-and-excitation context aggregation net for single image deraining［C］//Proceedings of the European Conference on Computer Vision. Munich：ECCV，2018：254-269. DOI：10.1007/978-3-030-01234-2_16
23 ZHANG H，PATEL V M. Density-aware single image de-raining using a multi-stream dense network［C］//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Salt Lake City：IEEE，2018：695-704. DOI： 10.1109/cvpr.2018.00079
24 FAN Z W，WU H F，FU X Y，et al. Residual-guide network for single image deraining［C］//Proceedings of the 26th ACM International Conference on Multimedia.Seoul：ACM，2018：1751-1759. DOI： 10.1145/3240508.3240694
25 LIU X，SUGANUMA M，SUN Z，et al. Dual residual networks leveraging the potential of paired operations for image restoration［C］//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Long Beach：IEEE，2019：7007-7016. DOI：10.1109/cvpr.2019.00717
26 LI R，CHEONG L F，TAN R T. Heavy rain image restoration：Integrating physics model and conditional adversarial learning［C］//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Long Beach：IEEE，2019：1633-1642. DOI：10.1109/cvpr.2019.00173
27 WANG T，YANG X，XU K，et al. Spatial attentive single-image deraining with a high quality real rain dataset［C］//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Long Beach：IEEE，2019：12270-12279. DOI：10.1109/cvpr.2019.01255
28 WEI W，MENG D，ZHAO Q，et al. Semi-supervised transfer learning for image rain removal ［C］//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Long Beach：IEEE，2019：3877-3886. DOI：10.1109/cvpr.2019.00400
29 WANG G，SUN C，SOWMYA A.ERL-Net：Entangled representation learning for single image de-raining［C］//Proceedings of the IEEE International Conference on Computer Vision. Seoul：IEEE，2019：5644-5652. DOI： 10.1109/iccv.2019.00574
30 YANG W，TAN R T，WANG S，et al. Single image deraining：From model-based to data-driven and beyond［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence，2020：1. DOI：10.1109/TPAVI.2020.2985190
31 GOODFELLOW I，POUGET-ABADIE J，MIRZA M，et al. Generative Adversarial Nets［C］//Proceedings of the 27th International Conference on Neural Information Processing Systems. Cambridge：MIT Press 2014：2672-2680. DOI：10. 1145/3422622
32 ISOLA P，ZHU J Y，ZHOU T，et al. Image-to-image translation with conditional adversarial networks［C］//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Hawaii：IEEE，2017：1125-1134. DOI：10.1109/cvpr.2017.632
33 QU Y Y，CHEN Y Z，HUANG J Y，et al. Enhanced pix2pix dehazing network［C］//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Long Beach：IEEE，2019：8160-8168. DOI：10.1109/cvpr.2019.00835
34 HE K M，ZHANG X Y，REN S Q，et al. Deep residual learning for image recognition［C］//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Las Vegas：IEEE，2016：770-778. DOI：10.1109/cvpr.2016.90
35 ZHU J Y，PARK T，ISOLA P，et al. Unpaired image-to-image translation using cycle-consistent adversarial networks［C］//Proceedings of the IEEE International Conference on Computer Vision.Venice：IEEE，2017：2223-2232. DOI：10.1109/iccv.2017.244
36 LI C，WAND M. Precomputed real-time texture synthesis with markovian generative adversarial networks［C］//Proceedings of the European Conference on Computer Vision. Switzerland：Springer，Cham，2016：702-716. DOI：10.1007/978-3-319-46487-9_43
37 YU J，LIN Z，YANG J，et al. Generative image inpainting with contextual attention［C］//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Salt Lake City：IEEE，2018：5505-5514. DOI：10.1109/cvpr.2018.00577
38 ZHANG Q，SHEN X Y，XU L，et al. Rolling guidance filter［C］//Proceedings of the European Conference on Computer Vision. Switerland：Springer，Cham，2014：815-830. DOI：10.1007/978-3-319-10578-9_53
39 WOFK D，MA F，YANG T J，et al. FastDepth：Fast monocular depth estimation on embedded systems［C］//Proceedings of the International Conference on Robotics and Automation. Montreal：IEEE，2019：6101-6108. DOI：10.1109/icra.2019. 8794182
40 HUANG D A，KANG L W，YANG M C，et al.Context-aware single image rain removal［C］//Proceedings of the IEEE International Conference on Multimedia and Expo. Melbourne：IEEE，2012：164-169. DOI：10.1109/icme.2012.92
41 RONNEBERGER O，FISCHER P，BROX T. U-Net：Convolutional networks for biomedical image segmentation［C］//Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention. Switerland：Springer，Cham，2015：234-241. DOI：10.1007/978-3-319-24574-4_28
42 BROOKS A C，ZHAO X，PAPPAS T N. Structural similarity quality metrics in a coding context：Exploring the space of realistic distortions［J］. IEEE Transactions on Image Processing，2008，17（8）：1261-1273. DOI：10.1109/tip.2008.926161
43 LIU G，REDA F A，SHIH K J，et al. Image inpainting for irregular holes using partial convolutions［C］//Proceedings of the European Conference on Computer Vision. Munich：ECCV，2018：85-100. DOI：10.1007/978-3-030-01252-6_6 |
|
Viewed |
|
|
|
Full text
|
|
|
|
|
Abstract
|
|
|
|
|
Cited |
|
|
|
|
| |
Shared |
|
|
|
|
| |
Discussed |
|
|
|
|
