Please wait a minute...
浙江大学学报（理学版）
浙江大学学报（理学版）  2021, Vol. 48 Issue (3): 270-281    DOI: 10.3785/j.issn.1008-9497.2021.03.002
图像处理算法     
基于先验知识的单幅图像雨雾去除方法
梁楚萍1,2, 冯一箪1,2, 谢浩然3, 魏明强1,2, 燕雪峰1
1.南京航空航天大学 计算机科学与技术学院,江苏 南京 210016
2.模式分析与机器智能工业和信息化部 重点实验室,江苏 南京 210016
3.香港岭南大学 电脑及决策科学学系,中国 香港 999077
Prior-based single image rain and haze removal
LIANG Chuping1,2, FENG Yidan1,2, XIE Haoran3, WEI Mingqiang1,2, YAN Xuefeng1
1.School of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, Nanjing 210016,China
2.MIIT Key Laboratory of Pattern Analysis and Machine Intelligence, Nanjing 210016, China
3.Department of Computer and Decision Sciences, Lingnan University, HongKong 999077, China
 全文: PDF(7499 KB)   HTML  
摘要： 单幅图像雨雾去除是计算机视觉研究的热点之一，现有的去雨方法往往需根据雨的特性恢复图像，忽视了场景中的雾以及场景深处雨累积的影响。提出了一种新型单幅图像去雨雾方法，设计了一种基于先验知识的三阶段单幅图像去雨雾框架。首先，将深度图作为图像透射率的引导，利用暗通道先验知识对输入图像的低频部分去雾，然后，采用残差网络学习高频雨痕特征，最后，引入条件生成对抗网络（conditional generative adversarial network，cGAN），对图像局部细节进行精细化修复，cGAN对图像的空间信息和纹理细节更为敏感，能有效恢复去雨雾过程中丢失或未能填补的细节。分别用高质量的合成雨雾数据集和真实数据集进行实验测试，结果表明，本文方法在峰值信噪比和结构相似度上均较基准方法优，能将雾浓度各异和雨分布变化情况下的图像恢复至细节丰富的干净场景图。
关键词： 单幅图像去雨雾;  暗通道先验;  残差网络;  条件生成对抗网络    
Abstract: Single image rain and haze removal is an important task in computer vision. Most state-of-the-art methods focus on the features of rain while ignoring the disturbance from haze and rain accumulation.To solve this problem,this paper proposes a novel strategy of rain and haze removal from a single image and develops a three-stage framework.First,only the low-frequency of the input image is dehazed based on dark channel prior,where a depth map is used as guidance of transmission to improve the result; Then the streaks in high-frequency can be learned through a residual network; At last,local details are refined by a conditional generative adversarial network (cGAN).A high-quality synthetic dataset and real photo data are used in testing.Experiment results show that this framework has better performance on rainy image restoration than baseline,by comparing the value of PSNR and SSIM with ground-truth images. Our method is robust in restoring clean scene from images with various haze density or rain distribution.
Key words: single image rain and haze removal    dark channel prior    residual network    conditional generative adversarial network (cGAN)
收稿日期: 2020-09-23 出版日期: 2021-05-20
CLC:  TP 391  
基金资助: 国家自然科学基金资助项目（62032011）；中央高校基本科研业务费专项资金项目(NJ2019010).
通讯作者: ORCID://https://orcid.org/0000-0003-0429-490X,E-mail:mqwei@nuaa.edu.cn.     E-mail: mqwei@nuaa.edu.cn
作者简介: 梁楚萍(1998—),ORCID://https://orcid.org/0000-0003-4079-387X,女,硕士研究生,主要从事计算机视觉研究,E-mail:liangcp@nuaa.edu.c；
服务　　
把本文推荐给朋友
加入引用管理器
E-mail Alert
RSS
作者相关文章　　
梁楚萍
冯一箪
谢浩然
魏明强
燕雪峰

引用本文:

梁楚萍, 冯一箪, 谢浩然, 魏明强, 燕雪峰. 基于先验知识的单幅图像雨雾去除方法[J]. 浙江大学学报（理学版）, 2021, 48(3): 270-281.

LIANG Chuping, FENG Yidan, XIE Haoran, WEI Mingqiang, YAN Xuefeng. Prior-based single image rain and haze removal. Journal of Zhejiang University (Science Edition), 2021, 48(3): 270-281.

链接本文:

https://www.zjujournals.com/sci/CN/10.3785/j.issn.1008-9497.2021.03.002    或    https://www.zjujournals.com/sci/CN/Y2021/V48/I3/270

1 HE K，SUN J，TANG X，et al. Single image haze removal using dark channel prior［J］.IEEE Transactions on Pattern Analysis & Machine Intelligence，2011，33（12）：2341-2353. DOI：10.1109/tpami.2010.168
2 GARG K，NAYAR S K.Vision and rain［J］.International Journal of Computer Vision，2007，75（1）：3-27. DOI：10.1007/s11263-006-0028-6
3 HU X，FU C W，ZHU L，et al.Depth-attentional features for single-image rain removal［C］//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Long Beach：IEEE，2019：8022-8031. DOI：10.1109/cvpr.2019.00821
4 MCCARTNEY E J.Optics of the Atmosphere：Scattering by Molecules and Particles［M］.New York：John Wiley and Sons，Inc，1976.
5 NARASIMHAN S G，NAYAR S K. Vision and the atmosphere［J］. International Journal of Computer Vision，2002，48（3）：233-254.
6 孙小明，孙俊喜，赵立荣，等. 暗原色先验单幅图像去雾改进算法［J］. 中国图像图形学报，2014，19（3）：381-385. SUN X M，SUN J X，ZHAO L R，et al. Improved algorithm for single image haze removing using dark channel prior［J］. Journal of Image and Graphics，2014，19（3）：381-385.
7 TAN R T.Visibility in bad weather from a single image［C］//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Alaska：IEEE，2008：1-8. DOI：10.1109/cvpr.2008.4587643
8 ZHU Q S，MAI J M，SHAO L. A fast single image haze removal algorithm using color attenuation prior［J］. IEEE Transactions on Image Processing，2015，24（11）：3522-3533. DOI：10.1109/tip.2015. 2446191
9 TANG K T，YANG J C，WANG J. Investigating haze-relevant features in a learning framework for image dehazing［C］//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Columbus：IEEE，2014：2995-3000. DOI：10.1109/cvpr.2014.383
10 CAI B L，XU X M，JIA K，et al. DehazeNet：An end-to-end system for single image haze removal［J］. IEEE Transactions on Image Processing，2016，25（11）：5187-5198. DOI：10.1109/tip.2016.2598681
11 REN W Q，LIU S，ZHANG H，et al. Single image dehazing via multi-scale convolutional neural networks［C］//Proceedings of the European Conference on Computer Vision. Switzerland：Springer，Cham，2016：154-169. DOI：10.1007/978-3-319-46475-6_10
12 LI B Y，PENG X L，WANG Z Y，et al. AOD-Net：All-in-one dehazing network［C］//Proceedings of the IEEE International Conference on Computer Vision.Venice：IEEE，2017：4770-4778. DOI：10.1109/ICCV.2017.511
13 REN W Q，MA L，ZHANG J W，et al. Gated fusion network for single image dehazing［C］//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Salt Lake City：IEEE，2018：3253-3261. DOI：10.1109/cvpr.2018.00343
14 KANG L W，LIN C W，FU Y H. Automatic single-image-based rain streaks removal via image decomposition［J］.IEEE Transactions on Image Processing，2011，21（4）：1742-1755. DOI：10.1109/TIP.2011.2179057
15 LUO Y，XU Y，JI H.Removing rain from a single image via discriminative sparse coding［C］//Proceedings of the IEEE International Conference on Computer Vision. Santiago：IEEE，2015：3397-3405. DOI：10.1109/iccv.2015.388
16 LI Y，TAN R T，GUO X，et al. Rain streak removal using layer priors［C］//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Salt Lake City：IEEE，2016：2736-2744. DOI：10.1109/cvpr.2016.299
17 ZHU L，FU C W，LISCHINSKI D，et al. Joint bi-layer optimization for single-image rain streak removal［C］//Proceedings of the IEEE International Conference on Computer Vision. Hawaii：IEEE，2017：2526-2534. DOI：10.1109/iccv.2017.276
18 ZHANG H，PATEL V M. Convolutional sparse and low-rank coding-based rain streak removal［C］//Proceedings of the IEEE Winter Conference on Applications of Computer Vision. Santa Rosa：IEEE，2017：1259-1267. DOI：10.1109/wacv. 2017.145
19 FU X Y，HUANG J B，ZENG D L，et al. Removing rain from single images via a deep detail network ［C］// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Honolulu：IEEE，2017：3855-3863. DOI：10.1109/cvpr.2017.186
20 ZHANG H，SINDAGI V，PATEL V M. Image de-raining using a conditional generative adversarial network［J］.IEEE Transactions on Circuits and Systems for Video Technology，2020，30（11）：3943-3956. DOI：10.1109/TCSVT.2019.2920407
21 YANG W H，TAN R T，FENG J S，et al. Joint rain detection and removal via iterative region dependent multi-task learning［EB/OL］. （2016-09-23）. https：//arxiv.org/abs/1609.07769v1
22 LI X，WU J L，LIN Z C，et al. Recurrent squeeze-and-excitation context aggregation net for single image deraining［C］//Proceedings of the European Conference on Computer Vision. Munich：ECCV，2018：254-269. DOI：10.1007/978-3-030-01234-2_16
23 ZHANG H，PATEL V M. Density-aware single image de-raining using a multi-stream dense network［C］//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Salt Lake City：IEEE，2018：695-704. DOI： 10.1109/cvpr.2018.00079
24 FAN Z W，WU H F，FU X Y，et al. Residual-guide network for single image deraining［C］//Proceedings of the 26th ACM International Conference on Multimedia.Seoul：ACM，2018：1751-1759. DOI： 10.1145/3240508.3240694
25 LIU X，SUGANUMA M，SUN Z，et al. Dual residual networks leveraging the potential of paired operations for image restoration［C］//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Long Beach：IEEE，2019：7007-7016. DOI：10.1109/cvpr.2019.00717
26 LI R，CHEONG L F，TAN R T. Heavy rain image restoration：Integrating physics model and conditional adversarial learning［C］//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Long Beach：IEEE，2019：1633-1642. DOI：10.1109/cvpr.2019.00173
27 WANG T，YANG X，XU K，et al. Spatial attentive single-image deraining with a high quality real rain dataset［C］//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Long Beach：IEEE，2019：12270-12279. DOI：10.1109/cvpr.2019.01255
28 WEI W，MENG D，ZHAO Q，et al. Semi-supervised transfer learning for image rain removal ［C］//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Long Beach：IEEE，2019：3877-3886. DOI：10.1109/cvpr.2019.00400
29 WANG G，SUN C，SOWMYA A.ERL-Net：Entangled representation learning for single image de-raining［C］//Proceedings of the IEEE International Conference on Computer Vision. Seoul：IEEE，2019：5644-5652. DOI： 10.1109/iccv.2019.00574
30 YANG W，TAN R T，WANG S，et al. Single image deraining：From model-based to data-driven and beyond［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence，2020：1. DOI：10.1109/TPAVI.2020.2985190
31 GOODFELLOW I，POUGET-ABADIE J，MIRZA M，et al. Generative Adversarial Nets［C］//Proceedings of the 27th International Conference on Neural Information Processing Systems. Cambridge：MIT Press 2014：2672-2680. DOI：10. 1145/3422622
32 ISOLA P，ZHU J Y，ZHOU T，et al. Image-to-image translation with conditional adversarial networks［C］//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Hawaii：IEEE，2017：1125-1134. DOI：10.1109/cvpr.2017.632
33 QU Y Y，CHEN Y Z，HUANG J Y，et al. Enhanced pix2pix dehazing network［C］//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Long Beach：IEEE，2019：8160-8168. DOI：10.1109/cvpr.2019.00835
34 HE K M，ZHANG X Y，REN S Q，et al. Deep residual learning for image recognition［C］//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Las Vegas：IEEE，2016：770-778. DOI：10.1109/cvpr.2016.90
35 ZHU J Y，PARK T，ISOLA P，et al. Unpaired image-to-image translation using cycle-consistent adversarial networks［C］//Proceedings of the IEEE International Conference on Computer Vision.Venice：IEEE，2017：2223-2232. DOI：10.1109/iccv.2017.244
36 LI C，WAND M. Precomputed real-time texture synthesis with markovian generative adversarial networks［C］//Proceedings of the European Conference on Computer Vision. Switzerland：Springer，Cham，2016：702-716. DOI：10.1007/978-3-319-46487-9_43
37 YU J，LIN Z，YANG J，et al. Generative image inpainting with contextual attention［C］//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Salt Lake City：IEEE，2018：5505-5514. DOI：10.1109/cvpr.2018.00577
38 ZHANG Q，SHEN X Y，XU L，et al. Rolling guidance filter［C］//Proceedings of the European Conference on Computer Vision. Switerland：Springer，Cham，2014：815-830. DOI：10.1007/978-3-319-10578-9_53
39 WOFK D，MA F，YANG T J，et al. FastDepth：Fast monocular depth estimation on embedded systems［C］//Proceedings of the International Conference on Robotics and Automation. Montreal：IEEE，2019：6101-6108. DOI：10.1109/icra.2019. 8794182
40 HUANG D A，KANG L W，YANG M C，et al.Context-aware single image rain removal［C］//Proceedings of the IEEE International Conference on Multimedia and Expo. Melbourne：IEEE，2012：164-169. DOI：10.1109/icme.2012.92
41 RONNEBERGER O，FISCHER P，BROX T. U-Net：Convolutional networks for biomedical image segmentation［C］//Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention. Switerland：Springer，Cham，2015：234-241. DOI：10.1007/978-3-319-24574-4_28
42 BROOKS A C，ZHAO X，PAPPAS T N. Structural similarity quality metrics in a coding context：Exploring the space of realistic distortions［J］. IEEE Transactions on Image Processing，2008，17（8）：1261-1273. DOI：10.1109/tip.2008.926161
43 LIU G，REDA F A，SHIH K J，et al. Image inpainting for irregular holes using partial convolutions［C］//Proceedings of the European Conference on Computer Vision. Munich：ECCV，2018：85-100. DOI：10.1007/978-3-030-01252-6_6
[1] 谭晓东,赵奇,文明珠,王小超. 基于BEMD、DCT和SVD的混合图像水印算法[J]. 浙江大学学报（理学版）, 2023, 50(4): 442-454.
[2] 方于华,叶枫. MFDC-Net：一种融合多尺度特征和注意力机制的乳腺癌病理图像分类算法[J]. 浙江大学学报（理学版）, 2023, 50(4): 455-464.
[3] 孔翔,陈军. 一类带4个形状参数的同次三角曲面构造算法[J]. 浙江大学学报（理学版）, 2023, 50(2): 153-159.
[4] 张远鹏, 陈鸿韬, 王伟娜. 基于非凸非光滑变分模型的灰度图像泊松噪声移除算法[J]. 浙江大学学报（理学版）, 2023, 50(2): 160-166.
[5] 李军成,刘成志,罗志军,龙志文. 空间参数曲线的双目标能量极小化方法及其应用[J]. 浙江大学学报（理学版）, 2023, 50(1): 63-68.
[6] 全浩荣,刘成志,李军成,杨炼,胡丽娟. 张量积型Said-Ball曲面的预处理渐近迭代逼近法[J]. 浙江大学学报（理学版）, 2022, 49(6): 682-690.
[7] 虞瑞麒,刘玉华,沈禧龙,翟如钰,张翔,周志光. 表征学习驱动的多重网络图采样[J]. 浙江大学学报（理学版）, 2022, 49(3): 271-279.
[8] 律睿慜,张陶洁,席旭,王濛濛,孟磊,张克俊. 笔法与结构对楷书文字美学品质影响的量化研究[J]. 浙江大学学报（理学版）, 2022, 49(3): 261-270.
[9] 钟颖,王松,吴浩,程泽鹏,李学俊. 基于SEMMA的网络安全事件可视探索[J]. 浙江大学学报（理学版）, 2022, 49(2): 131-140.
[10] 祝锦泰, 叶继华, 郭凤, 江蕗, 江爱文. FSAGN： 一种自主选择关键帧的表情识别方法[J]. 浙江大学学报（理学版）, 2022, 49(2): 141-150.
[11] 朱强,王超毅,张吉庆,尹宝才,魏小鹏,杨鑫. 基于事件相机的无人机目标跟踪算法[J]. 浙江大学学报（理学版）, 2022, 49(1): 10-18.
[12] 杨猛,丁曙,马云涛,谢佳翊,段瑞枫. 基于纹理特征的小麦锈病动态模拟方法[J]. 浙江大学学报（理学版）, 2022, 49(1): 1-9.
[13] 傅汝佳, 冼楚华, 李桂清, 万隽杰, 曹铖, 杨存义, 高月芳. 面向表型精确鉴定的豆株快速三维重建[J]. 浙江大学学报（理学版）, 2021, 48(5): 531-539.
[14] 余鹏, 刘兰, 蔡韵, 何煜, 张松海. 基于单目摄像头的自主健身监测系统[J]. 浙江大学学报（理学版）, 2021, 48(5): 521-530.
[15] 林俊聪, 陈萌, 施渝斌, 雷钧, 郭诗辉, 高星, 廖明宏, 金小刚. 面向高级定制的个性化虚拟服装展示[J]. 浙江大学学报（理学版）, 2021, 48(4): 418-426.