Please wait a minute...
浙江大学学报(工学版)
浙江大学学报(工学版)  2023, Vol. 57 Issue (12): 2456-2466    DOI: 10.3785/j.issn.1008-973X.2023.12.013
计算机技术     
基于特征融合和一致性损失的双目低光照增强
廖嘉文1(),庞彦伟1,2,*(),聂晶1,3,孙汉卿1,曹家乐1
1. 天津大学 电气自动化与信息工程学院，天津 300072
2. 上海人工智能实验室，上海 200232
3. 重庆大学 微电子与通信工程学院，重庆 401331
Stereo low-light enhancement based on feature fusion and consistency loss
Jia-wen LIAO1(),Yan-wei PANG1,2,*(),Jing NIE1,3,Han-qing SUN1,Jia-le CAO1
1. School of Electrical and Information Engineering, Tianjin University, Tianjin 300072, China
2. Shanghai Artificial Intelligence Laboratory, Shanghai 200232, China
3. School of Microelectronics and Communication Engineering, Chongqing University, Chongqing 401331, China
 全文: PDF(2642 KB)   HTML
摘要：

构建大规模真实场景的双目低光照图像数据集SLL10K. 该数据集包含12 658对无参考双目低照度图像和205对有参考双目图像，数据集图像涵盖丰富的光照、时间及场景. 提出基于特征融合和一致性损失的双目低光照图像增强网络FCNet，特征融合模块用于充分融合单目内和双目间的特征，一致性损失函数用于保持增强前和增强后图像间的一致性. 在SLL10K数据集和暗化KITTI数据集上的实验表明，使用FCNet增强后的图像获得了比单目增强方法更好的低光照图像增强效果和目标检测效果.
关键词： 图像增强;  低光照;  双目数据集;  无参考图像增强;  特征融合    
Abstract:

A large scale real scene stereo low-light image dataset SLL10K was proposed. There were 12 658 pairs of unreferenced stereo low-illumination images and 205 pairs of referenced stereo images contained in the dataset. The images in the SLL10K dataset cover a wealth of lighting, time, and scene. FCNet, a stereo low-light image enhancement network based on feature fusion and consistency loss was proposed. The feature fusion module was used to fully integrate intra-monocular and inter-stereo features, and the consistency loss function was used to maintain the consistency between images before and after enhancement. Experiments on the SLL10K dataset and darkening KITTI dataset show that the images with FCNet obtain better performance on low-light image enhancement and object detection than the monocular enhancement method.
Key words: image enhancement    low-light    stereo dataset    no-reference image enhancement    feature fusion
收稿日期: 2023-02-27 出版日期: 2023-12-27
CLC:  TP 391.4  
基金资助: 国家科技创新2030新一代人工智能重大项目（2022ZD0160400）；国家自然科学基金资助项目（62271346）
通讯作者: 庞彦伟     E-mail: gavin971209@tju.edu.cn;pyw@tju.edu.cn
作者简介: 廖嘉文（1997—），男，硕士生，从事图像处理、低光照图像增强研究. orcid.org/0009-0002-1419-9404. E-mail： gavin971209@tju.edu.cn
服务　　
把本文推荐给朋友
加入引用管理器
E-mail Alert
作者相关文章　　
廖嘉文
庞彦伟
聂晶
孙汉卿
曹家乐

引用本文:

廖嘉文,庞彦伟,聂晶,孙汉卿,曹家乐. 基于特征融合和一致性损失的双目低光照增强[J]. 浙江大学学报(工学版), 2023, 57(12): 2456-2466.

Jia-wen LIAO,Yan-wei PANG,Jing NIE,Han-qing SUN,Jia-le CAO. Stereo low-light enhancement based on feature fusion and consistency loss. Journal of ZheJiang University (Engineering Science), 2023, 57(12): 2456-2466.

链接本文:

https://www.zjujournals.com/eng/CN/10.3785/j.issn.1008-973X.2023.12.013    或    https://www.zjujournals.com/eng/CN/Y2023/V57/I12/2456

图 1  SLL10K的代表性图像
图 2  SLL10K室外场景的训练集和测试集图像对样例
图 3  SLL10K室内场景的测试集低光照图像对和参考图像对样例
图 4  FCNet的结构图
图 5  双目内外特征融合模块的结构图
测试集 SIIFF SC损失 BRISQUE NIQE PIQE LOE PSNR/dB SSIM LPIPS
室外 × × 25.684 8 2.832 7 32.569 0 807.7 — — —
√ × 23.546 3 2.687 0 30.476 3 696.3 — — —
√ √ 23.095 1 2.641 1 28.111 5 679.2 — — —
室内 × × 27.663 3 3.694 9 38.553 7 2 738.0 11.230 5 0.360 5 0.633 5
√ × 24.973 8 3.500 8 34.091 6 2 628.4 11.224 1 0.375 7 0.623 4
√ √ 22.531 7 3.390 1 31.279 4 2 593.4 11.222 1 0.406 2 0.609 2
表 1  FCNet的不同模块在2个测试集上的消融实验
测试集 nm BRISQUE NIQE PIQE LOE PSNR/dB SSIM LPIPS
室外 1 23.310 1 2.651 0 28.553 6 688.8 — — —
2 23.095 1 2.641 1 28.111 5 679.2 — — —
室内 1 22.656 7 3.414 7 32.062 5 2 609.4 11.119 0 0.363 7 0.639 8
2 22.531 7 3.390 1 31.279 4 2 593.4 11.222 1 0.406 2 0.609 2
表 2  FCNet的模块数量变化在2个测试集上的消融实验
测试集 卷积类别 BRISQUE NIQE PIQE LOE PSNR/dB SSIM LPIPS
室外 3×3 23.732 9 3.253 5 31.648 9 753.7 — — —
1×9和9×1 23.095 1 2.641 1 28.111 5 679.2 — — —
室内 3×3 24.625 2 3.816 4 41.876 4 2 655.2 11.068 3 0.317 7 0.651 6
1×9和9×1 22.531 7 3.390 1 31.279 4 2 593.4 11.222 1 0.406 2 0.609 2
表 3  模块卷积变化在2个测试集上的消融实验
方法 左目 右目
BRISQUE NIQE PIQE LOE BRISQUE NIQE PIQE LOE
RetinexNet[11] 24.640 7 4.257 3 36.454 5 1 741.5 24.420 5 4.270 1 36.276 1 1752.7
ISSR[24] 28.392 8 2.815 1 27.923 2 753.0 28.903 5 2.810 7 28.525 2 724.2
GLAD[25] 23.628 8 3.280 8 28.633 4 590.8 23.729 0 3.268 0 28.310 2 584.8
DVENet[16] 23.100 2 2.931 7 28.175 1 791.9 22.706 7 2.872 3 27.531 7 740.2
ZeroDCE++[17] 25.684 8 2.832 7 32.569 0 807.7 25.560 2 2.857 2 32.118 0 814.4
RUAS[26] 29.713 1 3.746 6 30.512 2 2 520.0 29.736 9 3.675 5 30.863 7 2 300.6
FCNet 23.095 1 2.641 1 28.111 5 679.2 22.689 8 2.632 3 27.374 5 669.9
表 4  不同图像增强方法在SLL10K室外数据集上的指标对比
方法 左目 右目
BRISQUE NIQE PIQE LOE PSNR SSIM LPIPS BRISQUE NIQE PIQE LOE PSNR SSIM LPIPS
RetinexNet[11] 34.042 9 5.592 2 49.155 6 3 241.0 11.640 2 0.222 1 0.812 2 33.562 7 5.596 0 48.305 5 3 102.3 11.211 8 0.238 0 0.798 2
ISSR[24] 23.900 7 2.751 8 29.193 1 2 599.6 8.858 8 0.258 8 0.674 5 24.626 0 2.950 9 28.848 0 2 577.8 8.199 8 0.261 9 0.660 3
GLAD[25] 23.139 6 3.673 4 42.574 7 2 547.2 12.875 1 0.229 0 0.666 0 25.854 5 3.581 8 41.646 0 2 515.4 12.174 0 0.247 5 0.648 3
DVENet[16] 23.075 7 3.405 7 32.115 1 2 595.8 9.026 0 0.246 0 0.656 9 22.189 2 3.391 6 30.236 3 2 543.2 8.595 1 0.248 9 0.643 7
ZeroDCE++[17] 27.663 3 3.694 9 38.553 7 2 738.0 11.230 5 0.360 5 0.725 4 27.262 2 3.649 7 38.050 0 2 671.1 10.410 9 0.363 1 0.713 0
RUAS[26] 24.693 4 3.361 7 35.548 8 2 671.3 9.864 7 0.380 4 0.710 2 24.375 1 3.305 0 33.077 0 2 581.7 9.234 6 0.369 1 0.693 1
FCNet 22.531 7 3.390 1 31.279 4 2 593.4 11.222 1 0.406 2 0.609 2 21.273 6 3.329 7 29.109 7 2 539.9 10.407 3 0.407 3 0.598 0
表 5  不同图像增强方法在SLL10K室内数据集上的指标对比
图 6  不同图像增强方法对4种场景图像的处理效果对比
图 7  室内图像单、双目增强效果对比
%
方法 $ {\mathrm{A}\mathrm{P}}_{{\rm{2d}}} $ $ {\mathrm{A}\mathrm{P}}_{{\rm{ori}}} $ $ {\mathrm{A}\mathrm{P}}_{{\rm{bev}}} $ $ {\mathrm{A}\mathrm{P}}_{{\rm{3d}}} $
原始 88.86 86.90 64.54 50.49
低光照 70.48 68.14 48.16 34.43
RetinexNet[11] 69.70 67.71 48.99 36.21
ISSR[23] 70.73 68.12 46.20 33.45
GLAD[24] 88.30 86.12 57.23 45.05
ZeroDCE++[17] 88.23 86.03 57.12 44.98
RUAS[25] 87.81 84.24 55.27 42.32
FCNet 88.51 86.55 57.72 45.45
表 6  不同图像增强方法的目标检测结果
图 8  单、双目图像增强方法的视差图对比
1 WANG Y, WAN R, YANG W, et al. Low-light image enhancement with normalizing flow [C]// Proceedings of the AAAI Conference on Artificial Intelligence. [S.l.]: AAAI, 2022: 2604-2612.
2 LV F, LI Y, LU F Attention guided low-light image enhancement with a large scale low-light simulation dataset[J]. International Journal of Computer Vision, 2021, 129: 2175- 2193
doi: 10.1007/s11263-021-01466-8
3 WANG L, LIU Z, SIU W, et al Lightening network for low-light image enhancement[J]. IEEE Transactions on Image Processing, 2020, 29: 7984- 7996
doi: 10.1109/TIP.2020.3008396
4 ZHOU S, LI C, LOY C C. LEDNet: joint low-light enhancement and deblurring in the dark [C]// Proceedings of the European Conference on Computer Vision. [S.l.]: Springer, 2022: 573-589.
5 ZHENG C, SHI D, SHI W. Adaptive unfolding total variation network for low-light image enhancement [C]// Proceedings of the IEEE/CVF International Conference on Computer Vision. Montreal: IEEE, 2021: 4439-4448.
6 WU W, WENG J, ZHANG P, et al. URetinex-Net: retinex-based deep unfolding network for low-light image enhancement [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans: IEEE, 2022: 5901-5910.
7 HAO S, HAN X, GUO Y, et al Low-light image enhancement with semi-decoupled decomposition[J]. IEEE Transactions on Multimedia, 2020, 22 (12): 3025- 3038
doi: 10.1109/TMM.2020.2969790
8 GUO C, LI C, GUO J, et al. Zero-reference deep curve estimation for low-light image enhancement [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020: 1777–1786.
9 MA L, MA T, LIU R, et al. Toward fast, flexible, and robust low-light image enhancement [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans: IEEE, 2022: 5627-5636.
10 ZHANG Z, ZHENG H, HONG R, et al. Deep color consistent network for low-light image enhancement [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans: IEEE, 2022: 1889-1898.
11 WEI C, WANG W, YANG W, et al. Deep retinex decomposition for low-light enhancement [C]// Proceedings of the 29th British Machine Vision Conference. Newcastle Upon Tyne: [s.n.], 2018.
12 CHEN C, CHEN Q, XU J, et al. Learning to see in the dark [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 3291-3300.
13 PANG Y, NIE J, XIE J, et al. BidNet: binocular image dehazing without explicit disparity estimation [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020: 5931-5940.
14 ZHOU S, ZHANG J, ZUO W, et al. DAVANet: stereo deblurring with view aggregation [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2019: 10996-11005.
15 ZHANG K, LUO W, REN W, et al. Beyond monocular deraining: stereo image deraining via semantic understanding [C]// Proceedings of the European Conference on Computer Vision. [S.l.]: Springer, 2020: 71-89.
16 HUANG J, FU X, XIAO Z, et al Low-light stereo image enhancement[J]. IEEE Transactions on Multimedia, 2022, 25: 2978- 2992
17 LI C, GUO C, LOY C C Learning to enhance low-light image via zero-reference deep curve estimation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44 (8): 4225- 4238
18 WANG L, WANG Y, LIANG Z, et al. Learning parallax attention for stereo image super-resolution [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2019: 12250-12259.
19 MITTAL A, MOORTHY A K, BOVIK A C No-reference image quality assessment in the spatial domain[J]. IEEE Transactions on Image Processing, 2012, 21 (12): 4695- 4708
doi: 10.1109/TIP.2012.2214050
20 MITTAL A, SOUNDARARAJAN R, BOVIK A C Making a “completely blind” image quality analyzer[J]. IEEE Signal Processing Letters, 2012, 20 (3): 209- 212
21 VENKATANATH N, PRANEETH D, BH M C, et al. Blind image quality evaluation using perception based features [C]// Proceedings of the Twenty First National Conference on Communications. Mumbai: IEEE, 2015: 1-6.
22 WANG S, ZHENG J, HU H, et al Naturalness preserved enhancement algorithm for non-uniform illumination images[J]. IEEE Transactions on Image Processing, 2013, 22 (9): 3538- 3548
doi: 10.1109/TIP.2013.2261309
23 ZHANG R, ISOLA P, EFROS A A, et al. The unreasonable effectiveness of deep features as a perceptual metric [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 586-595.
24 FAN M, WANG W, YANG W, et al. Integrating semantic segmentation and retinex model for low-light image enhancement [C]// Proceedings of the 28th ACM International Conference on Multimedia. [S.l.]: ACM, 2020: 2317-2325.
25 WANG W, WEI C, YANG W, et al. GLADNet: low-light enhancement network with global awareness [C]// Proceedings of the IEEE International Conference on Automatic Face and Gesture Recognition. Xi’an: IEEE, 2018: 751-755.
26 LIU R, MA L, ZHANG J, et al. Retinex-inspired unrolling with cooperative prior architecture search for low-light image enhancement [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Nashville: IEEE, 2021: 10561-10570.
27 GEIGER A, LENZ P, URTASUN R. Are we ready for autonomous driving? The KITTI vision benchmark suite [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Providence: IEEE, 2012: 3354-3361.
28 彭国福, 林正浩 图像处理中Gamma校正的研究和实现[J]. 电子工程师, 2006, 32 (2): 30- 32
PENG Guo-fu, LIN Zheng-hao A study on Gamma correction and its implementation in image processing[J]. Electronic Engineer, 2006, 32 (2): 30- 32
doi: 10.3969/j.issn.1674-4888.2006.02.008
[1] 韩俊,袁小平,王准,陈烨. 基于YOLOv5s的无人机密集小目标检测算法[J]. 浙江大学学报(工学版), 2023, 57(6): 1224-1233.
[2] 艾青林,崔景瑞,吕冰海,童桐. 基于损伤区域融合变换的轴承鼓形滚子表面损伤检测方法[J]. 浙江大学学报(工学版), 2023, 57(5): 1009-1020.
[3] 张剑钊,郭继昌,汪昱东. 基于融合逆透射率图的水下图像增强算法[J]. 浙江大学学报(工学版), 2023, 57(5): 921-929.
[4] 杨帆,宁博,李怀清,周新,李冠宇. 基于语义增强特征融合的多模态图像检索模型[J]. 浙江大学学报(工学版), 2023, 57(2): 252-258.
[5] 陈巧红,孙佳锦,漏杨波,方志坚. 基于多任务学习与层叠 Transformer 的多模态情感分析模型[J]. 浙江大学学报(工学版), 2023, 57(12): 2421-2429.
[6] 侯小虎,贾晓芬,赵佰亭. 眼底病变OCT图像的轻量化识别算法[J]. 浙江大学学报(工学版), 2023, 57(12): 2448-2455.
[7] 艾青林,杨佳豪,崔景瑞. 基于自适应增殖数据增强与全局特征融合的小目标行人检测[J]. 浙江大学学报(工学版), 2023, 57(10): 1933-1944.
[8] 莫建文,李晋,蔡晓东,陈锦威. 基于门控特征融合与中心损失的目标识别[J]. 浙江大学学报(工学版), 2023, 57(10): 2011-2017.
[9] 成宏达,骆海明,夏庆超,杨灿军. 基于改进γ-CLAHE算法的水下机器人图像识别[J]. 浙江大学学报(工学版), 2022, 56(8): 1648-1655.
[10] 张娜,戚旭磊,包晓安,吴彪,涂小妹,金瑜婷. 基于优化预测定位的单阶段目标检测算法[J]. 浙江大学学报(工学版), 2022, 56(4): 783-794.
[11] 温佩芝,陈君谋,肖雁南,温雅媛,黄文明. 基于生成式对抗网络和多级小波包卷积网络的水下图像增强算法[J]. 浙江大学学报(工学版), 2022, 56(2): 213-224.
[12] 黄新宇,游帆,张沛,张昭,张柏礼,吕建华,徐立臻. 基于多分类及特征融合的静默活体检测算法[J]. 浙江大学学报(工学版), 2022, 56(2): 263-270.
[13] 于楠晶,范晓飚,邓天民,冒国韬. 基于多头自注意力的复杂背景船舶检测算法[J]. 浙江大学学报(工学版), 2022, 56(12): 2392-2402.
[14] 张云佐,郭威,蔡昭权,李文博. 联合多尺度与注意力机制的遥感图像目标检测[J]. 浙江大学学报(工学版), 2022, 56(11): 2215-2223.
[15] 李明,段立娟,王文健,恩擎. 基于显著稀疏强关联的脑功能连接分类方法[J]. 浙江大学学报(工学版), 2022, 56(11): 2232-2240.