Please wait a minute...
浙江大学学报(工学版)
Journal of ZheJiang University (Engineering Science)  2023, Vol. 57 Issue (12): 2456-2466    DOI: 10.3785/j.issn.1008-973X.2023.12.013
    
Stereo low-light enhancement based on feature fusion and consistency loss
Jia-wen LIAO1(),Yan-wei PANG1,2,*(),Jing NIE1,3,Han-qing SUN1,Jia-le CAO1
1. School of Electrical and Information Engineering, Tianjin University, Tianjin 300072, China
2. Shanghai Artificial Intelligence Laboratory, Shanghai 200232, China
3. School of Microelectronics and Communication Engineering, Chongqing University, Chongqing 401331, China
Download: HTML     PDF(2642KB) HTML
Export: BibTeX | EndNote (RIS)      

Abstract  

A large scale real scene stereo low-light image dataset SLL10K was proposed. There were 12 658 pairs of unreferenced stereo low-illumination images and 205 pairs of referenced stereo images contained in the dataset. The images in the SLL10K dataset cover a wealth of lighting, time, and scene. FCNet, a stereo low-light image enhancement network based on feature fusion and consistency loss was proposed. The feature fusion module was used to fully integrate intra-monocular and inter-stereo features, and the consistency loss function was used to maintain the consistency between images before and after enhancement. Experiments on the SLL10K dataset and darkening KITTI dataset show that the images with FCNet obtain better performance on low-light image enhancement and object detection than the monocular enhancement method.

Key words： image enhancement      low-light      stereo dataset      no-reference image enhancement      feature fusion     
Received: 27 February 2023      Published: 27 December 2023
CLC:  TP 391.4  
Fund:  国家科技创新2030新一代人工智能重大项目（2022ZD0160400）；国家自然科学基金资助项目（62271346）
Corresponding Authors: Yan-wei PANG     E-mail: gavin971209@tju.edu.cn;pyw@tju.edu.cn
Service
E-mail this article
Add to my bookshelf
Add to citation manager
E-mail Alert
RSS
Articles by authors
Jia-wen LIAO
Yan-wei PANG
Jing NIE
Han-qing SUN
Jia-le CAO
Cite this article:

Jia-wen LIAO,Yan-wei PANG,Jing NIE,Han-qing SUN,Jia-le CAO. Stereo low-light enhancement based on feature fusion and consistency loss. Journal of ZheJiang University (Engineering Science), 2023, 57(12): 2456-2466.

URL:

https://www.zjujournals.com/eng/10.3785/j.issn.1008-973X.2023.12.013     OR     https://www.zjujournals.com/eng/Y2023/V57/I12/2456


基于特征融合和一致性损失的双目低光照增强

构建大规模真实场景的双目低光照图像数据集SLL10K. 该数据集包含12 658对无参考双目低照度图像和205对有参考双目图像，数据集图像涵盖丰富的光照、时间及场景. 提出基于特征融合和一致性损失的双目低光照图像增强网络FCNet，特征融合模块用于充分融合单目内和双目间的特征，一致性损失函数用于保持增强前和增强后图像间的一致性. 在SLL10K数据集和暗化KITTI数据集上的实验表明，使用FCNet增强后的图像获得了比单目增强方法更好的低光照图像增强效果和目标检测效果.


关键词： 图像增强,  低光照,  双目数据集,  无参考图像增强,  特征融合 
Fig.1 Some example images in SLL10K
Fig.2 Sample image pairs for training and testing sets in outdoor scene of SLL10K
Fig.3 Sample of low-light image pairs and reference image image pairs in indoor scene of SLL10K
Fig.4 Structure diagram of FCNet
Fig.5 Structure diagram of stereo inter-intra-feature fusion module
测试集 SIIFF SC损失 BRISQUE NIQE PIQE LOE PSNR/dB SSIM LPIPS
室外 × × 25.684 8 2.832 7 32.569 0 807.7 — — —
√ × 23.546 3 2.687 0 30.476 3 696.3 — — —
√ √ 23.095 1 2.641 1 28.111 5 679.2 — — —
室内 × × 27.663 3 3.694 9 38.553 7 2 738.0 11.230 5 0.360 5 0.633 5
√ × 24.973 8 3.500 8 34.091 6 2 628.4 11.224 1 0.375 7 0.623 4
√ √ 22.531 7 3.390 1 31.279 4 2 593.4 11.222 1 0.406 2 0.609 2
Tab.1 Ablation experiment of FCNet’s different modules on two test sets
测试集 nm BRISQUE NIQE PIQE LOE PSNR/dB SSIM LPIPS
室外 1 23.310 1 2.651 0 28.553 6 688.8 — — —
2 23.095 1 2.641 1 28.111 5 679.2 — — —
室内 1 22.656 7 3.414 7 32.062 5 2 609.4 11.119 0 0.363 7 0.639 8
2 22.531 7 3.390 1 31.279 4 2 593.4 11.222 1 0.406 2 0.609 2
Tab.2 Ablation experiment of FCNet’s module number variation on two test sets
测试集 卷积类别 BRISQUE NIQE PIQE LOE PSNR/dB SSIM LPIPS
室外 3×3 23.732 9 3.253 5 31.648 9 753.7 — — —
1×9和9×1 23.095 1 2.641 1 28.111 5 679.2 — — —
室内 3×3 24.625 2 3.816 4 41.876 4 2 655.2 11.068 3 0.317 7 0.651 6
1×9和9×1 22.531 7 3.390 1 31.279 4 2 593.4 11.222 1 0.406 2 0.609 2
Tab.3 Ablation experiment of convolution variation on two test sets
方法 左目 右目
BRISQUE NIQE PIQE LOE BRISQUE NIQE PIQE LOE
RetinexNet[11] 24.640 7 4.257 3 36.454 5 1 741.5 24.420 5 4.270 1 36.276 1 1752.7
ISSR[24] 28.392 8 2.815 1 27.923 2 753.0 28.903 5 2.810 7 28.525 2 724.2
GLAD[25] 23.628 8 3.280 8 28.633 4 590.8 23.729 0 3.268 0 28.310 2 584.8
DVENet[16] 23.100 2 2.931 7 28.175 1 791.9 22.706 7 2.872 3 27.531 7 740.2
ZeroDCE++[17] 25.684 8 2.832 7 32.569 0 807.7 25.560 2 2.857 2 32.118 0 814.4
RUAS[26] 29.713 1 3.746 6 30.512 2 2 520.0 29.736 9 3.675 5 30.863 7 2 300.6
FCNet 23.095 1 2.641 1 28.111 5 679.2 22.689 8 2.632 3 27.374 5 669.9
Tab.4 Indicators comparison of different image enhancement methods on SLL10K outdoor dataset
方法 左目 右目
BRISQUE NIQE PIQE LOE PSNR SSIM LPIPS BRISQUE NIQE PIQE LOE PSNR SSIM LPIPS
RetinexNet[11] 34.042 9 5.592 2 49.155 6 3 241.0 11.640 2 0.222 1 0.812 2 33.562 7 5.596 0 48.305 5 3 102.3 11.211 8 0.238 0 0.798 2
ISSR[24] 23.900 7 2.751 8 29.193 1 2 599.6 8.858 8 0.258 8 0.674 5 24.626 0 2.950 9 28.848 0 2 577.8 8.199 8 0.261 9 0.660 3
GLAD[25] 23.139 6 3.673 4 42.574 7 2 547.2 12.875 1 0.229 0 0.666 0 25.854 5 3.581 8 41.646 0 2 515.4 12.174 0 0.247 5 0.648 3
DVENet[16] 23.075 7 3.405 7 32.115 1 2 595.8 9.026 0 0.246 0 0.656 9 22.189 2 3.391 6 30.236 3 2 543.2 8.595 1 0.248 9 0.643 7
ZeroDCE++[17] 27.663 3 3.694 9 38.553 7 2 738.0 11.230 5 0.360 5 0.725 4 27.262 2 3.649 7 38.050 0 2 671.1 10.410 9 0.363 1 0.713 0
RUAS[26] 24.693 4 3.361 7 35.548 8 2 671.3 9.864 7 0.380 4 0.710 2 24.375 1 3.305 0 33.077 0 2 581.7 9.234 6 0.369 1 0.693 1
FCNet 22.531 7 3.390 1 31.279 4 2 593.4 11.222 1 0.406 2 0.609 2 21.273 6 3.329 7 29.109 7 2 539.9 10.407 3 0.407 3 0.598 0
Tab.5 Indicators comparison of different image enhancement methods on SLL10K indoor dataset
Fig.6 Processing effects comparison of different image enhancement methods on four scene images
Fig.7 Comparison of monocular and stereo enhancement of indoor images
%
方法 $ {\mathrm{A}\mathrm{P}}_{{\rm{2d}}} $ $ {\mathrm{A}\mathrm{P}}_{{\rm{ori}}} $ $ {\mathrm{A}\mathrm{P}}_{{\rm{bev}}} $ $ {\mathrm{A}\mathrm{P}}_{{\rm{3d}}} $
原始 88.86 86.90 64.54 50.49
低光照 70.48 68.14 48.16 34.43
RetinexNet[11] 69.70 67.71 48.99 36.21
ISSR[23] 70.73 68.12 46.20 33.45
GLAD[24] 88.30 86.12 57.23 45.05
ZeroDCE++[17] 88.23 86.03 57.12 44.98
RUAS[25] 87.81 84.24 55.27 42.32
FCNet 88.51 86.55 57.72 45.45
Tab.6 Object detection results of different image enhancement methods
Fig.8 Comparison of parallax map between monocular and stereo image enhancement methods
[1]   WANG Y, WAN R, YANG W, et al. Low-light image enhancement with normalizing flow [C]// Proceedings of the AAAI Conference on Artificial Intelligence. [S.l.]: AAAI, 2022: 2604-2612.
[2]   LV F, LI Y, LU F Attention guided low-light image enhancement with a large scale low-light simulation dataset[J]. International Journal of Computer Vision, 2021, 129: 2175- 2193
doi: 10.1007/s11263-021-01466-8
[3]   WANG L, LIU Z, SIU W, et al Lightening network for low-light image enhancement[J]. IEEE Transactions on Image Processing, 2020, 29: 7984- 7996
doi: 10.1109/TIP.2020.3008396
[4]   ZHOU S, LI C, LOY C C. LEDNet: joint low-light enhancement and deblurring in the dark [C]// Proceedings of the European Conference on Computer Vision. [S.l.]: Springer, 2022: 573-589.
[5]   ZHENG C, SHI D, SHI W. Adaptive unfolding total variation network for low-light image enhancement [C]// Proceedings of the IEEE/CVF International Conference on Computer Vision. Montreal: IEEE, 2021: 4439-4448.
[6]   WU W, WENG J, ZHANG P, et al. URetinex-Net: retinex-based deep unfolding network for low-light image enhancement [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans: IEEE, 2022: 5901-5910.
[7]   HAO S, HAN X, GUO Y, et al Low-light image enhancement with semi-decoupled decomposition[J]. IEEE Transactions on Multimedia, 2020, 22 (12): 3025- 3038
doi: 10.1109/TMM.2020.2969790
[8]   GUO C, LI C, GUO J, et al. Zero-reference deep curve estimation for low-light image enhancement [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020: 1777–1786.
[9]   MA L, MA T, LIU R, et al. Toward fast, flexible, and robust low-light image enhancement [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans: IEEE, 2022: 5627-5636.
[10]   ZHANG Z, ZHENG H, HONG R, et al. Deep color consistent network for low-light image enhancement [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans: IEEE, 2022: 1889-1898.
[11]   WEI C, WANG W, YANG W, et al. Deep retinex decomposition for low-light enhancement [C]// Proceedings of the 29th British Machine Vision Conference. Newcastle Upon Tyne: [s.n.], 2018.
[12]   CHEN C, CHEN Q, XU J, et al. Learning to see in the dark [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 3291-3300.
[13]   PANG Y, NIE J, XIE J, et al. BidNet: binocular image dehazing without explicit disparity estimation [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020: 5931-5940.
[14]   ZHOU S, ZHANG J, ZUO W, et al. DAVANet: stereo deblurring with view aggregation [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2019: 10996-11005.
[15]   ZHANG K, LUO W, REN W, et al. Beyond monocular deraining: stereo image deraining via semantic understanding [C]// Proceedings of the European Conference on Computer Vision. [S.l.]: Springer, 2020: 71-89.
[16]   HUANG J, FU X, XIAO Z, et al Low-light stereo image enhancement[J]. IEEE Transactions on Multimedia, 2022, 25: 2978- 2992
[17]   LI C, GUO C, LOY C C Learning to enhance low-light image via zero-reference deep curve estimation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44 (8): 4225- 4238
[18]   WANG L, WANG Y, LIANG Z, et al. Learning parallax attention for stereo image super-resolution [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2019: 12250-12259.
[19]   MITTAL A, MOORTHY A K, BOVIK A C No-reference image quality assessment in the spatial domain[J]. IEEE Transactions on Image Processing, 2012, 21 (12): 4695- 4708
doi: 10.1109/TIP.2012.2214050
[20]   MITTAL A, SOUNDARARAJAN R, BOVIK A C Making a “completely blind” image quality analyzer[J]. IEEE Signal Processing Letters, 2012, 20 (3): 209- 212
[21]   VENKATANATH N, PRANEETH D, BH M C, et al. Blind image quality evaluation using perception based features [C]// Proceedings of the Twenty First National Conference on Communications. Mumbai: IEEE, 2015: 1-6.
[22]   WANG S, ZHENG J, HU H, et al Naturalness preserved enhancement algorithm for non-uniform illumination images[J]. IEEE Transactions on Image Processing, 2013, 22 (9): 3538- 3548
doi: 10.1109/TIP.2013.2261309
[23]   ZHANG R, ISOLA P, EFROS A A, et al. The unreasonable effectiveness of deep features as a perceptual metric [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 586-595.
[24]   FAN M, WANG W, YANG W, et al. Integrating semantic segmentation and retinex model for low-light image enhancement [C]// Proceedings of the 28th ACM International Conference on Multimedia. [S.l.]: ACM, 2020: 2317-2325.
[25]   WANG W, WEI C, YANG W, et al. GLADNet: low-light enhancement network with global awareness [C]// Proceedings of the IEEE International Conference on Automatic Face and Gesture Recognition. Xi’an: IEEE, 2018: 751-755.
[26]   LIU R, MA L, ZHANG J, et al. Retinex-inspired unrolling with cooperative prior architecture search for low-light image enhancement [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Nashville: IEEE, 2021: 10561-10570.
[27]   GEIGER A, LENZ P, URTASUN R. Are we ready for autonomous driving? The KITTI vision benchmark suite [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Providence: IEEE, 2012: 3354-3361.
[28]   彭国福, 林正浩 图像处理中Gamma校正的研究和实现[J]. 电子工程师, 2006, 32 (2): 30- 32
PENG Guo-fu, LIN Zheng-hao A study on Gamma correction and its implementation in image processing[J]. Electronic Engineer, 2006, 32 (2): 30- 32
doi: 10.3969/j.issn.1674-4888.2006.02.008
[1] Jun HAN,Xiao-ping YUAN,Zhun WANG,Ye CHEN. UAV dense small target detection algorithm based on YOLOv5s[J]. Journal of ZheJiang University (Engineering Science), 2023, 57(6): 1224-1233.
[2] Jian-zhao ZHANG,Ji-chang GUO,Yu-dong WANG. Underwater image enhancement algorithm via fusing reverse medium transmission map[J]. Journal of ZheJiang University (Engineering Science), 2023, 57(5): 921-929.
[3] Qing-lin AI,Jing-rui CUI,Bing-hai LV,Tong TONG. Surface defect detection method for bearing drum-shaped rollers based on fusion transformation of defective area[J]. Journal of ZheJiang University (Engineering Science), 2023, 57(5): 1009-1020.
[4] Fan YANG,Bo NING,Huai-qing LI,Xin ZHOU,Guan-yu LI. Multimodal image retrieval model based on semantic-enhanced feature fusion[J]. Journal of ZheJiang University (Engineering Science), 2023, 57(2): 252-258.
[5] Qiao-hong CHEN,Jia-jin SUN,Yang-bo LOU,Zhi-jian FANG. Multimodal sentiment analysis model based on multi-task learning and stacked cross-modal Transformer[J]. Journal of ZheJiang University (Engineering Science), 2023, 57(12): 2421-2429.
[6] Xiao-hu HOU,Xiao-fen JIA,Bai-ting ZHAO. Lightweight recognition algorithm for OCT images of fundus lesions[J]. Journal of ZheJiang University (Engineering Science), 2023, 57(12): 2448-2455.
[7] Qing-lin AI,Jia-hao YANG,Jing-rui CUI. Small target pedestrian detection based on adaptive proliferation data enhancement and global feature fusion[J]. Journal of ZheJiang University (Engineering Science), 2023, 57(10): 1933-1944.
[8] Jian-wen MO,Jin LI,Xiao-dong CAI,Jin-wei CHEN. Target recognition based on gated feature fusion and center loss[J]. Journal of ZheJiang University (Engineering Science), 2023, 57(10): 2011-2017.
[9] Hong-da CHENG,Hai-ming LUO,Qing-chao XIA,Can-jun YANG. Recognition of images for underwater vehicle based on improved γ-CLAHE algorithm[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(8): 1648-1655.
[10] Na ZHANG,Xu-lei QI,Xiao-an BAO,Biao WU,Xiao-mei TU,Yu-ting JIN. Single-stage object detection algorithm based on optimizing position prediction[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(4): 783-794.
[11] Pei-zhi WEN,Jun-mou CHEN,Yan-nan XIAO,Ya-yuan WEN,Wen-ming HUANG. Underwater image enhancement algorithm based on GAN and multi-level wavelet CNN[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(2): 213-224.
[12] Xin-yu HUANG,Fan YOU,Pei ZHANG,Zhao ZHANG,Bai-li ZHANG,Jian-hua LV,Li-zhen XU. Silent liveness detection algorithm based on multi classification and feature fusion network[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(2): 263-270.
[13] Yun-zuo ZHANG,Wei GUO,Zhao-quan CAI,Wen-bo LI. Remote sensing image target detection combining multi-scale and attention mechanism[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(11): 2215-2223.
[14] Ming LI,Li-juan DUAN,Wen-jian WANG,Qing EN. Brain functional connections classification method based on significant sparse strong correlation[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(11): 2232-2240.
[15] Ze-xin XU,Li-juan DUAN,Wen-jian WANG,Qing EN. Code vulnerability detection method based on contextual feature fusion[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(11): 2260-2270.