基于Trans-nightSeg的夜间道路场景语义分割方法

doi:10.3785/j.issn.1008-973X.2024.02.008

浙江大学学报(工学版)

2024, Vol. 58

Issue (2): 294-303 DOI: 10.3785/j.issn.1008-973X.2024.02.008

计算机技术、通信技术

基于Trans-nightSeg的夜间道路场景语义分割方法

李灿林1(

),张文娇1,邵志文2,3,马利庄3,王新玥1

1. 郑州轻工业大学计算机与通信工程学院，河南郑州 450000
2. 中国矿业大学计算机科学与技术学院，江苏徐州 221116
3. 上海交通大学计算机科学与工程系，上海 200240

Semantic segmentation method on nighttime road scene based on Trans-nightSeg

Canlin LI1(

),Wenjiao ZHANG1,Zhiwen SHAO2,3,Lizhuang MA3,Xinyue WANG1

1. School of Computer and Communication Engineering, Zhengzhou University of Light Industry, Zhengzhou 450000, China
2. School of Computer Science and Technology, China University of Mining and Technology, Xuzhou 221116, China
3. Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai 200240, China

全文: PDF(4108 KB) HTML

摘要：

针对夜间道路场景图像亮度低及缺乏带标注的夜间道路场景语义分割数据集的问题，提出夜间道路场景语义分割方法Trans-nightSeg. 使用TransCartoonGAN，将带标注的白天道路场景语义分割数据集Cityscapes转换为低光条件下的道路场景图像，两者共用同一个语义分割标注，丰富夜间道路场景数据集. 将该结果和真实的道路场景数据集一并作为N-Refinenet的输入，N-Refinenet网络引入了低光图像自适应增强网络，提高夜间道路场景的语义分割性能. 该网络采用深度可分离卷积替代普通的卷积，降低了计算量. 实验结果表明，所提算法在Dark Zurich-test和Nighttime Driving-test数据集上的平均交并比（mIoU）分别达到56.0%和56.6%，优于其他的夜间道路场景语义分割算法.

关键词： 图像增强; 语义分割; 生成对抗网络(GAN); 风格转换; 道路场景

Abstract:

The semantic segmentation method Trans-nightSeg was proposed aiming at the issues of low brightness and lack of annotated semantic segmentation dataset in nighttime road scenes. The annotated daytime road scene semantic segmentation dataset Cityscapes was converted into low-light road scene images by TransCartoonGAN, which shared the same semantic segmentation annotation, thereby enriching the nighttime road scene dataset. The result together with the real road scene dataset was used as input of N-Refinenet. The N-Refinenet network introduced a low-light image adaptive enhancement network to improve the semantic segmentation performance of the nighttime road scene. Depth-separable convolution was used instead of normal convolution in order to reduce the computational complexity. The experimental results show that the mean intersection over union (mIoU) of the proposed algorithm on the Dark Zurich-test dataset and Nighttime Driving-test dataset reaches 56.0% and 56.6%, respectively, outperforming other semantic segmentation algorithms for nighttime road scene.

Key words: image enhancement semantic segmentation generative adversarial network (GAN) style transformation road scene

收稿日期: 2023-06-29 出版日期: 2024-01-23

CLC:

TP 391

基金资助: 国家自然科学基金资助项目(61972157，62106268)；河南省科技攻关项目(212102210097)；上海市科技创新行动计划人工智能科技支撑项目(21511101200)；江苏省“双创博士”人才资助项目(JSSCBS20211220)

作者简介: 李灿林（1976—），男，副教授，从事图像处理和计算机视觉的研究. orcid.org/0000-0003-0307-3910. E-mail：lcl_zju@aliyun.com

	服务
	把本文推荐给朋友
	加入引用管理器
	E-mail Alert
	作者相关文章
	李灿林
	张文娇
	邵志文
	马利庄
	王新玥

引用本文:

李灿林,张文娇,邵志文,马利庄,王新玥. 基于Trans-nightSeg的夜间道路场景语义分割方法[J]. 浙江大学学报(工学版), 2024, 58(2): 294-303.

Canlin LI,Wenjiao ZHANG,Zhiwen SHAO,Lizhuang MA,Xinyue WANG. Semantic segmentation method on nighttime road scene based on Trans-nightSeg. Journal of ZheJiang University (Engineering Science), 2024, 58(2): 294-303.

链接本文:

https://www.zjujournals.com/eng/CN/10.3785/j.issn.1008-973X.2024.02.008 或 https://www.zjujournals.com/eng/CN/Y2024/V58/I2/294

图 1 Trans-nightSeg的网络结构

图 2 原图与滤波处理后的图片对比

图 3 TransCartoonGAN的结构图

图 4 连续情况下的侧窗图

图 5 N-Refinenet的结构图

图 6 不同网络合成夜间道路场景图像的效果图

表 1 各方法的相似度比较

表 2 不同方法在Dark Zurich-test上的IoU结果

图 7 不同夜间语义分割算法在Dark Zurich-test上的对比效果图

表 3 Night Driving-test上的结果比较

图 8 不同夜间语义分割算法在Night Driving-test上的对比效果图

表 4 在Dark Zurich-test数据集上的消融实验结果

图 9 各部件在Dark Zurich-test数据集上的部分结果图

图 10 Trans-nightSeg在极端黑暗条件下的结果图

1	CORDTS M, OMRAN M, RAMOS S, et al. The Cityscapes dataset for semantic urban scene understanding [C]// IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 3213-3223.
2	SAKARIDIS C, DAI D, GOOL L V. ACDC: the adverse conditions dataset with correspondences for semantic driving scene understanding [C]// IEEE International Conference on Computer Vision. Montreal: IEEE, 2021: 10765-10775.
3	GOODFELLOW I, POUGET-ABADIE J, MIRZA M, et al Generative adversarial networks[J]. Communications of the ACM, 2020, 63 (11): 139- 144 doi: 10.1145/3422622
4	REED S, AKATA Z, YAN X, et al. Generative adversarial text to image synthesis [C]// International Conference on Machine Learning. New York: PMLR, 2016: 1060-1069.
5	YEH R, CHEN C, LIM T Y, et al. Semantic image inpainting with perceptual and contextual losses [EB/OL]. [2016-07-26]. https://arxiv.org/abs/1607.07539.
6	LEDIG C, THEIS L, HUSZAR F, et al. Photo-realistic single image super-resolution using a generative adversarial network [C]// IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017: 4681-4690.
7	ISOLA P, ZHU J Y, ZHOU T, et al. Image-to-image translation with conditional adversarial networks [C]// IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017: 1125-1134.
8	KARACAN L, AKATA Z, ERDEM A, et al. Learning to generate images of outdoor scenes from attributes and semantic layouts [EB/OL]. [2016-12-01]. https://arxiv.org/abs/1612.00215.
9	ZHU J Y, PARK T, ISOLA P, et al. Unpaired image-to-image translation using cycle-consistent adversarial networks [C]// IEEE International Conference on Computer Vision. Venice: IEEE, 2017: 2223-2232.
10	WANG H, CHEN Y, CAI Y, et al SFNet-N: an improved SFNet algorithm for semantic segmentation of low-light autonomous driving road scenes[J]. IEEE Transactions on Intelligent Transportation Systems, 2022, 23 (11): 21405- 21417 doi: 10.1109/TITS.2022.3177615
11	CHEN Y, LAI Y K, LIU Y J. Cartoongan: generative adversarial networks for photo cartoonization [C]// IEEE Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 9465-9474.
12	YIN H, GONG Y, QIU G. Side window filtering [C]// IEEE Conference on Computer Vision and Pattern Recognition. Los Angeles: IEEE, 2019: 8758-8766.
13	SHELHAMER E, LONG J, DARRELL T Fully convolutional networks for semantic segmentation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39 (4): 640- 651 doi: 10.1109/TPAMI.2016.2572683
14	LIN G, MILAN A, SHEN C, et al. Refinenet: multi-path refinement networks for high-resolution semantic segmentation [C]// IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017: 1925-1934.
15	BIJELIC M, GRUBER T, RITTER W. Benchmarking image sensors under adverse weather conditions for autonomous driving [C]// IEEE Intelligent Vehicles Symposium. Changshu: IEEE, 2018: 1773-1779.
16	WULFMEIER M, BEWLEY A, POSNER I. Addressing appearance change in outdoor robotics with adversarial domain adaptation [C] // IEEE International Conference on Intelligent Robots and Systems. Vancouver: IEEE, 2017: 1551-1558.
17	DAI D, GOOL L V. Dark model adaptation: semantic image segmentation from daytime to nighttime [C]// IEEE International Conference on Intelligent Transportation Systems. Hawaii: IEEE, 2018: 3819-3824.
18	SAKARIDIS C, DAI D, GOOL L V. Guided curriculum model adaptation and uncertainty-aware evaluation for semantic nighttime image segmentation [C]// IEEE International Conference on Computer Vision. Seoul: IEEE, 2019: 7374-7383.
19	SAKARIDIS C, DAI D, GOOL L V Map-guided curriculum domain adaptation and uncertainty-aware evaluation for semantic nighttime image segmentation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44 (6): 3139- 3153 doi: 10.1109/TPAMI.2020.3045882
20	XU Q, MA Y, WU J, et al. CDAda: a curriculum do-main adaptation for nighttime semantic segmentation[C]// IEEE International Conference on Computer Vision. Montreal: IEEE, 2021: 2962-2971.
21	ROMERA E, BERGASA L M, YANG K, et al. Bridging the day and night domain gap for semantic segmentation [C]// IEEE Intelligent Vehicles Symposium. Paris: IEEE, 2019: 1312-1318.
22	SUN L, WANG K, YANG K, et al. See clearer at night: towards robust nighttime semantic segmentation through day-night image conversion [C]// Artificial Intelligence and Machine Learning in Defense Applications. Bellingham: SPIE, 2019, 11169: 77-89.
23	WU X, WU Z, GUO H, et al. Dannet: a one-stage domain adaptation network for unsupervised nighttime semantic segmentation [C]// IEEE Conference on Computer Vision and Pattern Recognition. Nashville: IEEE, 2021: 15769-15778.
24	WU X, WU Z, JU L, et al A one-stage domain adaptation network with image alignment for unsupervised nighttime semantic segmentation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023, 45 (1): 58- 72 doi: 10.1109/TPAMI.2021.3138829
25	HU Y, HU H, XU C, et al Exposure: a white-box photo post-processing framework[J]. ACM Transactions on Graphics, 2018, 37 (2): 26.1- 26.17
26	CHOLLET F. Xception: deep learning with depthwise separable convolutions [C]// IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017: 1251-1258.
27	HOWARD A G, ZHU M, CHEN B, et al. Mobilenets: efficient convolutional neural networks for mobile vis-ion applications [EB/OL]. [2017-04-17]. https://arxiv.org/abs/1704.04861.
28	TAN X, XU K, CAO Y, et al Nighttime scene parsing with a large real dataset[J]. IEEE Transactions on Image Processing, 2021, (30): 9085- 9098

[1]	冯志成,杨杰,陈智超. 基于轻量级Transformer的城市路网提取方法[J]. 浙江大学学报(工学版), 2024, 58(1): 40-49.
[2]	郭浩然,郭继昌,汪昱东. 面向水下场景的轻量级图像语义分割网络[J]. 浙江大学学报(工学版), 2023, 57(7): 1278-1286.
[3]	刘春娟,乔泽,闫浩文,吴小所,王嘉伟,辛钰强. 基于多尺度互注意力的遥感图像语义分割网络[J]. 浙江大学学报(工学版), 2023, 57(7): 1335-1344.
[4]	张海波,蔡磊,任俊平,王汝言,刘富. 基于Transformer的高效自适应语义分割网络[J]. 浙江大学学报(工学版), 2023, 57(6): 1205-1214.
[5]	张剑钊,郭继昌,汪昱东. 基于融合逆透射率图的水下图像增强算法[J]. 浙江大学学报(工学版), 2023, 57(5): 921-929.
[6]	杨长春,叶赞挺,刘半藤,王柯,崔海东. 基于多源信息融合的医学图像分割方法[J]. 浙江大学学报(工学版), 2023, 57(2): 226-234.
[7]	廖嘉文,庞彦伟,聂晶,孙汉卿,曹家乐. 基于特征融合和一致性损失的双目低光照增强[J]. 浙江大学学报(工学版), 2023, 57(12): 2456-2466.
[8]	杨燕泽,王萌,刘诚,徐慧通,张小月. 基于语义分割的沥青路面裂缝智能识别[J]. 浙江大学学报(工学版), 2023, 57(10): 2094-2105.
[9]	成宏达,骆海明,夏庆超,杨灿军. 基于改进γ-CLAHE算法的水下机器人图像识别[J]. 浙江大学学报(工学版), 2022, 56(8): 1648-1655.
[10]	吴泽康,赵姗,李宏伟,姜懿芮. 遥感图像语义分割空间全局上下文信息网络[J]. 浙江大学学报(工学版), 2022, 56(4): 795-802.
[11]	程若然,赵晓丽,周浩军,叶翰辰. 基于深度学习的中文字体风格转换研究综述[J]. 浙江大学学报(工学版), 2022, 56(3): 510-519, 530.
[12]	温佩芝,陈君谋,肖雁南,温雅媛,黄文明. 基于生成式对抗网络和多级小波包卷积网络的水下图像增强算法[J]. 浙江大学学报(工学版), 2022, 56(2): 213-224.
[13]	陈彤,郭剑锋,韩心中,谢学立,席建祥. 基于生成对抗模型的可见光-红外图像匹配方法[J]. 浙江大学学报(工学版), 2022, 56(1): 63-74.
[14]	程训,余建波. 基于机器视觉的加工刀具磨损监测方法[J]. 浙江大学学报(工学版), 2021, 55(5): 896-904.
[15]	雍子叶,郭继昌,李重仪. 融入注意力机制的弱监督水下图像增强算法[J]. 浙江大学学报(工学版), 2021, 55(3): 555-562.

Viewed

Full text

Abstract

Cited

Shared

Discussed