Please wait a minute...
浙江大学学报(工学版)
浙江大学学报(工学版)  2022, Vol. 56 Issue (4): 795-802    DOI: 10.3785/j.issn.1008-973X.2022.04.019
计算机技术、信息工程     
遥感图像语义分割空间全局上下文信息网络
吴泽康1(),赵姗2,李宏伟2,*(),姜懿芮1
1. 郑州大学 信息工程学院，河南 郑州 450001
2. 郑州大学 地球科学与技术学院，河南 郑州 450001
Spatial global context information network for semantic segmentation of remote sensing image
Ze-kang WU1(),Shan ZHAO2,Hong-wei LI2,*(),Yi-rui JIANG1
1. College of Information Engineering, Zhengzhou University, Zhengzhou 450001, China
2. College of Geo-Science and Technology, Zhengzhou University, Zhengzhou 450001, China
 全文: PDF(1135 KB)   HTML
摘要：

为了解决卷积神经网络(CNN)在语义分割特征提取阶段容易丢失空间信息以及边界信息不明确的问题，基于U-Net基线网络提出空间全局上下文信息网络(NC-Net). 增加再编码阶段(ReEncoder)，以增强空间信息识别能力. 在Decoder阶段输出多尺度特征，与ReEncoder阶段结合获取全局上下文信息. 保留边界损失函数，设计多尺度损失函数级联方法，优化整体网络. 在GID以及WHDLD数据集上的实验结果表明，该方法的总体准确度达到最好成绩，明显优于其他基线模型.
关键词： 语义分割;  遥感影像;  空间信息;  全局上下文;  神经网络    
Abstract:

A spatial global context information network (NC-Net) was proposed based on the U-Net baseline network in order to solve the problem that the convolutional neural network (CNN) easily lost spatial information and the boundary information was unclear in the feature extraction stage of semantic segmentation. A re-encoding stage was added (ReEncoder) in order to enhance the ability of spatial information recognition. Multi-scale features were output in the Decoder stage, which was combined with the ReEncoder stage to obtain global context information. The boundary loss function was retained, and a multi-scale loss function cascade method was designed to optimize the overall network. The experimental results on the GID and WHDLD data sets show that the overall accuracy of the method achieves the best results, significantly outperforming other baseline models.
Key words: semantic segmentation    remote sensing image    spatial information    global context    neural network
收稿日期: 2021-07-13 出版日期: 2022-04-24
CLC:  TP 751  
基金资助: 国家自然科学基金面上项目(41571394)
通讯作者: 李宏伟     E-mail: 2471217214@qq.com;laob_811@sina.com
作者简介: 吴泽康（1997—），男，硕士生，从事数据挖掘的研究. orcid.org/0000-0003-1729-7157. E-mail： 2471217214@qq.com
服务　　
把本文推荐给朋友
加入引用管理器
E-mail Alert
作者相关文章　　
吴泽康
赵姗
李宏伟
姜懿芮

引用本文:

吴泽康,赵姗,李宏伟,姜懿芮. 遥感图像语义分割空间全局上下文信息网络[J]. 浙江大学学报(工学版), 2022, 56(4): 795-802.

Ze-kang WU,Shan ZHAO,Hong-wei LI,Yi-rui JIANG. Spatial global context information network for semantic segmentation of remote sensing image. Journal of ZheJiang University (Engineering Science), 2022, 56(4): 795-802.

链接本文:

https://www.zjujournals.com/eng/CN/10.3785/j.issn.1008-973X.2022.04.019    或    https://www.zjujournals.com/eng/CN/Y2022/V56/I4/795

图 1  U-Net网络结构
图 2  NC-Net网络结构
图 3  多尺度输出融合再采样过程
图 4  多尺度损失函数级联框架
方法 OA K FWIoU F1
U-Net[9] 70.721 51.857 56.135 70.131
DeepLabv3[18] 74.072 60.702 61.330 75.247
PSPNet[12] 72.513 59.147 60.880 75.782
DeepLabv3+[19] 75.982 61.958 63.021 77.920
NC-Net 77.331 62.214 64.550 78.694
表 1  GID的实验结果
图 5  NC-Net和其他模型的视觉整体对比
图 6  NC-Net同其他模型的局部视觉对比
标签类别 Pr R F1
U-
Net[9]		 Deep-
Labv3[18]		 PSP-
Net[12]		 Deep-
Labv3+[19]		 NC-
Net 		U-
Net[9]		 Deep-
Labv3[18]		 PSP-
Net[12]		 Deep-
Labv3+[19]		 NC-
Net 		U-
Net[9]		 Deep-
Labv3[18]		 PSP-
Net[12]		 Deep-
Labv3+[19]		 NC-
Net
水浇地 85.61 84.95 81.77 88.22 86.73 83.69 86.91 91.59 89.73 91.94 84.63 85.91 86.40 88.96 89.25
园地 33.25 19.71 24.58 37.08 38.83 1.17 1.32 1.58 5.51 8.12 2.26 2.47 2.96 9.59 13.43
工业区 54.01 62.69 55.73 79.15 74.27 43.07 62.85 55.73 50.47 51.39 47.92 62.76 55.73 61.52 60.74
村镇住宅 73.97 82.92 76.73 78.77 90.65 64.75 74.15 75.52 83.78 65.10 69.05 78.29 76.12 81.19 75.77
交通用地 66.45 72.15 45.53 68.21 67.05 17.31 42.56 35.58 61.63 47.45 27.46 53.53 39.94 64.75 55.57
河流 39.91 23.70 21.47 22.43 42.87 3.95 1.21 4.25 4.43 5.30 7.18 2.30 7.09 7.39 9.43
湖泊 87.77 89.48 88.97 85.93 86.09 16.22 20.40 3.56 39.31 74.89 27.38 33.22 6.84 53.94 80.10
其他 38.62 39.23 34.07 43.90 46.69 48.25 61.84 44.02 59.59 62.65 42.90 48.00 38.41 50.55 53.50
表 2  GID各个类别的定量指标
方法 OA K FWIoU F1
U-Net[9] 76.218 70.816 69.709 70.066
DeepLabv3[18] 80.053 69.380 71.252 72.391
PSPNet[12] 78.406 71.528 69.039 71.002
DeepLabv3+[19] 81.295 76.524 73.157 75.563
NC-Net 84.897 79.944 76.025 76.301
表 3  WHDLD的实验结果
图 7  WHDLD的可视化结果
标签类别 Pr R F1
U-
Net[9]		 Deep-
Labv3[18]		 PSP-
Net[12]		 Deep-
Labv3+[19]		 NC-
Net 		U-
Net[9]		 Deep-
Labv3[18]		 PSP-
Net[12]		 Deep-
Labv3+[19]		 NC-
Net 		U-
Net[9]		 Deep-
Labv3[18]		 PSP-
Net[12]		 Deep-
Labv3+[19]		 NC-
Net
建筑物 67.72 70.28 69.53 72.36 72.78 68.07 68.93 70.33 72.98 72.19 67.89 69.59 69.92 72.16 72.48
道路 68.65 73.06 68.86 72.20 75.71 73.66 82.61 75.26 87.01 90.36 71.06 77.54 71.91 78.91 82.78
人行道 70.88 69.51 71.74 75.74 70.54 61.28 60.02 63.85 62.55 60.36 65.73 64.41 67.56 67.68 65.04
植被 65.07 66.03 65.91 68.04 68.98 68.80 70.33 69.04 70.83 71.12 66.88 68.11 67.43 69.40 70.03
裸地 71.92 74.43 72.09 75.97 76.26 66.31 69.95 70.72 71.76 72.07 69.00 72.12 71.39 72.84 74.10
水体 80.53 85.82 84.25 89.61 91.01 81.56 84.61 83.78 84.06 86.47 81.04 85.21 84.01 86.74 89.15
表 4  WHDLD各个类别的定量指标
%
方法 OA K FWIoU F1
U-NetAtt[13] 82.602 75.484 73.474 69.622
U-Net++[21] 84.067 77.430 74.496 74.633
MACU-Net[20] 84.623 78.233 75.231 75.245
NC-Net 84.897 79.944 76.025 76.301
表 5  NC-Net在WHDLD上与其他改进网络的对比结果
1 NORONHA S, NEVATIA R Detection and modeling of buildings from multiple aerial images[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2001, 23 (5): 501- 518
doi: 10.1109/34.922708
2 COTE M, SAEEDI P Automatic rooftop extraction in nadir aerial imagery of suburban regions using corners and variational level set evolution[J]. IEEE Transactions on Geoscience and Remote Sensing, 2012, 51 (1): 313- 328
3 LI E, FEMIANI J, XU S, et al Robust rooftop extraction from visible band images using higher order CRF[J]. IEEE Transactions on Geoscience and Remote Sensing, 2015, 53 (8): 4483- 4495
doi: 10.1109/TGRS.2015.2400462
4 胡翔云, 巩晓雅, 张觅 变分法遥感影像人工地物自动检测[J]. 测绘学报, 2018, 47 (6): 780- 789
HU Xiang-yun, GONG Xiao-ya, ZHANG Mi A variational approach for automatic man-made object detection from remote sensing image[J]. Acta Geodaetica at Cartographica Sinica, 2018, 47 (6): 780- 789
doi: 10.11947/j.AGCS.2018.20170642
5 林祥国, 张继贤 面向对象的形态学建筑物指数及其高分辨率遥感影像建筑物提取应用[J]. 测绘学报, 2017, 46 (6): 724- 733
LIN Xiang-guo, ZHANG Ji-xian Object-based morphological building index for building extraction form high resolution remote sensing imagery[J]. Acta Geodaetica at Cartographica Sinica, 2017, 46 (6): 724- 733
doi: 10.11947/j.AGCS.2017.20170068
6 李道纪, 郭海涛, 卢俊, 等 遥感影像地物分类多注意力融和U型网络法[J]. 测绘学报, 2020, 49 (8): 1051- 1064
LI Dao-ji, GUO Hai-tao, LU Jun, et al A remote sensing image classification procedure based on multilevel attention fusion UNet[J]. Acta Geodaetica at Cartographica Sinica, 2020, 49 (8): 1051- 1064
doi: 10.11947/j.AGCS.2020.20190407
7 LIU Z, LUO P, WANG X, et al. Deep learning face attributes in the wild [C]//Proceedings of the IEEE International Conference on Computer Vision. Santiago: IEEE, 2015: 3730-3738.
8 LONG J, SHELHAMER E, DARRElLL T. Fully convolutional networks for semantic segmentation [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Santiago: IEEE, 2015: 3431-3440.
9 RONNEBERGER O, FISCHER P, BROX T. U-net: convolutional networks for biomedical image segmentation [C]//International Conference on Medical Image Computing and Computer-Assisted Intervention. Cham: Springer, 2015: 234-241.
10 BADRINARAYANAN V, KENDALL A, CIPOLLA R Segnet: a deep convolutional encoder-decoder architecture for image segmentation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39 (12): 2481- 2495
doi: 10.1109/TPAMI.2016.2644615
11 SIMONYAN K, VEDALDI A, ZISSERMAN A. Deep inside convolutional networks: visualizing image classification models and saliency maps [C]// Workshop at International Conference on Learning Representations. Banff: IEEE, 2014.
12 ZhAO H, SHI J, QI X, et al. Pyramid scene parsing network [C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017: 2881-2890.
13 OKTAY O, SCHLEMPER J, FOLGOC L L, et al. Attention u-net: learning where to look for the pancreas [EB/OL]. (2018-04-11). https://arxiv.org/abs/1804.03999.
14 LIU P, WEI Y, WANG Q, et al Research on post-earthquake landslide extraction algorithm based on improved U-Net model[J]. Remote Sensing, 2020, 12 (5): 894
doi: 10.3390/rs12050894
15 TONG X Y, XIA G S, LU Q, et al Land-cover classification with high-resolution remote sensing images using transferable deep models[J]. Remote Sensing of Environment, 2020, 237: 111322
16 SHAO Z, YANG K, ZHOU W Performance evaluation of single-label and multi-label remote sensing image retrieval using a dense labeling dataset[J]. Remote Sensing, 2018, 10 (6): 964
doi: 10.3390/rs10060964
17 SHAO Z, ZHOU W, DENG X, et al Multilabel remote sensing image retrieval based on fully convolutional network[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2020, 13 (1): 318- 328
doi: 10.1109/JSTARS.2019.2961634
18 ChEN L C, PAPANDREOU G, SCHROFF F, et al. Rethinking atrous convolution for semantic image segmentation [EB/OL]. (2017-06-17). https://arxiv.org/abs/1706.05587.
19 CHEN L C, ZHU Y, PAPANDREOU G, et al. Encoderdecoder with atrous separable convolution for semantic image segmentation [C]// Proceedings of the European Conference on Computer Vision. Munich: Springer, 2018: 801-818.
20 RUI L, CEHNXI D, SHUNYI Z. MACU-Net semantic segmentation from high-resolution remote sensing images [EB/OL]. [2020-07-26]. https://arxiv.org/abs/2007.13083.
[1] 王云灏,孙铭会,辛毅,张博宣. 基于压电薄膜传感器的机器人触觉识别系统[J]. 浙江大学学报(工学版), 2022, 56(4): 702-710.
[2] 程若然,赵晓丽,周浩军,叶翰辰. 基于深度学习的中文字体风格转换研究综述[J]. 浙江大学学报(工学版), 2022, 56(3): 510-519, 530.
[3] 王婷,朱小飞,唐顾. 基于知识增强的图卷积神经网络的文本分类[J]. 浙江大学学报(工学版), 2022, 56(2): 322-328.
[4] 温佩芝,陈君谋,肖雁南,温雅媛,黄文明. 基于生成式对抗网络和多级小波包卷积网络的水下图像增强算法[J]. 浙江大学学报(工学版), 2022, 56(2): 213-224.
[5] 李广龙,申德荣,聂铁铮,寇月. 数据库外基于多模型的学习式查询优化方法[J]. 浙江大学学报(工学版), 2022, 56(2): 288-296.
[6] 任松,朱倩雯,涂歆玥,邓超,王小书. 基于深度学习的公路隧道衬砌病害识别方法[J]. 浙江大学学报(工学版), 2022, 56(1): 92-99.
[7] 黄发明,潘李含,姚池,周创兵,姜清辉,常志璐. 基于半监督机器学习的滑坡易发性预测建模[J]. 浙江大学学报(工学版), 2021, 55(9): 1705-1713.
[8] 张楠,董红召,佘翊妮. 公交专用道条件下公交车辆轨迹的Seq2Seq预测[J]. 浙江大学学报(工学版), 2021, 55(8): 1482-1489.
[9] 王飞,徐维祥. 基于LSTM神经网络改进的路阻函数模型[J]. 浙江大学学报(工学版), 2021, 55(6): 1065-1071.
[10] 郝晏,丁雅斌,付津昇. 机器人加工系统累积误差逐级闭环优化策略[J]. 浙江大学学报(工学版), 2021, 55(6): 1142-1149.
[11] 许佳辉,王敬昌,陈岭,吴勇. 基于图神经网络的地表水水质预测模型[J]. 浙江大学学报(工学版), 2021, 55(4): 601-607.
[12] 程鸿,胡佳杰,刘勇,叶远青. 强度传输方程和神经网络融合的三维重构算法[J]. 浙江大学学报(工学版), 2021, 55(4): 658-664.
[13] 郑英杰,吴松荣,韦若禹,涂振威,廖进,刘东. 基于目标图像FCM算法的地铁定位点匹配及误报排除方法[J]. 浙江大学学报(工学版), 2021, 55(3): 586-593.
[14] 毛奕喆,龚国芳,周星海,王飞. 基于马尔可夫过程和深度神经网络的TBM围岩识别[J]. 浙江大学学报(工学版), 2021, 55(3): 448-454.
[15] 刘芳,汪震,刘睿迪,王锴. 基于组合损失函数的BP神经网络风力发电短期预测方法[J]. 浙江大学学报(工学版), 2021, 55(3): 594-600.