Please wait a minute...
浙江大学学报(工学版)
Journal of ZheJiang University (Engineering Science)  2022, Vol. 56 Issue (4): 795-802    DOI: 10.3785/j.issn.1008-973X.2022.04.019
    
Spatial global context information network for semantic segmentation of remote sensing image
Ze-kang WU1(),Shan ZHAO2,Hong-wei LI2,*(),Yi-rui JIANG1
1. College of Information Engineering, Zhengzhou University, Zhengzhou 450001, China
2. College of Geo-Science and Technology, Zhengzhou University, Zhengzhou 450001, China
Download: HTML     PDF(1135KB) HTML
Export: BibTeX | EndNote (RIS)      

Abstract  

A spatial global context information network (NC-Net) was proposed based on the U-Net baseline network in order to solve the problem that the convolutional neural network (CNN) easily lost spatial information and the boundary information was unclear in the feature extraction stage of semantic segmentation. A re-encoding stage was added (ReEncoder) in order to enhance the ability of spatial information recognition. Multi-scale features were output in the Decoder stage, which was combined with the ReEncoder stage to obtain global context information. The boundary loss function was retained, and a multi-scale loss function cascade method was designed to optimize the overall network. The experimental results on the GID and WHDLD data sets show that the overall accuracy of the method achieves the best results, significantly outperforming other baseline models.

Key words： semantic segmentation      remote sensing image      spatial information      global context      neural network     
Received: 13 July 2021      Published: 24 April 2022
CLC:  TP 751  
Fund:  国家自然科学基金面上项目(41571394)
Corresponding Authors: Hong-wei LI     E-mail: 2471217214@qq.com;laob_811@sina.com
Service
E-mail this article
Add to my bookshelf
Add to citation manager
E-mail Alert
RSS
Articles by authors
Ze-kang WU
Shan ZHAO
Hong-wei LI
Yi-rui JIANG
Cite this article:

Ze-kang WU,Shan ZHAO,Hong-wei LI,Yi-rui JIANG. Spatial global context information network for semantic segmentation of remote sensing image. Journal of ZheJiang University (Engineering Science), 2022, 56(4): 795-802.

URL:

https://www.zjujournals.com/eng/10.3785/j.issn.1008-973X.2022.04.019     OR     https://www.zjujournals.com/eng/Y2022/V56/I4/795


遥感图像语义分割空间全局上下文信息网络

为了解决卷积神经网络(CNN)在语义分割特征提取阶段容易丢失空间信息以及边界信息不明确的问题，基于U-Net基线网络提出空间全局上下文信息网络(NC-Net). 增加再编码阶段(ReEncoder)，以增强空间信息识别能力. 在Decoder阶段输出多尺度特征，与ReEncoder阶段结合获取全局上下文信息. 保留边界损失函数，设计多尺度损失函数级联方法，优化整体网络. 在GID以及WHDLD数据集上的实验结果表明，该方法的总体准确度达到最好成绩，明显优于其他基线模型.


关键词： 语义分割,  遥感影像,  空间信息,  全局上下文,  神经网络 
Fig.1 U-Net network structure
Fig.2 NC-Net network structure
Fig.3 Multi-scale output fusion re-sampling process
Fig.4 Multi-scale loss function cascade framework
方法 OA K FWIoU F1
U-Net[9] 70.721 51.857 56.135 70.131
DeepLabv3[18] 74.072 60.702 61.330 75.247
PSPNet[12] 72.513 59.147 60.880 75.782
DeepLabv3+[19] 75.982 61.958 63.021 77.920
NC-Net 77.331 62.214 64.550 78.694
Tab.1 Experimental results of GID %
Fig.5 Visual overall comparison between NC-Net and other models
Fig.6 Local visual comparison between NC-Net and other models
标签类别 Pr R F1
U-
Net[9]		 Deep-
Labv3[18]		 PSP-
Net[12]		 Deep-
Labv3+[19]		 NC-
Net 		U-
Net[9]		 Deep-
Labv3[18]		 PSP-
Net[12]		 Deep-
Labv3+[19]		 NC-
Net 		U-
Net[9]		 Deep-
Labv3[18]		 PSP-
Net[12]		 Deep-
Labv3+[19]		 NC-
Net
水浇地 85.61 84.95 81.77 88.22 86.73 83.69 86.91 91.59 89.73 91.94 84.63 85.91 86.40 88.96 89.25
园地 33.25 19.71 24.58 37.08 38.83 1.17 1.32 1.58 5.51 8.12 2.26 2.47 2.96 9.59 13.43
工业区 54.01 62.69 55.73 79.15 74.27 43.07 62.85 55.73 50.47 51.39 47.92 62.76 55.73 61.52 60.74
村镇住宅 73.97 82.92 76.73 78.77 90.65 64.75 74.15 75.52 83.78 65.10 69.05 78.29 76.12 81.19 75.77
交通用地 66.45 72.15 45.53 68.21 67.05 17.31 42.56 35.58 61.63 47.45 27.46 53.53 39.94 64.75 55.57
河流 39.91 23.70 21.47 22.43 42.87 3.95 1.21 4.25 4.43 5.30 7.18 2.30 7.09 7.39 9.43
湖泊 87.77 89.48 88.97 85.93 86.09 16.22 20.40 3.56 39.31 74.89 27.38 33.22 6.84 53.94 80.10
其他 38.62 39.23 34.07 43.90 46.69 48.25 61.84 44.02 59.59 62.65 42.90 48.00 38.41 50.55 53.50
Tab.2 Quantitative indicators for each category of GID %
方法 OA K FWIoU F1
U-Net[9] 76.218 70.816 69.709 70.066
DeepLabv3[18] 80.053 69.380 71.252 72.391
PSPNet[12] 78.406 71.528 69.039 71.002
DeepLabv3+[19] 81.295 76.524 73.157 75.563
NC-Net 84.897 79.944 76.025 76.301
Tab.3 WHDLD experimental results %
Fig.7 Visualization results of WHDLD
标签类别 Pr R F1
U-
Net[9]		 Deep-
Labv3[18]		 PSP-
Net[12]		 Deep-
Labv3+[19]		 NC-
Net 		U-
Net[9]		 Deep-
Labv3[18]		 PSP-
Net[12]		 Deep-
Labv3+[19]		 NC-
Net 		U-
Net[9]		 Deep-
Labv3[18]		 PSP-
Net[12]		 Deep-
Labv3+[19]		 NC-
Net
建筑物 67.72 70.28 69.53 72.36 72.78 68.07 68.93 70.33 72.98 72.19 67.89 69.59 69.92 72.16 72.48
道路 68.65 73.06 68.86 72.20 75.71 73.66 82.61 75.26 87.01 90.36 71.06 77.54 71.91 78.91 82.78
人行道 70.88 69.51 71.74 75.74 70.54 61.28 60.02 63.85 62.55 60.36 65.73 64.41 67.56 67.68 65.04
植被 65.07 66.03 65.91 68.04 68.98 68.80 70.33 69.04 70.83 71.12 66.88 68.11 67.43 69.40 70.03
裸地 71.92 74.43 72.09 75.97 76.26 66.31 69.95 70.72 71.76 72.07 69.00 72.12 71.39 72.84 74.10
水体 80.53 85.82 84.25 89.61 91.01 81.56 84.61 83.78 84.06 86.47 81.04 85.21 84.01 86.74 89.15
Tab.4 Quantitative indicators for each category of WHDLD %
%
方法 OA K FWIoU F1
U-NetAtt[13] 82.602 75.484 73.474 69.622
U-Net++[21] 84.067 77.430 74.496 74.633
MACU-Net[20] 84.623 78.233 75.231 75.245
NC-Net 84.897 79.944 76.025 76.301
Tab.5 Comparison of NC-Net with other improved networks on WHDLD
[1]   NORONHA S, NEVATIA R Detection and modeling of buildings from multiple aerial images[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2001, 23 (5): 501- 518
doi: 10.1109/34.922708
[2]   COTE M, SAEEDI P Automatic rooftop extraction in nadir aerial imagery of suburban regions using corners and variational level set evolution[J]. IEEE Transactions on Geoscience and Remote Sensing, 2012, 51 (1): 313- 328
[3]   LI E, FEMIANI J, XU S, et al Robust rooftop extraction from visible band images using higher order CRF[J]. IEEE Transactions on Geoscience and Remote Sensing, 2015, 53 (8): 4483- 4495
doi: 10.1109/TGRS.2015.2400462
[4]   胡翔云, 巩晓雅, 张觅 变分法遥感影像人工地物自动检测[J]. 测绘学报, 2018, 47 (6): 780- 789
HU Xiang-yun, GONG Xiao-ya, ZHANG Mi A variational approach for automatic man-made object detection from remote sensing image[J]. Acta Geodaetica at Cartographica Sinica, 2018, 47 (6): 780- 789
doi: 10.11947/j.AGCS.2018.20170642
[5]   林祥国, 张继贤 面向对象的形态学建筑物指数及其高分辨率遥感影像建筑物提取应用[J]. 测绘学报, 2017, 46 (6): 724- 733
LIN Xiang-guo, ZHANG Ji-xian Object-based morphological building index for building extraction form high resolution remote sensing imagery[J]. Acta Geodaetica at Cartographica Sinica, 2017, 46 (6): 724- 733
doi: 10.11947/j.AGCS.2017.20170068
[6]   李道纪, 郭海涛, 卢俊, 等 遥感影像地物分类多注意力融和U型网络法[J]. 测绘学报, 2020, 49 (8): 1051- 1064
LI Dao-ji, GUO Hai-tao, LU Jun, et al A remote sensing image classification procedure based on multilevel attention fusion UNet[J]. Acta Geodaetica at Cartographica Sinica, 2020, 49 (8): 1051- 1064
doi: 10.11947/j.AGCS.2020.20190407
[7]   LIU Z, LUO P, WANG X, et al. Deep learning face attributes in the wild [C]//Proceedings of the IEEE International Conference on Computer Vision. Santiago: IEEE, 2015: 3730-3738.
[8]   LONG J, SHELHAMER E, DARRElLL T. Fully convolutional networks for semantic segmentation [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Santiago: IEEE, 2015: 3431-3440.
[9]   RONNEBERGER O, FISCHER P, BROX T. U-net: convolutional networks for biomedical image segmentation [C]//International Conference on Medical Image Computing and Computer-Assisted Intervention. Cham: Springer, 2015: 234-241.
[10]   BADRINARAYANAN V, KENDALL A, CIPOLLA R Segnet: a deep convolutional encoder-decoder architecture for image segmentation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39 (12): 2481- 2495
doi: 10.1109/TPAMI.2016.2644615
[11]   SIMONYAN K, VEDALDI A, ZISSERMAN A. Deep inside convolutional networks: visualizing image classification models and saliency maps [C]// Workshop at International Conference on Learning Representations. Banff: IEEE, 2014.
[12]   ZhAO H, SHI J, QI X, et al. Pyramid scene parsing network [C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017: 2881-2890.
[13]   OKTAY O, SCHLEMPER J, FOLGOC L L, et al. Attention u-net: learning where to look for the pancreas [EB/OL]. (2018-04-11). https://arxiv.org/abs/1804.03999.
[14]   LIU P, WEI Y, WANG Q, et al Research on post-earthquake landslide extraction algorithm based on improved U-Net model[J]. Remote Sensing, 2020, 12 (5): 894
doi: 10.3390/rs12050894
[15]   TONG X Y, XIA G S, LU Q, et al Land-cover classification with high-resolution remote sensing images using transferable deep models[J]. Remote Sensing of Environment, 2020, 237: 111322
[16]   SHAO Z, YANG K, ZHOU W Performance evaluation of single-label and multi-label remote sensing image retrieval using a dense labeling dataset[J]. Remote Sensing, 2018, 10 (6): 964
doi: 10.3390/rs10060964
[17]   SHAO Z, ZHOU W, DENG X, et al Multilabel remote sensing image retrieval based on fully convolutional network[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2020, 13 (1): 318- 328
doi: 10.1109/JSTARS.2019.2961634
[18]   ChEN L C, PAPANDREOU G, SCHROFF F, et al. Rethinking atrous convolution for semantic image segmentation [EB/OL]. (2017-06-17). https://arxiv.org/abs/1706.05587.
[19]   CHEN L C, ZHU Y, PAPANDREOU G, et al. Encoderdecoder with atrous separable convolution for semantic image segmentation [C]// Proceedings of the European Conference on Computer Vision. Munich: Springer, 2018: 801-818.
[20]   RUI L, CEHNXI D, SHUNYI Z. MACU-Net semantic segmentation from high-resolution remote sensing images [EB/OL]. [2020-07-26]. https://arxiv.org/abs/2007.13083.
[1] Yun-hao WANG,Ming-hui SUN,Yi XIN,Bo-xuan ZHANG. Robot tactile recognition system based on piezoelectric film sensor[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(4): 702-710.
[2] Ruo-ran CHENG,Xiao-li ZHAO,Hao-jun ZHOU,Han-chen YE. Review of Chinese font style transfer research based on deep learning[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(3): 510-519, 530.
[3] Ting WANG,Xiao-fei ZHU,Gu TANG. Knowledge-enhanced graph convolutional neural networks for text classification[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(2): 322-328.
[4] Pei-zhi WEN,Jun-mou CHEN,Yan-nan XIAO,Ya-yuan WEN,Wen-ming HUANG. Underwater image enhancement algorithm based on GAN and multi-level wavelet CNN[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(2): 213-224.
[5] Guang-long LI,De-rong SHEN,Tie-zheng NIE,Yue KOU. Learning query optimization method based on multi model outside database[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(2): 288-296.
[6] Song REN,Qian-wen ZHU,Xin-yue TU,Chao DENG,Xiao-shu WANG. Lining disease identification of highway tunnel based on deep learning[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(1): 92-99.
[7] Fa-ming HUANG,Li-han PAN,Chi YAO,Chuang-bing ZHOU,Qing-hui JIANG,Zhi-lu CHANG. Landslide susceptibility prediction modelling based on semi-supervised machine learning[J]. Journal of ZheJiang University (Engineering Science), 2021, 55(9): 1705-1713.
[8] Nan ZHANG,Hong-zhao DONG,Yi-ni SHE. Seq2Seq prediction of bus trajectory on exclusive bus lanes[J]. Journal of ZheJiang University (Engineering Science), 2021, 55(8): 1482-1489.
[9] Fei WANG,Wei-xiang XU. Improved model of road impedance function based on LSTM neural network[J]. Journal of ZheJiang University (Engineering Science), 2021, 55(6): 1065-1071.
[10] Yan HAO,Ya-bin DING,Jin-sheng FU. Hierarchical closed-loop optimization strategy for cumulative error of robot machining system[J]. Journal of ZheJiang University (Engineering Science), 2021, 55(6): 1142-1149.
[11] Jia-hui XU,Jing-chang WANG,Ling CHEN,Yong WU. Surface water quality prediction model based on graph neural network[J]. Journal of ZheJiang University (Engineering Science), 2021, 55(4): 601-607.
[12] Hong CHENG,Jia-jie HU,Yong LIU,Yuan-qing YE. Three-dimensional reconstruction algorithm based on fusion of transport of intensity equation and neural network[J]. Journal of ZheJiang University (Engineering Science), 2021, 55(4): 658-664.
[13] Ying-jie ZHENG,Song-rong WU,Ruo-yu WEI,Zhen-wei TU,Jin LIAO,Dong LIU. Metro location point matching and false alarm elimination based on FCM algorithm of target image[J]. Journal of ZheJiang University (Engineering Science), 2021, 55(3): 586-593.
[14] Yi-zhe MAO,Guo-fang GONG,Xing-hai ZHOU,Fei WANG. Identification of TBM surrounding rock based on Markov process and deep neural network[J]. Journal of ZheJiang University (Engineering Science), 2021, 55(3): 448-454.
[15] Fang LIU,Zhen WANG,Rui-di LIU,Kai WANG. Short-term forecasting method of wind power generation based on BP neural network with combined loss function[J]. Journal of ZheJiang University (Engineering Science), 2021, 55(3): 594-600.