Please wait a minute...
Journal of ZheJiang University (Engineering Science)  2022, Vol. 56 Issue (11): 2251-2259    DOI: 10.3785/j.issn.1008-973X.2022.11.016
    
3D point cloud super-resolution with dynamic residual graph convolutional networks
Fan ZHONG(),Zheng-yao BAI*()
School of Information Science and Engineering, Yunnan University, Kunming 650500, China
Download: HTML     PDF(1454KB) HTML
Export: BibTeX | EndNote (RIS)      

Abstract  

A 3D point cloud super-resolution network with dynamic residual graph convolution (PSR-DRGCN) was proposed to efficiently extract of local information from 3D point clouds of non-European data in super-resolution. The network includes feature extraction module, DRGCN module and upsampling module. For the input point cloud, the feature extraction module locates k nearest points of each point in 3D space by k-NN algorithm and then converts the local geometry information into the high dimensional feature space through a multi-layer pointwise convolution. The DRGCN module converts the local geometry feature of each point into the semantic feature through a multi-layer graph convolution. It dynamically adjusts the neighbor space of the point in each layer to increase the receptive field range and effectively fuse the semantic information of different levels through residual connection, which makes the extraction of local geometric information efficient. The upsampling module adds the number of points and maps them from feature space to 3D space. The results showed that at 2× magnification of the high-resolution point cloud generated by PSR-DRGCN, the similarity indexes CD, EMD and F-score compared with the second network were increased by 10.00%, 4.76% and 16.84% respectively. Compared with the second network, the similarity indexes at 6× magnification were increased by 2.35%, 40.00% and 0.58% respectively. In all cases, the optimal effect was achieved on the mean and the std indicators and the generated high-resolution point cloud quality was high.



Key words3D point cloud      super-resolution      dynamic GCN      semantic feature      deep learning     
Received: 29 November 2021      Published: 02 December 2022
CLC:  TP 391.4  
Fund:  云南省重大科技专项计划资助项目(202002AD080001)
Corresponding Authors: Zheng-yao BAI     E-mail: zffhost@mail.ynu.edu.cn;baizhy@ynu.edu.cn
Cite this article:

Fan ZHONG,Zheng-yao BAI. 3D point cloud super-resolution with dynamic residual graph convolutional networks. Journal of ZheJiang University (Engineering Science), 2022, 56(11): 2251-2259.

URL:

https://www.zjujournals.com/eng/10.3785/j.issn.1008-973X.2022.11.016     OR     https://www.zjujournals.com/eng/Y2022/V56/I11/2251


采用动态残差图卷积的3D点云超分辨率

为了在超分辨率中对非欧数据的3D点云进行局部信息高效提取,提出采用动态残差图卷积的3D点云超分辨率网络(PSR-DRGCN),该网络包括特征提取模块、DRGCN模块及上采样模块. 对于输入的3D点云, 特征提取模块采用k-近邻(k-NN)算法在3D空间中找到每个点对应的k个邻居,通过逐点卷积把局部几何信息转换到高维特征空间中;DRGCN模块利用多层图卷积操作将3D空间中每个点的局部几何特征抽象为语义特征,在每一层对点的近邻空间进行动态调整以增加感受野范围,并通过残差连接融合多层次语义信息,从而对局部几何信息高效提取;上采样模块将特征空间中的点进行上采样并转换到3D空间中. 实验结果表明,PSR-DRGCN生成的高分辨率点云在放大尺度为2倍时,相似性指标CD、EMD、F-score相比第2网络分别优化了10.00%,4.76%,16.84%;当放大尺度为6倍时,相似性指标相比第2网络分别优化了2.35%,40.00%,0.58%;在所有情况下的均值与标准差指标上达到最优效果,生成的高分辨率点云质量高.


关键词: 3D点云,  超分辨率,  动态图卷积网络,  语义特征,  深度学习 
Fig.1 llustration of PSR-DRGCN architecture
Fig.2 DRGCN module
Fig.3 P-DRGCN module
Fig.4 Scale predict network
Fig.5 Upsampling module
方法 CD EMD F-score NUC mean std CD EMD F-score NUC mean std
注:CD、EMD、NUC、mean、std指标均为10?3数量级
AR-GCN ? ? ? ? ? ? 8.6 18.0 70.09% 339.0 2.9 3.3
PU-GAN 16.0 9.0 32.17% 249.0 12.0 15.0 9.7 16.0 69.75% 202.0 3.0 3.1
AR-GCN x16sample 15.0 13.0 36.98% 273.0 6.7 8.2 13.0 13.0 54.05% 288.0 6.6 8.0
PC-SR 15.0 12.0 52.82% 188.0 2.8 3.3 14.0 19.0 72.18% 211.0 2.8 3.1
Meta-PU 10.0 6.3 53.20% 163.0 2.6 2.9 8.7 7.8 74.05% 192.0 2.6 2.7
PSR-DRGCN 9.0 6.0 70.06% 154.0 2.4 2.3 8.8 7.6 70.12% 183.0 2.4 2.2
方法 CD EMD F-score NUC mean std CD EMD F-score NUC mean std
AR-GCN ? ? ? ? ? ? 8.1 22.0 74.63% 344.0 3.4 4.4
PU-GAN 12.0 13.0 58.56% 287.0 11.0 18.0 9.1 8.5 70.61% 212.0 4.7 5.7
AR-GCN x16sample 12.0 14.0 59.41% 293.0 6.5 7.9 11 14.0 62.70% 298.0 6.5 7.8
PC-SR 14.0 22.0 70.02% 225.0 2.7 2.9 9.3 25.0 72.92% 232.0 2.6 2.8
Meta-PU 8.5 14.0 72.98% 267.0 2.5 3.0 8.3 16.0 73.74% 274.0 3.0 3.4
PSR-DRGCN 8.3 8.4 73.56% 243.0 2.3 2.1 7.9 9.8 74.23% 210.0 2.2 2.0
Tab.1 Performance comparison in point cloud super-resolution
网络 D/(10?2) k t/ms
mean std
Pointnet++ 3.13 3.74 1 480 25.30
PU-net 0.46 0.55 777 10.40
AR-GCN 0.29 0.33 785 15.60
PU-GAN 0.30 0.31 684 14.30
DGCNN 2.90 3.20 1 842 41.60
PC-SR 2.64 2.90 844 21.30
Meta-PU 0.29 0.27 2 756 90.20
PSR-DRGCN 0.24 0.22 1 282 12.30
Tab.2 Performance comparison in parameters and inference time
方法 CD EMD F-score NUC mean std
注:CD、EMD、NUC、mean、std指标均为10?3数量级
RGCN 11.0 15.0 63.36% 224.0 55.0 57.0
DRGCN 9.0 6.0 70.06% 154.0 2.4 2.3
Tab.3 Performance comparison in 2× dynamic neighbor and residual connection
Fig.6 Visual performance compression in 2×, 4×, 6×, 9×, 16× scale
Fig.7 visual performance compression in different SR networks(pig)
Fig.8 error visual performance compression in different SR networks
[1]   YANG J, WRIGHT J, HUANG T S, et al Image super-resolution via sparse representation[J]. IEEE Transactions on Image Processing, 2010, 19 (11): 2861- 2873
doi: 10.1109/TIP.2010.2050625
[2]   QIAO C, LI D, GUO Y, et al Evaluation and development of deep neural networks for image super-resolution in optical microscopy[J]. Nature Methods, 2021, 18 (2): 194- 202
doi: 10.1038/s41592-020-01048-5
[3]   段然, 周登文, 赵丽娟, 等 基于多尺度特征映射网络的图像超分辨率重建[J]. 浙江大学学报: 工学版, 2019, 53 (7): 1331- 1339
DUAN Ran, ZHOU Deng-Wen, ZHAO Li-Juan, et al Image super-resolution reconstruction based on multi-scale feature mapping network[J]. Journal of Zhejiang University: Engineering Science, 2019, 53 (7): 1331- 1339
[4]   DONG C, LOY C C, HE K, et al Image super-resolution using deep convolutional networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015, 38 (2): 295- 307
[5]   CHEN Y, LIU L, PHONEVILAY V, et al Image super-resolution reconstruction based on feature map attention mechanism[J]. Applied Intelligence, 2021, (8): 1- 14
[6]   WU H, ZHANG J, HUANG K. Point cloud super resolution with adversarial residual graph networks [EB/OL]. [2019-08-06]. https://arxiv.org/pdf/1908.0211.pdf .
[7]   YU L, LI X, FU C W, et al. Pu-net: point cloud upsampling network [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 2790-2799.
[8]   LI R, LI X, FU C W, et al. Pu-gan: a point cloud upsampling adversarial network [C]// Proceedings of the IEEE/CVF International Conference on Computer Vision. Seoul: IEEE, 2019: 7203-7212.
[9]   QI C R, SU H, MO K, et al. Pointnet: deep learning on point sets for 3d classification and segmentation [C]// Proceedings of the IEEE conference on computer vision and pattern recognition. Hawaii: IEEE, 2017: 652-660.
[10]   LI R, LI X, HENG P A, et al. Point cloud upsampling via Disentangled Refinement [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Online: IEEE, 2021: 344-353.
[11]   QI C R, YI L, SU H, et al. Pointnet++: deep hierarchical feature learning on point sets in a metric space [C]// Proceedings of Neural Information Processing Systems. Long Beach: NIPS, 2017: 5099-5108.
[12]   彭秀平, 仝其胜, 林洪彬, 等 一种面向散乱点云语义分割的深度残差特征金字塔网络框架[J]. 自动化学报, 2021, 47 (12): 2831- 2840
PENG Xiu-Ping, TONG Qi-Sheng, LIN Hong-Bin, et al A deep residual-feature pyramid network for scattered point cloud semantic segmentation[J]. Acta Automatica Sinica, 2021, 47 (12): 2831- 2840
[13]   MATURANA D, SCHERER S. Voxnet: a 3d convolutional neural network for real-time object recognition [C]// Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems. Hamburg: IEEE, 2015: 922-928.
[14]   MAO J, XUE Y, NIU M, et al. Voxel transformer for 3d object detection [C]// Proceedings of the IEEE/CVF International Conference on Computer Vision. Online: IEEE, 2021: 3164-3173.
[15]   WANG G, XU G, WU Q, et al Two-stage point cloud super resolution with local interpolation and readjustment via outer-product neural network[J]. Journal of Systems Science and Complexity, 2021, 34 (1): 68- 82
doi: 10.1007/s11424-020-9266-x
[16]   SIMONOYSKY M, KOMODAKIS N. Dynamic edge-conditioned flters in convolutional neural networks on graphs [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Hawaii: IEEE, 2017: 3693-3702.
[17]   TE G, HU W, ZHENG A, et al. Rgcnn: regularized graph cnn for point cloud segmentation [C]// Proceedings of the 26th ACM International Conference on Multimedia. Lisboa: ACM, 2018: 746-754.
[18]   QIAN G, ABUALSHOUR A, LI G, et al. Pu-gcn: point cloud upsampling using graph convolutional networks [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Online: IEEE, 2021: 11683-11692.
[19]   GUO M H, CAI J X, LIU Z N, et al Pct: point cloud transformer[J]. Computational Visual Media, 2021, 7 (2): 187- 199
doi: 10.1007/s41095-021-0229-5
[20]   YU X, RAO Y, WANG Z, et al. Pointr: diverse point cloud completion with geometry-aware transformers [C]// Proceedings of the IEEE/CVF International Conference on Computer Vision. Online: IEEE, 2021: 12498-12507.
[21]   HE K, ZHANG X, REN S, et al. Deep residual learning for image recognition [C]// Proceedings of the IEEE conference on computer vision and pattern recognition. Las Vegas: IEEE, 2016: 770-778.
[22]   KIPF T N, WELLING M. Semi-supervised classification with graph convolutional networks [EB/OL]. [2016-09-09]. https://arxiv.org/pdf/1609.02907.pdf.
[23]   LUO C, CHEN Y, WANG N, et al. Spectral feature transformation for person re-identification [C]// Proceedings of the IEEE/CVF International Conference on Computer Vision. Seoul: IEEE, 2019: 4976-4985.
[24]   HU X, MU H, ZHANG X, et al. Meta-SR: a magnification-arbitrary network for super-resolution [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2019: 1575-1584.
[25]   BUTT M A, MARAGOS P Optimum design of chamfer distance transforms[J]. IEEE Transactions on Image Processing, 1998, 7 (10): 1477- 1484
doi: 10.1109/83.718487
[26]   CUTURI M. Sinkhorn distances: lightspeed computation of optimal transport [C]// Proceedings of the 26th International Conference on Neural Information Processing Systems. Lake Tahoe: NIPS, 2013: 2292-2300.
[27]   YE S, CHEN D, HAN S, et al Meta-PU: an arbitrary-scale upsampling network for point cloud[J]. IEEE Transactions on Visualization and Computer Graphics, 2022, 28 (9): 3206- 3218
doi: 10.1109/TVCG.2021.3058311
[28]   BOLTZ S, NIELSEN F, SOATTO S. Earth mover distance on superpixels [C]// IEEE International Conference on Image Processing. Hong Kong: IEEE, 2010: 4597-4600.
[29]   SOKOLOVA M, JAPKOWICZ N, SZPAKOWICZ S. Beyond accuracy, F-score and ROC: a family of discriminant measures for performance evaluation [C]// Australasian Joint Conference on Artificial Intelligence. Berlin: Springer, 2006: 1015-1021.
[1] Jin-zhen LIU,Fei CHEN,Hui XIONG. Open electrical impedance imaging algorithm based on multi-scale residual network model[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(9): 1789-1795.
[2] Wan-liang WANG,Tie-jun WANG,Jia-cheng CHEN,Wen-bo YOU. Medical image segmentation method combining multi-scale and multi-head attention[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(9): 1796-1805.
[3] Kun HAO,Kuo WANG,Bei-bei WANG. Lightweight underwater biological detection algorithm based on improved Mobilenet-YOLOv3[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(8): 1622-1632.
[4] Yong-sheng ZHAO,Rui-xiang LI,Na-na NIU,Zhi-yong ZHAO. Shape control method of fuselage driven by digital twin[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(7): 1457-1463.
[5] Li HE,Shan-min PANG. Face reconstruction from voice based on age-supervised learning and face prior information[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(5): 1006-1016.
[6] Xue-qin ZHANG,Tian-ren LI. Breast cancer pathological image classification based on Cycle-GAN and improved DPN network[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(4): 727-735.
[7] Jing-hui CHU,Li-dong SHI,Pei-guang JING,Wei LV. Context-aware knowledge distillation network for object detection[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(3): 503-509.
[8] Ruo-ran CHENG,Xiao-li ZHAO,Hao-jun ZHOU,Han-chen YE. Review of Chinese font style transfer research based on deep learning[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(3): 510-519, 530.
[9] Xiang-dong PENG,Cong-cheng PAN,Ze-jun KE,Hua-qiang ZHU,Xiao ZHOU. Classification method for electrocardiograph signals based on parallel architecture model and spatial-temporal attention mechanism[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(10): 1912-1923.
[10] Dong-jie YANG,Xian-jun GAO,Shu-hao RAN,Guang-bin ZHANG,Ping WANG,Yuan-wei YANG. Building extraction based on multiple multiscale-feature fusion attention network[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(10): 1924-1934.
[11] Wei-hang CHEN,Qiang LUO,Teng-fei WANG,Wen-sheng ZHANG,Liang-wei JIANG. Reliability analysis of post-construction settlement of DMC composite foundation and design optimization[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(10): 2019-2027.
[12] Tong CHEN,Jian-feng GUO,Xin-zhong HAN,Xue-li XIE,Jian-xiang XI. Visible and infrared image matching method based on generative adversarial model[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(1): 63-74.
[13] Song REN,Qian-wen ZHU,Xin-yue TU,Chao DENG,Xiao-shu WANG. Lining disease identification of highway tunnel based on deep learning[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(1): 92-99.
[14] Xing LIU,Jian-bo YU. Attention convolutional GRU-based autoencoder and its application in industrial process monitoring[J]. Journal of ZheJiang University (Engineering Science), 2021, 55(9): 1643-1651.
[15] Xue-yun CHEN,Xiao-qiao HUANG,Li XIE. Classification and detection method of blood cells images based on multi-scale conditional generative adversarial network[J]. Journal of ZheJiang University (Engineering Science), 2021, 55(9): 1772-1781.