Please wait a minute...
浙江大学学报(工学版)  2022, Vol. 56 Issue (11): 2251-2259    DOI: 10.3785/j.issn.1008-973X.2022.11.016
计算机技术     
采用动态残差图卷积的3D点云超分辨率
钟帆(),柏正尧*()
云南大学 信息学院,云南 昆明 650500
3D point cloud super-resolution with dynamic residual graph convolutional networks
Fan ZHONG(),Zheng-yao BAI*()
School of Information Science and Engineering, Yunnan University, Kunming 650500, China
 全文: PDF(1454 KB)   HTML
摘要:

为了在超分辨率中对非欧数据的3D点云进行局部信息高效提取,提出采用动态残差图卷积的3D点云超分辨率网络(PSR-DRGCN),该网络包括特征提取模块、DRGCN模块及上采样模块. 对于输入的3D点云, 特征提取模块采用k-近邻(k-NN)算法在3D空间中找到每个点对应的k个邻居,通过逐点卷积把局部几何信息转换到高维特征空间中;DRGCN模块利用多层图卷积操作将3D空间中每个点的局部几何特征抽象为语义特征,在每一层对点的近邻空间进行动态调整以增加感受野范围,并通过残差连接融合多层次语义信息,从而对局部几何信息高效提取;上采样模块将特征空间中的点进行上采样并转换到3D空间中. 实验结果表明,PSR-DRGCN生成的高分辨率点云在放大尺度为2倍时,相似性指标CD、EMD、F-score相比第2网络分别优化了10.00%,4.76%,16.84%;当放大尺度为6倍时,相似性指标相比第2网络分别优化了2.35%,40.00%,0.58%;在所有情况下的均值与标准差指标上达到最优效果,生成的高分辨率点云质量高.

关键词: 3D点云超分辨率动态图卷积网络语义特征深度学习    
Abstract:

A 3D point cloud super-resolution network with dynamic residual graph convolution (PSR-DRGCN) was proposed to efficiently extract of local information from 3D point clouds of non-European data in super-resolution. The network includes feature extraction module, DRGCN module and upsampling module. For the input point cloud, the feature extraction module locates k nearest points of each point in 3D space by k-NN algorithm and then converts the local geometry information into the high dimensional feature space through a multi-layer pointwise convolution. The DRGCN module converts the local geometry feature of each point into the semantic feature through a multi-layer graph convolution. It dynamically adjusts the neighbor space of the point in each layer to increase the receptive field range and effectively fuse the semantic information of different levels through residual connection, which makes the extraction of local geometric information efficient. The upsampling module adds the number of points and maps them from feature space to 3D space. The results showed that at 2× magnification of the high-resolution point cloud generated by PSR-DRGCN, the similarity indexes CD, EMD and F-score compared with the second network were increased by 10.00%, 4.76% and 16.84% respectively. Compared with the second network, the similarity indexes at 6× magnification were increased by 2.35%, 40.00% and 0.58% respectively. In all cases, the optimal effect was achieved on the mean and the std indicators and the generated high-resolution point cloud quality was high.

Key words: 3D point cloud    super-resolution    dynamic GCN    semantic feature    deep learning
收稿日期: 2021-11-29 出版日期: 2022-12-02
CLC:  TP 391.4  
基金资助: 云南省重大科技专项计划资助项目(202002AD080001)
通讯作者: 柏正尧     E-mail: zffhost@mail.ynu.edu.cn;baizhy@ynu.edu.cn
作者简介: 钟帆(1997—),男,硕士生,从事基于深度学习3D建模研究. orcid.org/0000-0002-5392-5926. E-mail: zffhost@mail.ynu.edu.cn
服务  
把本文推荐给朋友
加入引用管理器
E-mail Alert
作者相关文章  
钟帆
柏正尧

引用本文:

钟帆,柏正尧. 采用动态残差图卷积的3D点云超分辨率[J]. 浙江大学学报(工学版), 2022, 56(11): 2251-2259.

Fan ZHONG,Zheng-yao BAI. 3D point cloud super-resolution with dynamic residual graph convolutional networks. Journal of ZheJiang University (Engineering Science), 2022, 56(11): 2251-2259.

链接本文:

https://www.zjujournals.com/eng/CN/10.3785/j.issn.1008-973X.2022.11.016        https://www.zjujournals.com/eng/CN/Y2022/V56/I11/2251

图 1  PSR-DRGCN框架图
图 2  DRGCN模块
图 3  P-DRGCN模块
图 4  尺度预测网络
图 5  上采样模块
方法 CD EMD F-score NUC mean std CD EMD F-score NUC mean std
注:CD、EMD、NUC、mean、std指标均为10?3数量级
AR-GCN ? ? ? ? ? ? 8.6 18.0 70.09% 339.0 2.9 3.3
PU-GAN 16.0 9.0 32.17% 249.0 12.0 15.0 9.7 16.0 69.75% 202.0 3.0 3.1
AR-GCN x16sample 15.0 13.0 36.98% 273.0 6.7 8.2 13.0 13.0 54.05% 288.0 6.6 8.0
PC-SR 15.0 12.0 52.82% 188.0 2.8 3.3 14.0 19.0 72.18% 211.0 2.8 3.1
Meta-PU 10.0 6.3 53.20% 163.0 2.6 2.9 8.7 7.8 74.05% 192.0 2.6 2.7
PSR-DRGCN 9.0 6.0 70.06% 154.0 2.4 2.3 8.8 7.6 70.12% 183.0 2.4 2.2
方法 CD EMD F-score NUC mean std CD EMD F-score NUC mean std
AR-GCN ? ? ? ? ? ? 8.1 22.0 74.63% 344.0 3.4 4.4
PU-GAN 12.0 13.0 58.56% 287.0 11.0 18.0 9.1 8.5 70.61% 212.0 4.7 5.7
AR-GCN x16sample 12.0 14.0 59.41% 293.0 6.5 7.9 11 14.0 62.70% 298.0 6.5 7.8
PC-SR 14.0 22.0 70.02% 225.0 2.7 2.9 9.3 25.0 72.92% 232.0 2.6 2.8
Meta-PU 8.5 14.0 72.98% 267.0 2.5 3.0 8.3 16.0 73.74% 274.0 3.0 3.4
PSR-DRGCN 8.3 8.4 73.56% 243.0 2.3 2.1 7.9 9.8 74.23% 210.0 2.2 2.0
表 1  点云超分辨率对比
网络 D/(10?2) k t/ms
mean std
Pointnet++ 3.13 3.74 1 480 25.30
PU-net 0.46 0.55 777 10.40
AR-GCN 0.29 0.33 785 15.60
PU-GAN 0.30 0.31 684 14.30
DGCNN 2.90 3.20 1 842 41.60
PC-SR 2.64 2.90 844 21.30
Meta-PU 0.29 0.27 2 756 90.20
PSR-DRGCN 0.24 0.22 1 282 12.30
表 2  参数与推理时间对比结果
方法 CD EMD F-score NUC mean std
注:CD、EMD、NUC、mean、std指标均为10?3数量级
RGCN 11.0 15.0 63.36% 224.0 55.0 57.0
DRGCN 9.0 6.0 70.06% 154.0 2.4 2.3
表 3  2倍动态调整近邻空间与残差连接消融对比
图 6  2×、4×、6×、9×、16×上采样尺度可视化对比图
图 7  不同网络超分辨率可视化对比图(猪)
图 8  不同网络超分辨率误差图
1 YANG J, WRIGHT J, HUANG T S, et al Image super-resolution via sparse representation[J]. IEEE Transactions on Image Processing, 2010, 19 (11): 2861- 2873
doi: 10.1109/TIP.2010.2050625
2 QIAO C, LI D, GUO Y, et al Evaluation and development of deep neural networks for image super-resolution in optical microscopy[J]. Nature Methods, 2021, 18 (2): 194- 202
doi: 10.1038/s41592-020-01048-5
3 段然, 周登文, 赵丽娟, 等 基于多尺度特征映射网络的图像超分辨率重建[J]. 浙江大学学报: 工学版, 2019, 53 (7): 1331- 1339
DUAN Ran, ZHOU Deng-Wen, ZHAO Li-Juan, et al Image super-resolution reconstruction based on multi-scale feature mapping network[J]. Journal of Zhejiang University: Engineering Science, 2019, 53 (7): 1331- 1339
4 DONG C, LOY C C, HE K, et al Image super-resolution using deep convolutional networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015, 38 (2): 295- 307
5 CHEN Y, LIU L, PHONEVILAY V, et al Image super-resolution reconstruction based on feature map attention mechanism[J]. Applied Intelligence, 2021, (8): 1- 14
6 WU H, ZHANG J, HUANG K. Point cloud super resolution with adversarial residual graph networks [EB/OL]. [2019-08-06]. https://arxiv.org/pdf/1908.0211.pdf .
7 YU L, LI X, FU C W, et al. Pu-net: point cloud upsampling network [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 2790-2799.
8 LI R, LI X, FU C W, et al. Pu-gan: a point cloud upsampling adversarial network [C]// Proceedings of the IEEE/CVF International Conference on Computer Vision. Seoul: IEEE, 2019: 7203-7212.
9 QI C R, SU H, MO K, et al. Pointnet: deep learning on point sets for 3d classification and segmentation [C]// Proceedings of the IEEE conference on computer vision and pattern recognition. Hawaii: IEEE, 2017: 652-660.
10 LI R, LI X, HENG P A, et al. Point cloud upsampling via Disentangled Refinement [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Online: IEEE, 2021: 344-353.
11 QI C R, YI L, SU H, et al. Pointnet++: deep hierarchical feature learning on point sets in a metric space [C]// Proceedings of Neural Information Processing Systems. Long Beach: NIPS, 2017: 5099-5108.
12 彭秀平, 仝其胜, 林洪彬, 等 一种面向散乱点云语义分割的深度残差特征金字塔网络框架[J]. 自动化学报, 2021, 47 (12): 2831- 2840
PENG Xiu-Ping, TONG Qi-Sheng, LIN Hong-Bin, et al A deep residual-feature pyramid network for scattered point cloud semantic segmentation[J]. Acta Automatica Sinica, 2021, 47 (12): 2831- 2840
13 MATURANA D, SCHERER S. Voxnet: a 3d convolutional neural network for real-time object recognition [C]// Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems. Hamburg: IEEE, 2015: 922-928.
14 MAO J, XUE Y, NIU M, et al. Voxel transformer for 3d object detection [C]// Proceedings of the IEEE/CVF International Conference on Computer Vision. Online: IEEE, 2021: 3164-3173.
15 WANG G, XU G, WU Q, et al Two-stage point cloud super resolution with local interpolation and readjustment via outer-product neural network[J]. Journal of Systems Science and Complexity, 2021, 34 (1): 68- 82
doi: 10.1007/s11424-020-9266-x
16 SIMONOYSKY M, KOMODAKIS N. Dynamic edge-conditioned flters in convolutional neural networks on graphs [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Hawaii: IEEE, 2017: 3693-3702.
17 TE G, HU W, ZHENG A, et al. Rgcnn: regularized graph cnn for point cloud segmentation [C]// Proceedings of the 26th ACM International Conference on Multimedia. Lisboa: ACM, 2018: 746-754.
18 QIAN G, ABUALSHOUR A, LI G, et al. Pu-gcn: point cloud upsampling using graph convolutional networks [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Online: IEEE, 2021: 11683-11692.
19 GUO M H, CAI J X, LIU Z N, et al Pct: point cloud transformer[J]. Computational Visual Media, 2021, 7 (2): 187- 199
doi: 10.1007/s41095-021-0229-5
20 YU X, RAO Y, WANG Z, et al. Pointr: diverse point cloud completion with geometry-aware transformers [C]// Proceedings of the IEEE/CVF International Conference on Computer Vision. Online: IEEE, 2021: 12498-12507.
21 HE K, ZHANG X, REN S, et al. Deep residual learning for image recognition [C]// Proceedings of the IEEE conference on computer vision and pattern recognition. Las Vegas: IEEE, 2016: 770-778.
22 KIPF T N, WELLING M. Semi-supervised classification with graph convolutional networks [EB/OL]. [2016-09-09]. https://arxiv.org/pdf/1609.02907.pdf.
23 LUO C, CHEN Y, WANG N, et al. Spectral feature transformation for person re-identification [C]// Proceedings of the IEEE/CVF International Conference on Computer Vision. Seoul: IEEE, 2019: 4976-4985.
24 HU X, MU H, ZHANG X, et al. Meta-SR: a magnification-arbitrary network for super-resolution [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2019: 1575-1584.
25 BUTT M A, MARAGOS P Optimum design of chamfer distance transforms[J]. IEEE Transactions on Image Processing, 1998, 7 (10): 1477- 1484
doi: 10.1109/83.718487
26 CUTURI M. Sinkhorn distances: lightspeed computation of optimal transport [C]// Proceedings of the 26th International Conference on Neural Information Processing Systems. Lake Tahoe: NIPS, 2013: 2292-2300.
27 YE S, CHEN D, HAN S, et al Meta-PU: an arbitrary-scale upsampling network for point cloud[J]. IEEE Transactions on Visualization and Computer Graphics, 2022, 28 (9): 3206- 3218
doi: 10.1109/TVCG.2021.3058311
28 BOLTZ S, NIELSEN F, SOATTO S. Earth mover distance on superpixels [C]// IEEE International Conference on Image Processing. Hong Kong: IEEE, 2010: 4597-4600.
29 SOKOLOVA M, JAPKOWICZ N, SZPAKOWICZ S. Beyond accuracy, F-score and ROC: a family of discriminant measures for performance evaluation [C]// Australasian Joint Conference on Artificial Intelligence. Berlin: Springer, 2006: 1015-1021.
[1] 刘近贞,陈飞,熊慧. 多尺度残差网络模型的开放式电阻抗成像算法[J]. 浙江大学学报(工学版), 2022, 56(9): 1789-1795.
[2] 王万良,王铁军,陈嘉诚,尤文波. 融合多尺度和多头注意力的医疗图像分割方法[J]. 浙江大学学报(工学版), 2022, 56(9): 1796-1805.
[3] 郝琨,王阔,王贝贝. 基于改进Mobilenet-YOLOv3的轻量级水下生物检测算法[J]. 浙江大学学报(工学版), 2022, 56(8): 1622-1632.
[4] 夏杰锋,唐武勤,杨强. 光伏航拍红外图像的热斑自动检测方法[J]. 浙江大学学报(工学版), 2022, 56(8): 1640-1647.
[5] 赵永胜,李瑞祥,牛娜娜,赵志勇. 数字孪生驱动的机身形状控制方法[J]. 浙江大学学报(工学版), 2022, 56(7): 1457-1463.
[6] 何立,庞善民. 结合年龄监督和人脸先验的语音-人脸图像重建[J]. 浙江大学学报(工学版), 2022, 56(5): 1006-1016.
[7] 张雪芹,李天任. 基于Cycle-GAN和改进DPN网络的乳腺癌病理图像分类[J]. 浙江大学学报(工学版), 2022, 56(4): 727-735.
[8] 褚晶辉,史李栋,井佩光,吕卫. 适用于目标检测的上下文感知知识蒸馏网络[J]. 浙江大学学报(工学版), 2022, 56(3): 503-509.
[9] 程若然,赵晓丽,周浩军,叶翰辰. 基于深度学习的中文字体风格转换研究综述[J]. 浙江大学学报(工学版), 2022, 56(3): 510-519, 530.
[10] 彭向东,潘从成,柯泽浚,朱华强,周肖. 基于并行架构和时空注意力机制的心电分类方法[J]. 浙江大学学报(工学版), 2022, 56(10): 1912-1923.
[11] 杨栋杰,高贤君,冉树浩,张广斌,王萍,杨元维. 基于多重多尺度融合注意力网络的建筑物提取[J]. 浙江大学学报(工学版), 2022, 56(10): 1924-1934.
[12] 陈伟航,罗强,王腾飞,张文生,蒋良潍. DMC复合地基工后沉降可靠性分析及设计优化[J]. 浙江大学学报(工学版), 2022, 56(10): 2019-2027.
[13] 陈彤,郭剑锋,韩心中,谢学立,席建祥. 基于生成对抗模型的可见光-红外图像匹配方法[J]. 浙江大学学报(工学版), 2022, 56(1): 63-74.
[14] 任松,朱倩雯,涂歆玥,邓超,王小书. 基于深度学习的公路隧道衬砌病害识别方法[J]. 浙江大学学报(工学版), 2022, 56(1): 92-99.
[15] 刘兴,余建波. 注意力卷积GRU自编码器及其在工业过程监控的应用[J]. 浙江大学学报(工学版), 2021, 55(9): 1643-1651.