Occluded human pose estimation network based on knowledge sharing

doi:10.3785/j.issn.1008-973X.2024.10.003

Journal of ZheJiang University (Engineering Science)

2024, Vol. 58

Issue (10): 2001-2010 DOI: 10.3785/j.issn.1008-973X.2024.10.003

Occluded human pose estimation network based on knowledge sharing

Jiahong JIANG(

),Nan XIA*(

),Changwu LI,Xinmiao YU

School of Information Science and Engineering, Dalian Polytechnic University, Dalian 116034, China

Download:

HTML

PDF(1801KB) HTML
Export: BibTeX | EndNote (RIS)

Abstract

A new estimation network was proposed for improving the insufficient occlusion handling ability of existing human pose estimation methods. An occluded parts enhanced convolutional network (OCNN) and an occluded features compensation graph convolutional network (OGCN) were included in the proposed network. A high-low order feature matching attention was designed to strengthen the occlusion area features, and high-adaptation weights were extracted by OCNN, achieving enhanced detection of the occluded parts with a small amount of occlusion data. OGCN strengthened the shared and private attribute compensation node features by eliminating the obstacle features. The adjacency matrix was importance-weighted to enhance the quality of the occlusion area features and to improve the detection accuracy. The proposed network achieved detection accuracy of 78.5%, 67.1%, and 77.8% in the datasets COCO2017, COCO-Wholebody, and CrowdPose, respectively, outperforming the comparative algorithms. The proposed network saved 75% of the training data usage in the self-built occlusion dataset.

Key words： human pose estimation occlusion handling high-low order feature matching node feature compensation adjacency matrix weighting

Received: 26 March 2024 Published: 27 September 2024

CLC:

TP 391.4

Fund: 教育部产学合作协同育人资助项目（220603231024713）.

Corresponding Authors: Nan XIA E-mail: jjh19990901@163.com;xianan@dlpu.edu.cn

	Service
	E-mail this article
	Add to my bookshelf
	Add to citation manager
	E-mail Alert
	RSS
	Articles by authors
	Jiahong JIANG
	Nan XIA
	Changwu LI
	Xinmiao YU

Cite this article:

Jiahong JIANG,Nan XIA,Changwu LI,Xinmiao YU. Occluded human pose estimation network based on knowledge sharing. Journal of ZheJiang University (Engineering Science), 2024, 58(10): 2001-2010.

URL:

https://www.zjujournals.com/eng/10.3785/j.issn.1008-973X.2024.10.003 OR https://www.zjujournals.com/eng/Y2024/V58/I10/2001

基于知识共享的遮挡人体姿态估计网络

现有人体姿态估计方法处理遮挡情况时性能较差，为此提出新的估计网络，包含遮挡区域强化卷积网络(OCNN)和遮挡特征补偿图卷积网络(OGCN). 设计高低阶特征匹配注意力以强化遮挡区域特征，由OCNN提取高适配权重，通过少量遮挡数据的方式实现遮挡部位的强化检测. 由OGCN消除障碍物特征，通过强化关键点共有及专有属性的方式补偿节点特征；进行邻接矩阵重要性加权以改善遮挡部位特征质量，提升检测精度. 所提网络在数据集COCO2017、COCO-Wholebody、CrowdPose上的检测精度分别为78.5%、67.1%、77.8%，优于对比算法. 在自建遮挡数据集上所提网络节约了75%的训练数据使用.

关键词： 人体姿态估计, 遮挡处理, 高低阶特征匹配, 节点特征补偿, 邻接矩阵加权

Fig.1 Flowchart of occluded human pose estimation network based on knowledge sharing

Fig.2 Flowchart of high-low order feature matching attention

Fig.3 Flowchart of keypoint feature compensation and adjacency matrix importance weighting

Tab.1 Performance comparison of different algorithms in COCO 2017 dataset

Tab.2 Performance comparison of different algorithms in COCO-Wholebody dataset

Tab.3 Performance comparison of different algorithms in CrowdPose dataset

Tab.4 Detection performance comparison of different algorithms

Fig.4 Comparison of keypoints heatmaps for different algorithms

Fig.5 Pose estimation results of occluded human pose estimation network based on knowledge sharing

Fig.6 Comparison of pose estimation results for different algorithms


[1]	孙雪菲, 张瑞峰, 关欣, 等强化先验骨架结构的轻量型高效人体姿态估计[J]. 浙江大学学报: 工学版, 2024, 58 (1): 50- 60 SUN Xuefei, ZHANG Ruifeng, GUAN Xin, et al Lightweight and efficient human pose estimation with enhanced priori skeleton structure[J]. Journal of Zhejiang University: Engineering Science, 2024, 58 (1): 50- 60

[2]	YU X W, CHEN G S. HRPoseFormer: high-resolution Transformer for human pose estimation via multi-scale token aggregation [C]// IEEE 16th International Conference on Solid-State and Integrated Circuit Technology . Nanjing: IEEE, 2022: 1–3.

[3]	ZHOU L, CHEN Y, WANG J Progressive direction-aware pose grammar for human pose estimation[J]. IEEE Transactions on Biometrics, Behavior, and Identity Science, 2023, 5 (4): 593- 605 doi: 10.1109/TBIOM.2023.3315509

[4]	ZHANG Z, LIU M, SHEN J, et al Lightweight whole body human pose estimation with two-stage refinement training strategy[J]. IEEE Transactions on Human-Machine Systems, 2024, 54 (1): 121- 130 doi: 10.1109/THMS.2024.3349652

[5]	LIN J, ZHENG Z, ZHONG Z, et al. Joint representation learning and keypoint detection for cross-view geo-localization [J]. IEEE Transactions on Image Processing , 2022, 31: 3780–3792.

[6]	MENG Q, QIN C, BAI W, et al MulViMotion: shape-aware 3D myocardial motion tracking from multi-view cardiac MRI[J]. IEEE Transactions on Medical Imaging, 2022, 41 (8): 1961- 1974 doi: 10.1109/TMI.2022.3154599

[7]	CHAKRAVARTHI B, PATIL A K, RYU J Y, et al Scenario-based sensed human motion editing and validation through the motion-sphere[J]. IEEE Access, 2022, 10: 28295- 28307 doi: 10.1109/ACCESS.2022.3157939

[8]	MARQUES B, SILVA S, ALVES J, et al A conceptual model and taxonomy for collaborative augmented reality[J]. IEEE Transactions on Visualization and Computer Graphics, 2022, 28 (12): 5113- 5133 doi: 10.1109/TVCG.2021.3101545

[9]	LI N, CHEN X, FENG Y, et al Human–computer interaction cognitive behavior modeling of command and control systems[J]. IEEE Internet of Things Journal, 2022, 9 (14): 12723- 12736 doi: 10.1109/JIOT.2021.3138247

[10]	SUN K, XIAO B, LIU D, et al. Deep high-resolution representation learning for human pose estimation [C]// IEEE/CVF Conference on Computer Vision and Pattern Recognition . Long Beach: IEEE, 2019: 5693–5703.

[11]	WANG J, SUN K, CHENG T, et al Deep high-resolution representation learning for visual recognition[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021, 43 (10): 3349- 3364 doi: 10.1109/TPAMI.2020.2983686

[12]	KE L, CHANG M C, QI H, et al DetPoseNet: improving multi-person pose estimation via coarse-pose filtering[J]. IEEE Transactions on Image Processing, 2022, 31: 2782- 2795 doi: 10.1109/TIP.2022.3161081

[13]	CAO Z, HIDALGO G, SIMON T, et al OpenPose: realtime multi-person 2D pose estimation using part affinity fields[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021, 43 (1): 172- 186 doi: 10.1109/TPAMI.2019.2929257

[14]	LI Q, ZHANG Z, ZHANG F, et al HRNeXt: high-resolution context network for crowd pose estimation[J]. IEEE Transactions on Multimedia, 2023, 25 (46): 1521- 1528

[15]	KIM G, KIM H, KONG K, et al Human body aware feature extractor using attachable feature corrector for human pose estimation[J]. IEEE Transactions on Multimedia, 2023, 25: 5789- 5799 doi: 10.1109/TMM.2022.3199098

[16]	LIN T Y, MAIRE M, BELONGIE S, et al. Microsoft COCO: common objects in context [C]// European Conference on Computer Vision . [S. l.]: Springer, 2014: 740–755.

[17]	BANZI J, BULUGU I, YE Z Learning a deep predictive coding network for a semi-supervised 3D-hand pose estimation[J]. IEEE/CAA Journal of Automatica Sinica, 2020, 7 (5): 1371- 1379 doi: 10.1109/JAS.2020.1003090

[18]	KIM S, KANG S, CHOI H, et al Keypoint aware robust representation for transformer-based re-identification of occluded person[J]. IEEE Signal Processing Letters, 2023, 30: 65- 69 doi: 10.1109/LSP.2023.3240596

[19]	WANG Y J, LUO Y M, BAI G H, et al UformPose: a U-shaped hierarchical multi-scale keypoint-aware framework for human pose estimation[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2023, 33 (4): 1697- 1709 doi: 10.1109/TCSVT.2022.3213206

[20]	PENG S, ZHOU X, LIU Y, et al PVNet: pixel-wise voting network for 6DoF object pose estimation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44 (1): 3212- 3223

[21]	ARTACHO B, SAVAKIS A UniPose+: a unified framework for 2D and 3D human pose estimation in images and videos[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44 (12): 9641- 9653 doi: 10.1109/TPAMI.2021.3124736

[22]	GAI D, FENG R Y, MIN W, et al Spatiotemporal learning transformer for video-based human pose estimation[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2023, 33 (9): 4564- 4576 doi: 10.1109/TCSVT.2023.3269666

[23]	YIN Y, LIU M, ZHU Q, et al Multibranch attention graph convolutional networks for 3-D human pose estimation[J]. IEEE Transactions on Instrumentation and Measurement, 2023, 72: 2520412.

[24]	FAN J, ZHENG P, LI S, et al An integrated hand-object dense pose estimation approach with explicit occlusion awareness for human-robot collaborative disassembly[J]. IEEE Transactions on Automation Science and Engineering, 2024, 21 (1): 147- 156 doi: 10.1109/TASE.2022.3215584

[25]	PASA L, NAVARIN N, SPERDUTI A Multiresolution reservoir graph neural network[J]. IEEE Transactions on Neural Networks and Learning Systems, 2022, 33 (6): 2642- 2653 doi: 10.1109/TNNLS.2021.3090503

[26]	MORSHED M G, SULTANA T, LEE Y K LeL-GNN: learnable edge sampling and line based graph neural network for link prediction[J]. IEEE Access, 2023, 11: 56083- 56097 doi: 10.1109/ACCESS.2023.3283029

[27]	ISUFI E, GAMA F, RIBEIRO A EdgeNets: edge varying graph neural networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44 (11): 7457- 7473 doi: 10.1109/TPAMI.2021.3111054

[28]	XU L, JIN S, LIU W, et al ZoomNAS: searching for whole-body human pose estimation in the wild[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023, 45 (8): 5296- 5313

[29]	LEE K, KIM W, LEE S From human pose similarity metric to 3D human pose estimator: temporal propagating LSTM networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023, 45 (2): 1781- 1797 doi: 10.1109/TPAMI.2022.3164344

[1]	Xuefei SUN,Ruifeng ZHANG,Xin GUAN,Qiang LI. Lightweight and efficient human pose estimation with enhanced priori skeleton structure[J]. Journal of ZheJiang University (Engineering Science), 2024, 58(1): 50-60.

Viewed

Full text

Abstract

Cited

Shared

Discussed