3D human pose estimation based on multi-scale encoder fusion

doi:10.3785/j.issn.1008-973X.2026.03.012

Journal of ZheJiang University (Engineering Science)

2026, Vol. 60

Issue (3): 565-573 DOI: 10.3785/j.issn.1008-973X.2026.03.012

3D human pose estimation based on multi-scale encoder fusion

Xiaoan BAO1(

),Enlin CHEN1,Na ZHANG1,Xiaomei TU2,Biao WU3,Qingqi ZHANG4,*(

)

1. School of Computer Science and Technology, Zhejiang Sci-Tech University, Hangzhou 310018, China
2. School of Civil Engineering and Architecture, Zhejiang Guangsha Vocational and Technical University of Construction, Dongyang 322100, China
3. School of Science, Zhejiang Sci-Tech University, Hangzhou 310018, China
4. Graduate School of East Asian Studies, Yamaguchi University, Yamaguchi 753-8514, Japan

Download:

HTML

PDF(1735KB) HTML
Export: BibTeX | EndNote (RIS)

Abstract

A 3D human pose estimation method based on multi-scale encoder fusion was proposed in order to address the contradiction between redundant information interference and the need for information completeness. The method consisted of a key-frame spatial-temporal encoder (KFSTE) and a global retention self-attention encoder (GRSAE). The skeletal feature sequence was filtered by using a key-frame selector in KFSTE. Then local spatial-temporal dependencies were modeled through a temporal encoder. Global single-stage encoding was performed via a retention encoder in GRSAE to capture global skeletal sequence feature, thereby avoiding information loss caused by key-frame selection bias. The 3D human pose coordinates were predicted by concatenating the features from the two encoders followed by a regression module. The experimental results on the large-scale Human3.6M dataset demonstrated that the proposed method reduced the mean per-joint position error (MPJPE) by 3% compared with MixSTE and achieved the best performance on 11 actions.

Key words： three-dimensional human pose estimation spatial-temporal encoder key-frame extraction retentive self-attention encoding multi-encoder feature fusion

Received: 13 March 2025 Published: 04 February 2026

CLC:

TP 393

Fund: 国家自然科学基金资助项目 (6207050141)；浙江省重点研发计划资助项目(2020C03094)；浙江省教育厅一般科研资助项目(Y202147659)；浙江省教育厅资助项目(Y202250706，Y202250677)；浙江省基础公益研究计划资助项目(QY19E050003).

Corresponding Authors: Qingqi ZHANG E-mail: baoxiaoan@zstu.edu.cn;c503snw@yamaguchi-u.ac.jp

	Service
	E-mail this article
	Add to my bookshelf
	Add to citation manager
	E-mail Alert
	RSS
	Articles by authors
	Xiaoan BAO
	Enlin CHEN
	Na ZHANG
	Xiaomei TU
	Biao WU
	Qingqi ZHANG

Cite this article:

Xiaoan BAO,Enlin CHEN,Na ZHANG,Xiaomei TU,Biao WU,Qingqi ZHANG. 3D human pose estimation based on multi-scale encoder fusion. Journal of ZheJiang University (Engineering Science), 2026, 60(3): 565-573.

URL:

https://www.zjujournals.com/eng/10.3785/j.issn.1008-973X.2026.03.012 OR https://www.zjujournals.com/eng/Y2026/V60/I3/565

基于多尺度编码器融合的三维人体姿态估计算法

针对冗余信息干扰与信息完整性需求之间的矛盾，提出基于多尺度编码器融合的三维人体姿态估计方法. 该方法由关键帧时空编码器（KFSTE）和全局保留自注意力编码器（GRSAE）构成. KFSTE通过关键帧选择器对骨架特征序列进行筛选后，由时间编码器获取局部时空建模. GRSAE通过保留编码器进行全局单阶段编码来获取全局骨架序列特征，避免因关键帧筛选偏差导致的信息损失. 通过对双编码器的特征拼接及回归处理，预测得到三维人体姿态坐标. 实验结果表明，在较大规模的Human3.6M数据集上，所提方法的平均关节位置误差(MPJPE)比MixSTE低3%，有11个动作获得最佳.

关键词： 三维人体姿态估计, 时空编码器, 关键帧提取, 保留自注意力编码, 多编码特征融合

Fig.1 Network structure of three-dimensional human pose estimation algorithm based on multi-scale encoding fusion

Fig.2 Schematic diagram of key-frame spatial-temporal encoder

Fig.3 Structure of spatial-temporal transformer encoder

Fig.4 Structure of retentive self-attention encoder

CPN 协议1	会议名	MPJPE/mm
CPN 协议1	会议名	Dir.	Disc.	Eat.	Grceet	Phone	Photo	Pose	Punch.	Sit	SitD.	Somke	Wait	WalkD.	Walk	WalkT.	平均值
文献[32] (f= 243)	CVPR’18	45.2	46.7	43.3	45.6	48.1	55.1	44.6	44.3	57.3	65.8	47.1	44	49	32.8	33.9	46.8
文献[33]	NeurIPS’19	44.8	46.1	43.3	46.4	49.0	55.2	44.6	44	58.3	62.7	47.1	43.9	48.6	32.7	33.3	46.7
文献[9] (f = 243)	CVPR’20	41.8	44.8	41.1	44.9	47.4	54.1	43.4	42.2	56.2	63.6	45.3	43.5	45.3	31.3	32.2	45.1
SRNet^[11]	ECCV’20	46.6	47.1	43.9	41.6	45.8	49.6	46.5	40.0	53.4	61.1	46.1	42.6	43.1	31.5	32.6	44.8
UGCN^[34] (f= 96)	ECCV’20	41.3	43.9	44.0	42.2	48.0	57.1	42.2	43.2	57.3	61.3	47.0	43.5	47.0	32.6	31.8	45.6
文献[8] (f = 81)	TCSVT’21	42.1	43.8	41.0	43.8	46.1	53.5	42.4	43.1	53.9	60.5	45.7	42.1	46.2	32.2	33.8	44.6
PoseFormer^[13] (f =81)	ICCV’21	41.5	44.8	39.8	42.5	46.5	51.6	42	42	53.3	60.7	45.5	43.3	46.1	31.8	32.2	44.3
MHFormer^[21] (f =351)	CVPR’22	39.2	43.1	40.1	40.9	44.9	51.2	40.6	41.3	53.5	60.3	43.7	41.1	43.8	29.8	30.6	43.0
MixSTE^[15] (f =243)	CVPR’22	37.6	40.9	37.3	39.7	42.3	49.9	40.1	39.8	51.7	55.0	42.1	39.8	41.0	27.9	27.9	40.9
KTPFormer^[30] (f =243)	CVPR’24	37.3	39.2	35.9	37.6	42.5	48.2	38.6	39.0	51.4	55.9	41.6	39.0	40.0	27.0	27.4	40.1
TCPFormer^[31] (f =81)	CVPR’25	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	40.5
MSEFN (f =81，k_f = 27)	—	36.8	39.9	35.8	39.1	41.0	47.5	37.5	37.6	51.5	47.9	39.8	38.2	42.1	25.6	26.8	39.8

CPN 协议2	会议名	MPJPE/mm
CPN 协议2	会议名	Dir.	Disc.	Eat.	Grceet	Phone	Photo	Pose	Punch.	Sit	SitD.	Somke	Wait	WalkD.	Walk	WalkT.	平均值
文献[7]	CVPR’18	34.7	39.8	41.8	38.6	42.5	47.5	38.0	36.6	50.7	56.8	42.6	39.6	43.9	32.1	36.5	41.8
文献[35] (f = 7)	ICCV’19	35.7	37.8	36.9	40.7	39.6	45.2	37.4	34.5	46.9	50.1	40.5	36.1	41.0	29.6	32.3	39.0
文献[9] (f = 243)	CVPR’20	32.3	35.2	33.3	35.8	35.9	41.5	33.2	32.7	44.6	50.9	37.0	32.4	37.0	25.2	27.2	35.6
UGCN^[34] (f = 96)	ECCV’20	32.9	35.,2	35.6	34.4	364	42.7	31.2	32.5	45.6	50.2	37.3	32.8	36.3	26.0	23.9	35.5
PoseFormer^[13] (f =81)	ICCV’21	32.5	34,8	32.6	34.6	35.3	39.5	32.1	32.0	42.8	48.5	34.8	32.4	35.3	24.5	26	34.6
MHFormer^[21] (f =351)	CVPR’22	31.5	34.9	32.8	33.6	35.3	39.6	32.0	32.2	43.5	48.1	36.4	32.6	34.3	23.9	25.1	34.4
MixSTE^[15] (f =243)	CVPR’22	30.8	33.1	30.3	31.8	33.1	39.1	31.1	30.5	42.5	44.5	34.0	30.8	32.7	22.1	22.9	32.6
KTPFormer^[30] (f =243)	CVPR’24	30.1	32.3	29.6	30.8	32.3	37.3	30.0	30.2	41.0	45.3	33.6	29.9	31.4	21.5	22.6	31.9
TCPFormer^[31] (f =81)	CVPR’25	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	33.7
MSEFN (f =81，k_f = 27)	—	27.5	30.5	28.5	31.8	31.5	36.4	28.5	28	41.7	46.6	32	28.3	31.8	19.8	21.3	31.0

Tab.1 Comparison of MPJPE result of MSEFN and different methods under protocol 1 and protocol 2 on Human3.6M dataset (based on CPN-detected input)

Tab.2 Comparison of MPJPE result of MSEFN and different methods under protocol 1 on Human3.6M dataset (based on GT-detected pose input)

Tab.3 Detailed quantitative comparison result of three indicators under MPI-INF-3DHP

Tab.4 Quantitative comparison result of FLOPs, N_p and MPJPE indicator on Human3.6M

Tab.5 Ablation study of different modules of MSEFN

Fig.5 3D HPE visual evaluation of proposed method on Human3.6M dataset

Tab.6 Quantitative comparison result of ablation study on selection of frame rate

Tab.7 Ablation study result under different dimension and layer

Fig.6 Visualization result of proposed method on Human3.6M dataset


[1]	ZHANG C, YANG T, WENG J, et al. Unsupervised pre-training for temporal action localization tasks [C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans: IEEE, 2022: 14011–14021.

[2]	CHEN H, HE J Y, XIANG W, et al. Hdformer: high-order directed transformer for 3d human pose estimation [C]//Proceedings of the 32nd International Joint Conference on Artificial Intelligence. Macao: ACM, 2023: 581-589.

[3]	LIU M, YUAN J. Recognizing human actions as the evolution of pose estimation maps [C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 1159–1168.

[4]	ZHANG Q, BAO X, WU R, et al A skeleton temporal fusion graph convolutional network for elderly action recognition[J]. IEICE Transactions on Fundamentals of Electronics, Communications and Computer Sciences, 2025, 108 (5): 704- 713

[5]	MEHTA D, SRIDHAR S, SOTNYCHENKO O, et al VNect: real-time 3D human pose estimation with a single RGB camera[J]. ACM Transactions on Graphics, 2017, 36 (4): 1- 14

[6]	MOON G, LEE K M. I2L-MeshNet: image-to-lixel prediction network for accurate 3D human pose and mesh estimation from a single RGB image [C]//Proceedings of the European Conference on Computer Vision. Cham: Springer, 2020: 752–768.

[7]	PAVLAKOS G, ZHOU X, DANIILIDIS K. Ordinal depth supervision for 3D human pose estimation [C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 7307–7316.

[8]	CHEN T, FANG C, SHEN X, et al Anatomy-aware 3D human pose estimation with bone-based pose decomposition[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2022, 32 (1): 198- 209 doi: 10.1109/TCSVT.2021.3057267

[9]	LIU R, SHEN J, WANG H, et al. Attention mechanism exploits temporal contexts: real-time 3D human pose reconstruction [C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020: 5063–5072.

[10]	WANG J, YAN S, XIONG Y, et al. Motion guided 3D pose estimation from videos [C]// Proceedings of the European Conference on Computer Vision. Cham: Springer, 2020: 764–780.

[11]	ZENG A, SUN X, HUANG F, et al. SRNet: improving generalization in 3D human pose estimation with a split-and-recombine approach [C]// Proceedings of the European Conference on Computer Vision. Cham: Springer, 2020: 507–523.

[12]	CAO Z, SIMON T, WEI S E, et al. Realtime multi-person 2d pose estimation using part affinity fields [C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017: 7291-7299.

[13]	ZHENG C, ZHU S, MENDIETA M, et al. 3D human pose estimation with spatial and temporal transformers [C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. Montreal: IEEE, 2022: 11636–11645.

[14]	LI W, LIU H, DING R, et al Exploiting temporal contexts with strided transformer for 3D human pose estimation[J]. IEEE Transactions on Multimedia, 2023, 25: 1282- 1293 doi: 10.1109/TMM.2022.3141231

[15]	ZHANG J, TU Z, YANG J, et al. MixSTE: seq2seq mixed spatio-temporal encoder for 3D human pose estimation in video [C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans: IEEE, 2022: 13222–13232.

[16]	ZHU W, MA X, LIU Z, et al. MotionBERT: a unified perspective on learning human motion representations [C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. Paris: IEEE, 2024: 15039–15053.

[17]	TANG Z, QIU Z, HAO Y, et al. 3D human pose estimation with spatio-temporal criss-cross attention [C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Vancouver: IEEE, 2023: 4790–4799.

[18]	CHEN X, HAN Y, WANG X, et al Action keypoint network for efficient video recognition[J]. IEEE Transactions on Image Processing, 2022, 31: 4980- 4993 doi: 10.1109/TIP.2022.3191461

[19]	EINFALT M, LUDWIG K, LIENHART R. Uplift and upsample: efficient 3D human pose estimation with uplifting transformers [C]//Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. Waikoloa: IEEE, 2023: 2902–2912.

[20]	FAN Q, HUANG H, CHEN M, et al. Rmt: retentive networks meet vision transformers [C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2024: 5641-5651.

[21]	LI W, LIU H, TANG H, et al. MHFormer: multi-hypothesis transformer for 3D human pose estimation [C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans: IEEE, 2022: 13137–13146.

[22]	SHAN W, LIU Z, ZHANG X, et al. P-STMO: pre-trained spatial temporal many-to-one model for3D human pose estimation [C]//European Conference on Computer Vision. Cham: Springer, 2022: 461–478.

[23]	FAN Q, HUANG H, CHEN M, et al. RMT: retentive networks meet vision transformers [C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2024: 5641–5651.

[24]	IONESCU C, PAPAVA D, OLARU V, et al Human3.6M: large scale datasets and predictive methods for 3D human sensing in natural environments[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2014, 36 (7): 1325- 1339 doi: 10.1109/TPAMI.2013.248

[25]	MEHTA D, RHODIN H, CASAS D, et al. Monocular 3D human pose estimation in the wild using improved CNN supervision [C]//Proceedings of the International Conference on 3D Vision. Qingdao: IEEE, 2018: 506–516.

[26]	ZHENG C, WU W, CHEN C, et al Deep learning-based human pose estimation: a survey[J]. ACM Computing Surveys, 2024, 56 (1): 1- 37

[27]	MARGOSSIAN C C A review of automatic differentiation and its efficient implementation[J]. WIREs Data Mining and Knowledge Discovery, 2019, 9 (4): e1305 doi: 10.1002/widm.1305

[28]	FINDER S E, AMOYAL R, TREISTER E, et al. Wavelet convolutions for large receptive fields [C]//European Conference on Computer Vision. Cham: Springer, 2024: 363-380.

[29]	CHEN Y, WANG Z, PENG Y, et al. Cascaded pyramid network for multi-person pose estimation [C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 7103–7112.

[30]	PENG J, ZHOU Y, MOK P Y. KTPFormer: kinematics and trajectory prior knowledge-enhanced transformer for 3D human pose estimation [C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2024: 1123–1132.

[31]	LIU J, LIU M, LIU H, et al. Tcpformer: Learning temporal correlation with implicit pose proxy for 3d human pose estimation [C]//Proceedings of the AAAI Conference on Artificial Intelligence. Washington: AAAI, 2025, 39(5): 5478−5486.

[32]	PAVLLO D, FEICHTENHOFER C, GRANGIER D, et al. 3D human pose estimation in video with temporal convolutions and semi-supervised training [C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2020: 7745–7754.

[33]	YEH R, HU Y T, SCHWING A. Chirality nets for human pose regression [J]. Advances in Neural Information Processing Systems, 2019, 32: 8161–8171.

[34]	WANG J, YAN S, XIONG Y, et al. Motion guided 3d pose estimation from videos [C]//European Conference on Computer Vision. Cham: Springer, 2020: 764−780.

[35]	CAI Y, GE L, LIU J, et al. Exploiting spatial-temporal relationships for 3D pose estimation via graph convolutional networks [C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. Seoul: IEEE, 2020: 2272–2281.

[36]	LI H, SHI B, DAI W, et al Pose-oriented transformer with uncertainty-guided refinement for 2D-to-3D human pose estimation[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2023, 37 (1): 1296- 1304 doi: 10.1609/aaai.v37i1.25213

[1]	Juntao LV,Jueyu QI,Haochen YU,Lei MA,Huimin MA,Tianyu HU. Current status and future prospect of integrated simulation platform for autonomous driving[J]. Journal of ZheJiang University (Engineering Science), 2026, 60(3): 513-526.

[2]	Wenqiang CHEN,Linyue FENG,Dongdan WANG,Yulei GU,Xuan ZHAO. Vehicle trajectory prediction model integrating dynamic risk map and multivariate attention mechanism[J]. Journal of ZheJiang University (Engineering Science), 2026, 60(3): 455-467.

[3]	Xiaoyan KUI,Min ZHANG,Ling XIAO,Qinsong LI,Liming CHEN,Wensheng ZHANG,Beiji ZOU. Systematic classification and performance analysis of data deduplication and reduction techniques[J]. Journal of ZheJiang University (Engineering Science), 2026, 60(2): 287-302.

[4]	Yanle WANG,Ruifeng ZHANG,Qiang LI. Graph neural network recommendation model integrating global information and contrastive learning[J]. Journal of ZheJiang University (Engineering Science), 2026, 60(2): 351-359.

[5]	Yue WU,Zheng LIANG,Wei GAO,Maoda YANG,Peisen ZHAO,Hongxia DENG,Yuanyuan CHANG. Multi-modal gait recognition based on SMPL model decomposition and embedding fusion[J]. Journal of ZheJiang University (Engineering Science), 2026, 60(1): 52-60.

[6]	Huhang CHEN,Quan LV,Zihang SU,Junqiao ZHANG,Zhu CHEN,Xu HAN. Operating mode selection model for heating unit considering capacity charge[J]. Journal of ZheJiang University (Engineering Science), 2026, 60(1): 169-178.

[7]	Hao HE,Yongdong SHU,Yonggang LIN,Fuquan DAI,Ju ZHANG. hydrodynamic and hydraulic-control simulation of high-power ship azimuth thruster under variable load[J]. Journal of ZheJiang University (Engineering Science), 2026, 60(1): 191-198.

[8]	Yiming YU,Wei CAI,Yi LI,Wei FU,Xu YAO,Tingyi ZHANG,Shangqi DIAO,Dan LI,Songqing LIN,Yongshun CHEN. Review of environmental heavy metal detection based on microelectrode arrays[J]. Journal of ZheJiang University (Engineering Science), 2026, 60(1): 217-230.

[9]	Linghao ZHANG,Haibo TAN,He ZHAO,Zhong CHEN,Haotian CHENG,Zhiyu MA. CompuDEX: blockchain-based large model fine-tuning compute-power sharing platform[J]. Journal of ZheJiang University (Engineering Science), 2026, 60(1): 1-18.

[10]	Yanpu YANG,Zhihong WU,Wenhao MENG,Yueming ZHUO,Jialing LIU. Upper-limb muscle fatigue assessment in overhead work based on Hammerstein model and surface electromyography[J]. Journal of ZheJiang University (Engineering Science), 2025, 59(12): 2483-2494.

[11]	Yuxuan LIU,Yizhi LIU,Zhuhua LIAO,Zhengbiao ZOU,Jingxin TANG. Adaptive graph attention Transformer for dynamic traffic flow prediction[J]. Journal of ZheJiang University (Engineering Science), 2025, 59(12): 2585-2592.

[12]	Xiongxiong ZHOU,Qiujiang HE,Jichun HE,Jing ZHOU. Deformation decoupling and parameter inversion for high core wall dams considering mechanical causes[J]. Journal of ZheJiang University (Engineering Science), 2025, 59(12): 2616-2626.

[13]	Jiawei TANG,Tiezheng GUO,Yingyou WEN. Reinforcement learning-based scheduling algorithm for cloud-edge collaborative computing on Kubernetes[J]. Journal of ZheJiang University (Engineering Science), 2025, 59(11): 2400-2408.

[14]	Qinxue WANG,Wenfang ZHANG. Maritime positioning sharing scheme based on compressing zero-knowledge proof[J]. Journal of ZheJiang University (Engineering Science), 2025, 59(11): 2409-2417.

[15]	Zhaolong DONG,He HUANG,Zhanyi LI,Lan YANG,Huifeng WANG. Snowy scene integration construction method in autonomous driving sample library[J]. Journal of ZheJiang University (Engineering Science), 2025, 59(10): 2078-2085.

Viewed

Full text

Abstract

Cited

Shared

Discussed