Object pose estimation based on attention enhancement cross-modal multilayer fusion network

doi:10.3785/j.issn.1008-973X.2026.09.004

Journal of ZheJiang University (Engineering Science)

2026, Vol. 60

Issue (9): 1872-1880 DOI: 10.3785/j.issn.1008-973X.2026.09.004

Object pose estimation based on attention enhancement cross-modal multilayer fusion network

Heng YANG(

),Shaohan WANG,Qing DONG,Keyuan ZHAO,Mingliang YANG

School of Mechanical Engineering, Taiyuan University of Science and Technology, Taiyuan 030024, China

Download:

HTML

PDF(1443KB) HTML
Export: BibTeX | EndNote (RIS)

Abstract

A robust end-to-end 6D pose estimation network was proposed. A hierarchical feature fusion mechanism was used to effectively integrate RGB-D data. The pixel-level feature was extracted via an attention enhancement module. Then a multi-scale network was employed to process the fused pixel-level multimodal feature, yielding multi-scale multimodal features. The multimodal features were subsequently fed into subsequent modules to obtain corresponding global features. The global features were concatenated with the multimodal features to form multi-scale dense discriminative features that encode global information. The final pose was derived from multi-scale dense discriminative features, enabling accurate 6D pose estimation. Extensive experiments on the LineMOD and YCB-Video datasets demonstrate that the proposed method achieves superior performance, particularly in handling challenging smooth and textureless objects in the YCB-Video dataset. The method outperforms the representative approach DenseFusion in terms of accuracy on the LineMOD dataset, while achieving significantly higher processing speed.

Key words： 6D pose deep learning neural network cross-modal fusion attention mechanism

Received: 17 July 2025 Published: 20 July 2026

CLC:

TP 183

Fund: 国家自然科学基金资助项目（52105269）；省部级科技计划资助项目（202402150101006）.

	Service
	E-mail this article
	Add to my bookshelf
	Add to citation manager
	E-mail Alert
	RSS
	Articles by authors
	Heng YANG
	Shaohan WANG
	Qing DONG
	Keyuan ZHAO
	Mingliang YANG

Cite this article:

Heng YANG,Shaohan WANG,Qing DONG,Keyuan ZHAO,Mingliang YANG. Object pose estimation based on attention enhancement cross-modal multilayer fusion network. Journal of ZheJiang University (Engineering Science), 2026, 60(9): 1872-1880.

URL:

https://www.zjujournals.com/eng/10.3785/j.issn.1008-973X.2026.09.004 OR https://www.zjujournals.com/eng/Y2026/V60/I9/1872

基于注意力增强的跨模态多层融合网络的物体位姿估计

提出鲁棒的端到端6D位姿估计网络. 利用层次化特征融合机制，使得RGB-D数据有效融合. 网络通过注意力增强模块提取像素级特征，通过多尺度网络对融合后的像素级多模态特征进行处理，得到多尺度多模态特征. 将多模态特征分别送至后续模块中，得到对应的全局特征. 将多模态特征与全局特征进行拼接，形成包含全局信息的多尺度密集判别特征. 根据多尺度密集判别特征得到最终位姿，实现精确的6D位姿估计. 在LineMOD和YCB-Video数据集上开展多项实验，所提方法的表现出色，特别是在处理YCB-Video数据集中具有挑战性的光滑和无纹理物体时表现突出. 在LineMOD数据集上，所提方法的精度优于现有的代表性方法DenseFusion，处理速度显著提升.

关键词： 6D位姿, 深度学习, 神经网络, 跨模态融合, 注意力机制

Fig.1 Extracting pixel-level feature from RGB-D image

Fig.2 Overall structure of DeepFusion6D network

Fig.3 Overall structure of AFE

Fig.4 Output size of each module in extracted multi-scale dense discriminative features

Tab.1 Quantitative assessment of ADD(-S) score on linenmod dataset

方法	ADD(-S)分数/%
方法	master- chef- can	cracker- box	sugar- box	tomato- soup- can	mustard- bottle	tuna- fish- can	pudding- box	gelatin- box	potted- meat- can	Banana	pitcher- base	bleach- cleaner	Bowl	Mug	power- drill	wood- block	Scissor	large- marker	large- clamp	extra- large- clamp	foam- brick	平均值
PointFusion^[12]	99.8	62.2	95.4	96.9	84.0	99.8	96.7	100	88.5	70.5	79.8	65.0	24.1	99.8	22.8	18.2	35.9	80.4	50.0	20.1	100	74.1
PVN3D^[31]	100	91.6	100	96.9	100	100	100	100	93.6	99.7	100	99.4	54.9	99.8	99.6	80.2	95.6	99.7	74.9	48.8	100	93.2
DenseFusion^[13]	100	99.5	100	96.9	100	100	100	100	91.3	100	100	100	98.8	100	98.7	94.6	100	100	79.2	76.3	100	96.8
CFFM+CWTM^[23]	100	95.3	100	95.7	100	99.0	93.9	100	90.8	93.4	100	97.8	53.2	98.2	97.4	81.3	94.7	96.7	68.3	62.5	99.6	93.5
FoundationPose^[14]	99.1	98.7	99.3	98.9	99.5	99.0	98.8	99.4	98.5	99.2	98.6	98.3	98.0	98.7	98.4	97.5	97.1	98.2	96.9	96.5	99.1	99.0
本文方法（仅使用多尺度模块）	98.5	93.1	98.0	89.1	94.3	97.3	93.9	99.9	92.4	95.1	97.4	93.6	92.1	94.1	94.6	93.2	91.0	96.2	88.1	83.2	99.4	94.3
本文方法（仅使用注意力增强模块）	99.1	94.9	98.7	90.6	95.7	97.8	94.7	100	93.7	96.3	98.1	95.1	93.5	95.3	95.8	94.7	92.5	97.5	90.3	85.8	99.7	95.2
本文方法	100	97.5	99.0	93.5	97.5	98.5	96.5	100	96.0	97.5	98.0	97.0	99.2	97.0	96.5	98.0	96.5	98.0	85.0	82.0	100	97.1

方法	AUC/%
方法	master- chef- can	cracker- box	sugar- box	tomato- soup- can	mustard- bottle	tuna- fish- can	pudding- box	gelatin- box	potted- meat- can	Banana	pitcher- base	bleach- cleaner	Bowl	Mug	power- drill	wood- block	Scissor	large- marker	large- clamp	extra- large- clamp	foam- brick	平均值
PointFusion^[12]	90.9	80.5	90.4	91.9	88.5	93.8	87.5	95.0	86.4	84.7	85.5	81.0	75.7	94.2	71.5	68.1	76.7	87.9	65.9	60.4	91.8	83.9
PVN3D^[31]	95.8	92.7	98.2	94.5	98.6	97.1	97.9	98.8	92.7	97.1	97.8	96.9	81.0	95.0	98.2	87.6	91.7	97.2	75.2	64.4	97.2	93.0
DenseFusion^[13]	96.4	95.5	97.5	94.6	97.2	96.6	96.5	98.1	91.3	96.6	97.1	95.8	88.2	97.1	96.0	89.7	95.2	97.5	72.9	69.8	92.5	93.1
CFFM+CWTM^[23]	94.3	92.2	98.7	93.9	97.1	94.8	93.3	99.1	91.6	91.1	95.2	86.5	83.8	90.5	94.8	84.3	92.8	88.6	73.2	62.3	93.3	90.5
FoundationPose^[14]	96.5	94.2	96.8	95.0	97.4	96.1	94.5	97.0	93.8	96.3	94.0	92.5	91.8	94.3	93.1	89.5	88.2	92.0	87.5	85.4	95.9	93.2
本文方法（仅使用多尺度模块）	93.9	84.3	93.7	80.9	90.8	91.3	89.2	97.8	85.9	87.9	93.6	89.4	79.5	89.1	87.2	85.4	78.7	89.3	73.5	68.7	94.0	88.5
本文方法（仅使用注意力增强模块）	94.7	86.7	94.4	82.7	92.0	92.7	90.3	98.3	88.2	89.7	94.8	91.0	82.7	90.6	88.9	87.7	82.3	91.0	75.2	71.5	95.5	89.7
本文方法	96.9	98.6	98.6	95.0	96.5	94.5	93.1	99.0	90.3	91.8	95.5	93.4	88.3	92.5	90.5	100	85.5	98.4	82.5	80.5	97.0	93.7

Tab.2 Quantitative evaluation of ADD(-S) score and ADD(-S) AUC on YCB-Video dataset

Tab.3 Ablation study on efficiency and accuracy on YCB-Video dataset

Fig.5 ADD(-S) score-threshold curve for 'large_clamp' and 'extra_large_clamp' split by PoseCNN and Ground Truth

Fig.6 Performance change of different methods on YCB-Video dataset with increasing level of occlusion

Fig.7 Visualization of DeepFusion6D’s estimated 6D object pose on LINEMOD dataset


[1]	JIN D, ZHANG R, LI Y, et al. Mitigating latency effects on subjective experience in robot teleoperation using a VR-enabled virtual spring [C]//Proceedings of the IEEE International Symposium on Mixed and Augmented Reality. Bellevue: IEEE, 2024: 1276–1282.

[2]	ZHAO Y, WANG B, YU H, et al. Research on the self-powered flexible non-contact distance/contact pressure dual-mode sensor for manipulator based on triboelectric nanogenerator [C]//Proceedings of the China Automation Congress. Qingdao: IEEE, 2024: 437–441.

[3]	PAWAR U S, KULKARNI R A. Navigation with augmented reality [C]//Proceedings of the 1st International Conference on AIML: Applications for Engineering and Technology. Pune: IEEE, 2025: 1–6.

[4]	DIAS P, MARQUES B, FELIX I, et al. Creating asynchronous augmented reality instructions through a virtual reality framework [C]//Proceedings of the IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops. Saint Malo: IEEE, 2025: 1065–1071.

[5]	KIM J, JEON J, PARK J, et al. Continuous performance improvement of infrastructure guidance service for autonomous cooperative driving: focusing on data-centric AI [C]//Proceedings of the International Conference on Electronics, Information, and Communication. Taipei: IEEE, 2024: 1–4.

[6]	FENG Y, SHEN H, SHAN Z, et al Semantic communication for edge intelligence enabled autonomous driving system[J]. IEEE Network, 2025, 39 (2): 149- 157 doi: 10.1109/MNET.2024.3468328

[7]	CHEN J, LEI B, SONG Q, et al. A hierarchical graph network for 3D object detection on point clouds [C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020: 389–398.

[8]	LU P, SHI W, QIAO X Multi-view image-based 3D reconstruction in indoor scenes: a survey[J]. ZTE Communications, 2024, 22 (3): 91- 98

[9]	BESL P J, MCKAY N D Method for registration of 3-D shapes[J]. Sensor Fusion IV: Control Paradigms and Data Structures, 1992, 1611: 586- 606 doi: 10.1109/cisp.2012.6469977

[10]	ZHAO R, ZHANG Y, WEN W, et al E-TPE: EfficientThumbnail: preserving encryption for privacy protection in visual sensor networks[J]. ACM Transactions on Sensor Networks, 2024, 20 (4): 1- 26

[11]	HANH N T, CUONG V D, TUNG D D, et al. H-LSHADE: an efficient hybrid approach for solving heterogeneous target coverage in visual sensor networks [C]//International Symposium on Information and Communication Technology. Singapore: Springer, 2024: 297–307.

[12]	XU D, ANGUELOV D, JAIN A. PointFusion: deep sensor fusion for 3D bounding box estimation [C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 244–253.

[13]	WANG C, XU D, ZHU Y, et al. DenseFusion: 6D object pose estimation by iterative dense fusion [C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2020: 3338–3347.

[14]	WEN B, YANG W, KAUTZ J, et al. FoundationPose: unified 6D pose estimation and tracking of novel objects [C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2024: 17868–17879.

[15]	GOPAL G Y, AMER M A. Separable self and mixed attention transformers for efficient object tracking [C]//Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. Waikoloa: IEEE, 2024: 6708–6717.

[16]	LEI X, LU W, YONG J, et al RFF-PoseNet: a 6D object pose estimation network based on robust feature fusion in complex scenes[J]. Electronics, 2024, 13 (17): 3518 doi: 10.3390/electronics13173518

[17]	LV Z, GUO Y, YANG S, et al Multiscale feature fusion with self-attention for efficient 6D pose estimation[J]. Algorithms, 2025, 18 (4): 207 doi: 10.3390/a18040207

[18]	XIANG Y, SCHMIDT T, NARAYANAN V, et al. PoseCNN: a convolutional neural network for 6D object pose estimation in cluttered scenes [C]//Proceedings of the Robotics: Science and Systems XIV. Robotics: Science and Systems Foundation, 2018.

[19]	RAD M, LEPETIT V. BB8: a scalable, accurate, robust to partial occlusion method for predicting the 3D poses of challenging objects without using depth [C]//Proceedings of the IEEE International Conference on Computer Vision. Venice: IEEE, 2017: 3848–3856.

[20]	KEHL W, MANHARDT F, TOMBARI F, et al. SSD-6D: making RGB-based 3D detection and 6D pose estimation great again [C]//Proceedings of the IEEE International Conference on Computer Vision. Venice: IEEE, 2017: 1530–1538.

[21]	PENG S, LIU Y, HUANG Q, et al. PVNet: pixel-wise voting network for 6DoF pose estimation [C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2020: 4556–4565.

[22]	LEPETIT V, MORENO-NOGUER F, FUA P EPnP: an accurate O(n) solution to the PnP problem[J]. International Journal of Computer Vision, 2009, 81 (2): 155- 166 doi: 10.1007/s11263-008-0152-6

[23]	JIANG M, ZHANG L, WANG X, et al 6D object pose estimation based on cross-modality feature fusion[J]. Sensors, 2023, 23 (19): 8088 doi: 10.3390/s23198088

[24]	ZENG C, KWONG S. Dual swin-transformer based mutual interactive network for RGB-D salient object detection [EB/OL]. [2025-08-20]. https://arxiv.org/abs/2206.03105

[25]	WU P, LI W, YAN M 3D scene reconstruction based on improved ICP algorithm[J]. Microprocessors and Microsystems, 2020, 75: 103064 doi: 10.1016/j.micpro.2020.103064

[26]	WOHLHART P, LEPETIT V. Learning descriptors for object recognition and 3D pose estimation [C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Boston: IEEE, 2015: 3109–3118.

[27]	KENDALL A, GRIMES M, CIPOLLA R. PoseNet: a convolutional network for real-time 6-DOF camera relocalization [C]//Proceedings of the IEEE International Conference on Computer Vision. Santiago: IEEE, 2016: 2938–2946.

[28]	LI Y, WANG G, JI X, et al DeepIM: deep iterative matching for 6D pose estimation[J]. International Journal of Computer Vision, 2020, 128 (3): 657- 678 doi: 10.1007/s11263-019-01250-9

[29]	PARK K, PATTEN T, VINCZE M. Pix2Pose: pixel-wise coordinate regression of objects for 6D pose estimation [C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. Seoul: IEEE, 2019: 7667–7676.

[30]	HOSSEINI JAFARI O, MUSTIKOVELA S K, PERTSCH K, et al. iPose: instance-aware 6D pose estimation of partly occluded objects [C]// 2018 Asian Conference on Computer Vision. Cham: Springer, 2019: 477–492.

[31]	HE Y, SUN W, HUANG H, et al. PVN3D: a deep point-wise 3D keypoints voting network for 6DoF pose estimation [C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020: 11629–11638.

[32]	QI C R, YI L, SU H, et al. PointNet++: deep hierarchical feature learning on point sets in a metric space [C]//Advances in Neural Information Processing Systems. Long Beach: NIPS, 2017: 5100–5109.

[33]	HU J, SHEN L, SUN G. Squeeze-and-excitation networks [C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 7132–7141.

[34]	WOO S, PARK J, LEE J Y, et al. CBAM: convolutional block attention module [C]//European Conference on Computer Vision. Munich: Springer, 2018: 3–19.

[35]	HINTERSTOISSER S, CAGNIAT C, IIII V L, et al. Multimodal templates for real-time detection of texture-less objects in heavily cluttered scenes [C]//International Conference on Computer Vision. Barcelona: IEEE, 2011: 858–865.

[1]	Minghua ZHAO,Yuxuan LYU,Jiahao LYU,Yifei CHEN,Cheng SHI,Jing HU. Video anomaly detection based on multi-scale appearance and motion fusion[J]. Journal of ZheJiang University (Engineering Science), 2026, 60(8): 1730-1738.

[2]	Yebo FAN,Yicheng LI,Yong LIU,Wei ZHANG. Efficient attack model targeting GNN-based rumor detectors[J]. Journal of ZheJiang University (Engineering Science), 2026, 60(8): 1739-1748.

[3]	Huixue LIU,Xinqian LIU,Shuai WANG,Chuan ZHAO,Jianguo DING. Graph-adversarial learning framework integrating log semantics for APT detection[J]. Journal of ZheJiang University (Engineering Science), 2026, 60(8): 1749-1759.

[4]	Huixia LIU,Rongjing WANG,Yuxuan GAO,Meng CAO,Suying SHENG,Youpeng MA. Adaptive neural network-based deadlock-free vehicle scheduling algorithm for unsignalized intersections[J]. Journal of ZheJiang University (Engineering Science), 2026, 60(8): 1832-1840.

[5]	Kaiwei XU,Hafiz KHIZER BIN TALIB,Yanlong CAO,Yuanping XU,Zhijie XU,Jingchun SONG. Lightweight micro-expression recognition based on optical flow and convolutional vision Transformer[J]. Journal of ZheJiang University (Engineering Science), 2026, 60(7): 1381-1391.

[6]	Yanchun YANG,Jialong LI. Multi-path collaboration-based and spatial-spectral prior-based hyperspectral and multispectral image fusion[J]. Journal of ZheJiang University (Engineering Science), 2026, 60(7): 1427-1437.

[7]	Ting SU,Yueping XU,Quanjun WANG,Hua ZHONG,Jianqun JIANG. Physics-informed multi-stage surrogate model for urban flooding[J]. Journal of ZheJiang University (Engineering Science), 2026, 60(7): 1557-1566.

[8]	Naizhou ZHANG,Yunchao ZHAO,Wei CAO,Xiaojian ZHANG. Image captioning generation based on multiple-view cross-modal feature fusion[J]. Journal of ZheJiang University (Engineering Science), 2026, 60(6): 1205-1212.

[9]	Yunhong LI,Qiqi ZHANG,Jinni CHEN,Weichong CHEN,Xueping SU,Chengming LIANG. Text-to-image generation algorithm based on generative adversarial network and coordinate attention mechanism[J]. Journal of ZheJiang University (Engineering Science), 2026, 60(6): 1213-1220.

[10]	Wenjun ZHENG,Zhikun LI,Shoufei HAN. Aspect-based sentiment analysis via knowledge-enhanced graph Transformer[J]. Journal of ZheJiang University (Engineering Science), 2026, 60(6): 1269-1276.

[11]	Guoyan LI,Wei YU,Yupeng MEI,Minghui ZHANG,Xinqiang WANG. Building extraction from remote sensing images with global-local feature fusion[J]. Journal of ZheJiang University (Engineering Science), 2026, 60(5): 1100-1108.

[12]	Hongbin LIN,Sijin LV,Chenyang WANG,Tianfang CAI,Pengwei LUO. Radial basis network based on residual/gradient Gaussian adaptive sampling[J]. Journal of ZheJiang University (Engineering Science), 2026, 60(5): 1119-1127.

[13]	Zixiang LI,Kecheng LU,Haibing CAI,Weishuai XIE,Guangdong ZHANG. Concrete crack detection in dark environments based on biomimetic tactile technology[J]. Journal of ZheJiang University (Engineering Science), 2026, 60(5): 915-925.

[14]	Jie WU,Beilin HAN,Yihang ZHANG,Chao ZOU,Lifeng XIN,Shiping HUANG. Method for augmenting crack image datasets via fine-tuning of stable diffusion models[J]. Journal of ZheJiang University (Engineering Science), 2026, 60(5): 926-934.

[15]	Yaolian SONG,Chi PENG,Jingmin TANG,Xuanzhi ZHAO,Guicai YU. Small object detection algorithm for optical remote sensing images based on fusion attention mechanism[J]. Journal of ZheJiang University (Engineering Science), 2026, 60(4): 763-771.

Viewed

Full text

Abstract

Cited

Shared

Discussed