Please wait a minute...
浙江大学学报(工学版)  2026, Vol. 60 Issue (12): 2602-2612    DOI: 10.3785/j.issn.1008-973X.2026.12.006
计算机技术     
基于双阶段注意力机制与可逆变换的医学图像融合
沈瑜(),袁祯(),刘佳英,白珊,强振凯
兰州交通大学 电子与信息工程学院 甘肃 兰州 730070
Medical image fusion based on dual-stage attention mechanism and invertible transformation
Yu SHEN(),Zhen YUAN(),Jiaying LIU,Shan BAI,Zhenkai QIANG
School of Electronic and Information Engineering, Lanzhou Jiaotong University, Lanzhou 730070, China
 全文: PDF(7997 KB)   HTML
摘要:

针对现有医学图像融合方法中全局信息易丢失、互补信息融合不充分及融合图像对比度与清晰度差的问题,提出基于双阶段注意力机制(DSAM)与可逆变换的新型融合方法. 在小波分解后的低频与高频分支中引入双阶段注意力模块,分别用于增强浅层全局特征与深层局部特征. 采用可逆瓶颈残差耦合模块(IBRC Block)实现双向可逆的互补信息整合. 通过多模态特征加权融合单元(MM_FWFU)完成特征整合. 通过边缘细节增强网络(EDEN)提升融合图像的细节清晰度和边缘对比度. 实验结果表明,所提方法在标准差(SD)、平均梯度(AG)、相关系数(CC)和多尺度结构相似性(MS-SSIM)等指标上较9种主流方法中的最优值分别提升了1.99%、13.75%、0.34%和0.10%. 利用所提方法,能够有效地保留源图像的深层局部特征,融合结果的轮廓清晰,对比度高,细节信息充分,且具有较强的泛化能力.

关键词: 医学图像融合;  双阶段注意力机制(DSAM);  可逆瓶颈残差;  边缘细节增强    
Abstract:

A new fusion method based on a dual-stage attention mechanism (DSAM) and invertible transformation was proposed aiming at the problem that existing medical image fusion methods suffer from the loss of global information, insufficient complementary information integration, and poor contrast and clarity of fused images. A DSAM was introduced into the low-frequency and high-frequency branches after wavelet decomposition in order to enhance shallow global features and deep local features, respectively. An IBRC Block was employed to achieve bidirectional and invertible integration of complementary information. A multi-modal feature weighted fusion unit (MM_FWFU) was used to complete feature aggregation. An edge detail enhancement network (EDEN) was applied to improve the detail clarity and edge contrast of the fused images. The experimental results demonstrated that the proposed method achieved improvements of 1.99%, 13.75%, 0.34% and 0.10% in standard deviation (SD), average gradient (AG), correlation coefficient (CC) and multi-scale structural similarity index measure (MS-SSIM), respectively, compared with the best-performing results among nine state-of-the-art methods. The proposed method effectively preserved deep local features from the source images, producing fused images with clear contours, high contrast and rich detail information, and exhibited strong generalization capability.

Key words: medical image fusion    dual-stage attention mechanism (DSAM)    invertible bottleneck residual    edge detail enhancement
收稿日期: 2025-09-16 出版日期: 2026-10-21
CLC:  TP 391  
基金资助: 国家自然科学基金资助项目(62241106,61861025); 甘肃省重点研发计划资助项目(甘科计[2024]10号-24YFGA037);甘肃省科技专员专项资助项目(甘科计[2023]18号-23CXGA0008);“智慧天路”建设重大专项资助项目(2023QZzhtl1102);兰州局集团公司科技研究开发计划资助项目(LZJKY2025107-1);中国国家铁路集团有限公司重点课题(N2023X050);兰州交通大学重点研发资助项目(LZJTU-ZDYF2305).
作者简介: 沈瑜(1982—),女,教授,博导,从事医学图像融合研究. orcid.org/0000-0001-5396-9657. E-mail:18609311366@163.com
服务  
把本文推荐给朋友
加入引用管理器
E-mail Alert
作者相关文章  
沈瑜
袁祯
刘佳英
白珊
强振凯

引用本文:

沈瑜,袁祯,刘佳英,白珊,强振凯. 基于双阶段注意力机制与可逆变换的医学图像融合[J]. 浙江大学学报(工学版), 2026, 60(12): 2602-2612.

Yu SHEN,Zhen YUAN,Jiaying LIU,Shan BAI,Zhenkai QIANG. Medical image fusion based on dual-stage attention mechanism and invertible transformation. Journal of ZheJiang University (Engineering Science), 2026, 60(12): 2602-2612.

链接本文:

https://www.zjujournals.com/eng/CN/10.3785/j.issn.1008-973X.2026.12.006    或    https://www.zjujournals.com/eng/CN/Y2026/V60/I12/2602

图 1  基于双阶段注意力机制与可逆变换的多模态医学图像融合网络
图 2  双阶段注意力模块
图 3  可逆瓶颈残差耦合模块
图 4  纹理-对比多尺度融合模块
图 5  “转移性支气管癌”MRI-SPECT融合结果对比
图 6  “阿尔茨海默病”MRI-PET融合结果对比
图 7  “神经胶质瘤”MRI-SPECT融合结果对比
病例方法SDSFAGCCSCDPSNR/dBVIFMS-SSIM
病例1CNP43.691213.43522.51460.75130.360414.79310.30620.6036
NSST44.691515.23643.62350.77920.406215.36420.40650.5931
MATR45.000913.42183.71400.85850.303317.80220.63810.2624
CDDFuse55.030415.64474.14720.86401.058619.37830.89570.8061
EMMA63.945412.59923.38490.90181.703618.22850.52670.8148
SwinFusion52.073616.02394.35610.86840.598618.56320.69310.2813
MMAE58.286513.43993.36970.89551.423019.97690.62470.8164
BSAFusion66.880018.18304.79090.86321.403716.45990.54230.3911
FATFusion31.544622.50425.38120.79901.750812.23010.14940.6179
MBHFuse67.671217.50064.40930.90671.790116.89580.54020.3959
CrossFuse66.474511.22392.90350.90831.693914.80620.55640.2261
本文方法69.020625.59965.25580.90980.650017.53850.76660.8172
病例2CNP45.279029.44504.93730.69270.872710.12460.31910.5961
NSST39.230316.25985.22480.72340.399212.81970.22570.2060
MATR45.425923.93036.27870.68730.280314.01410.62830.2358
CDDFuse65.207129.45477.64310.78961.638415.51310.66290.7536
EMMA72.931420.99766.20790.78041.730014.23550.52840.7335
SwinFusion59.215030.04458.01290.77091.459315.06730.60020.2524
MMAE70.848225.29826.75220.78341.629814.76330.62350.7463
BSAFusion66.870833.33458.46010.74291.403914.27140.53430.3321
FATFusion56.579030.32447.74340.72681.380415.16290.71960.7555
MBHFuse73.266331.87907.93700.78881.694214.09940.61660.3147
CrossFuse67.862213.97374.19440.75651.603213.58810.54640.1962
本文方法67.144535.22248.85160.79641.617315.21540.65950.7621
病例3CNP43.824916.34853.57930.70140.858911.58290.38530.5715
NSST40.578213.19433.68390.66190.492812.34620.41230.4812
MATR50.110316.34694.61630.64750.578613.59290.74380.2504
CDDFuse58.760019.55575.31710.66801.071415.02290.85880.7630
EMMA59.540017.28635.00760.70381.396514.55860.62430.7335
SwinFusion56.126819.80785.53980.68901.177114.46770.86070.2619
MMAE57.468517.69964.90850.70791.456715.27740.73350.7576
BSAFusion62.485622.46486.64070.66591.163413.72030.68700.4304
FATFusion67.891722.17936.03030.65071.175313.49800.86240.7528
MBHFuse58.241620.25795.56030.70531.394714.71260.83360.3152
CrossFuse39.081011.35173.19580.74801.187815.55920.66580.1936
本文方法65.052125.00536.05770.78961.663714.76140.80980.7698
表 1  各病例融合图像的客观评价结果
方法设备Sm/kBFLOPs/109tt/s
MATRGPU861.9480.046
CDDFuseGPU471239.1660.050
EMMAGPU60438.8620.048
SwinFusionGPU7747127.4612.386
MMAEGPU91420.3271.313
BSAFusionGPU3934851.4320.270
FATFusionGPU3043888.9302.724
MBHFuseGPU12020.7370.513
CrossFuseGPU49145.0371.180
本文方法GPU11370.2080.039
表 2  模型大小、FLOPs与推理时间的比较
图 8  消融实验的主观对比
方法SDSFAGCCPSNR/dBSCDVIFMS-SSIM
DSAMIBRCEDEN
—√√62.222219.70615.10530.903118.63761.77710.59840.8012
√—√64.230319.85565.35210.895318.11741.85280.62270.7751
√√—61.972819.60715.12060.881419.30511.75150.64720.8007
√——53.156919.43904.24880.893911.28351.32870.28570.8029
—√—62.953716.73384.25120.826118.13591.77760.25790.3367
——√58.462416.25145.26340.834216.45431.23450.46240.4672
———56.490020.25315.03060.705315.57531.66720.06630.5890
√√√66.244920.94475.44020.918919.32951.86460.84690.8105
表 3  消融实验的客观比较
1 沈瑜, 白珊, 魏子易, 等 基于跨模态感知和空频交叉的医学图像融合[J]. 中国激光, 2025, 52 (9): 0810001
Shen Yu, Bai Shan, Wei Ziyi, et al Medical image fusion network for Cross-Modality perception and Spatial-Frequency interaction[J]. Chinese Journal of Lasers, 2025, 52 (9): 0810001
2 沈瑜, 刘佳英, 闫佳荣, 等 基于局部与全局信息协同的双分支多模态医学图像融合[J]. 光学学报, 2025, 45 (8): 117- 131
Shen Yu, Liu Jiaying, Yan Jiarong, et al Dual-branch multimodal medical image fusion based on local and global information collaboration[J]. Acta Optica Sinica, 2025, 45 (8): 117- 131
3 Qiang Z, Shen Y, Yuan Y, et al DWSFusion: dual weight supervision for lightweight infrared and visible image fusion[J]. Pattern Recognition, 2026, 179: 113520
doi: 10.1016/j.patcog.2026.113520
4 Satyanarayana V, Mohanaiah P Multimodal medical image fusion in NSST domain in coupled neural systems[J]. International Journal of Computational Intelligence Systems, 2025, 18 (1): 111
doi: 10.1007/s44196-025-00841-4
5 沈瑜, 强振凯, 魏子易, 等 基于深度学习的红外与可见光图像融合综述: 发展与展望[J]. 控制与决策, 2025, 40 (6): 1793- 1806
Shen Yu, Qiang Zhenkai, Wei Ziyi, et al Review of infrared and visible image fusion based on deep learning: developments and prospects[J]. Control and Decision, 2025, 40 (6): 1793- 1806
doi: 10.13195/j.kzyjc.2024.1037
6 Wang Y M, Wang C Y, Liu K Y, et al CNN-based cross-modality fusion for enhanced breast cancer detection using mammography and ultrasound[J]. Tomography, 2024, 10 (12): 2038- 2057
doi: 10.3390/tomography10120145
7 Xie X Y, Zhang X Z, Tang X L, et al MACTFusion: lightweight cross transformer for adaptive multimodal medical image fusion[J]. IEEE Journal of Biomedical and Health Informatics, 2025, 29 (5): 3317- 3328
doi: 10.1109/JBHI.2024.3391620
8 Wang Y H, Liu R R, Li Z H, et al Variable augmented network for invertible modality synthesis and fusion[J]. IEEE Journal of Biomedical and Health Informatics, 2023, 27 (6): 2898- 2909
doi: 10.1109/JBHI.2023.3257544
9 Zhao Z X, Bai H W, Zhang J S, et al. CDDFuse: correlation-driven dual-branch feature decomposition for multi-modality image fusion [C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Vancouver: IEEE, 2023: 5906–5916.
10 He D, He X P, Xiang H L, et al Wavelet transform-based two-stream convolutional networks for face anti-spoofing[J]. Journal of Electronic Imaging, 2023, 32 (1): 013015
doi: 10.1117/1.jei.32.1.013015
11 Wang Z, Simoncelli E P, Bovik A C. Multiscale structural similarity for image quality assessment [C]//Proceedings of the 37th Asilomar Conference on Signals, Systems and Computers. Pacific Grove: IEEE, 2004: 1398–1402.
12 Ma J Y, Tang L F, Fan F, et al SwinFusion: cross-domain long-range learning for general image fusion via swin transformer[J]. IEEE/CAA Journal of Automatica Sinica, 2022, 9 (7): 1200- 1217
doi: 10.1109/JAS.2022.105686
13 Li B, Peng H, Luo X H, et al Medical image fusion method based on coupled neural P systems in nonsubsampled shearlet transform domain[J]. International Journal of Neural Systems, 2021, 31 (1): 2050050
doi: 10.1142/S0129065720500501
14 Kong W W, Zhang L J, Lei Y Novel fusion method for visible light and infrared images based on NSST-SF-PCNN[J]. Infrared Physics and Technology, 2014, 65: 103- 112
doi: 10.1016/j.infrared.2014.04.003
15 Li H, Su D, Cai Q, et al BSAFusion: a bidirectional stepwise feature alignment network for unaligned medical image fusion[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2025, 39 (5): 4725- 4733
doi: 10.1609/aaai.v39i5.32499
16 Wang X X, Fang L X, Zhao J L, et al MMAE: a universal image fusion method via mask attention mechanism[J]. Pattern Recognition, 2025, 158: 111041
doi: 10.1016/j.patcog.2024.111041
17 Sun Y C, Dong M L, Yu M X, et al MBHFuse: a multi- branch heterogeneous global and local infrared and visible image fusion with differential convolutional amplification features[J]. Optics and Laser Technology, 2025, 181: 111666
doi: 10.1016/j.optlastec.2024.111666
18 Zhao Z, Bai H, Zhang J, et al. Equivariant multi-modality image fusion [C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2024: 25912–25921.
19 Li H, Wu X J CrossFuse: a novel cross attention mechanism based infrared and visible image fusion approach[J]. Information Fusion, 2024, 103: 102147
doi: 10.1016/j.inffus.2023.102147
20 Tang W, He F Z FATFusion: a functional–anatomical transformer for medical image fusion[J]. Information Processing and Management, 2024, 61 (4): 103687
doi: 10.1016/j.ipm.2024.103687
[1] 陈可嘉,曾威文,王朕卿. 基于条件生成对抗和泊松融合的图像数据增强方法[J]. 浙江大学学报(工学版), 2026, 60(12): 2592-2601.
[2] 黄腾飞,火久元,曹芳,武泽宇,孔垂乐. 面向小目标与模糊目标的轨道缺陷检测框架[J]. 浙江大学学报(工学版), 2026, 60(11): 2453-2461.
[3] 彭艺,李恒,杨青青,李辉,王健明. 基于Swin Transformer的图像多级语义MIMO传输[J]. 浙江大学学报(工学版), 2026, 60(11): 2462-2471.
[4] 杨泽毅,周正东,袁晓曦,蔡智,袁梦瑶. 基于3D点云复杂曲面的个性化图案喷涂轨迹规划方法[J]. 浙江大学学报(工学版), 2026, 60(11): 2364-2372.
[5] 谭澳龙,付文杰,沈奥,向召伟,马靖华,殷勤. 融合多通道特征与词权聚类的机加工艺文本层级分类方法[J]. 浙江大学学报(工学版), 2026, 60(11): 2373-2382.
[6] 刘春娟,包易航,吴小所,闫浩文,朱铁栓,赵浩然. 基于细节增强和跨尺度注意力融合的PCB缺陷检测算法[J]. 浙江大学学报(工学版), 2026, 60(11): 2431-2441.
[7] 马文煜,夏鸿斌,王晓锋. 面向去噪与用户偏好建模的多模态推荐模型[J]. 浙江大学学报(工学版), 2026, 60(10): 2215-2226.
[8] 蔡晓东,方晟,李婷. 基于信息瓶颈的社交推荐优化模型[J]. 浙江大学学报(工学版), 2026, 60(10): 2227-2235.
[9] 牛宏侠,冯鼎超,侯涛. 基于改进RT-DETR的复杂天气下铁路异物实时检测算法[J]. 浙江大学学报(工学版), 2026, 60(10): 2165-2175.
[10] 卜玉真,余家斌,马道滨,陈梁宇,孙龙,杨力,章东平. 基于扩散合成与特征挖掘的工业图像异常检测[J]. 浙江大学学报(工学版), 2026, 60(10): 2186-2195.
[11] 武晓春,李梓宁. 基于YOLOv11n的改进铁路工人安全穿戴检测模型[J]. 浙江大学学报(工学版), 2026, 60(9): 1901-1911.
[12] 火久元,姜灏,常琛. 基于改进EAST的高速公路交通标识文本检测[J]. 浙江大学学报(工学版), 2026, 60(9): 1881-1889.
[13] 付红圣,曹衍龙,罗钧鼎,张宗政,李涂鲲,黄芳. 白车身基准设计空间-语义分层知识图谱建模[J]. 浙江大学学报(工学版), 2026, 60(9): 1841-1850.
[14] 李珺,丁彬彬,史维娟,杨琳. 基于多尺度特征聚合的航拍图像检测算法[J]. 浙江大学学报(工学版), 2026, 60(9): 1890-1900.
[15] 汤毅杰,钟铭恩,袁彬淦,范康,谭佳威,林志强. 联合正交特征融合与大核可分离注意力的道路分割算法[J]. 浙江大学学报(工学版), 2026, 60(9): 1942-1952.