Please wait a minute...
Journal of ZheJiang University (Engineering Science)  2026, Vol. 60 Issue (12): 2602-2612    DOI: 10.3785/j.issn.1008-973X.2026.12.006
    
Medical image fusion based on dual-stage attention mechanism and invertible transformation
Yu SHEN(),Zhen YUAN(),Jiaying LIU,Shan BAI,Zhenkai QIANG
School of Electronic and Information Engineering, Lanzhou Jiaotong University, Lanzhou 730070, China
Download: HTML     PDF(7997KB) HTML
Export: BibTeX | EndNote (RIS)      

Abstract  

A new fusion method based on a dual-stage attention mechanism (DSAM) and invertible transformation was proposed aiming at the problem that existing medical image fusion methods suffer from the loss of global information, insufficient complementary information integration, and poor contrast and clarity of fused images. A DSAM was introduced into the low-frequency and high-frequency branches after wavelet decomposition in order to enhance shallow global features and deep local features, respectively. An IBRC Block was employed to achieve bidirectional and invertible integration of complementary information. A multi-modal feature weighted fusion unit (MM_FWFU) was used to complete feature aggregation. An edge detail enhancement network (EDEN) was applied to improve the detail clarity and edge contrast of the fused images. The experimental results demonstrated that the proposed method achieved improvements of 1.99%, 13.75%, 0.34% and 0.10% in standard deviation (SD), average gradient (AG), correlation coefficient (CC) and multi-scale structural similarity index measure (MS-SSIM), respectively, compared with the best-performing results among nine state-of-the-art methods. The proposed method effectively preserved deep local features from the source images, producing fused images with clear contours, high contrast and rich detail information, and exhibited strong generalization capability.



Key words: medical image fusion      dual-stage attention mechanism (DSAM)      invertible bottleneck residual      edge detail enhancement     
Received: 16 September 2025      Published: 21 October 2026
CLC:  TP 391  
Fund:  国家自然科学基金资助项目(62241106,61861025); 甘肃省重点研发计划资助项目(甘科计[2024]10号-24YFGA037);甘肃省科技专员专项资助项目(甘科计[2023]18号-23CXGA0008);“智慧天路”建设重大专项资助项目(2023QZzhtl1102);兰州局集团公司科技研究开发计划资助项目(LZJKY2025107-1);中国国家铁路集团有限公司重点课题(N2023X050);兰州交通大学重点研发资助项目(LZJTU-ZDYF2305).
Cite this article:

Yu SHEN,Zhen YUAN,Jiaying LIU,Shan BAI,Zhenkai QIANG. Medical image fusion based on dual-stage attention mechanism and invertible transformation. Journal of ZheJiang University (Engineering Science), 2026, 60(12): 2602-2612.

URL:

https://www.zjujournals.com/eng/10.3785/j.issn.1008-973X.2026.12.006     OR     https://www.zjujournals.com/eng/Y2026/V60/I12/2602


基于双阶段注意力机制与可逆变换的医学图像融合

针对现有医学图像融合方法中全局信息易丢失、互补信息融合不充分及融合图像对比度与清晰度差的问题,提出基于双阶段注意力机制(DSAM)与可逆变换的新型融合方法. 在小波分解后的低频与高频分支中引入双阶段注意力模块,分别用于增强浅层全局特征与深层局部特征. 采用可逆瓶颈残差耦合模块(IBRC Block)实现双向可逆的互补信息整合. 通过多模态特征加权融合单元(MM_FWFU)完成特征整合. 通过边缘细节增强网络(EDEN)提升融合图像的细节清晰度和边缘对比度. 实验结果表明,所提方法在标准差(SD)、平均梯度(AG)、相关系数(CC)和多尺度结构相似性(MS-SSIM)等指标上较9种主流方法中的最优值分别提升了1.99%、13.75%、0.34%和0.10%. 利用所提方法,能够有效地保留源图像的深层局部特征,融合结果的轮廓清晰,对比度高,细节信息充分,且具有较强的泛化能力.


关键词: 医学图像融合,  双阶段注意力机制(DSAM),  可逆瓶颈残差,  边缘细节增强 
Fig.1 Multi-modal medical image fusion network based on dual-stage attention mechanism and invertible transformation
Fig.2 Dual-stage attention module
Fig.3 Invertible bottleneck residual coupling block
Fig.4 Texture-contrast multi-scale fusion module
Fig.5 Comparison of MRI-SPECT fusion result for “metastatic bronchial carcinoma”
Fig.6 Comparison of MRI-PET fusion result for Alzheimer's disease
Fig.7 Comparison of MRI-SPECT fusion result for “Glioma”
病例方法SDSFAGCCSCDPSNR/dBVIFMS-SSIM
病例1CNP43.691213.43522.51460.75130.360414.79310.30620.6036
NSST44.691515.23643.62350.77920.406215.36420.40650.5931
MATR45.000913.42183.71400.85850.303317.80220.63810.2624
CDDFuse55.030415.64474.14720.86401.058619.37830.89570.8061
EMMA63.945412.59923.38490.90181.703618.22850.52670.8148
SwinFusion52.073616.02394.35610.86840.598618.56320.69310.2813
MMAE58.286513.43993.36970.89551.423019.97690.62470.8164
BSAFusion66.880018.18304.79090.86321.403716.45990.54230.3911
FATFusion31.544622.50425.38120.79901.750812.23010.14940.6179
MBHFuse67.671217.50064.40930.90671.790116.89580.54020.3959
CrossFuse66.474511.22392.90350.90831.693914.80620.55640.2261
本文方法69.020625.59965.25580.90980.650017.53850.76660.8172
病例2CNP45.279029.44504.93730.69270.872710.12460.31910.5961
NSST39.230316.25985.22480.72340.399212.81970.22570.2060
MATR45.425923.93036.27870.68730.280314.01410.62830.2358
CDDFuse65.207129.45477.64310.78961.638415.51310.66290.7536
EMMA72.931420.99766.20790.78041.730014.23550.52840.7335
SwinFusion59.215030.04458.01290.77091.459315.06730.60020.2524
MMAE70.848225.29826.75220.78341.629814.76330.62350.7463
BSAFusion66.870833.33458.46010.74291.403914.27140.53430.3321
FATFusion56.579030.32447.74340.72681.380415.16290.71960.7555
MBHFuse73.266331.87907.93700.78881.694214.09940.61660.3147
CrossFuse67.862213.97374.19440.75651.603213.58810.54640.1962
本文方法67.144535.22248.85160.79641.617315.21540.65950.7621
病例3CNP43.824916.34853.57930.70140.858911.58290.38530.5715
NSST40.578213.19433.68390.66190.492812.34620.41230.4812
MATR50.110316.34694.61630.64750.578613.59290.74380.2504
CDDFuse58.760019.55575.31710.66801.071415.02290.85880.7630
EMMA59.540017.28635.00760.70381.396514.55860.62430.7335
SwinFusion56.126819.80785.53980.68901.177114.46770.86070.2619
MMAE57.468517.69964.90850.70791.456715.27740.73350.7576
BSAFusion62.485622.46486.64070.66591.163413.72030.68700.4304
FATFusion67.891722.17936.03030.65071.175313.49800.86240.7528
MBHFuse58.241620.25795.56030.70531.394714.71260.83360.3152
CrossFuse39.081011.35173.19580.74801.187815.55920.66580.1936
本文方法65.052125.00536.05770.78961.663714.76140.80980.7698
Tab.1 Objective evaluation result of fused images for each case
方法设备Sm/kBFLOPs/109tt/s
MATRGPU861.9480.046
CDDFuseGPU471239.1660.050
EMMAGPU60438.8620.048
SwinFusionGPU7747127.4612.386
MMAEGPU91420.3271.313
BSAFusionGPU3934851.4320.270
FATFusionGPU3043888.9302.724
MBHFuseGPU12020.7370.513
CrossFuseGPU49145.0371.180
本文方法GPU11370.2080.039
Tab.2 Comparison of model size, FLOPs and inference time
Fig.8 Subjective comparison of ablation experiment
方法SDSFAGCCPSNR/dBSCDVIFMS-SSIM
DSAMIBRCEDEN
—√√62.222219.70615.10530.903118.63761.77710.59840.8012
√—√64.230319.85565.35210.895318.11741.85280.62270.7751
√√—61.972819.60715.12060.881419.30511.75150.64720.8007
√——53.156919.43904.24880.893911.28351.32870.28570.8029
—√—62.953716.73384.25120.826118.13591.77760.25790.3367
——√58.462416.25145.26340.834216.45431.23450.46240.4672
———56.490020.25315.03060.705315.57531.66720.06630.5890
√√√66.244920.94475.44020.918919.32951.86460.84690.8105
Tab.3 Objective comparison of ablation experiment
[1]   沈瑜, 白珊, 魏子易, 等 基于跨模态感知和空频交叉的医学图像融合[J]. 中国激光, 2025, 52 (9): 0810001
Shen Yu, Bai Shan, Wei Ziyi, et al Medical image fusion network for Cross-Modality perception and Spatial-Frequency interaction[J]. Chinese Journal of Lasers, 2025, 52 (9): 0810001
[2]   沈瑜, 刘佳英, 闫佳荣, 等 基于局部与全局信息协同的双分支多模态医学图像融合[J]. 光学学报, 2025, 45 (8): 117- 131
Shen Yu, Liu Jiaying, Yan Jiarong, et al Dual-branch multimodal medical image fusion based on local and global information collaboration[J]. Acta Optica Sinica, 2025, 45 (8): 117- 131
[3]   Qiang Z, Shen Y, Yuan Y, et al DWSFusion: dual weight supervision for lightweight infrared and visible image fusion[J]. Pattern Recognition, 2026, 179: 113520
doi: 10.1016/j.patcog.2026.113520
[4]   Satyanarayana V, Mohanaiah P Multimodal medical image fusion in NSST domain in coupled neural systems[J]. International Journal of Computational Intelligence Systems, 2025, 18 (1): 111
doi: 10.1007/s44196-025-00841-4
[5]   沈瑜, 强振凯, 魏子易, 等 基于深度学习的红外与可见光图像融合综述: 发展与展望[J]. 控制与决策, 2025, 40 (6): 1793- 1806
Shen Yu, Qiang Zhenkai, Wei Ziyi, et al Review of infrared and visible image fusion based on deep learning: developments and prospects[J]. Control and Decision, 2025, 40 (6): 1793- 1806
doi: 10.13195/j.kzyjc.2024.1037
[6]   Wang Y M, Wang C Y, Liu K Y, et al CNN-based cross-modality fusion for enhanced breast cancer detection using mammography and ultrasound[J]. Tomography, 2024, 10 (12): 2038- 2057
doi: 10.3390/tomography10120145
[7]   Xie X Y, Zhang X Z, Tang X L, et al MACTFusion: lightweight cross transformer for adaptive multimodal medical image fusion[J]. IEEE Journal of Biomedical and Health Informatics, 2025, 29 (5): 3317- 3328
doi: 10.1109/JBHI.2024.3391620
[8]   Wang Y H, Liu R R, Li Z H, et al Variable augmented network for invertible modality synthesis and fusion[J]. IEEE Journal of Biomedical and Health Informatics, 2023, 27 (6): 2898- 2909
doi: 10.1109/JBHI.2023.3257544
[9]   Zhao Z X, Bai H W, Zhang J S, et al. CDDFuse: correlation-driven dual-branch feature decomposition for multi-modality image fusion [C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Vancouver: IEEE, 2023: 5906–5916.
[10]   He D, He X P, Xiang H L, et al Wavelet transform-based two-stream convolutional networks for face anti-spoofing[J]. Journal of Electronic Imaging, 2023, 32 (1): 013015
doi: 10.1117/1.jei.32.1.013015
[11]   Wang Z, Simoncelli E P, Bovik A C. Multiscale structural similarity for image quality assessment [C]//Proceedings of the 37th Asilomar Conference on Signals, Systems and Computers. Pacific Grove: IEEE, 2004: 1398–1402.
[12]   Ma J Y, Tang L F, Fan F, et al SwinFusion: cross-domain long-range learning for general image fusion via swin transformer[J]. IEEE/CAA Journal of Automatica Sinica, 2022, 9 (7): 1200- 1217
doi: 10.1109/JAS.2022.105686
[13]   Li B, Peng H, Luo X H, et al Medical image fusion method based on coupled neural P systems in nonsubsampled shearlet transform domain[J]. International Journal of Neural Systems, 2021, 31 (1): 2050050
doi: 10.1142/S0129065720500501
[14]   Kong W W, Zhang L J, Lei Y Novel fusion method for visible light and infrared images based on NSST-SF-PCNN[J]. Infrared Physics and Technology, 2014, 65: 103- 112
doi: 10.1016/j.infrared.2014.04.003
[15]   Li H, Su D, Cai Q, et al BSAFusion: a bidirectional stepwise feature alignment network for unaligned medical image fusion[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2025, 39 (5): 4725- 4733
doi: 10.1609/aaai.v39i5.32499
[16]   Wang X X, Fang L X, Zhao J L, et al MMAE: a universal image fusion method via mask attention mechanism[J]. Pattern Recognition, 2025, 158: 111041
doi: 10.1016/j.patcog.2024.111041
[17]   Sun Y C, Dong M L, Yu M X, et al MBHFuse: a multi- branch heterogeneous global and local infrared and visible image fusion with differential convolutional amplification features[J]. Optics and Laser Technology, 2025, 181: 111666
doi: 10.1016/j.optlastec.2024.111666
[18]   Zhao Z, Bai H, Zhang J, et al. Equivariant multi-modality image fusion [C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2024: 25912–25921.
[19]   Li H, Wu X J CrossFuse: a novel cross attention mechanism based infrared and visible image fusion approach[J]. Information Fusion, 2024, 103: 102147
doi: 10.1016/j.inffus.2023.102147
[20]   Tang W, He F Z FATFusion: a functional–anatomical transformer for medical image fusion[J]. Information Processing and Management, 2024, 61 (4): 103687
doi: 10.1016/j.ipm.2024.103687
[1] Kejia CHEN,Weiwen ZENG,Zhenqing WANG. Image data enhancement method based on conditional generative adversarial network and Poisson blending[J]. Journal of ZheJiang University (Engineering Science), 2026, 60(12): 2592-2601.
[2] Yi PENG,Heng LI,Qingqing YANG,Hui LI,Jianming WANG. Multilevel semantic MIMO transmission of images based on Swin Transformer[J]. Journal of ZheJiang University (Engineering Science), 2026, 60(11): 2462-2471.
[3] Zeyi YANG,Zhengdong ZHOU,Xiaoxi YUAN,Zhi CAI,Mengyao YUAN. 3D point cloud-based trajectory planning method for personalized pattern spraying on complex surfaces[J]. Journal of ZheJiang University (Engineering Science), 2026, 60(11): 2364-2372.
[4] Aolong TAN,Wenjie FU,Ao SHEN,Zhaowei XIANG,Jinghua MA,Qin YIN. Hierarchical classification method for machining process texts integrating multi-channel features and word-weighted clustering[J]. Journal of ZheJiang University (Engineering Science), 2026, 60(11): 2373-2382.
[5] Chunjuan LIU,Yihang BAO,Xiaosuo WU,Haowen YAN,Tieshuan ZHU,Haoran ZHAO. PCB defect detection algorithm based on detail enhancement and cross-scale attention fusion[J]. Journal of ZheJiang University (Engineering Science), 2026, 60(11): 2431-2441.
[6] Tengfei HUANG,Jiuyuan HUO,Fang CAO,Zeyu WU,Chuile KONG. Framework for railway defect detection oriented to small and blurry object[J]. Journal of ZheJiang University (Engineering Science), 2026, 60(11): 2453-2461.
[7] Yuzhen BU,Jiabin YU,Daobin MA,Liangyu CHEN,Long SUN,Li YANG,Dongping ZHANG. Industrial image anomaly detection via diffusion synthesis and feature mining[J]. Journal of ZheJiang University (Engineering Science), 2026, 60(10): 2186-2195.
[8] Hongxia NIU,Dingchao FENG,Tao HOU. Real-time detection algorithm for railway foreign objects in complex weather conditions based on improved RT-DETR[J]. Journal of ZheJiang University (Engineering Science), 2026, 60(10): 2165-2175.
[9] Xiaodong CAI,Sheng FANG,Ting LI. Information bottleneck-based social recommendation optimization model[J]. Journal of ZheJiang University (Engineering Science), 2026, 60(10): 2227-2235.
[10] Wenyu MA,Hongbin XIA,Xiaofeng WANG. Multi-modal recommendation model for denoising and user preference modeling[J]. Journal of ZheJiang University (Engineering Science), 2026, 60(10): 2215-2226.
[11] Hongsheng FU,Yanlong CAO,Junding LUO,Zongzheng ZHANG,Tukun LI,Fang HUANG. Spatial-semantic hierarchical knowledge graph modeling for body-in-white datum design[J]. Journal of ZheJiang University (Engineering Science), 2026, 60(9): 1841-1850.
[12] Jiuyuan HUO,Hao JIANG,Chen CHANG. Text detection of highway traffic sign based on improved EAST[J]. Journal of ZheJiang University (Engineering Science), 2026, 60(9): 1881-1889.
[13] Rui CAI,Qinhan GUO,Zihao ZHANG,Jianfeng DONG,Rong ZHANG,Baolong LIU,Xun WANG. Large model-assisted noise-robust unsupervised skeleton representation learning[J]. Journal of ZheJiang University (Engineering Science), 2026, 60(9): 1972-1979.
[14] Jun LI,Binbin DING,Weijuan SHI,Lin YANG. Aerial image detection algorithm based on multiscale feature aggregation[J]. Journal of ZheJiang University (Engineering Science), 2026, 60(9): 1890-1900.
[15] Yijie TANG,Mingen ZHONG,Bingan YUAN,Kang FAN,Jiawei TAN,Zhiqiang LIN. Road segmentation algorithm based on joint orthogonal feature fusion and large kernel separable attention[J]. Journal of ZheJiang University (Engineering Science), 2026, 60(9): 1942-1952.