Please wait a minute...
浙江大学学报(工学版)  2025, Vol. 59 Issue (1): 39-48    DOI: 10.3785/j.issn.1008-973X.2025.01.004
计算机与控制工程     
阿尔茨海默病辅助诊断的多模态数据融合轻量级网络
王光明(),柏正尧*(),宋帅,徐月娥
云南大学 信息学院,云南 昆明 650504
Lightweight multimodal data fusion network for auxiliary diagnosis of Alzheimer’s disease
Guangming WANG(),Zhengyao BAI*(),Shuai SONG,Yue’e XU
School of Information Science and Engineering, Yunnan University, Kunming 650504, China
 全文: PDF(1364 KB)   HTML
摘要:

单模态阿尔茨海默病辅助诊断方法缺少专业标注的影像数据,特征提取不稳定且要求高计算能力,为此融合核磁成像、正电子发射断层扫描影像数据和精神认知评分数据,提出多模态轻量级阿尔茨海默病辅助诊断网络(LightMoDAD). 在影像特征提取模块中,去冗余卷积以提取局部特征,引入全局滤波用于提取全局特征,通过配准并相加实现多模态影像特征融合. 在文本特征提取模块中,由可分离深度卷积提取精神认知评分数据特征与多模态影像特征融合,通过迁移学习增强特征判别性. 采用多层感知器识别复杂的模式和特征,提高所提网络的分类准确率. 在ADNI数据库中开展有效性验证实验,LightMoDAD的分类准确率、敏感性和特异性分别为0.980、0.985和0.975. 实验结果表明,所提网络有助于提高医生诊断效率,具有移动端部署潜力.

关键词: 阿尔茨海默病多模态数据轻量级网络融合算法迁移学习    
Abstract:

The auxiliary diagnostic methods for Alzheimer’s disease (AD) using single-modal data suffer from a lack of professionally annotated imaging data, unstable feature extraction and high requirements for computing power. A multimodal lightweight Alzheimer’s disease auxiliary diagnostic network (LightMoDAD) was proposed, and magnetic resonance imaging (MRI), positron emission tomography (PET) imaging data, and psychometric score data were utilized in the network. In the image feature extraction module, spatial and channel reconstruction convolution was employed to extract local features, global filtering to extract global features, and feature registration and addition to achieve multimodal feature fusion. In the text feature extraction module, the features extracted by separable convolution from psychometric score data were integrated with multimodal image feature, and transfer learning was used to improve feature discrimination. A multi-layer perceptron was applied to recognize complex patterns and features, and the classification accuracy of the proposed network was improved. Experiments were conducted on the ADNI dataset, and LightMoDAD’s accuracy, sensitivity and specificity were 0.980, 0.985 and 0.975, respectively. Experimental results show that the proposed network enhances physicians’ diagnosis efficiency and holds potential for mobile platform deployment.

Key words: Alzheimer’s disease    multimodal data    lightweight network    fusion algorithm    transfer learning
收稿日期: 2024-01-22 出版日期: 2025-01-18
CLC:  TP 391  
基金资助: 云南省重大科技专项计划资助项目(202002AD080001).
通讯作者: 柏正尧     E-mail: 1615710321@qq.com;biazhy@ynu.edu.cn
作者简介: 王光明(1997—),男,硕士生,从事阿尔茨海默病医学图像处理研究. orcid.org/0009-0008-3946-3578. E-mail:1615710321@qq.com
服务  
把本文推荐给朋友
加入引用管理器
E-mail Alert
作者相关文章  
王光明
柏正尧
宋帅
徐月娥

引用本文:

王光明,柏正尧,宋帅,徐月娥. 阿尔茨海默病辅助诊断的多模态数据融合轻量级网络[J]. 浙江大学学报(工学版), 2025, 59(1): 39-48.

Guangming WANG,Zhengyao BAI,Shuai SONG,Yue’e XU. Lightweight multimodal data fusion network for auxiliary diagnosis of Alzheimer’s disease. Journal of ZheJiang University (Engineering Science), 2025, 59(1): 39-48.

链接本文:

https://www.zjujournals.com/eng/CN/10.3785/j.issn.1008-973X.2025.01.004        https://www.zjujournals.com/eng/CN/Y2025/V59/I1/39

类型NMNF年龄SMMSESGDSSCDRSFAQSNPI
AD644175.76±24.2521.48±8.521.93±6.120.93±0.4016.68±4.234.59±4.86
CN8210475.92±12.1328.88±4.211.09±3.650.07±0.120.67±0.760.71±0.61
MCI757775.95±11.2925.68±4.861.96±4.530.59±0.357.53±2.622.75±4.85
表 1  ADNI数据库中的受试者信息
图 1  图像预处理过程
图 2  多模态图像融合过程
图 3  多模态图像数据的降维过程
图 4  多模态轻量级阿尔兹海默病辅助诊断网络
图 5  所提网络的特征提取模块
图 6  迁移学习的示意图
模型$ {{\mathrm{ACC}}_3} $$ {{\mathrm{ACC}}_2} $$ {{\mathrm{SEN}}_2} $$ {{\mathrm{SPE}}_2} $
MRI0.7460.8820.8750.848
PET0.6850.9280.9460.927
MRI+PET0.8220.9430.9580.934
文本0.8800.9480.9540.944
本研究0.9120.9800.9850.975
表 2  不同数据模型的分类结果
网络ACCSENSPEFLOPS8NP5MAC8
MobileViT[15]0.9470.9680.9473.39.99.2
Mobileswin0.9230.9450.9522.14.97.4
图像特征0.9430.9580.9341.33.37.6
文本特征0.9480.9540.9443.22.71.6
本研究0.9800.9850.9751.33.37.6
表 3  不同分类网络的性能参数
损失函数$ {{\mathrm{ACC}}_3} $$ {{\mathrm{ACC}}_2} $$ {{\mathrm{SEN}}_2} $$ {{\mathrm{SPE}}_2} $
0.8560.9610.9670.956
焦点0.8910.9720.9880.968
多项式0.9120.9800.9850.975
表 4  不同损失函数的分类结果
图 7  不同分类网络的性能评估参数对比
图 8  不同成像技术得到的海马体位置图像
图 9  所提网络对脑部图像的训练可视化
36 LI Changjun, TIAN Miaoyuan, LI Qiwei, et al Action mechanism of acupuncture intervention on hippocampus of Alzheimer’s disease[J]. Journal of Clinical Acupuncture and Moxibustion, 2023, 39 (7): 102- 106
37 SELVARAJU R R, COGSWELL M, DAS A, et al. Grad-CAM: visual explanations from deep networks via gradient-based localization [C]// Proceedings of the IEEE International Conference on Computer Vision . Venice: IEEE, 2017: 618–626.
1 YAO Z, WANG H, YAN W, et al Artificial intelligence-based diagnosis of Alzheimer’s disease with brain MRI images[J]. European Journal of Radiology, 2023, 165: 110934
doi: 10.1016/j.ejrad.2023.110934
2 MENG Q, LIN M S, TZENG I S Relationship between exercise and Alzheimer’s disease: a narrative literature review[J]. Frontiers in Neuroscience, 2020, 14: 131
doi: 10.3389/fnins.2020.00131
3 GAUTHIER S, ISMAIL Z, GOODARZI Z, et al Clinicians’ perspectives on how disease modifying drugs for Alzheimer’s disease impact specialty care[J]. The Journal of Prevention of Alzheimer’s Disease, 2023, 10: 339- 341
4 REN R, QI J, LIN S, et al The China Alzheimer report 2022[J]. Gen Psychiatr, 2022, 35 (1): e100751
doi: 10.1136/gpsych-2022-100751
5 LIU S, SONG Y, CAI W, et al. Multifold Bayesian kernelization in Alzheimer’s diagnosis [J] Medical Image Computing and Computer-Assisted Intervention , 2013, 16(3): 303–310.
6 ZHANG D, WANG Y, ZHOU L, et al Multimodal classification of Alzheimer’s disease and mild cognitive impairment[J]. Neuroimage, 2011, 55 (3): 856- 867
doi: 10.1016/j.neuroimage.2011.01.008
7 LIN W, GAO Q, DU M, et al Multiclass diagnosis of stages of Alzheimer’s disease using linear discriminant analysis scoring for multimodal data[J]. Computers in Biology and Medicine, 2021, 134: 104478
doi: 10.1016/j.compbiomed.2021.104478
8 FENG W, HALM-LUTTERODT N V, TANG H, et al Automated MRI-based deep learning model for detection of Alzheimer’s disease process[J]. International Journal of Neural Systems, 2020, 30 (6): 2050032
doi: 10.1142/S012906572050032X
9 LIU M, CHENG D, YAN W Classification of Alzheimer’s disease by combination of convolutional and recurrent neural networks using FDG-PET images[J]. Frontiers in Neuroinformatics, 2018, 12: 35
doi: 10.3389/fninf.2018.00035
10 KONG Z, ZHANG M, ZHU W, et al Multi-modal data Alzheimer’s disease detection based on 3D convolution[J]. Biomedical Signal Processing and Control, 2022, 75: 103565
doi: 10.1016/j.bspc.2022.103565
11 韩坤, 潘海为, 张伟, 等 基于多模态医学图像的Alzheimer病分类方法[J]. 清华大学学报: 自然科学版, 2020, 60 (8): 664- 671
HAN Kun, PAN Haiwei, ZHANG Wei, et al Alzheimer’s disease classification method based on multimodal medical imaging[J]. Journal of Tsinghua University: Science and Technology, 2020, 60 (8): 664- 671
12 LIANG C, LAO H, WEI T, et al Alzheimer’s disease classification from hippocampal atrophy based on PCANet-BLS[J]. Multimedia Tools and Applications, 2022, 81: 11187- 11203
doi: 10.1007/s11042-022-12228-0
13 DEEPA N, CHOKKALINGAM S P Optimization of VGG16 utilizing the arithmetic optimization algorithm for early detection of Alzheimer’s disease[J]. Biomedical Signal Processing and Control, 2022, 74: 103455
doi: 10.1016/j.bspc.2021.103455
14 张默文. 基于结构磁共振成像和神经网络的阿尔兹海默症诊断方法研究[D]. 济南: 山东师范大学, 2023:1–60.
ZHANG Mowen. Diagnosis of Alzheimer’s disease based on structural magnetic resonance imaging and neural network [D]. Jinan: Shandong Normal University, 2023: 1–60.
15 MEHTA S C, RASTEGARI M. MobileViT: light-weight, general-purpose, and mobile-friendly vision transformer [EB/OL]. (2022−03−04)[2024−04−16], https://arxiv.org/pdf/2110.02178.
16 JACK C R JR, BERNSTEIN M A, FOX N C, et al The Alzheimer’s disease neuroimaging initiative (ADNI): MRI methods[J]. Journal of Magnetic Resonance Imaging, 2008, 27 (4): 685- 691
doi: 10.1002/jmri.21049
17 JENKINSON M, BECKMANN C F, BEHRENS T E J, et al FSL[J]. Neuroimage, 2012, 62 (2): 782- 790
doi: 10.1016/j.neuroimage.2011.09.015
18 XU Z, DENG H, LIU J, et al Diagnosis of Alzheimer’s disease based on the modified Tresnet[J]. Electronics, 2021, 10 (16): 1908
doi: 10.3390/electronics10161908
19 ODUSAMI M, MASKELIŪNAS R, DAMAŠEVIČIUS R Pixel-level fusion approach with vision transformer for early detection of Alzheimer’s disease[J]. Electronics, 2023, 12 (5): 1218
doi: 10.3390/electronics12051218
20 陈蕊森, 夏军, 陈建良, 等 基于VBM的AD患者整体认知水平与脑灰质容积关系的横断面研究[J]. CT理论与应用研究, 2018, 27 (6): 709- 717
CHEN Ruisen, XIA Jun, CHEN Jianliang, et al Cross-sectional study on the relationship between integral cognitive level and gray matter volume in patients with Alzheimer’s disease based on VBM[J]. CT Theory and Applications, 2018, 27 (6): 709- 717
21 SANDLER M, HOWARD A, ZHU M, et al. MobileNetV2: inverted residuals and linear bottlenecks [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . Salt Lake City: IEEE, 2018: 4510–4520.
22 RUSSAKOVSKY O, DENG J, SU H, et al ImageNet large scale visual recognition challenge[J]. International Journal of Computer Vision, 2015, 115 (3): 211- 252
doi: 10.1007/s11263-015-0816-y
23 LI J, WEN Y, HE L. SCConv: spatial and channel reconstruction convolution for feature redundancy [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . Vancouver: IEEE, 2023: 6153–6162.
24 VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need [C]// Proceedings of the 31st International Conference on Neural Information Processing Systems . Long Beach: [s.n.], 2017: 6000–6010.
25 HUANG Z, ZHANG Z, LAN C, et al. Adaptive frequency filters as efficient global token mixers [C]// Proceedings of the IEEE/CVF International Conference on Computer Vision . Paris: IEEE, 2023: 6049–6059.
26 DOSOVITSKIY A, BEYER L, KOLESNIKOV A, et al. An image is worth 16x16 words: transformers for image recognition at scale [EB/OL]. (2021−06−03)[2024−04−16]. https://arxiv.org/pdf/2010.11929.
27 CHOLLET F. Xception: deep learning with depth wise separable convolutions [C]// IEEE Conference on Computer Vision and Pattern Recognition . Honolulu: IEEE, 2017: 1800−1807.
28 LENG Z Q, TAN M, LIU C, et al. PloyLoss: a polynomial expansion perspective of classification loss functions [EB/OL]. (2022−05−10)[2024−04−16]. https://arxiv.org/pdf/2204.12511.
29 HERINGTON J, MCCRADDEN M D, CREEL K, et al Ethical considerations for artificial intelligence in medical imaging: data collection, development, and evaluation[J]. Journal of Nuclear Medicine, 2023, 64 (12): 1848- 1854
doi: 10.2967/jnumed.123.266080
30 ZHU X, SUK H, SHEN D A novel matrix-similarity based loss function for joint regression and classification in AD diagnosis[J]. Neuroimage, 2014, 100: 91- 105
doi: 10.1016/j.neuroimage.2014.05.078
31 HAO X, BAO Y, GUO Y, et al Multi-modal neuroimaging feature selection with consistent metric constraint for diagnosis of Alzheimer’s disease[J]. Medical Image Analysis, 2020, 60: 101625
doi: 10.1016/j.media.2019.101625
32 ZHU Q, YUAN N, HUANG J, et al Multi-modal AD classification via self-paced latent correlation analysis[J]. Neurocomputing, 2019, 355: 143- 154
doi: 10.1016/j.neucom.2019.04.066
33 TONG T, GRAY K, GAO Q, et al Multi-modal classification of Alzheimer’s disease using nonlinear graph fusion[J]. Pattern Recognition, 2017, 63: 171- 181
doi: 10.1016/j.patcog.2016.10.009
34 ZHANG F, LI Z, ZHANG B, et al Multi-modal deep learning model for auxiliary diagnosis of Alzheimer’s disease[J]. Neurocomputing, 2019, 361: 185- 195
doi: 10.1016/j.neucom.2019.04.093
35 李伟汉, 侯北平, 胡飞阳, 等 阿尔茨海默症的多模态分类方法[J]. 应用科学学报, 2023, 41 (6): 1004- 1018
LI Weihan, HOU Beiping, HU Feiyang, et al Multi-modal diagnosis method for Alzheimer’s disease[J]. Journal of Applied Sciences, 2023, 41 (6): 1004- 1018
[1] 陈珂,张文浩. 基于对比学习的零样本对象谣言检测[J]. 浙江大学学报(工学版), 2024, 58(9): 1790-1800.
[2] 刘宇庭,郭世杰,唐术锋,张学炜,李田田. 改进A*与ROA-DWA融合的机器人路径规划[J]. 浙江大学学报(工学版), 2024, 58(2): 360-369.
[3] 周姿含,王叙萌,陈为. 基于迁移学习的交互时序数据可视化生成方法[J]. 浙江大学学报(工学版), 2024, 58(2): 239-246.
[4] 赵蕴龙,赵敏喆,朱文强,查星宇. 基于轻量化迁移学习的云边协同自然语言处理方法[J]. 浙江大学学报(工学版), 2024, 58(12): 2531-2539.
[5] 郭浩然,郭继昌,汪昱东. 面向水下场景的轻量级图像语义分割网络[J]. 浙江大学学报(工学版), 2023, 57(7): 1278-1286.
[6] 华夏,王新晴,芮挺,邵发明,王东. 视觉感知的无人机端到端目标跟踪控制技术[J]. 浙江大学学报(工学版), 2022, 56(7): 1464-1472.
[7] 高一聪,王彦坤,费少梅,林琼. 基于迁移学习的机械制图智能评阅方法[J]. 浙江大学学报(工学版), 2022, 56(5): 856-863, 889.
[8] 付晓峰,牛力. 基于深度卷积和自编码器增强的微表情判别[J]. 浙江大学学报(工学版), 2022, 56(10): 1948-1957.
[9] 陈智超,焦海宁,杨杰,曾华福. 基于改进MobileNet v2的垃圾图像分类算法[J]. 浙江大学学报(工学版), 2021, 55(8): 1490-1499.
[10] 康庄,杨杰,郭濠奇. 基于机器视觉的垃圾自动分类系统设计[J]. 浙江大学学报(工学版), 2020, 54(7): 1272-1280.
[11] 沈宗礼,余建波. 基于迁移学习与深度森林的晶圆图缺陷识别[J]. 浙江大学学报(工学版), 2020, 54(6): 1228-1239.
[12] 付晓峰,牛力,胡卓群,李建军,吴卿. 基于过渡帧概念训练的微表情检测深度网络[J]. 浙江大学学报(工学版), 2020, 54(11): 2128-2137.