Please wait a minute...
浙江大学学报(工学版)  2021, Vol. 55 Issue (11): 2045-2053    DOI: 10.3785/j.issn.1008-973X.2021.11.004
生物医学工程     
基于异构低秩多模态融合网络的后囊膜混浊预测
陈志刚1(),万永菁1,*(),王于蓝2,蒋翠玲1,陈霞2
1. 华东理工大学 信息科学与工程学院,上海 200237
2. 上海市眼病防治中心,上海 200041
Prediction of posterior capsular opacification based on heterogeneous low-rank multimodal fusion network
Zhi-gang CHEN1(),Yong-jing WAN1,*(),Yu-lan WANG2,Cui-ling JIANG1,Xia CHEN2
1. School of Information Science and Engineering, East China University of Science and Technology, Shanghai 200237, China
2. Shanghai Eye Disease Prevention and Control Center, Shanghai 200041, China
 全文: PDF(1284 KB)   HTML
摘要:

针对后囊膜混浊并发症发病周期长、筛查范围广的问题,提出利用多模态机器学习预测后囊膜混浊并发症的计算机辅助诊断方法. 对后照影像进行感兴趣区域(ROI)提取和白色反光区域填充,所构建的异构低秩多模态融合网络(HLMF)能同时输入后照影像和视觉质量参数进行特征提取与融合,HLMF模型基于通道积融合多模态信息;采用卷积核参数低秩分解解决过拟合问题;选用Focal Loss损失函数解决类别不均衡的问题;在训练过程中还采用预训练和模态腐蚀的训练方法,使模型更好地提取单一模态的特征并进行融合. 该算法在后囊膜混浊数据集上的十折交叉验证准确率为95.63%,F1分数为96.72%. 实验结果表明,所提算法能较好地提取单模态特征并进行特征融合,相比于其他多模态融合模型有更好的性能.

关键词: 异构低秩分解多模态融合后照影像后囊膜混浊计算机辅助诊断    
Abstract:

A computer-aided diagnosis method for posterior capsular opacification using multimodal machine learning was proposed for the long incidence cycle and wide screening range of the complication of posterior capsular opacification. The region of interest (ROI) was extracted and the white reflective region was filled on the retro-illumination image. Retro-illumination image and visual quality data can be input into constructed heterogeneous low-rank multimodal fusion network (HLMF) simultaneously for performing feature extraction and fusion, and multimodal information was fused based on channel product. Overfitting problem was solved by low-rank decomposition of convolution kernel parameters, and the Focal Loss was chosen to solve the problem of uneven category. The pre-training and corrupted augmentation methods were used in the training process to better extract and fuse the features of single modality. The accuracy and F1 score of 10-fold cross-validation of the algorithm on the posterior capsule opacification dataset were 95.63% and 96.72%, respectively. Experimental results show that the proposed algorithm can extract single modality features and perform feature fusion well, and has better performance compared with other multimodal fusion models.

Key words: heterogeneity    low-rank decomposition    multimodal fusion    retro-illumination image    posterior capsular opacification    computer-aided diagnosis
收稿日期: 2020-12-30 出版日期: 2021-11-05
CLC:  TP 391  
基金资助: 国家自然科学基金资助项目(61872143);上海市申康医院发展中心临床科技创新资助项目(SHDC2018X16);上海市卫生健康委员会卫生行业临床研究专项课题资助项目(20204Y0218)
通讯作者: 万永菁     E-mail: zhigang_xs@163.com;wanyongjing@ecust.edu.cn
作者简介: 陈志刚(1996—),男,硕士生,从事医学图像处理、多模态机器学习研究. orcid.org/0000-0002-8791-8310. E-mail: zhigang_xs@163.com
服务  
把本文推荐给朋友
加入引用管理器
E-mail Alert
作者相关文章  
陈志刚
万永菁
王于蓝
蒋翠玲
陈霞

引用本文:

陈志刚,万永菁,王于蓝,蒋翠玲,陈霞. 基于异构低秩多模态融合网络的后囊膜混浊预测[J]. 浙江大学学报(工学版), 2021, 55(11): 2045-2053.

Zhi-gang CHEN,Yong-jing WAN,Yu-lan WANG,Cui-ling JIANG,Xia CHEN. Prediction of posterior capsular opacification based on heterogeneous low-rank multimodal fusion network. Journal of ZheJiang University (Engineering Science), 2021, 55(11): 2045-2053.

链接本文:

https://www.zjujournals.com/eng/CN/10.3785/j.issn.1008-973X.2021.11.004        https://www.zjujournals.com/eng/CN/Y2021/V55/I11/2045

图 1  后照影像预处理流程示例
图 2  数据增强示例
图 3  HLMF网络架构
图 4  通道积示意图
图 5  卷积核参数低秩分解示意图
图 6  不同秩下十折交叉验证准确率
图 7  不同秩下HLMF网络使用不同多模态训练方法结果对比
模型 Acc/% PR/% RE/% F1/%
TFN 91.35 95.60 91.17 93.23
LMF 92.51 95.77 93.01 94.23
PTP 93.46 96.78 93.42 94.95
HTFN 94.71 96.13 96.23 96.04
HLMF 95.63 96.88 96.73 96.72
表 1  不同多模态融合网络性能对比
模型 模型复杂度
TFN $ O\left( {W \times H \times M \times N \times K'} \right) $
LMF $ O\left( {r \times (W \times H \times M + N) \times K'} \right) $
PTP $ O\left( {r \times (W \times H \times M + N) \times K'} \right) $
HTFN $ O\left( {{f^2} \times M \times N \times K} \right) $
HLMF $ O\left( {r \times ({f^2} \times M + N) \times K} \right) $
表 2  不同多模态融合网络模型复杂度对比
模态组合 Acc/% SHAP值 IMP/%
无数据 50.00 ? ?
视觉质量参数 72.99 0.1227 26.89
后照影像 94.08 0.3336 73.11
多模态 95.63 ? ?
表 3  4种模态组合下HLMF模型准确率
1 ALLEN D, VASAVADA A Cataract and surgery for cataract[J]. BMJ, 2006, 333 (7559): 128- 132
doi: 10.1136/bmj.333.7559.128
2 PASCOLINI D, MARIOTTI S P Global estimates of visual impairment: 2010[J]. British Journal of Ophthalmology, 2012, 96 (5): 614- 618
doi: 10.1136/bjophthalmol-2011-300539
3 VIVEKANAND A, WERGHI N, AL-AHMAD H. Automated image assessment of posterior capsule opacification using Hölder exponents [C]// 2013 IEEE 20th International Conference on Electronics, Circuits, and Systems (ICECS). Abu Dhabi: IEEE, 2013: 538-541.
4 AWASTHI N, GUO S, WAGNER B J Posterior capsular opacification: a problem reduced but not yet eradicated[J]. Archives of Ophthalmology, 2009, 127 (4): 555- 562
doi: 10.1001/archophthalmol.2009.3
5 严宏, 陈曦, 陈颖 白内障术后并发症: 现状与对策[J]. 眼科新进展, 2019, 39 (1): 1- 7
YAN Hong, CHEN Xi, CHEN Ying Postoperative complications of cataract: current status and countermeasures[J]. Recent Advances in Ophthalmology, 2019, 39 (1): 1- 7
6 XU Y, HE J, LIN S, et al General analysis of factors influencing cataract surgery practice in Shanghai residents[J]. BMC Ophthalmology, 2018, 18 (1): 102
doi: 10.1186/s12886-018-0767-5
7 SZIGIATO A A, SCHLENKER M B, AHMED I I K Population-based analysis of intraocular lens exchange and repositioning[J]. Journal of Cataract and Refractive Surgery, 2017, 43 (6): 754- 760
doi: 10.1016/j.jcrs.2017.03.040
8 MOHAMMADI S F, SABBAGHI M, HADI Z, et al Using artificial intelligence to predict the risk for posterior capsule opacification after phacoemulsification[J]. Journal of Cataract and Refractive Surgery, 2012, 38 (3): 403- 408
doi: 10.1016/j.jcrs.2011.09.036
9 章佳. 基于模式识别的小儿白内障红反图像诊断研究及术后并发症预测 [D]. 西安: 西安电子科技大学, 2017.
ZHANG Jia. Research on automatic diagnosis of pediatric cataract retro-illumination image and postoperative complications prediction based on pattern recognition [D]. Xi'an: Xidian University, 2017.
10 WERGHI N, SAMMOUDA R, ALKIRBI F An unsupervised learning approach based on a Hopfield-like network for assessing posterior capsule opacification[J]. Pattern Analysis and Applications, 2010, 13 (4): 383- 396
doi: 10.1007/s10044-010-0181-y
11 VIVEKANAND A, WERGHI N, AL-AHMAD H Multiscale roughness approach for assessing posterior capsule opacification[J]. IEEE Journal of Biomedical and Health Informatics, 2014, 18 (6): 1923- 1931
doi: 10.1109/JBHI.2014.2304965
12 刘琳. 基于时序红反影像的后发性白内障预测问题研究 [D]. 西安: 西安电子科技大学, 2019.
LIU Lin. Research on the prediction of posterior capsular opacification based on time series retro-illumination images [D]. Xi'an: Xidian University, 2019.
13 KRONSCHLGER M, SIEGL H, PINZ A, et al Automated qualitative and quantitative assessment of posterior capsule opacification by Automated Quantification of After-Cataract II (AQUA II) system[J]. BMC Ophthalmology, 2019, 19 (1): 114
doi: 10.1186/s12886-019-1116-z
14 JIANG J, LIU X, ZHANG K, et al Automatic diagnosis of imbalanced ophthalmic images using a cost-sensitive deep convolutional neural network[J]. Biomedical Engineering Online, 2017, 16 (1): 132
doi: 10.1186/s12938-017-0420-1
15 BALTRUŠAITIS T, AHUJA C, MORENCY L P Multimodal machine learning: a survey and taxonomy[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 41 (2): 423- 443
16 KIROS R, POPURI K, COBZAS D, et al. Stacked multiscale feature learning for domain independent medical image segmentation [C]// International Workshop on Machine Learning in Medical Imaging. Boston: Springer, 2014: 25-32.
17 GU Y, VYAS K, SHEN M, et al Deep graph-based multimodal feature embedding for endomicroscopy image retrieval[J]. IEEE Transactions on Neural Networks and Learning Systems, 2020, 32 (2): 481- 492
18 SIMONOVSKY M, GUTIÉRREZ-BECKER B, MATEUS D, et al. A deep metric for multimodal registration [C]// International Conference on Medical Image Computing and Computer-assisted Intervention. Athens: Springer, 2016: 10-18.
19 LIU S, LIU S, CAI W, et al Multimodal neuroimaging feature learning for multiclass diagnosis of Alzheimer's disease[J]. IEEE Transactions on Biomedical Engineering, 2014, 62 (4): 1132- 1140
20 ZADEH A, CHEN M, PORIA S, et al. Tensor fusion network for multimodal sentiment analysis [C]// Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. Copenhagen: ACL, 2017: 1103-1114.
21 LIU Z, SHEN Y, LAKSHMINARASIMHAN V B, et al. Efficient low-rank multimodal fusion with modality-specific factors [C]// Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics. Melbourne: ACL, 2018: 2247-2256.
22 LIN T Y, GOYAL P, GIRSHICK R, et al. Focal loss for dense object detection [C]// Proceedings of the IEEE International Conference on Computer Vision. Venice: IEEE, 2017: 2980-2988.
23 NGIAM J, KHOSLA A, KIM M, et al. Multimodal deep learning [C]// International Conference on Machine Learning. Bellevue: ACM, 2011: 689-696.
24 HOU M, TANG J, ZHANG J, et al Deep multimodal multilinear fusion with high-order polynomial pooling[J]. Advances in Neural Information Processing Systems, 2019, 32: 12136- 12145
[1] 李伟达,李娟,李想,张虹淼,顾洪,史逸鹏,张浩杰,孙立宁. 欠驱动异构式下肢康复机器人动力学分析及参数优化[J]. 浙江大学学报(工学版), 2021, 55(2): 222-228.
[2] 赵廷廷,王喆,卢奕南. 基于传播概率矩阵的异构信息网络表示学习[J]. 浙江大学学报(工学版), 2019, 53(3): 548-554.
[3] 杨熠,钱旭升,周志勇,朱建兵,沈钧康,戴亚康. 采用影像组学的肾肿瘤组织学亚型分类[J]. 浙江大学学报(工学版), 2019, 53(12): 2381-2388.
[4] 王洪凯, 陈中华, 周纵苇, 李迎辞, 陆佩欧, 王文志, 刘宛予, 于丽娟. 机器学习算法诊断PET/CT纵膈淋巴结性能评估[J]. 浙江大学学报(工学版), 2018, 52(4): 788-797.
[5] 李建丽, 丁丁, 李涛. 基于二次聚类的多目标混合云任务调度算法[J]. 浙江大学学报(工学版), 2017, 51(6): 1233-1241.
[6] 陈霞, 陈超, 刘凯. 多源异构众包数据风景旅行路线规划[J]. 浙江大学学报(工学版), 2016, 50(6): 1183-1188.
[7] 李敏丹,沈晔,章东平,殷海兵. 基于混合类别标记新技术的小样本学习算法[J]. 浙江大学学报(工学版), 2016, 50(1): 137-143.
[8] 方锡武,刘振宇,谭建荣,程丰备. 异构有限元网格多场信息的等效集成方法[J]. 浙江大学学报(工学版), 2014, 48(6): 973-879.
[9] 尹红, 俞安, 盛财平, 袁慎峰, 陈志荣. 三丁酸甘油酯中维生素A热处理反应产物研究[J]. J4, 2013, 47(12): 2170-2175.
[10] 李丽娟, 熊路, 刘君, 徐欧官. 基于AP-LSSVM的多模型预测控制[J]. J4, 2013, 47(10): 1741-1746.
[11] 沈晔 ,李敏丹,夏顺仁. 计算机辅助乳腺癌诊断中的非平衡学习技术[J]. J4, 2013, 47(1): 1-7.
[12] 沈晔, 李敏丹, 夏顺仁. 计算机辅助乳腺癌诊断中的非平衡学习技术[J]. J4, 2013, 47(1): 1-7.
[13] 刘冰辉, 陈志荣, 尹红, 袁慎峰. 异丁烯齐聚反应研究[J]. J4, 2013, 47(1): 188-192.
[14] 曹颖, 郝欣, 朱晓恩, 夏顺仁. 基于自动随机游走的乳腺肿块分割算法[J]. J4, 2011, 45(10): 1753-1760.
[15] 吕艳, 倪益华, 顾新建, 等. 基于本体的面向业务协作的异构系统互操作关键技术[J]. J4, 2009, 43(8): 1485-1491.