Please wait a minute...
Journal of ZheJiang University (Engineering Science)  2021, Vol. 55 Issue (11): 2045-2053    DOI: 10.3785/j.issn.1008-973X.2021.11.004
    
Prediction of posterior capsular opacification based on heterogeneous low-rank multimodal fusion network
Zhi-gang CHEN1(),Yong-jing WAN1,*(),Yu-lan WANG2,Cui-ling JIANG1,Xia CHEN2
1. School of Information Science and Engineering, East China University of Science and Technology, Shanghai 200237, China
2. Shanghai Eye Disease Prevention and Control Center, Shanghai 200041, China
Download: HTML     PDF(1284KB) HTML
Export: BibTeX | EndNote (RIS)      

Abstract  

A computer-aided diagnosis method for posterior capsular opacification using multimodal machine learning was proposed for the long incidence cycle and wide screening range of the complication of posterior capsular opacification. The region of interest (ROI) was extracted and the white reflective region was filled on the retro-illumination image. Retro-illumination image and visual quality data can be input into constructed heterogeneous low-rank multimodal fusion network (HLMF) simultaneously for performing feature extraction and fusion, and multimodal information was fused based on channel product. Overfitting problem was solved by low-rank decomposition of convolution kernel parameters, and the Focal Loss was chosen to solve the problem of uneven category. The pre-training and corrupted augmentation methods were used in the training process to better extract and fuse the features of single modality. The accuracy and F1 score of 10-fold cross-validation of the algorithm on the posterior capsule opacification dataset were 95.63% and 96.72%, respectively. Experimental results show that the proposed algorithm can extract single modality features and perform feature fusion well, and has better performance compared with other multimodal fusion models.



Key wordsheterogeneity      low-rank decomposition      multimodal fusion      retro-illumination image      posterior capsular opacification      computer-aided diagnosis     
Received: 30 December 2020      Published: 05 November 2021
CLC:  TP 391  
Fund:  国家自然科学基金资助项目(61872143);上海市申康医院发展中心临床科技创新资助项目(SHDC2018X16);上海市卫生健康委员会卫生行业临床研究专项课题资助项目(20204Y0218)
Corresponding Authors: Yong-jing WAN     E-mail: zhigang_xs@163.com;wanyongjing@ecust.edu.cn
Cite this article:

Zhi-gang CHEN,Yong-jing WAN,Yu-lan WANG,Cui-ling JIANG,Xia CHEN. Prediction of posterior capsular opacification based on heterogeneous low-rank multimodal fusion network. Journal of ZheJiang University (Engineering Science), 2021, 55(11): 2045-2053.

URL:

https://www.zjujournals.com/eng/10.3785/j.issn.1008-973X.2021.11.004     OR     https://www.zjujournals.com/eng/Y2021/V55/I11/2045


基于异构低秩多模态融合网络的后囊膜混浊预测

针对后囊膜混浊并发症发病周期长、筛查范围广的问题,提出利用多模态机器学习预测后囊膜混浊并发症的计算机辅助诊断方法. 对后照影像进行感兴趣区域(ROI)提取和白色反光区域填充,所构建的异构低秩多模态融合网络(HLMF)能同时输入后照影像和视觉质量参数进行特征提取与融合,HLMF模型基于通道积融合多模态信息;采用卷积核参数低秩分解解决过拟合问题;选用Focal Loss损失函数解决类别不均衡的问题;在训练过程中还采用预训练和模态腐蚀的训练方法,使模型更好地提取单一模态的特征并进行融合. 该算法在后囊膜混浊数据集上的十折交叉验证准确率为95.63%,F1分数为96.72%. 实验结果表明,所提算法能较好地提取单模态特征并进行特征融合,相比于其他多模态融合模型有更好的性能.


关键词: 异构,  低秩分解,  多模态融合,  后照影像,  后囊膜混浊,  计算机辅助诊断 
Fig.1 Example of retro-illumination image preprocessing flow
Fig.2 Example of data augmentation
Fig.3 HLMF network architecture
Fig.4 Schematic diagram of channel product
Fig.5 Schematic diagram of convolution kernel parameter low-rank decomposition
Fig.6 Accuracy of 10-fold cross-validation under different ranks
Fig.7 Comparison of HLMF network using different multimodal training methods under different ranks
模型 Acc/% PR/% RE/% F1/%
TFN 91.35 95.60 91.17 93.23
LMF 92.51 95.77 93.01 94.23
PTP 93.46 96.78 93.42 94.95
HTFN 94.71 96.13 96.23 96.04
HLMF 95.63 96.88 96.73 96.72
Tab.1 Performance comparison of multimodal fusion networks
模型 模型复杂度
TFN $ O\left( {W \times H \times M \times N \times K'} \right) $
LMF $ O\left( {r \times (W \times H \times M + N) \times K'} \right) $
PTP $ O\left( {r \times (W \times H \times M + N) \times K'} \right) $
HTFN $ O\left( {{f^2} \times M \times N \times K} \right) $
HLMF $ O\left( {r \times ({f^2} \times M + N) \times K} \right) $
Tab.2 Comparison of model complexity of different multimodal fusion networks
模态组合 Acc/% SHAP值 IMP/%
无数据 50.00 ? ?
视觉质量参数 72.99 0.1227 26.89
后照影像 94.08 0.3336 73.11
多模态 95.63 ? ?
Tab.3 Accuracy of HLMF network under four modality combination
[1]   ALLEN D, VASAVADA A Cataract and surgery for cataract[J]. BMJ, 2006, 333 (7559): 128- 132
doi: 10.1136/bmj.333.7559.128
[2]   PASCOLINI D, MARIOTTI S P Global estimates of visual impairment: 2010[J]. British Journal of Ophthalmology, 2012, 96 (5): 614- 618
doi: 10.1136/bjophthalmol-2011-300539
[3]   VIVEKANAND A, WERGHI N, AL-AHMAD H. Automated image assessment of posterior capsule opacification using Hölder exponents [C]// 2013 IEEE 20th International Conference on Electronics, Circuits, and Systems (ICECS). Abu Dhabi: IEEE, 2013: 538-541.
[4]   AWASTHI N, GUO S, WAGNER B J Posterior capsular opacification: a problem reduced but not yet eradicated[J]. Archives of Ophthalmology, 2009, 127 (4): 555- 562
doi: 10.1001/archophthalmol.2009.3
[5]   严宏, 陈曦, 陈颖 白内障术后并发症: 现状与对策[J]. 眼科新进展, 2019, 39 (1): 1- 7
YAN Hong, CHEN Xi, CHEN Ying Postoperative complications of cataract: current status and countermeasures[J]. Recent Advances in Ophthalmology, 2019, 39 (1): 1- 7
[6]   XU Y, HE J, LIN S, et al General analysis of factors influencing cataract surgery practice in Shanghai residents[J]. BMC Ophthalmology, 2018, 18 (1): 102
doi: 10.1186/s12886-018-0767-5
[7]   SZIGIATO A A, SCHLENKER M B, AHMED I I K Population-based analysis of intraocular lens exchange and repositioning[J]. Journal of Cataract and Refractive Surgery, 2017, 43 (6): 754- 760
doi: 10.1016/j.jcrs.2017.03.040
[8]   MOHAMMADI S F, SABBAGHI M, HADI Z, et al Using artificial intelligence to predict the risk for posterior capsule opacification after phacoemulsification[J]. Journal of Cataract and Refractive Surgery, 2012, 38 (3): 403- 408
doi: 10.1016/j.jcrs.2011.09.036
[9]   章佳. 基于模式识别的小儿白内障红反图像诊断研究及术后并发症预测 [D]. 西安: 西安电子科技大学, 2017.
ZHANG Jia. Research on automatic diagnosis of pediatric cataract retro-illumination image and postoperative complications prediction based on pattern recognition [D]. Xi'an: Xidian University, 2017.
[10]   WERGHI N, SAMMOUDA R, ALKIRBI F An unsupervised learning approach based on a Hopfield-like network for assessing posterior capsule opacification[J]. Pattern Analysis and Applications, 2010, 13 (4): 383- 396
doi: 10.1007/s10044-010-0181-y
[11]   VIVEKANAND A, WERGHI N, AL-AHMAD H Multiscale roughness approach for assessing posterior capsule opacification[J]. IEEE Journal of Biomedical and Health Informatics, 2014, 18 (6): 1923- 1931
doi: 10.1109/JBHI.2014.2304965
[12]   刘琳. 基于时序红反影像的后发性白内障预测问题研究 [D]. 西安: 西安电子科技大学, 2019.
LIU Lin. Research on the prediction of posterior capsular opacification based on time series retro-illumination images [D]. Xi'an: Xidian University, 2019.
[13]   KRONSCHLGER M, SIEGL H, PINZ A, et al Automated qualitative and quantitative assessment of posterior capsule opacification by Automated Quantification of After-Cataract II (AQUA II) system[J]. BMC Ophthalmology, 2019, 19 (1): 114
doi: 10.1186/s12886-019-1116-z
[14]   JIANG J, LIU X, ZHANG K, et al Automatic diagnosis of imbalanced ophthalmic images using a cost-sensitive deep convolutional neural network[J]. Biomedical Engineering Online, 2017, 16 (1): 132
doi: 10.1186/s12938-017-0420-1
[15]   BALTRUŠAITIS T, AHUJA C, MORENCY L P Multimodal machine learning: a survey and taxonomy[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 41 (2): 423- 443
[16]   KIROS R, POPURI K, COBZAS D, et al. Stacked multiscale feature learning for domain independent medical image segmentation [C]// International Workshop on Machine Learning in Medical Imaging. Boston: Springer, 2014: 25-32.
[17]   GU Y, VYAS K, SHEN M, et al Deep graph-based multimodal feature embedding for endomicroscopy image retrieval[J]. IEEE Transactions on Neural Networks and Learning Systems, 2020, 32 (2): 481- 492
[18]   SIMONOVSKY M, GUTIÉRREZ-BECKER B, MATEUS D, et al. A deep metric for multimodal registration [C]// International Conference on Medical Image Computing and Computer-assisted Intervention. Athens: Springer, 2016: 10-18.
[19]   LIU S, LIU S, CAI W, et al Multimodal neuroimaging feature learning for multiclass diagnosis of Alzheimer's disease[J]. IEEE Transactions on Biomedical Engineering, 2014, 62 (4): 1132- 1140
[20]   ZADEH A, CHEN M, PORIA S, et al. Tensor fusion network for multimodal sentiment analysis [C]// Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. Copenhagen: ACL, 2017: 1103-1114.
[21]   LIU Z, SHEN Y, LAKSHMINARASIMHAN V B, et al. Efficient low-rank multimodal fusion with modality-specific factors [C]// Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics. Melbourne: ACL, 2018: 2247-2256.
[22]   LIN T Y, GOYAL P, GIRSHICK R, et al. Focal loss for dense object detection [C]// Proceedings of the IEEE International Conference on Computer Vision. Venice: IEEE, 2017: 2980-2988.
[23]   NGIAM J, KHOSLA A, KIM M, et al. Multimodal deep learning [C]// International Conference on Machine Learning. Bellevue: ACM, 2011: 689-696.
[24]   HOU M, TANG J, ZHANG J, et al Deep multimodal multilinear fusion with high-order polynomial pooling[J]. Advances in Neural Information Processing Systems, 2019, 32: 12136- 12145
[1] Yi YANG,Xu-sheng QIAN,Zhi-yong ZHOU,Jian-bing ZHU,Jun-kang SHEN,Ya-kang DAI. Classification of renal tumor histology subtypes using radiomics[J]. Journal of ZheJiang University (Engineering Science), 2019, 53(12): 2381-2388.
[2] WANG Hong-kai, CHEN Zhong-hua, ZHOU Zong-wei, LI Ying-ci, LU Pei-ou, WANG Wen-zhi, LIU Wan-yu, YU Li-juan. Evaluation of machine learning classifiers for diagnosing mediastinal lymph node metastasis of lung cancer from PET/CT images[J]. Journal of ZheJiang University (Engineering Science), 2018, 52(4): 788-797.
[3] LI Jian-li, DING Ding, LI Tao. Multi-objective hybrid cloud task scheduling using twice clustering[J]. Journal of ZheJiang University (Engineering Science), 2017, 51(6): 1233-1241.
[4] YUE Wen-ze, GAO Jia-bin, XIONG Chang-sheng. Cellular automata model for urban land expansion simulation based on heterogeneity adjacency relationship[J]. Journal of ZheJiang University (Engineering Science), 2017, 51(11): 2232-2238.
[5] SHEN Ye, LI Min-dan, XIA Shun-ren. Learning algorithm with non-balanced data for computer-aided
diagnosis of breast cancer
[J]. Journal of ZheJiang University (Engineering Science), 2013, 47(1): 1-7.
[6] SHEN Ye, LI Min-dan, XIA Shun-ren. Learning algorithm with non-balanced data for computer-aided
diagnosis of breast cancer
[J]. Journal of ZheJiang University (Engineering Science), 2013, 47(1): 1-7.
[7] CAO Ying, HAO Xin, ZHU Xiao-en, XIA Shun-ren. Mammographic mass segmentation algorithm based on
automatic random walks
[J]. Journal of ZheJiang University (Engineering Science), 2011, 45(10): 1753-1760.
[8] ZHANG Ta-Hua, GU Xin-Jian, LIU Hai-Jiang, et al. Part ontology mapping in cooperative enterprises[J]. Journal of ZheJiang University (Engineering Science), 2009, 43(12): 2287-2292.