Please wait a minute...
浙江大学学报(理学版)  2021, Vol. 48 Issue (1): 18-29    DOI: 10.3785/j.issn.1008-9497.2021.01.003
图像理解与数据分析     
医学影像处理的深度学习可解释性研究进展
陈园琼1,2,3,4, 邹北骥1,3,4, 张美华2, 廖望旻1,3,4, 黄嘉儿1,3,4, 朱承璋3,4,5
1.中南大学 计算机学院,湖南 长沙 410083
2.吉首大学 软件学院,湖南 张家界 427000
3.移动医疗”教育部-中国移动联合实验室,湖南 长沙 410083
4.机器视觉与智慧医疗工程技术中心,湖南 长沙 410083
5.中南大学 文学与新闻传播学院,湖南 长沙 410083
A review on deep learning interpretability in medical image processing
CHEN Yuanqiong1,2,3,4, ZOU Beiji1,3,4, ZHANG Meihua2, LIAO Wangmin1,3,4, HUANG Jiaer1,3,4, ZHU Chengzhang3,4,5
1.School of Computer Science and Engineering, Central South University, Changsha 410083, China
2.Software College, Jishou University, Zhangjiajie 427000, Hunan Province, China
3.Mobile Health Ministry of Education-China Mobile Joint Laboratory, Changsha 410083, China
4.Hunan Engineering Research Center of Machine Vision and Intelligent Medicine, Central South University, Changsha 410083, China
5.The College of Literature and Journalism, Central South University, Changsha 410083, China
 全文: PDF(2102 KB)   HTML  
摘要: 随着医学影像数据的迅速增长,传统的影像分析方法给医生带来巨大挑战。利用计算机视觉技术提供自动或半自动辅助诊断,可大大缓解人工阅片压力,提高诊断的准确性,促进医疗流程的标准化建设等。目前,深度学习卷积神经网络在医学影像处理中已取得不俗表现,但深度学习“黑匣子”的不可解释性阻碍了智能医疗诊断的发展。为增强对医学影像数据处理的深度学习可解释性的了解,对近几年相关研究进展进行了综述。首先,综述了深度学习在医学领域的应用现状及面临的问题,对神经网络的可解释性内涵进行了讨论;然后,从现有深度学习可解释性的常见方法出发,重点讨论了医学影像处理的深度学习可解释性研究进展;最后,探讨了医学影像处理的深度学习可解释性的发展趋势。
关键词: 医学影像可解释性深度学习    
Abstract: Medical image data are rapidly accumulating and traditional image analysis methods based on manual approaches has imposed a heavy burden on doctors. Computer vision has played an important role in alleviating the pressure of manual reading,improving the accuracy of diagnosis and promoting the standardization of medical procedures by providing automatic or semi-automatic auxiliary diagnostic methods. At present,deep learning convolutional neural network has achieved outstanding performance in various medical image processing tasks,but the unexplainability of deep learning "black box" has become a major obstacle to further explore the full potentials of intelligent medical diagnosis.This paper summarizes the research progress of deep learning interpretability in medical image processing in recent years.Firstly,we clarify the application status and problems of deep learning in the medical field,and discusses the interpretable connotation of neural networks. Then,we focus on the research progress of deep learning interpretability in medical image data processing starting from the common methods of deep learning interpretability. Finally,the interpretable development trend of medical image processing is discussed.
Key words: medical image processing    interpretability    deep learning
收稿日期: 2020-09-23 出版日期: 2021-01-20
CLC:  TP 391.41  
基金资助: 国家自然科学基金资助项目(61702559);国家重大科技专项(2018AAA0102102);国家重点研发计划项目(2017YFC0909901)湖南省科技计划项目(2017WK2074);湖南省自然科学基金资助项目(2018JJ3686);高等学校学科创新引智计划项目(B18059);湖南省教育厅科学研究项目(19C1535).
通讯作者: ORCID:http://orcid.org/0000-0001-8825-0992,E-mail:anandawork@126.com.     E-mail: anandawork@126.com
作者简介: 陈园琼(1985—),ORCID:http://orcid.org/0000-0002-9889-8853,女,博士研究生,讲师,主要从事计算机视觉、深度学习、图像处理研;
服务  
把本文推荐给朋友
加入引用管理器
E-mail Alert
RSS
作者相关文章  
陈园琼
邹北骥
张美华
廖望旻
黄嘉儿
朱承璋

引用本文:

陈园琼, 邹北骥, 张美华, 廖望旻, 黄嘉儿, 朱承璋. 医学影像处理的深度学习可解释性研究进展[J]. 浙江大学学报(理学版), 2021, 48(1): 18-29.

CHEN Yuanqiong, ZOU Beiji, ZHANG Meihua, LIAO Wangmin, HUANG Jiaer, ZHU Chengzhang. A review on deep learning interpretability in medical image processing. Journal of Zhejiang University (Science Edition), 2021, 48(1): 18-29.

链接本文:

https://www.zjujournals.com/sci/CN/10.3785/j.issn.1008-9497.2021.01.003        https://www.zjujournals.com/sci/CN/Y2021/V48/I1/18

1 LECUN Y,BENGIO Y,HINTON G.Deep learning[J].Nature,2015,521(7553):436.DOI:10.3969/J.ISSN.1672-7274.2019.06.017
2 张世豪,冼丽英,高敏,等.基于深度学习的人工智能在病理诊断的应用进展与展望[J].中国医学创新,2018,15(25):130-133.DOI:10.3969/j.issn.1674-4985.2018.25.035 ZHANG S H,XIAN L Y,GAO M,et al.Application and development of artificial intelligence based on deep learning in pathological diagnosis[J].Medical Innovation of China,2018,15(25):130-133.DOI:10.3969/j.issn.1674-4985.2018.25.035
3 龙明盛.迁移学习问题与方法研究[D].北京:清华大学,2014. LONG M S.Transfer Learning:Problems and Methods[D].Beijing:Tsinghua University,2014.
4 张长水.机器学习面临的挑战[J].中国科学(信息科学),2013,43(12):1612-1623. ZHANG C S.Challenges in machine learning[J].SCIENTIN Sinica Informations,2013,43(12):1612-1623.
5 HOSSEINIASL E,GIMELFARB G,ELBAZ A.Alzheimer’s disease diagnostics by a deeply supervised adaptable 3D convolutional network[J].arXiv:Learning,2016. DOI:10.2741/4606
6 SUK H L,LEE S W,SHEN D G.Hierarchical feature representation and multimodal fusion with deep learning for AD/MCI diagnosis[J].NeuroImage,2014,101:569-582. DOI:10.1016/j.neuroimage. 2014.06.077
7 HAO P,ZHENG X,HUANG J Z.An effective approach for robust lung cancer cell detection[C]//Proceedings of International Workshop on Patch based Techniques in Medical Imaging.Munich:MICCAIS,2015.
8 SPANHOL F A ,OLIVEIRA L S,PETITJEAN C,et al. Breast cancer histopathological image classification using convolutional neural networks[C]//Proceedings of International Joint Conference on Neural Networks.Vancouver:IEEE,2016.DOI:10.1109/ijcnn.2016. 7727519
9 XU T,ZHANG H,HUANG X L,et al. Multimodal deep learning for cervical dysplasia diagnosis[C]//Proceedings of Medical Image Computing and Computer-Assisted Intervention. Athens:MICCAIS,2016,9901:115-123. DOI:10.1007/978-3-319-46723-8_14
10 HALOI M.Improved microaneurysm detection using deep neural networks[J].arXiv:Computer Vision and Pattern Recognition,2015.
11 WANG S L,YIN Y L,CAO G B,et al.Hierarchical retinal blood vessel segmentation based on feature and ensemble learning[J].Neuro Computing,2015,149:708-717. DOI:10.1016/j.neucom.2014.07.059
12 CHANDRAKUMAR T,KATHIRVEL R.Classifying diabetic retinopathy using deep learning architecture[J].International Journal of Engineering Research and Technology,2016,5(6):19-24.DOI:10.17577/IJERTV5IS060055
13 BIRAN O,COTTON C.Explanation and justification in machine learning:A survey[C]//Proceedings of IJCAI-17 Workshop on Explainable AI (XAI).Melbourn:IJCAI,2017,8:1-5.
14 MILLER T.Explanation in artificial intelligence:Insights from the social sciences[J].Artificial Intelligence,2019:267:1-38.
15 WANG X T,CHEN Y R,YANG J,et al.A reinforcement learning framework for explainable recommendation[C]//Proceedings of 2018 IEEE International Conference on Data Mining(ICDM). Singapore:IEEE,2018:587-596.DOI:10.1109/icdm. 2018.00074
16 ADOMAVICIUS G,TUZHILIN A.Toward the next generation of recommender systems:A survey of the state-of-the-art and possible extensions[J].IEEE Transactions on Knowledge & Data Engineering,2005(6):734-749. DOI:10.1109/tkde.2005.99
17 ZHANG Y F,CHEN X. Explainable recommendation:A survey and new perspectives[J].arXiv:Information Retrieval,2018.
18 TEACH R L,SHORTLIFFE E H.An analysis of physician attitudes regarding computer-based clinical consultation systems[J].Computers and Biomedical Research,1981,14(6):542-558. DOI:10.1016/0010-4809(81)90012-4
19 SMITH L B,SLONE L K. A developmental approach to machine learning?[J].Frontiers in Psychology,2017(8):2124. DOI:10.3389/fpsyg. 2017.02124
20 BINKOWSKI M,SUTHERL D J,ARBEL M,et al.Demystifying MMD GANs[C]//Proceedings of International Conference on Learning Representations.Vancouver:ICLR,2018.
21 ZEILER M D,FERGUS R.Visualizing and understanding convolutional networks[C]//Proceedings of European Conference on Computer Vision. Munich:ECCV,2014:818-833.
22 MAHENDRAN A,VEDALDI A.Understanding deep image representations by inverting them[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Boston:IEEE,2015:5188-5196. DOI:10.1109/CVPR.2015.7299155
23 LIU M C ,SHI J X,LI Z ,et al.Towards better analysis of deep convolutional neural networks[J].IEEE Transactions on Visualization and Computer Graphics,2017,23(1):91-100. DOI:10.1109/TVCG.2016.2598831
24 ERHAN D,BENGIO Y,COURVILLE A,et al.Visualizing higher-layer features of a deep network[J].University of Montreal,2009,1341(3):1-13.
25 OLAH C,MORDVINTSEV A,SCHUBERT L.Feature visualization[J]. Distill,2017,2(11):e7. DOI:10.23915/distill.00007
26 BAU D,ZHOU B L,KHOSLA A,et al.Network dissection:Quantifying interpretability of deep visual representations[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Hawaii:IEEE,2017:6541-6549.
27 FONG R,VEDALDI A.Net2Vec:Quantifying and explaining how concepts are encoded by filters in deep neural networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Salt Lake City:IEEE,2018:8730-8738. DOI:10.1109/cvpr.2018.00910
28 KIM B,WATTENBERG M,GILMER J ,et al.Interpretability beyond feature attribution:Quantitative testing with concept activation vectors (TCAV)[J]. arXiv:Machine Learning,2017.
29 OLAH C,SATYANARAYAN A,JOHNSON I,et al.The building blocks of interpretability[J].Distill,2018,3(3):e10. DOI:10.23915/distill.00010
30 SABOUR S,FROSST N,HINTON G.Dynamic routing between capsules[J].arXiv:Computer Vision and Pattern Recognition,2017.
31 CHEN X,DUAN Y,HOUTHOOFT R,et al.InfoGAN:Interpretable representation learning by information maximizing generative adversarial nets[J].arXiv:Learning,2016.
32 LECUN Y,BOTTOU L.Gradient-Based learning applied to document recognition[J].Proceeding of the IEEE,1998,86(11):2278-2324.
33 LIU Z W,LUO P,WANG X G,et al.Deep learning face attributes in the wild[C]//Proceedings of the IEEE International Conference on Computer Vision.Santiago:IEEE,2015:3730-3738. DOI:10.1109/ICCV.2015.425
34 NETZER Y,WANG T,COATES A,et al.Reading digits in natural images with unsupervised feature learning[C]//Conference and Workshop on Neural Information Processing Systems. Granda:NIPS,2011.
35 PAYSAN P,KNOTHE R,AMBERG B,et al. A 3D face model for pose and illumination invariant face recognition[C]//Proceedings of 2009 Sixth IEEE International Conference on Advanced Video and Signal Based Surveillance. Genova:IEEE,2009:296-301. DOI:10.1109/avss.2009.58
36 AUBRY M,MATURANA D,EFROS A A,et al.Seeing 3D chairs:Exemplar part-based 2D-3D alignment using a large dataset of CAD models[C]//Proceedings of the IEEE Conference onComputer Vision and Pattern Recognition.Columbus:IEEE, 2014:3762-3769. DOI:10.1109/CVPR.2014.487
37 ZHOU B L,KHOSLA A,LAPEDRIZA A,et al.Learning deep features for discriminative localization[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Las Vegas:IEEE,2016:2921-2929. DOI:10.1109/cvpr. 2016.319
38 SELVARAJU R R,COGSWELL M,DAS A,et al.Grad-CAM:Visual explanations from deep networks via gradient-based localization[C]//Proceedings of the IEEE International Conference on Computer Vision.Venice:IEEE,2017:618-626.
39 WAGNER J,KOHLER J M,GINDELE T,et al.Interpretable and finegrained visual explanations for convolutional neural networks[C] //Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.California:IEEE,2019:9097-9107. DOI:10.1109/cvpr.2019.00931
40 FONG R C,VEDALDI A.Interpretable explanations of black boxes by meaningfulperturbation[C]//Proceedings of the IEEE International Conference on Computer Vision.Venice:IEEE,2017:3429-3437.
41 FONG R,PATRICK M,VEDALDI A.Understanding deep networks via extremal perturbations and smooth masks[C]//Proceedings of the IEEE International Conference on Computer Vision.Seoul:IEEE,2019:2950-2958. DOI:10.1109/iccv.2019.00304
42 KOH P W, LIANG P.Understanding black-box predictions via influence functions[C]//Proceedings of the 34th International Conference on Machine Learning. Sydney:IMLS,2017,70:1885-1894.
43 RIBEIRO M T,SAMEERSINGH,GUESTRIN A ."Why should I trust you?":Explaining the predictions of any classifier[C]//Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining.San Francisco:ACM, 2016:1135-1144.
44 WU M,HUGHES M C,PARBHOO S,et al. Beyond sparsity:Tree regularization of deep models for interpretability[C]//Proceedings of Thirty Second AAAI Conference on Artificial Intelligence.Louisiana:AAAI,2018.
45 ZHANG Q S,WU Y N,ZHU S C.Interpretable convolutional neural networks[C] //Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Salt Lake City:IEEE,2018:8827-8836. DOI:10.1109/cvpr.2018. 00920
46 ZHANG Q S,YANG Y,MA H T,et al.Interpreting CNNs via decision trees[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. California:IEEE,2019:6261-6270. DOI:10.1109/cvpr.2019.00642
47 HOU B J, ZHOU Z H.Learning with interpretable structure from RNN[J].arXiv:Neural and Evolutionary Computing,2018.
48 WU T F,SUN W,LI X L,et al.Towards interpretable R-CNN by unfolding latent structures[J].arXiv:Computer Vision and Pattern Recognition,2017.
49 MADUMAL P,MILLER T,SONENBERG L,et al.Explainable reinforcement learning through a causal lens[C]//Proceedings of the AAAI Conference on Artificial Intelligence,New York:IEEE,2020,34(3):2493-2500. DOI:10.1609/aaai.v34i03.5631
50 WANG Y L,SU H,ZHANG B,et al.Interpret neural networks by identifying critical data routing paths[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Salt Lake City:IEEE,2018:8906-8914. DOI:10.1109/cvpr. 2018.00928
51 CHU L,HU X,HU J,et al.Exact and consistent interpretation for piecewise linear neural networks:A closed form solution[C]//Proceedings of Knowledge Discovery and Data Mining.Alaska:ACM,2018:1244-1253. DOI:10.1145/3219819.3220063
52 ZHANG Q S,CAO R M,SHI F,et al.Interpreting CNN knowledge via an explanatory graph[C]//Proceedings of Thirty-Second AAAI Conference on Artificial Intelligence. Louisiana:AAAI,2018.
53 ZHANG Q S,WANG X,CAO R M,et al.Explanatory graphs for CNNs[J].arXiv:Computer Vision and Pattern Recognition,2018.
54 SELVARAJU R R,CHATTOPADHYAY P,ELHOSEINY M,et al.Choose your neuron:Incorporating domain knowledge through neuron-importance[C]//Proceedings of European Conference on Computer Vision.Munich:ECCV,2018:540-556. DOI:10.1007/978-3-030-01261-8_32
55 BAU D,ZHU J Y,STROBELT H,et al .GAN dissection:Visualizing and understanding generative adversarial networks[C]//Proceedings of International Conference on Learning Representations. Louisiana:ICLR,2019.
56 PASCHALI M,FERJADNAEEM M,SIMSON W ,et al.Improving the interpretability of medical imaging neural networks[J].arXiv:Computer Vision and Pattern Recognition,2019.
57 LEE H K,YUNE S,MANSOURI M,et al.An explainable deep-learning algorithm for the detection of acute intracranial haemorrhage from small datasets[J].Nature Biomedical Engineering,2019,3(3):173-182. DOI:10.1038/s41551-018-0324-9
58 LIAO W M,ZOU B J,ZHAO R C ,et al.Clinical interpretable deep learning model for glaucoma diagnosis[J].IEEE Journal of Biomedical and Health Informatics,2020:24(5)1405-1412. DOI:10.1109/JBHI.2019.2949075
59 GARCIA-PERAZA-HERRERA L C,EVERSON M,LI W Q,et al.Interpretable fully convolutional classification of intrapapillary capillary loops for real-time detection of early squamous neoplasia[J].arXiv:Computer Vision and Pattern Recognition,2018.
60 CRUZROA A,OVALLE J E,MADABHUSHI A,et al.A deep learning architecture for image representation,visual interpretability and automated basal-cell carcinoma cancer detection[C]//Proceedings of Medical Image Computing and Computer Assisted Intervention.Nagoya:MICCAIS,2013:403-410.
61 BIFFI C,OKTAY O,TARRONI G,et al.Learning interpretable Anatomical features through deep generative models:Application to cardiac remodeling[C]//Proceedings of International Conference on Medical Image Computing and Computer- Assisted Intervention. Granada:MICCAIS,2018:464-471. DOI:10.1007/978-3-030-00934-2_52
62 ZHANG Z Z,CHEN P J,SAPKOTA M,et al.TandemNet:Distilling knowledge from medical images using diagnostic reports as optional semantic references[C]//Proceedings of International Conference on Medical Image Computing and Computer-Assisted Intervention. Quebe:MICCAIS,2017:320-328. DOI:10.1007/978-3-319-66179-7_37
63 WANG X S,PENG Y F,LU L,et al.TieNet:Text-image embedding network for common thorax disease classification and reporting in chest X-rays[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Salt Lake City:IEEE,2018:9049-9058. DOI:10.1109/CVPR. 2018.00943
64 SHEN S W,HAN S X,ABERLE D R,et al.An interpretable deep hierarchical semantic convolutional neural network for lung nodule malignancy classification[J].Expert Systems with Applications,2019,128:84-95. DOI:10.1016/j.eswa.2019. 01.048
65 KIM S T,LEE H K,KIM H G,et al.ICADx:Interpretable computer aided diagnosis of breast masses[J].arXiv:Computer Vision and Pattern Recognition,2018. DOI:10.1117/12.2293570
66 ZHANG Z Z,XIE Y P,XING F Y,et al.MDNet:A semantically and visually interpretable medical image diagnosis network[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Hawaii:IEEE,2017:6428-6436. DOI:10.1109/CVPR.2017.378
67 FAUW J D,LEDSAM J,ROMERA-PAREDES B,et al.Clinically applicable deep learning for diagnosis and referral in retinal disease[J].Nature Medicine,2018,24(9):1342-1350. DOI:10.1038/s41591-018-0107-6
68 NIU Y H,GU L,LU F ,et al.Pathological evidence exploration in deep retinal image diagnosis[C]//Proceedings of the AAAI Conference on Artificial Intelligence.Hawaii:AAAI,2019,33:1093-1101. DOI:10.1609/aaai.v33i01.33011093
69 LI X X,DVORNEK N C,ZHOU Y,et al.Efficient interpretation of deep learning models using graph structure and cooperative game theory:Application to ASD biomarker discovery[C]//Proceedings of International Conference Information Processing in Medical Imaging. Hong Kong:Springer,2019:718-730. DOI:10.1007/978-3-030-20351-1_56
70 ALAA A M,DER SCHAAR M V.Forecasting individualized disease trajectories using interpretable deep learning[J]. arXiv:Learning,2018.
71 GOHORBANI A,WEXLER J,ZOU J,et al.Towards automatic concept-Based explanations[C]//Proceedings of Neural Information Processing Systems. Vancouver:NIPS,2019:9273-9282.
[1] 徐圣嘉,苏程,朱孔阳,章孝灿. 基于深度学习的岩石薄片矿物自动识别方法[J]. 浙江大学学报(理学版), 2022, 49(6): 743-752.
[2] 刘华玲,张国祥,马俊. 图嵌入算法研究进展[J]. 浙江大学学报(理学版), 2022, 49(4): 443-456.
[3] 钱立辉, 王斌, 郑云飞, 章佳杰, 李马丁, 于冰. 基于图像深度预测的景深视频分类算法[J]. 浙江大学学报(理学版), 2021, 48(3): 282-288.
[4] 傅颖颖, 张丰, 杜震洪, 刘仁义. 融合图卷积神经网络和注意力机制的PM2.5小时浓度多步预测[J]. 浙江大学学报(理学版), 2021, 48(1): 74-83.
[5] 李君轶, 任涛, 陆路正. 游客情感计算的文本大数据挖掘方法比较研究[J]. 浙江大学学报(理学版), 2020, 47(4): 507-520.
[6] 陈善雄, 王小龙, 韩旭, 刘云, 王明贵. 一种基于深度学习的古彝文识别方法[J]. 浙江大学学报(理学版), 2019, 46(3): 261-269.
[7] 黄婕, 张丰, 杜震洪, 刘仁义, 曹晓裴. 基于RNN-CNN集成深度学习模型的PM2.5小时浓度预测[J]. 浙江大学学报(理学版), 2019, 46(3): 370-379.
[8] 胡伟俭, 陈为, 冯浩哲, 张天平, 朱正茂, 潘巧明. 应用于平扫CT图像肺结节检测的深度学习方法综述[J]. 浙江大学学报(理学版), 2017, 44(4): 379-384.