Please wait a minute...
浙江大学学报（理学版）
Journal of Zhejiang University (Science Edition)  2021, Vol. 48 Issue (1): 18-29    DOI: 10.3785/j.issn.1008-9497.2021.01.003
Image Understanding and Data Sorting     
A review on deep learning interpretability in medical image processing
CHEN Yuanqiong1,2,3,4, ZOU Beiji1,3,4, ZHANG Meihua2, LIAO Wangmin1,3,4, HUANG Jiaer1,3,4, ZHU Chengzhang3,4,5
1.School of Computer Science and Engineering, Central South University, Changsha 410083, China
2.Software College, Jishou University, Zhangjiajie 427000, Hunan Province, China
3.Mobile Health Ministry of Education-China Mobile Joint Laboratory, Changsha 410083, China
4.Hunan Engineering Research Center of Machine Vision and Intelligent Medicine, Central South University, Changsha 410083, China
5.The College of Literature and Journalism, Central South University, Changsha 410083, China
Download: HTML ( )    PDF(2102KB)
Export: BibTeX | EndNote (RIS)      

Abstract  Medical image data are rapidly accumulating and traditional image analysis methods based on manual approaches has imposed a heavy burden on doctors. Computer vision has played an important role in alleviating the pressure of manual reading,improving the accuracy of diagnosis and promoting the standardization of medical procedures by providing automatic or semi-automatic auxiliary diagnostic methods. At present,deep learning convolutional neural network has achieved outstanding performance in various medical image processing tasks,but the unexplainability of deep learning "black box" has become a major obstacle to further explore the full potentials of intelligent medical diagnosis.This paper summarizes the research progress of deep learning interpretability in medical image processing in recent years.Firstly,we clarify the application status and problems of deep learning in the medical field,and discusses the interpretable connotation of neural networks. Then,we focus on the research progress of deep learning interpretability in medical image data processing starting from the common methods of deep learning interpretability. Finally,the interpretable development trend of medical image processing is discussed.

Key words： medical image processing      interpretability      deep learning     
Received: 23 September 2020      Published: 20 January 2021
CLC:  TP 391.41  
Service
E-mail this article
Add to my bookshelf
Add to citation manager
E-mail Alert
RSS
Articles by authors
CHEN Yuanqiong
ZOU Beiji
ZHANG Meihua
LIAO Wangmin
HUANG Jiaer
ZHU Chengzhang
Cite this article:

CHEN Yuanqiong, ZOU Beiji, ZHANG Meihua, LIAO Wangmin, HUANG Jiaer, ZHU Chengzhang. A review on deep learning interpretability in medical image processing. Journal of Zhejiang University (Science Edition), 2021, 48(1): 18-29.

URL:

https://www.zjujournals.com/sci/EN/Y2021/V48/I1/18


医学影像处理的深度学习可解释性研究进展

随着医学影像数据的迅速增长，传统的影像分析方法给医生带来巨大挑战。利用计算机视觉技术提供自动或半自动辅助诊断，可大大缓解人工阅片压力，提高诊断的准确性，促进医疗流程的标准化建设等。目前，深度学习卷积神经网络在医学影像处理中已取得不俗表现，但深度学习“黑匣子”的不可解释性阻碍了智能医疗诊断的发展。为增强对医学影像数据处理的深度学习可解释性的了解，对近几年相关研究进展进行了综述。首先，综述了深度学习在医学领域的应用现状及面临的问题，对神经网络的可解释性内涵进行了讨论；然后，从现有深度学习可解释性的常见方法出发，重点讨论了医学影像处理的深度学习可解释性研究进展；最后，探讨了医学影像处理的深度学习可解释性的发展趋势。

关键词： 医学影像,  可解释性,  深度学习 
1 LECUN Y，BENGIO Y，HINTON G.Deep learning［J］.Nature，2015，521（7553）：436.DOI：10.3969/J.ISSN.1672-7274.2019.06.017
2 张世豪，冼丽英，高敏，等.基于深度学习的人工智能在病理诊断的应用进展与展望［J］.中国医学创新，2018，15（25）：130-133.DOI：10.3969/j.issn.1674-4985.2018.25.035 ZHANG S H，XIAN L Y，GAO M，et al.Application and development of artificial intelligence based on deep learning in pathological diagnosis［J］.Medical Innovation of China，2018，15（25）：130-133.DOI：10.3969/j.issn.1674-4985.2018.25.035
3 龙明盛.迁移学习问题与方法研究［D］.北京：清华大学，2014. LONG M S.Transfer Learning：Problems and Methods［D］.Beijing：Tsinghua University，2014.
4 张长水.机器学习面临的挑战［J］.中国科学（信息科学），2013，43（12）：1612-1623. ZHANG C S.Challenges in machine learning［J］.SCIENTIN Sinica Informations，2013，43（12）：1612-1623.
5 HOSSEINIASL E，GIMELFARB G，ELBAZ A.Alzheimer’s disease diagnostics by a deeply supervised adaptable 3D convolutional network［J］.arXiv：Learning，2016. DOI：10.2741/4606
6 SUK H L，LEE S W，SHEN D G.Hierarchical feature representation and multimodal fusion with deep learning for AD/MCI diagnosis［J］.NeuroImage，2014，101：569-582. DOI：10.1016/j.neuroimage. 2014.06.077
7 HAO P，ZHENG X，HUANG J Z.An effective approach for robust lung cancer cell detection［C］//Proceedings of International Workshop on Patch based Techniques in Medical Imaging.Munich：MICCAIS，2015.
8 SPANHOL F A ，OLIVEIRA L S，PETITJEAN C，et al. Breast cancer histopathological image classification using convolutional neural networks［C］//Proceedings of International Joint Conference on Neural Networks.Vancouver：IEEE，2016.DOI：10.1109/ijcnn.2016. 7727519
9 XU T，ZHANG H，HUANG X L，et al. Multimodal deep learning for cervical dysplasia diagnosis［C］//Proceedings of Medical Image Computing and Computer-Assisted Intervention. Athens：MICCAIS，2016，9901：115-123. DOI：10.1007/978-3-319-46723-8_14
10 HALOI M.Improved microaneurysm detection using deep neural networks［J］.arXiv：Computer Vision and Pattern Recognition，2015.
11 WANG S L，YIN Y L，CAO G B，et al.Hierarchical retinal blood vessel segmentation based on feature and ensemble learning［J］.Neuro Computing，2015，149：708-717. DOI：10.1016/j.neucom.2014.07.059
12 CHANDRAKUMAR T，KATHIRVEL R.Classifying diabetic retinopathy using deep learning architecture［J］.International Journal of Engineering Research and Technology，2016，5（6）：19-24.DOI：10.17577/IJERTV5IS060055
13 BIRAN O，COTTON C.Explanation and justification in machine learning：A survey［C］//Proceedings of IJCAI-17 Workshop on Explainable AI （XAI）.Melbourn：IJCAI，2017，8：1-5.
14 MILLER T.Explanation in artificial intelligence：Insights from the social sciences［J］.Artificial Intelligence，2019：267：1-38.
15 WANG X T，CHEN Y R，YANG J，et al.A reinforcement learning framework for explainable recommendation［C］//Proceedings of 2018 IEEE International Conference on Data Mining（ICDM）. Singapore：IEEE，2018：587-596.DOI：10.1109/icdm. 2018.00074
16 ADOMAVICIUS G，TUZHILIN A.Toward the next generation of recommender systems：A survey of the state-of-the-art and possible extensions［J］.IEEE Transactions on Knowledge & Data Engineering，2005（6）：734-749. DOI：10.1109/tkde.2005.99
17 ZHANG Y F，CHEN X. Explainable recommendation：A survey and new perspectives［J］.arXiv：Information Retrieval，2018.
18 TEACH R L，SHORTLIFFE E H.An analysis of physician attitudes regarding computer-based clinical consultation systems［J］.Computers and Biomedical Research，1981，14（6）：542-558. DOI：10.1016/0010-4809（81）90012-4
19 SMITH L B，SLONE L K. A developmental approach to machine learning？［J］.Frontiers in Psychology，2017（8）：2124. DOI：10.3389/fpsyg. 2017.02124
20 BINKOWSKI M，SUTHERL D J，ARBEL M，et al.Demystifying MMD GANs［C］//Proceedings of International Conference on Learning Representations.Vancouver：ICLR，2018.
21 ZEILER M D，FERGUS R.Visualizing and understanding convolutional networks［C］//Proceedings of European Conference on Computer Vision. Munich：ECCV，2014：818-833.
22 MAHENDRAN A，VEDALDI A.Understanding deep image representations by inverting them［C］//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Boston：IEEE，2015：5188-5196. DOI：10.1109/CVPR.2015.7299155
23 LIU M C ，SHI J X，LI Z ，et al.Towards better analysis of deep convolutional neural networks［J］.IEEE Transactions on Visualization and Computer Graphics，2017，23（1）：91-100. DOI：10.1109/TVCG.2016.2598831
24 ERHAN D，BENGIO Y，COURVILLE A，et al.Visualizing higher-layer features of a deep network［J］.University of Montreal，2009，1341（3）：1-13.
25 OLAH C，MORDVINTSEV A，SCHUBERT L.Feature visualization［J］. Distill，2017，2（11）：e7. DOI：10.23915/distill.00007
26 BAU D，ZHOU B L，KHOSLA A，et al.Network dissection：Quantifying interpretability of deep visual representations［C］//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Hawaii：IEEE，2017：6541-6549.
27 FONG R，VEDALDI A.Net2Vec：Quantifying and explaining how concepts are encoded by filters in deep neural networks［C］//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Salt Lake City：IEEE，2018：8730-8738. DOI：10.1109/cvpr.2018.00910
28 KIM B，WATTENBERG M，GILMER J ，et al.Interpretability beyond feature attribution：Quantitative testing with concept activation vectors （TCAV）［J］. arXiv：Machine Learning，2017.
29 OLAH C，SATYANARAYAN A，JOHNSON I，et al.The building blocks of interpretability［J］.Distill，2018，3（3）：e10. DOI：10.23915/distill.00010
30 SABOUR S，FROSST N，HINTON G.Dynamic routing between capsules［J］.arXiv：Computer Vision and Pattern Recognition，2017.
31 CHEN X，DUAN Y，HOUTHOOFT R，et al.InfoGAN：Interpretable representation learning by information maximizing generative adversarial nets［J］.arXiv：Learning，2016.
32 LECUN Y，BOTTOU L.Gradient-Based learning applied to document recognition［J］.Proceeding of the IEEE，1998，86（11）：2278-2324.
33 LIU Z W，LUO P，WANG X G，et al.Deep learning face attributes in the wild［C］//Proceedings of the IEEE International Conference on Computer Vision.Santiago：IEEE，2015：3730-3738. DOI：10.1109/ICCV.2015.425
34 NETZER Y，WANG T，COATES A，et al.Reading digits in natural images with unsupervised feature learning［C］//Conference and Workshop on Neural Information Processing Systems. Granda：NIPS，2011.
35 PAYSAN P，KNOTHE R，AMBERG B，et al. A 3D face model for pose and illumination invariant face recognition［C］//Proceedings of 2009 Sixth IEEE International Conference on Advanced Video and Signal Based Surveillance. Genova：IEEE，2009：296-301. DOI：10.1109/avss.2009.58
36 AUBRY M，MATURANA D，EFROS A A，et al.Seeing 3D chairs：Exemplar part-based 2D-3D alignment using a large dataset of CAD models［C］//Proceedings of the IEEE Conference onComputer Vision and Pattern Recognition.Columbus：IEEE， 2014：3762-3769. DOI：10.1109/CVPR.2014.487
37 ZHOU B L，KHOSLA A，LAPEDRIZA A，et al.Learning deep features for discriminative localization［C］//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Las Vegas：IEEE，2016：2921-2929. DOI：10.1109/cvpr. 2016.319
38 SELVARAJU R R，COGSWELL M，DAS A，et al.Grad-CAM：Visual explanations from deep networks via gradient-based localization［C］//Proceedings of the IEEE International Conference on Computer Vision.Venice：IEEE，2017：618-626.
39 WAGNER J，KOHLER J M，GINDELE T，et al.Interpretable and finegrained visual explanations for convolutional neural networks［C］ //Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.California：IEEE，2019：9097-9107. DOI：10.1109/cvpr.2019.00931
40 FONG R C，VEDALDI A.Interpretable explanations of black boxes by meaningfulperturbation［C］//Proceedings of the IEEE International Conference on Computer Vision.Venice：IEEE，2017：3429-3437.
41 FONG R，PATRICK M，VEDALDI A.Understanding deep networks via extremal perturbations and smooth masks［C］//Proceedings of the IEEE International Conference on Computer Vision.Seoul：IEEE，2019：2950-2958. DOI：10.1109/iccv.2019.00304
42 KOH P W， LIANG P.Understanding black-box predictions via influence functions［C］//Proceedings of the 34th International Conference on Machine Learning. Sydney：IMLS，2017，70：1885-1894.
43 RIBEIRO M T，SAMEERSINGH，GUESTRIN A ."Why should I trust you？"：Explaining the predictions of any classifier［C］//Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining.San Francisco：ACM， 2016：1135-1144.
44 WU M，HUGHES M C，PARBHOO S，et al. Beyond sparsity：Tree regularization of deep models for interpretability［C］//Proceedings of Thirty Second AAAI Conference on Artificial Intelligence.Louisiana：AAAI，2018.
45 ZHANG Q S，WU Y N，ZHU S C.Interpretable convolutional neural networks［C］ //Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Salt Lake City：IEEE，2018：8827-8836. DOI：10.1109/cvpr.2018. 00920
46 ZHANG Q S，YANG Y，MA H T，et al.Interpreting CNNs via decision trees［C］//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. California：IEEE，2019：6261-6270. DOI：10.1109/cvpr.2019.00642
47 HOU B J， ZHOU Z H.Learning with interpretable structure from RNN［J］.arXiv：Neural and Evolutionary Computing，2018.
48 WU T F，SUN W，LI X L，et al.Towards interpretable R-CNN by unfolding latent structures［J］.arXiv：Computer Vision and Pattern Recognition，2017.
49 MADUMAL P，MILLER T，SONENBERG L，et al.Explainable reinforcement learning through a causal lens［C］//Proceedings of the AAAI Conference on Artificial Intelligence，New York：IEEE，2020，34（3）：2493-2500. DOI：10.1609/aaai.v34i03.5631
50 WANG Y L，SU H，ZHANG B，et al.Interpret neural networks by identifying critical data routing paths［C］//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Salt Lake City：IEEE，2018：8906-8914. DOI：10.1109/cvpr. 2018.00928
51 CHU L，HU X，HU J，et al.Exact and consistent interpretation for piecewise linear neural networks：A closed form solution［C］//Proceedings of Knowledge Discovery and Data Mining.Alaska：ACM，2018：1244-1253. DOI：10.1145/3219819.3220063
52 ZHANG Q S，CAO R M，SHI F，et al.Interpreting CNN knowledge via an explanatory graph［C］//Proceedings of Thirty-Second AAAI Conference on Artificial Intelligence. Louisiana：AAAI，2018.
53 ZHANG Q S，WANG X，CAO R M，et al.Explanatory graphs for CNNs［J］.arXiv：Computer Vision and Pattern Recognition，2018.
54 SELVARAJU R R，CHATTOPADHYAY P，ELHOSEINY M，et al.Choose your neuron：Incorporating domain knowledge through neuron-importance［C］//Proceedings of European Conference on Computer Vision.Munich：ECCV，2018：540-556. DOI：10.1007/978-3-030-01261-8_32
55 BAU D，ZHU J Y，STROBELT H，et al .GAN dissection：Visualizing and understanding generative adversarial networks［C］//Proceedings of International Conference on Learning Representations. Louisiana：ICLR，2019.
56 PASCHALI M，FERJADNAEEM M，SIMSON W ，et al.Improving the interpretability of medical imaging neural networks［J］.arXiv：Computer Vision and Pattern Recognition，2019.
57 LEE H K，YUNE S，MANSOURI M，et al.An explainable deep-learning algorithm for the detection of acute intracranial haemorrhage from small datasets［J］.Nature Biomedical Engineering，2019，3（3）：173-182. DOI：10.1038/s41551-018-0324-9
58 LIAO W M，ZOU B J，ZHAO R C ，et al.Clinical interpretable deep learning model for glaucoma diagnosis［J］.IEEE Journal of Biomedical and Health Informatics，2020：24（5）1405-1412. DOI：10.1109/JBHI.2019.2949075
59 GARCIA-PERAZA-HERRERA L C，EVERSON M，LI W Q，et al.Interpretable fully convolutional classification of intrapapillary capillary loops for real-time detection of early squamous neoplasia［J］.arXiv：Computer Vision and Pattern Recognition，2018.
60 CRUZROA A，OVALLE J E，MADABHUSHI A，et al.A deep learning architecture for image representation，visual interpretability and automated basal-cell carcinoma cancer detection［C］//Proceedings of Medical Image Computing and Computer Assisted Intervention.Nagoya：MICCAIS，2013：403-410.
61 BIFFI C，OKTAY O，TARRONI G，et al.Learning interpretable Anatomical features through deep generative models：Application to cardiac remodeling［C］//Proceedings of International Conference on Medical Image Computing and Computer- Assisted Intervention. Granada：MICCAIS，2018：464-471. DOI：10.1007/978-3-030-00934-2_52
62 ZHANG Z Z，CHEN P J，SAPKOTA M，et al.TandemNet：Distilling knowledge from medical images using diagnostic reports as optional semantic references［C］//Proceedings of International Conference on Medical Image Computing and Computer-Assisted Intervention. Quebe：MICCAIS，2017：320-328. DOI：10.1007/978-3-319-66179-7_37
63 WANG X S，PENG Y F，LU L，et al.TieNet：Text-image embedding network for common thorax disease classification and reporting in chest X-rays［C］//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Salt Lake City：IEEE，2018：9049-9058. DOI：10.1109/CVPR. 2018.00943
64 SHEN S W，HAN S X，ABERLE D R，et al.An interpretable deep hierarchical semantic convolutional neural network for lung nodule malignancy classification［J］.Expert Systems with Applications，2019，128：84-95. DOI：10.1016/j.eswa.2019. 01.048
65 KIM S T，LEE H K，KIM H G，et al.ICADx：Interpretable computer aided diagnosis of breast masses［J］.arXiv：Computer Vision and Pattern Recognition，2018. DOI：10.1117/12.2293570
66 ZHANG Z Z，XIE Y P，XING F Y，et al.MDNet：A semantically and visually interpretable medical image diagnosis network［C］//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Hawaii：IEEE，2017：6428-6436. DOI：10.1109/CVPR.2017.378
67 FAUW J D，LEDSAM J，ROMERA-PAREDES B，et al.Clinically applicable deep learning for diagnosis and referral in retinal disease［J］.Nature Medicine，2018，24（9）：1342-1350. DOI：10.1038/s41591-018-0107-6
68 NIU Y H，GU L，LU F ，et al.Pathological evidence exploration in deep retinal image diagnosis［C］//Proceedings of the AAAI Conference on Artificial Intelligence.Hawaii：AAAI，2019，33：1093-1101. DOI：10.1609/aaai.v33i01.33011093
69 LI X X，DVORNEK N C，ZHOU Y，et al.Efficient interpretation of deep learning models using graph structure and cooperative game theory：Application to ASD biomarker discovery［C］//Proceedings of International Conference Information Processing in Medical Imaging. Hong Kong：Springer，2019：718-730. DOI：10.1007/978-3-030-20351-1_56
70 ALAA A M，DER SCHAAR M V.Forecasting individualized disease trajectories using interpretable deep learning［J］. arXiv：Learning，2018.
71 GOHORBANI A，WEXLER J，ZOU J，et al.Towards automatic concept-Based explanations［C］//Proceedings of Neural Information Processing Systems. Vancouver：NIPS，2019：9273-9282.
[1] Shengjia XU,Cheng SU,Kongyang ZHU,Xiaocan ZHANG. Automatic identification of mineral in petrographic thin sections based on images using a deep learning method[J]. Journal of Zhejiang University (Science Edition), 2022, 49(6): 743-752.
[2] Hualing LIU,Guoxiang ZHANG,Jun MA. Research progress of graph embedding algorithms[J]. Journal of Zhejiang University (Science Edition), 2022, 49(4): 443-456.
[3] QIAN Lihui, WANG Bin, ZHENG Yunfei, ZHANG Jiajie, LI Mading, YU Bing. Depth of field videos classification based on image depth prediction[J]. Journal of Zhejiang University (Science Edition), 2021, 48(3): 282-288.
[4] FU Yingying, ZHANG Feng, DU Zhenhong, LIU Renyi. Multi-step prediction of PM2.5 hourly concentration by fusing graph convolution neural network and attention mechanism[J]. Journal of Zhejiang University (Science Edition), 2021, 48(1): 74-83.
[5] LI Junyi, REN Tao, LU Luzheng. A comparative study of big text data mining methods on tourist emotion computing[J]. Journal of Zhejiang University (Science Edition), 2020, 47(4): 507-520.
[6] Shanxiong CHEN, Xiaolong WANG, Xu HAN, Yun LIU, Minggui WANG. A recognition method of Ancient Yi character based on deep learning[J]. Journal of Zhejiang University (Science Edition), 2019, 46(3): 261-269.
[7] Jie HUANG, Feng ZHANG, Zhenhong DU, Renyi LIU, Xiaopei CAO. Hourly concentration prediction of PM2.5 based on RNN-CNN ensemble deep learning model[J]. Journal of Zhejiang University (Science Edition), 2019, 46(3): 370-379.
[8] HU Weijian, CHEN Wei, FENG Haozhe, ZHANG Tianping, ZHU Zhengmao, PAN Qiaoming. A survey of depth learning methods for detecting lung nodules by CT images[J]. Journal of Zhejiang University (Science Edition), 2017, 44(4): 379-384.