Please wait a minute...
浙江大学学报(工学版)
浙江大学学报(工学版)  2023, Vol. 57 Issue (10): 1923-1932    DOI: 10.3785/j.issn.1008-973X.2023.10.002
计算机技术、自动化技术     
可形变Transformer辅助的胸部X光影像疾病诊断模型
胡锦波1(),聂为之1,宋丹1,*(),高卓2,白云鹏3,赵丰3
1. 天津大学 电气自动化与信息工程学院，天津 300072
2. 长春职业技术学院 信息学院，吉林 长春 130033
3. 天津市胸科医院 心血管外科，天津 300222
Chest X-ray imaging disease diagnosis model assisted by deformable Transformer
Jin-bo HU1(),Wei-zhi NIE1,Dan SONG1,*(),Zhuo GAO2,Yun-peng BAI3,Feng ZHAO3
1. School of Electrical and Information Engineering, Tianjin University, Tianjin 300072, China
2. School of Information, Changchun Polytechnic, Changchun 130033, China
3. Department of Cardiovascular Surgery, Tianjin Chest Hospital, Tianjin 300222, China
 全文: PDF(1778 KB)   HTML
摘要：

针对胸部X光影像中的灰雾现象、病变区域重叠等问题，提出可形变Transformer辅助的胸部X光影像疾病诊断模型. 将扩展后的ResNet50作为特征提取网络，添加压缩型双注意力模块，增强病变区域与非病变区域之间的特征差异，降低冗余信息的干扰，提高图像数据的特征提取效果；通过可形变Transformer解码器内部的交叉注意力模块，引入类别表征作为先验知识，引导影像特征进一步融合，提高不同疾病在影像区域重叠情况下的特征区分度；将解码器的输出传入分类器中以获得最终的诊断结果. 压缩型双注意力模块和可形变Transformer均起到降低模型计算复杂度的作用，引入非对称损失函数可以更好地解决正负样本不均衡. 利用所提模型在公开数据集ChestX-Ray14和CheXpert上进行多组实验，在2个数据集上的受试者操作的特征曲线下面积值（AUC）分别达到0.839 8和0.906 1，表明该模型在胸部X光影像的疾病诊断方面具有正确性和有效性.
关键词： 胸部X光图像分类;  可形变Transformer;  压缩型双注意力;  非对称损失函数;  先验知识    
Abstract:

A disease diagnosis model for chest X-ray images assisted by deformable Transformer was proposed, aiming at the problems of gray fog phenomenon and overlapping lesion areas in chest X-ray images. The extended residual network ResNet50 was used as a feature extraction network. A compressed dual attention module was added to enhance the feature difference between the lesion area and the non-lesion area, further reduced the interference of redundant information and improved the feature extraction of image data. Through the cross-attention module inside the deformable Transformer decoder, category representations were introduced as the priori knowledge to guide further fusion of image features and improve the feature discrimination of different diseases in the case of overlapping image regions. Output of the decoder was passed into the classifier to obtain the final diagnosis. Both the compressed dual attention module and the deformable Transformer can reduce the computational complexity of the model. The asymmetric loss function was introduced to solve the imbalance of positive and negative samples. The proposed model was subjected to multiple sets of experiments on public datasets ChestX-Ray14 and CheXpert. The area under curve (AUC) on two datasets reached 0.839 8 and 0.906 1 respectively, indicating the correctness and validity of the model for disease diagnosis on chest X-ray images.
Key words: chest X-ray image classification    deformable transformer    compressed dual attention    asymmetric loss function    priori knowledge
收稿日期: 2022-09-01 出版日期: 2023-10-18
CLC:  TP 391  
基金资助: 国家自然科学基金资助项目（61902277，62272337）
通讯作者: 宋丹     E-mail: hjb@tju.edu.cn;dan.song@ tju.edu.cn
作者简介: 胡锦波（1999—），男，硕士生，从事计算机视觉研究. orcid.org /0000-0002-5467-0910. E-mail： hjb@tju.edu.cn
服务　　
把本文推荐给朋友
加入引用管理器
E-mail Alert
作者相关文章　　
胡锦波
聂为之
宋丹
高卓
白云鹏
赵丰

引用本文:

胡锦波,聂为之,宋丹,高卓,白云鹏,赵丰. 可形变Transformer辅助的胸部X光影像疾病诊断模型[J]. 浙江大学学报(工学版), 2023, 57(10): 1923-1932.

Jin-bo HU,Wei-zhi NIE,Dan SONG,Zhuo GAO,Yun-peng BAI,Feng ZHAO. Chest X-ray imaging disease diagnosis model assisted by deformable Transformer. Journal of ZheJiang University (Engineering Science), 2023, 57(10): 1923-1932.

链接本文:

https://www.zjujournals.com/eng/CN/10.3785/j.issn.1008-973X.2023.10.002    或    https://www.zjujournals.com/eng/CN/Y2023/V57/I10/1923

图 1  胸部X光图像分类模型框架
图 2  ResNet系列主要使用的2种残差结构
图 3  可形变注意力模块
图 4  压缩型位置注意力模块
图 5  压缩型通道注意力模块
图 6  8种胸部常见疾病病变区域图
疾病种类 $\overline{{\rm{AUC}}} $/%
Wang等[13] Yao等[15] CheXNet[19] Guendel等[17] Yan等[24] Ma等[25] DuaLAnet[26] Luo等[27] DAM Deformab-
CDAM-D
肺不张 0.700 3 0.733 0.779 5 0.767 0.792 4 0.777 0.783 0.789 1 0.803 6 0.820 1
心脏肿大 0.810 0 0.856 0.881 6 0.883 0.881 4 0.894 0.884 0.906 9 0.884 7 0.911 5
积液 0.758 5 0.806 0.826 8 0.828 0.841 5 0.829 0.832 0.841 8 0.879 8 0.890 2
渗透 0.661 4 0.673 0.689 4 0.709 0.709 5 0.696 0.708 0.718 4 0.704 1 0.714 4
肿块 0.693 3 0.718 0.830 7 0.821 0.847 0 0.838 0.837 0.837 6 0.828 4 0.864 9
肺结节 0.668 7 0.777 0.781 4 0.758 0.810 5 0.771 0.800 0.798 5 0.732 6 0.772 5
肺炎 0.658 0 0.689 0.735 4 0.731 0.737 9 0.722 0.735 0.741 9 0.745 4 0.762 1
气胸 0.799 3 0.805 0.851 3 0.846 0.875 9 0.862 0.866 0.906 3 0.884 6 0.903 3
肺实变 0.703 2 0.711 0.754 2 0.745 0.759 8 0.750 0.746 0.768 1 0.796 6 0.810 0
水肿 0.805 2 0.806 0.849 6 0.835 0.847 8 0.846 0.841 0.861 0 0.883 9 0.895 8
肺气肿 0.833 0 0.842 0.924 9 0.895 0.942 2 0.908 0.937 0.939 6 0.920 5 0.914 2
纤维变性 0.785 9 0.743 0.821 9 0.818 0.832 6 0.827 0.820 0.838 1 0.800 6 0.808 2
胸膜增厚 0.683 5 0.724 0.792 5 0.761 0.808 3 0.779 0.796 0.803 6 0.784 2 0.814 6
疝气 0.871 7 0.775 0.932 3 0.896 0.934 1 0.934 0.895 0.937 1 0.862 1 0.875 7
$\overline{{\rm{AUC}}}_{\rm{all}} $ 0.745 1 0.761 0.818 0 0.807 0.830 2 0.817 0.820 0.834 9 0.822 1 0.839 8
表 1  ChestX-ray14数据集上模型对各类疾病分类的性能对比
疾病种类 $\overline { {\text{AUC} } } $/%
U-Ignore U-Zeros U-Ones Guan等[28] Pham等[29] Irvin等[30] Deformab-CDAM-D
肺不张 0.818 0.811 0.858 0.847 0.825 0 0.858 0 0.863 5
心脏肿大 0.828 0.840 0.832 0.868 0.855 0 0.832 0 0.865 5
肺实变 0.938 0.932 0.899 0.923 0.937 0 0.899 0 0.907 9
水肿 0.934 0.929 0.941 0.924 0.930 0 0.941 0 0.942 9
胸膜增厚 0.928 0.931 0.934 0.926 0.923 0 0.934 0 0.951 1
$ \overline {{\text{AUC}}}_{\rm{all}} $ 0.889 2 0.888 6 0.892 8 0.898 0.894 0 0.893 0 0.906 1
表 2  CheXpert数据集上模型对各类疾病分类的性能对比
模型 $ \overline {{\text{AUC}}} $/%
ChestX-ray14 CheXpert
Deformab-CDAM 0.823 3 0.891 9
Deformab-CDAM-D 0.839 8 0.906 1
表 3  特征图分辨率对模型性能的影响
模型 $ \overline {{\text{AUC}}} $/%
ChestX-ray14 CheXpert
null
Query 		0.834 2
0.838 4 		0.899 3
0.902 6
Query+MLP 0.839 8 0.906 1
表 4  先验知识对模型性能的影响
模型 $ \overline {{\text{AUC}}} $/%
ChestX-ray14 CheXpert
CDAM-D 0.830 1 0.896 5
Deformable-DAM-D 0.829 7 0.896 4
Deformable-CDAM-D 0.839 8 0.906 1
Lable-random 0.823 6 0.893 2
表 5  不同模块对模型分类效果的影响
模型 T/s
ChestX-ray14 CheXpert
CDAM-D 0.042 0.033
Deformab-DAM-D 0.034 0.028
Deformable-CDAM-D 0.032 0.025
表 6  消融实验不同模型处理时长对比
损失函数 $ \overline {{\text{AUC}}} $/%
ChestX-ray14 CheXpert
交叉熵损失函数 0.825 1 0.890 3
焦点损失函数 0.828 6 0.894 2
非对称损失函数 0.839 8 0.906 1
表 7  不同损失函数对模型分类效果的影响
图 7  医生标注病变区域（左）与Grad-CAM热力图（右）对比图
1 石连红 放射科医生的透视眼—CT与核磁共振[J]. 特别健康, 2020, 8: 35
SHI Lian-hong The fluoroscopy eye of the radiologist -CI and MRI[J]. Special Health, 2020, 8: 35
2 ELDAHSHAN E S A, MOHSEN H M, REVETT K, et al Computer-aided diagnosis of human brain tumor through MRI: a survey and a new algorithm[J]. Expert Systems with Applications, 2014, 41 (11): 5526- 5545
doi: 10.1016/j.eswa.2014.01.021
3 CHEN J, YU H, FENG R, et al. Flow-Mixup: classifying multi-labeled medical images with corrupted labels [C]// BIBM. Piscataway: IEEE, 2020: 534-541.
4 于玉海, 林鸿飞, 孟佳娜, 等 跨模态多标签生物医学图像分类建模识别[J]. 中国图像图形学报, 2018, 23 (6): 917- 927
YU Yu-hai, LIN Hong-fei, MENG Jia-na, et al Cross- modal multi-label biomedical image classification modeling and recognition[J]. Journal of Image and Graphics, 2018, 23 (6): 917- 927
5 LUO Y, TAO D, GENG B, et al Manifold regularized multitask learning for semi-supervised multilabel image classification[J]. IEEE Transactions on Image Processing, 2013, 22 (2): 523- 536
doi: 10.1109/TIP.2012.2218825
6 WEI YUNCHAO, XIA WEI, LIN MIN, et al HCP: a flexible CNN framework for multi-label image classification[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, 38 (9): 1901- 1907
doi: 10.1109/TPAMI.2015.2491929
7 BENBARUCH E, RIDNIK T, ZAMIR N, et al. Asymmetric loss for multi-label classification [C]// ICCV. Piscataway: IEEE, 2021: 82- 91.
8 GUO G R, ARNALDO M, ELIE B A, et al Artificial intelligence in healthcare: review and prediction case studies[J]. Engineering, 2020, 6 (3): 291- 301
doi: 10.1016/j.eng.2019.08.015
9 MILLER D D, BROWN E W Artificial intelligence in medical practice: the question to the answer[J]. The American Journal of Medicine, 2018, 131 (2): 129- 133
doi: 10.1016/j.amjmed.2017.10.035
10 ESTEVA A, KUPREL B, NOVOA R A Dermatologist level classification of skin cancer with deep neural networks[J]. Oncologie, 2017, 19 (11/12): 407- 408
doi: 10.1007/s10269-017-2730-4
11 MCKINNEY S M, SIENIEK M, GODBOLE V, et al International evaluation of an AI system for breast cancer screening[J]. Nature, 2020, 577 (7788): 89- 94
doi: 10.1038/s41586-019-1799-6
12 ZHANG J W, HE J T, CHEN T F, et al Abnormal region detection in cervical smear images based on fully convolutional network[J]. IET Image Processing, 2019, 13 (4): 583- 590
doi: 10.1049/iet-ipr.2018.6032
13 WANG X S, PENG Y F, LU L, et al. ChestX-ray8: hospital-scale chest X-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases [C]// 30th IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2017: 3462-3471.
14 HE K M, ZHANG X Y, REN S Q. Deep residual learning for image recognition [C]// IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2016: 770-778.
15 LI Y, ERIC P, DMITRY D, et al. Learning to diagnose from scratch by exploiting dependencies among labels [C]// IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2017: 7925-7937.
16 HUANG G, LIU Z, LAURENS V D M, et al. Densely connected convolutional networks [C]// IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2017: 2261-2269.
17 GUENDEL S, GRBIC S, GEORGESCU B, et al. Learning to recognize abnormalities in chest X-rays with location-aware Dense Networks [C]// Iberoamerican Congress on Pattern Recognition. Berlin: Springer, 2018: 757-765.
18 CHEN X L, GUPTA ABHINAV. Webly supervised learning of convolutional networks [C]// IEEE International Conference on Computer Vision. Piscataway: IEEE, 2016: 1431- 1439.
19 RAJPURKAR P, IRVIN J, ZHU K, et al. CheXNet: radiologist-level pneumonia detection on chest X-Rays with deep learning [C]// IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2017: 2698-2705.
20 LIU S L, ZHANG L, YANG X, et al. Query2Label: a simple transformer way to multi-label classification [C]// IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2021: 391-407.
21 BELLO I, FEDUS W, DU X, et al Revisiting ResNets: improved training and scaling strategies[J]. Advances in Neural Information Processing Systems, 2021, 34: 22614- 22627
22 LIN T Y, GOYAL P, GIRSHICK R, et al Focal loss for dense object detection[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 42 (2): 318- 327
doi: 10.1109/TPAMI.2018.2858826
23 IRVIN J, RAJPURKAR P, KO M, et al. CheXpert: a large chest radiograph dataset with uncertainty labels and expert comparison [C]// 33rd AAAI Conference on Artificial Intelligence. Menlo Park: AAAI, 2019: 590-597.
24 YAN C, YAO J, LI R, et al. Weakly supervised deep learning for thoracic disease classification and localization on chest X-Rays [C]// Proc of ICBCB. New York: ACM, 2018: 103-110.
25 MA C, WANG H, HOI S C H. Multi-label thoracic disease image classification with cross-attention networks. international conference on medical image computing and computer-assisted intervention [C]// International Conference on Medical Image Computing and Computer Assisted Intervention. Berlin: Springer, 2019: 730-738.
26 TEIXEIRA V, BRAZ L, PEDRINI H, et al. DuaLAnet: dual lesion attention network for thoracic disease classification in chest X-rays [C]// International Conference on Systems, Signals and Image Processing. Piscataway: IEEE, 2020: 69-74.
27 LUO L, YU L, CHEN H, et al Deep mining external imperfect data for chest x-ray disease screening[J]. IEEE Trans Med Image, 2020, 39 (11): 3583- 3594
doi: 10.1109/TMI.2020.3000949
28 GUAN Q, HUANG Y, LUO Y, et al Discriminative feature learning for thorax disease classification in chest X-ray images[J]. IEEE Transactions on Image Processing, 2021, 99: 1- 2
29 PHAM H H, LE T T, TRAN D Q, et al Interpreting chest x-rays via cnns that exploit hierarchical disease dependencies and uncertainty labels[J]. Neurocomputing, 2021, 437: 186- 194
doi: 10.1016/j.neucom.2020.03.127
30 KAMAL U, ZUNAED M, NIZAM N B, et al Anatomy X-Net: a semi-supervised anatomy aware convolutional neural network for thoracic disease classification[J]. IEEE Journal of Biomedical and Health Informatics, 2021, 26 (11): 5518- 5528
[1] 赖小波,张学群,许茂盛. 胶质母细胞瘤多模态磁共振图像自动分割[J]. 浙江大学学报(工学版), 2019, 53(2): 355-363.