Please wait a minute...
浙江大学学报(工学版)
Journal of ZheJiang University (Engineering Science)  2023, Vol. 57 Issue (10): 1923-1932    DOI: 10.3785/j.issn.1008-973X.2023.10.002
    
Chest X-ray imaging disease diagnosis model assisted by deformable Transformer
Jin-bo HU1(),Wei-zhi NIE1,Dan SONG1,*(),Zhuo GAO2,Yun-peng BAI3,Feng ZHAO3
1. School of Electrical and Information Engineering, Tianjin University, Tianjin 300072, China
2. School of Information, Changchun Polytechnic, Changchun 130033, China
3. Department of Cardiovascular Surgery, Tianjin Chest Hospital, Tianjin 300222, China
Download: HTML     PDF(1778KB) HTML
Export: BibTeX | EndNote (RIS)      

Abstract  

A disease diagnosis model for chest X-ray images assisted by deformable Transformer was proposed, aiming at the problems of gray fog phenomenon and overlapping lesion areas in chest X-ray images. The extended residual network ResNet50 was used as a feature extraction network. A compressed dual attention module was added to enhance the feature difference between the lesion area and the non-lesion area, further reduced the interference of redundant information and improved the feature extraction of image data. Through the cross-attention module inside the deformable Transformer decoder, category representations were introduced as the priori knowledge to guide further fusion of image features and improve the feature discrimination of different diseases in the case of overlapping image regions. Output of the decoder was passed into the classifier to obtain the final diagnosis. Both the compressed dual attention module and the deformable Transformer can reduce the computational complexity of the model. The asymmetric loss function was introduced to solve the imbalance of positive and negative samples. The proposed model was subjected to multiple sets of experiments on public datasets ChestX-Ray14 and CheXpert. The area under curve (AUC) on two datasets reached 0.839 8 and 0.906 1 respectively, indicating the correctness and validity of the model for disease diagnosis on chest X-ray images.

Key words： chest X-ray image classification      deformable transformer      compressed dual attention      asymmetric loss function      priori knowledge     
Received: 01 September 2022      Published: 18 October 2023
CLC:  TP 391  
Fund:  国家自然科学基金资助项目（61902277，62272337）
Corresponding Authors: Dan SONG     E-mail: hjb@tju.edu.cn;dan.song@ tju.edu.cn
Service
E-mail this article
Add to my bookshelf
Add to citation manager
E-mail Alert
RSS
Articles by authors
Jin-bo HU
Wei-zhi NIE
Dan SONG
Zhuo GAO
Yun-peng BAI
Feng ZHAO
Cite this article:

Jin-bo HU,Wei-zhi NIE,Dan SONG,Zhuo GAO,Yun-peng BAI,Feng ZHAO. Chest X-ray imaging disease diagnosis model assisted by deformable Transformer. Journal of ZheJiang University (Engineering Science), 2023, 57(10): 1923-1932.

URL:

https://www.zjujournals.com/eng/10.3785/j.issn.1008-973X.2023.10.002     OR     https://www.zjujournals.com/eng/Y2023/V57/I10/1923


可形变Transformer辅助的胸部X光影像疾病诊断模型

针对胸部X光影像中的灰雾现象、病变区域重叠等问题，提出可形变Transformer辅助的胸部X光影像疾病诊断模型. 将扩展后的ResNet50作为特征提取网络，添加压缩型双注意力模块，增强病变区域与非病变区域之间的特征差异，降低冗余信息的干扰，提高图像数据的特征提取效果；通过可形变Transformer解码器内部的交叉注意力模块，引入类别表征作为先验知识，引导影像特征进一步融合，提高不同疾病在影像区域重叠情况下的特征区分度；将解码器的输出传入分类器中以获得最终的诊断结果. 压缩型双注意力模块和可形变Transformer均起到降低模型计算复杂度的作用，引入非对称损失函数可以更好地解决正负样本不均衡. 利用所提模型在公开数据集ChestX-Ray14和CheXpert上进行多组实验，在2个数据集上的受试者操作的特征曲线下面积值（AUC）分别达到0.839 8和0.906 1，表明该模型在胸部X光影像的疾病诊断方面具有正确性和有效性.


关键词： 胸部X光图像分类,  可形变Transformer,  压缩型双注意力,  非对称损失函数,  先验知识 
Fig.1 Framework of chest X-ray image classification model
Fig.2 Two main residual structures used by ResNet family
Fig.3 Deformable attention module
Fig.4 Compact position attention module
Fig.5 Compact channel attention module
Fig.6 Lesion area maps of 8 common chest diseases
疾病种类 $\overline{{\rm{AUC}}} $/%
Wang等[13] Yao等[15] CheXNet[19] Guendel等[17] Yan等[24] Ma等[25] DuaLAnet[26] Luo等[27] DAM Deformab-
CDAM-D
肺不张 0.700 3 0.733 0.779 5 0.767 0.792 4 0.777 0.783 0.789 1 0.803 6 0.820 1
心脏肿大 0.810 0 0.856 0.881 6 0.883 0.881 4 0.894 0.884 0.906 9 0.884 7 0.911 5
积液 0.758 5 0.806 0.826 8 0.828 0.841 5 0.829 0.832 0.841 8 0.879 8 0.890 2
渗透 0.661 4 0.673 0.689 4 0.709 0.709 5 0.696 0.708 0.718 4 0.704 1 0.714 4
肿块 0.693 3 0.718 0.830 7 0.821 0.847 0 0.838 0.837 0.837 6 0.828 4 0.864 9
肺结节 0.668 7 0.777 0.781 4 0.758 0.810 5 0.771 0.800 0.798 5 0.732 6 0.772 5
肺炎 0.658 0 0.689 0.735 4 0.731 0.737 9 0.722 0.735 0.741 9 0.745 4 0.762 1
气胸 0.799 3 0.805 0.851 3 0.846 0.875 9 0.862 0.866 0.906 3 0.884 6 0.903 3
肺实变 0.703 2 0.711 0.754 2 0.745 0.759 8 0.750 0.746 0.768 1 0.796 6 0.810 0
水肿 0.805 2 0.806 0.849 6 0.835 0.847 8 0.846 0.841 0.861 0 0.883 9 0.895 8
肺气肿 0.833 0 0.842 0.924 9 0.895 0.942 2 0.908 0.937 0.939 6 0.920 5 0.914 2
纤维变性 0.785 9 0.743 0.821 9 0.818 0.832 6 0.827 0.820 0.838 1 0.800 6 0.808 2
胸膜增厚 0.683 5 0.724 0.792 5 0.761 0.808 3 0.779 0.796 0.803 6 0.784 2 0.814 6
疝气 0.871 7 0.775 0.932 3 0.896 0.934 1 0.934 0.895 0.937 1 0.862 1 0.875 7
$\overline{{\rm{AUC}}}_{\rm{all}} $ 0.745 1 0.761 0.818 0 0.807 0.830 2 0.817 0.820 0.834 9 0.822 1 0.839 8
Tab.1 Comparison of model performance on ChestX-ray14 dataset for classification of various diseases
疾病种类 $\overline { {\text{AUC} } } $/%
U-Ignore U-Zeros U-Ones Guan等[28] Pham等[29] Irvin等[30] Deformab-CDAM-D
肺不张 0.818 0.811 0.858 0.847 0.825 0 0.858 0 0.863 5
心脏肿大 0.828 0.840 0.832 0.868 0.855 0 0.832 0 0.865 5
肺实变 0.938 0.932 0.899 0.923 0.937 0 0.899 0 0.907 9
水肿 0.934 0.929 0.941 0.924 0.930 0 0.941 0 0.942 9
胸膜增厚 0.928 0.931 0.934 0.926 0.923 0 0.934 0 0.951 1
$ \overline {{\text{AUC}}}_{\rm{all}} $ 0.889 2 0.888 6 0.892 8 0.898 0.894 0 0.893 0 0.906 1
Tab.2 Comparison of model performance on CheXpert dataset for classification of various diseases
模型 $ \overline {{\text{AUC}}} $/%
ChestX-ray14 CheXpert
Deformab-CDAM 0.823 3 0.891 9
Deformab-CDAM-D 0.839 8 0.906 1
Tab.3 Effect of feature map resolution on model performance
模型 $ \overline {{\text{AUC}}} $/%
ChestX-ray14 CheXpert
null
Query 		0.834 2
0.838 4 		0.899 3
0.902 6
Query+MLP 0.839 8 0.906 1
Tab.4 Effect of prior knowledge on model performance
模型 $ \overline {{\text{AUC}}} $/%
ChestX-ray14 CheXpert
CDAM-D 0.830 1 0.896 5
Deformable-DAM-D 0.829 7 0.896 4
Deformable-CDAM-D 0.839 8 0.906 1
Lable-random 0.823 6 0.893 2
Tab.5 Effect of different modules on model classification performance
模型 T/s
ChestX-ray14 CheXpert
CDAM-D 0.042 0.033
Deformab-DAM-D 0.034 0.028
Deformable-CDAM-D 0.032 0.025
Tab.6 Comparison of different models processing time in ablation experiments
损失函数 $ \overline {{\text{AUC}}} $/%
ChestX-ray14 CheXpert
交叉熵损失函数 0.825 1 0.890 3
焦点损失函数 0.828 6 0.894 2
非对称损失函数 0.839 8 0.906 1
Tab.7 Effect of different loss functions on model classification performance
Fig.7 Comparison between doctor’s marked lesion area (left) and Grad-CAM heat map (right)
[1]   石连红 放射科医生的透视眼—CT与核磁共振[J]. 特别健康, 2020, 8: 35
SHI Lian-hong The fluoroscopy eye of the radiologist -CI and MRI[J]. Special Health, 2020, 8: 35
[2]   ELDAHSHAN E S A, MOHSEN H M, REVETT K, et al Computer-aided diagnosis of human brain tumor through MRI: a survey and a new algorithm[J]. Expert Systems with Applications, 2014, 41 (11): 5526- 5545
doi: 10.1016/j.eswa.2014.01.021
[3]   CHEN J, YU H, FENG R, et al. Flow-Mixup: classifying multi-labeled medical images with corrupted labels [C]// BIBM. Piscataway: IEEE, 2020: 534-541.
[4]   于玉海, 林鸿飞, 孟佳娜, 等 跨模态多标签生物医学图像分类建模识别[J]. 中国图像图形学报, 2018, 23 (6): 917- 927
YU Yu-hai, LIN Hong-fei, MENG Jia-na, et al Cross- modal multi-label biomedical image classification modeling and recognition[J]. Journal of Image and Graphics, 2018, 23 (6): 917- 927
[5]   LUO Y, TAO D, GENG B, et al Manifold regularized multitask learning for semi-supervised multilabel image classification[J]. IEEE Transactions on Image Processing, 2013, 22 (2): 523- 536
doi: 10.1109/TIP.2012.2218825
[6]   WEI YUNCHAO, XIA WEI, LIN MIN, et al HCP: a flexible CNN framework for multi-label image classification[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, 38 (9): 1901- 1907
doi: 10.1109/TPAMI.2015.2491929
[7]   BENBARUCH E, RIDNIK T, ZAMIR N, et al. Asymmetric loss for multi-label classification [C]// ICCV. Piscataway: IEEE, 2021: 82- 91.
[8]   GUO G R, ARNALDO M, ELIE B A, et al Artificial intelligence in healthcare: review and prediction case studies[J]. Engineering, 2020, 6 (3): 291- 301
doi: 10.1016/j.eng.2019.08.015
[9]   MILLER D D, BROWN E W Artificial intelligence in medical practice: the question to the answer[J]. The American Journal of Medicine, 2018, 131 (2): 129- 133
doi: 10.1016/j.amjmed.2017.10.035
[10]   ESTEVA A, KUPREL B, NOVOA R A Dermatologist level classification of skin cancer with deep neural networks[J]. Oncologie, 2017, 19 (11/12): 407- 408
doi: 10.1007/s10269-017-2730-4
[11]   MCKINNEY S M, SIENIEK M, GODBOLE V, et al International evaluation of an AI system for breast cancer screening[J]. Nature, 2020, 577 (7788): 89- 94
doi: 10.1038/s41586-019-1799-6
[12]   ZHANG J W, HE J T, CHEN T F, et al Abnormal region detection in cervical smear images based on fully convolutional network[J]. IET Image Processing, 2019, 13 (4): 583- 590
doi: 10.1049/iet-ipr.2018.6032
[13]   WANG X S, PENG Y F, LU L, et al. ChestX-ray8: hospital-scale chest X-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases [C]// 30th IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2017: 3462-3471.
[14]   HE K M, ZHANG X Y, REN S Q. Deep residual learning for image recognition [C]// IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2016: 770-778.
[15]   LI Y, ERIC P, DMITRY D, et al. Learning to diagnose from scratch by exploiting dependencies among labels [C]// IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2017: 7925-7937.
[16]   HUANG G, LIU Z, LAURENS V D M, et al. Densely connected convolutional networks [C]// IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2017: 2261-2269.
[17]   GUENDEL S, GRBIC S, GEORGESCU B, et al. Learning to recognize abnormalities in chest X-rays with location-aware Dense Networks [C]// Iberoamerican Congress on Pattern Recognition. Berlin: Springer, 2018: 757-765.
[18]   CHEN X L, GUPTA ABHINAV. Webly supervised learning of convolutional networks [C]// IEEE International Conference on Computer Vision. Piscataway: IEEE, 2016: 1431- 1439.
[19]   RAJPURKAR P, IRVIN J, ZHU K, et al. CheXNet: radiologist-level pneumonia detection on chest X-Rays with deep learning [C]// IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2017: 2698-2705.
[20]   LIU S L, ZHANG L, YANG X, et al. Query2Label: a simple transformer way to multi-label classification [C]// IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2021: 391-407.
[21]   BELLO I, FEDUS W, DU X, et al Revisiting ResNets: improved training and scaling strategies[J]. Advances in Neural Information Processing Systems, 2021, 34: 22614- 22627
[22]   LIN T Y, GOYAL P, GIRSHICK R, et al Focal loss for dense object detection[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 42 (2): 318- 327
doi: 10.1109/TPAMI.2018.2858826
[23]   IRVIN J, RAJPURKAR P, KO M, et al. CheXpert: a large chest radiograph dataset with uncertainty labels and expert comparison [C]// 33rd AAAI Conference on Artificial Intelligence. Menlo Park: AAAI, 2019: 590-597.
[24]   YAN C, YAO J, LI R, et al. Weakly supervised deep learning for thoracic disease classification and localization on chest X-Rays [C]// Proc of ICBCB. New York: ACM, 2018: 103-110.
[25]   MA C, WANG H, HOI S C H. Multi-label thoracic disease image classification with cross-attention networks. international conference on medical image computing and computer-assisted intervention [C]// International Conference on Medical Image Computing and Computer Assisted Intervention. Berlin: Springer, 2019: 730-738.
[26]   TEIXEIRA V, BRAZ L, PEDRINI H, et al. DuaLAnet: dual lesion attention network for thoracic disease classification in chest X-rays [C]// International Conference on Systems, Signals and Image Processing. Piscataway: IEEE, 2020: 69-74.
[27]   LUO L, YU L, CHEN H, et al Deep mining external imperfect data for chest x-ray disease screening[J]. IEEE Trans Med Image, 2020, 39 (11): 3583- 3594
doi: 10.1109/TMI.2020.3000949
[28]   GUAN Q, HUANG Y, LUO Y, et al Discriminative feature learning for thorax disease classification in chest X-ray images[J]. IEEE Transactions on Image Processing, 2021, 99: 1- 2
[29]   PHAM H H, LE T T, TRAN D Q, et al Interpreting chest x-rays via cnns that exploit hierarchical disease dependencies and uncertainty labels[J]. Neurocomputing, 2021, 437: 186- 194
doi: 10.1016/j.neucom.2020.03.127
[30]   KAMAL U, ZUNAED M, NIZAM N B, et al Anatomy X-Net: a semi-supervised anatomy aware convolutional neural network for thoracic disease classification[J]. IEEE Journal of Biomedical and Health Informatics, 2021, 26 (11): 5518- 5528
[1] Dong LIN,Yong-qiang LI,Xiang QIU,Yuan-jing FENG,Bi-feng XIE. Dynamic knowledge graph inference method combining static facts and repeated historical facts[J]. Journal of ZheJiang University (Engineering Science), 2023, 57(10): 1915-1922.
[2] Hai-feng LI,Xue-ying ZHANG,Shu-fei DUAN,Hai-rong JIA,Hui-zhi LIANG. Fusing generative adversarial network and temporal convolutional network for Mandarin emotion recognition[J]. Journal of ZheJiang University (Engineering Science), 2023, 57(9): 1865-1875.
[3] Lan WU,Quan-long CUI. Heterogeneous domain adaptation based on pseudo label refinement and semantic alignment[J]. Journal of ZheJiang University (Engineering Science), 2023, 57(9): 1876-1884.
[4] Zhao-yang SONG,Xiao-qiang ZHAO,Yong-yong HUI,Hong-mei JIANG. Image super-resolution reconstruction algorithm based on multi-level continuous encoding and decoding[J]. Journal of ZheJiang University (Engineering Science), 2023, 57(9): 1885-1893.
[5] Wei-long HE,Ping WANG,Ai-hua ZHANG,Ting-ting LIANG,Qiang-jie MA. On-line quantitative detection of welding arc shape based on object detection[J]. Journal of ZheJiang University (Engineering Science), 2023, 57(9): 1903-1914.
[6] Xin JIN,Jian-jun ZHUANG,Zi-heng XU. Lightweight YOLOv5s network-based algorithm for identifying hazardous objects under vehicles[J]. Journal of ZheJiang University (Engineering Science), 2023, 57(8): 1516-1526.
[7] Hui-xin WANG,Xiang-rong TONG. Research progress of recommendation system based on knowledge graph[J]. Journal of ZheJiang University (Engineering Science), 2023, 57(8): 1527-1540.
[8] Qiang ZHANG,Kun-peng DU,Hui LI,Zong-fa LIN,Qing WANG. Aircraft assembly data integration based on virtual central data source[J]. Journal of ZheJiang University (Engineering Science), 2023, 57(8): 1667-1679.
[9] Hao-ran GUO,Ji-chang GUO,Yu-dong WANG. Lightweight semantic segmentation network for underwater image[J]. Journal of ZheJiang University (Engineering Science), 2023, 57(7): 1278-1286.
[10] Hao-jie FANG,Hong-zhao DONG,Shao-xuan LIN,Jian-yu LUO,Yong FANG. Driver fatigue state detection method based on multi-feature fusion[J]. Journal of ZheJiang University (Engineering Science), 2023, 57(7): 1287-1296.
[11] Xiao-yan LI,Peng WANG,Jia GUO,Xue LI,Meng-yu SUN. Multi branch Siamese network target tracking based on double attention mechanism[J]. Journal of ZheJiang University (Engineering Science), 2023, 57(7): 1307-1316.
[12] Zhe YANG,Hong-wei GE,Ting LI. Framework of feature fusion and distribution with mixture of experts for parallel recommendation algorithm[J]. Journal of ZheJiang University (Engineering Science), 2023, 57(7): 1317-1325.
[13] Yun-hong LI,Jiao-jiao DUAN,Xue-ping SU,Lei-tao ZHANG,Hui-kang YU,Xing-rui LIU. Calligraphy generation algorithm based on improved generative adversarial network[J]. Journal of ZheJiang University (Engineering Science), 2023, 57(7): 1326-1334.
[14] Wei QUAN,Yong-qing CAI,Chao WANG,Jia SONG,Hong-kai SUN,Lin-xuan LI. VR sickness estimation model based on 3D-ResNet two-stream network[J]. Journal of ZheJiang University (Engineering Science), 2023, 57(7): 1345-1353.
[15] Wen-jie JIAO,Shuai-xu JI,Hui-min HAO,Jia-hai HUANG,Li-na LI,Shi-yu LI. New method and application of inverse kinematic solution for spherical wrist rehabilitation mechanism[J]. Journal of ZheJiang University (Engineering Science), 2023, 57(7): 1365-1373.