Please wait a minute...
浙江大学学报(工学版)  2020, Vol. 54 Issue (3): 566-573    DOI: 10.3785/j.issn.1008-973X.2020.03.017
计算机技术与图像处理     
多模态多维信息融合的鼻咽癌MR图像肿瘤深度分割方法
洪炎佳1(),孟铁豹2,黎浩江2,刘立志2,李立2,徐硕瑀2,郭圣文1,*()
1. 华南理工大学 生物医学工程系,广东 广州 510006
2. 中山大学 肿瘤防治中心,广东 广州 510060
Deep segmentation method of tumor boundaries from MR images of patients with nasopharyngeal carcinoma using multi-modality and multi-dimension fusion
Yan-jia HONG1(),Tie-bao MENG2,Hao-jiang LI2,Li-zhi LIU2,Li LI2,Shuo-yu XU2,Sheng-wen GUO1,*()
1. Department of Biomedical Engineering, South China University of Technology, Guangzhou 510006, China
2. Medical Image Center, Sun Yat-sen University Cancer Center, Guangzhou 510060, China
 全文: PDF(1082 KB)   HTML
摘要:

收集421名鼻咽癌患者头颈部水平位T1加权(T1W)、T2加权(T2W)以及T1增强(T1C)三种模态MR图像,并由2名经验丰富的临床医生对图像中的肿瘤区域进行勾画,将其中346位患者的多模态图像及其标签作为训练集,将剩余75位患者的多模态图像及其标签作为独立测试集;分别构建单模态多维信息融合、两模态多维信息融合以及多模态多维信息融合(MMMDF)的卷积神经网络(CNN),并对模型进行训练和测试;使用Dice、豪斯多夫距离(HD)与面积差占比(PAD)评估3种模型的性能,结果表明,多模态多维融合模型的性能最优,两模态多维信息融合模型性能次之,单模态多维信息融合模型性能最差. 结果证明,多模态二维与三维特征融合的深度卷积网络能够准确有效地分割鼻咽癌MR图像中的肿瘤.

关键词: 鼻咽癌MR图像分割多模态多维度深度学习    
Abstract:

First, T1-weighted (T1W), T2-weighted (T2W) and T1 enhanced structural MR images of 421 patients were collected, the tumor boundaries of all images were delineated manually by two experienced doctors as the ground truth, the images and ground truth of 346 patients were considered as training set and the remaining images and corresponding ground truth of 75 patients were selected as independent testing set. Second, three single modality, based multi-dimension deep convolutional neural networks (CNN) and three two-modality multi-dimension fusion deep convolutional networks and a multi-modality multi-dimension fusion (MMMDF) deep convolutional neural network were constructed, and the networks were trained and tested, respectively. Finally, the performance of the three methods were evaluated by using three indexes, including Dice, Hausdorff distance (HD) and percentage area difference (PAD). The experimental results show that the MMMDF CNNs can acquire the best performances, followed by the two-modality multi-dimental fusion CNNs, while the single modlity multi-dimension CNNs achieves the worst measures.. This study demonstrates that the MMMDF-CNN combining multi-modality images and incorporating 2D with 3D images features can effectively fulfill accurate segmentation on tumors of MR images from NPC patients.

Key words: nasopharyngeal carcinoma    MR images    segmentation    multi-modality multi-dimension    deep learning
收稿日期: 2019-03-02 出版日期: 2020-03-05
CLC:  R 318.04  
基金资助: 广东省科技计划资助项目(2015A02024006);广州市产学研协同创新重大专项资助(201604020170);广州市科技计划资助项目(201907010043)
通讯作者: 郭圣文     E-mail: 531679559@qq.com;shwguo@scut.edu.cn
作者简介: 洪炎佳(1993—),男,硕士生,从事鼻咽癌智能分割及其预后研究. orcid.org/0000-0002-3953-7935. E-mail: 531679559@qq.com
服务  
把本文推荐给朋友
加入引用管理器
E-mail Alert
作者相关文章  
洪炎佳
孟铁豹
黎浩江
刘立志
李立
徐硕瑀
郭圣文

引用本文:

洪炎佳,孟铁豹,黎浩江,刘立志,李立,徐硕瑀,郭圣文. 多模态多维信息融合的鼻咽癌MR图像肿瘤深度分割方法[J]. 浙江大学学报(工学版), 2020, 54(3): 566-573.

Yan-jia HONG,Tie-bao MENG,Hao-jiang LI,Li-zhi LIU,Li LI,Shuo-yu XU,Sheng-wen GUO. Deep segmentation method of tumor boundaries from MR images of patients with nasopharyngeal carcinoma using multi-modality and multi-dimension fusion. Journal of ZheJiang University (Engineering Science), 2020, 54(3): 566-573.

链接本文:

http://www.zjujournals.com/eng/CN/10.3785/j.issn.1008-973X.2020.03.017        http://www.zjujournals.com/eng/CN/Y2020/V54/I3/566

图 1  多模态、多维度融合的卷积神经网络结构
网络层 2D-ResUNet 3D-ResUNet
特征图大小 网络层大小 特征图大小 网络层大小
输入 384×384 384×384×8
残差结构1 384×384 [3×3,16]×5 384×384×8 [3×3×3,16]×5
最大池化层1 192×192 2×2最大池化 192×192×4 2×2×2最大池化
残差结构2 192×192 [3×3,32]×5 192×192×4 [3×3×3,32]×5
最大池化层2 96×96 2×2最大池化 96×96×4 2×2×1最大池化
残差结构3 96×96 [3×3,64]×5 96×96×4 [3×3×3,64]×5
最大池化层3 48×48 2×2最大池化 48×48×2 2×2×2最大池化
残差结构4 48×48 [3×3,128]×5 48×48×2 [3×3×1,128]×5
最大池化层4 24×24 2×2最大池化 24×24×2 2×2×1最大池化
残差结构5 24×24 [3×3,256]×5 24×24×2 [3×3×1,256]×5
反卷积1 48×48 3×3,2×2-[残差结构4] 48×48×2 3×3×1,2×2×1-[残差结构4]
反卷积2 96×96 3×3,2×2-[残差结构3] 96×96×4 3×3×3,2×2×2-[残差结构3]
反卷积3 192×192 3×3,2×2-[残差结构2] 192×192×4 3×3×1,2×2×1-[残差结构2]
反卷积4 384×384 3×3,2×2-[残差结构1] 384×384×8 3×3×3,2×2×2-[残差结构1]
卷积层 384×384 1×1,2 384×384×8 1×1×1,2
表 1  2D-ResUNet与3D-ResUNet网络结构
图 2  多模态2D-ResUNet结构
数据集 被试数量 人数(男/女) 年龄(均值±标准差)
训练集 346 254/92 45.5±11.9
测试集 75 55/20 44.9±11.6
表 2  鼻咽癌(NPC)分割模型的训练集和测试集信息
鼻咽癌分割模型 Dice HD/mm PAD/%
T1W-MDF 0.759 6.51 20.0
T2W-MDF 0.763 6.37 17.9
T1C-MDF 0.747 6.41 19.8
T1W+T2W-MDF 0.781 5.84 16.5
T1W+T1C-MDF 0.773 6.02 17.1
T2W+T1C-MDF 0.775 5.93 16.8
Men等[8] 0.726 6.82 23.8
Li等[10] 0.718 6.91 25.1
Zhao等[15] 0.731 6.75 22.7
MMMDF 0.805 5.56 15.5
表 3  不同鼻咽癌分割模型的性能比较
图 3  7种鼻咽癌分割模型性能箱形图比较
图 4  7种不同鼻咽癌分割模型的分割结果(部分二维断面)比较
图 5  7种不同鼻咽癌分割模型的分割结果(部分三维断面)比较
1 CHANG E T, ADAMI H O The enigmatic epidemiology of nasopharyngeal carcinoma[J]. Cancer Epidemiology and Prevention Biomarkers, 2006, 15 (10): 1765- 1777
doi: 10.1158/1055-9965.EPI-06-0353
2 STEWART B W, WILD C. World cancer report 2014 [M]. Lyon: International Agency for Research on Cancer , 2014.
3 邓伟, 黄天壬, 陈万青, 等 中国2003—2007年鼻咽癌发病与死亡分析[J]. 肿瘤, 2012, 32 (3): 189- 193
DENG Wei, HUANG Tian-Ren, CHEN Wan-Qing, et al Analysis of the incidence and mortality of nasopharyngeal carcinoma in China from 2003 to 2007[J]. Tumor, 2012, 32 (3): 189- 193
doi: 10.3781/j.issn.1000-7431.2012.03.007
4 HUANG K W, ZHAO Z Y, GONG Q, et al. Nasopharyngeal carcinoma segmentation via HMRF-EM with maximum entropy [C] // 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). Milan: IEEE, 2015: 2968-2972.
5 RITTHIPRAVAT P, TATANUM C, BHONGMAKAPAT T, et al. Automatic segmentation of nasopharyngeal carcinoma from CT images [C] // 2008 International Conference on BioMedical Engineering and Informatics. Sanya: IEEE Computer Society, 2008, 2: 18-22.
6 ZHOU J, CHAN K L, XU P, et al. Nasopharyngeal carcinoma lesion segmentation from MR images by support vector machine [C] // 3rd IEEE International Symposium on Biomedical Imaging: Nano to Macro, 2006. Arlington: IEEE, 2006: 1364-1367.
7 MOHAMMED M A, GHANI M K A, HAMED R I, et al Artificial neural networks for automatic segmentation and identification of nasopharyngeal carcinoma[J]. Journal of Computer Science, 2017, 21: 263- 274
doi: 10.1016/j.jocs.2017.03.026
8 MEN K, CHEN X, ZHANG Y, et al Deep deconvolutional neural network for Target segmentation of nasopharyngeal cancer in Planning computed Tomography images[J]. Frontiers in Oncology, 2017, 7: 315
doi: 10.3389/fonc.2017.00315
9 SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition[J]. arXiv Preprint arXiv: 1409.1556, 2014.
10 LI Q L, XU Y, CHEN Z, et al Tumor segmentation in contrast-enhanced magnetic resonance imaging for nasopharyngeal carcinoma: deep learning with convolutional neural network[J]. BioMed Research International, 2018, 2018, 5: 1- 7
11 MA Z Q, WU X, SONG Q, et al Automated nasopharyngeal carcinoma segmentation in magnetic resonance images by combination of convolutional neural networks and graph cut[J]. Experimental and Therapeutic Medicine, 2018, 16 (3): 2511- 2521
12 LI X M, CHEN H, QI X J, et al H-DenseUNet: hybrid densely connected unet for liver and tumor segmentation from ct volumes[J]. IEEE Transactions on Medical Imaging, 2018, 37 (12): 2663- 2674
doi: 10.1109/TMI.2018.2845918
13 MILLETARI F, NAVAB N, AHMADI S A. V-net: Fully convolutional neural networks for volumetric medical image segmentation[C] // 2016 Fourth International Conference on 3D Vision (3DV). California: IEEE, 2016: 565-571.
14 ABADI M, BARHAM P, CHEN J, et al. Tensorflow: A system for large-scale machine learning[C] // 12th Symposium on Operating Systems Design and Implementation. Savannah, GA: OSDI, 2016: 265-283.
[1] 许佳辉,王敬昌,陈岭,吴勇. 基于图神经网络的地表水水质预测模型[J]. 浙江大学学报(工学版), 2021, 55(4): 601-607.
[2] 王虹力,郭斌,刘思聪,刘佳琪,仵允港,於志文. 边端融合的终端情境自适应深度感知模型[J]. 浙江大学学报(工学版), 2021, 55(4): 626-638.
[3] 张腾,蒋鑫龙,陈益强,陈前,米涛免,陈彪. 基于腕部姿态的帕金森病用药后开-关期检测[J]. 浙江大学学报(工学版), 2021, 55(4): 639-647.
[4] 曾凌霄,刘宏伟,李伟,林勇刚,顾亚京. 基于变桨系统的功率与载荷双目标协同控制[J]. 浙江大学学报(工学版), 2021, 55(4): 750-756.
[5] 徐利锋,黄海帆,丁维龙,范玉雷. 基于改进DenseNet的水果小目标检测[J]. 浙江大学学报(工学版), 2021, 55(2): 377-385.
[6] 许豪灿,李基拓,陆国栋. 由LeNet-5从单张着装图像重建三维人体[J]. 浙江大学学报(工学版), 2021, 55(1): 153-161.
[7] 黄毅鹏,胡冀苏,钱旭升,周志勇,赵文露,马麒,沈钧康,戴亚康. SE-Mask-RCNN:多参数MRI前列腺癌分割方法[J]. 浙江大学学报(工学版), 2021, 55(1): 203-212.
[8] 陈巧红,陈翊,李文书,贾宇波. 多尺度SE-Xception服装图像分类[J]. 浙江大学学报(工学版), 2020, 54(9): 1727-1735.
[9] 郑浦,白宏阳,李伟,郭宏伟. 复杂背景下的小目标检测算法[J]. 浙江大学学报(工学版), 2020, 54(9): 1777-1784.
[10] 周登文,田金月,马路遥,孙秀秀. 基于多级特征并联的轻量级图像语义分割[J]. 浙江大学学报(工学版), 2020, 54(8): 1516-1524.
[11] 明涛,王丹,郭继昌,李锵. 基于多尺度通道重校准的乳腺癌病理图像分类[J]. 浙江大学学报(工学版), 2020, 54(7): 1289-1297.
[12] 闫旭,范晓亮,郑传潘,臧彧,王程,程明,陈龙彪. 基于图卷积神经网络的城市交通态势预测算法[J]. 浙江大学学报(工学版), 2020, 54(6): 1147-1155.
[13] 汪周飞,袁伟娜. 基于深度学习的多载波系统信道估计与检测[J]. 浙江大学学报(工学版), 2020, 54(4): 732-738.
[14] 杨冰,莫文博,姚金良. 融合局部特征与深度学习的三维掌纹识别[J]. 浙江大学学报(工学版), 2020, 54(3): 540-545.
[15] 周金海,王依川,佟京鲆,周世镒,吴翔飞. 基于慢时间分割的超宽带雷达步态识别[J]. 浙江大学学报(工学版), 2020, 54(2): 283-290.