## 基于多尺度条件生成对抗网络血细胞图像分类检测方法

,, ,

1. 广西大学 电气工程学院，广西 南宁 530004

2. 广西医科大学第二附属医院 医学检验科，广西 南宁 530007

## Classification and detection method of blood cells images based on multi-scale conditional generative adversarial network

,, ,

1. School of Electrical Engineering, Guangxi University, Nanning 530004, China

2. Department of Clinical Laboratory, The Second Affiliated Hospital of Guangxi Medical University, Nanning 530007, China

 基金资助: 国家自然科学基金资助项目（62061002）

 Fund supported: 国家自然科学基金资助项目（62061002）

Abstract

A conditional generative adversarial network structure was proposed based on multi-scale discriminator to generate a large number of realistic leukocyte images, aiming at the problem of low detection accuracy due to insufficient leukocyte samples and unclear details of the generated cell images. The leukocyte images were added to the training set of the classification detection network to realize the generation and classification of blood cell images. A multi-scale convolution kernel and a multi-scale pooling domain were introduced into the authenticity discriminator of the generated confrontation network, and the channel connection which also improved the discriminator's ability to distinguish micro-detail texture features and macro-geometric features. The gradient similarity loss function was introduced to improve the brightness and edge clarity of the generated cell image to enhance the authenticity of the image. Results show that the quality of cell image is improved by the addition of multi-scale discriminator and gradient similarity loss function during the image generation stage. Compared with the case of real data training, the average accuracy of cell classification and detection is improved from 90.4% to 94.7% through increasing the diversity of cell samples during the image classification and detection stage.

Keywords： deep learning ; classification and detection of blood cell images ; conditional generative adversarial network ; gradient similarity ; multi-scale discriminator

CHEN Xue-yun, HUANG Xiao-qiao, XIE Li. Classification and detection method of blood cells images based on multi-scale conditional generative adversarial network. Journal of Zhejiang University(Engineering Science)[J], 2021, 55(9): 1772-1781 doi:10.3785/j.issn.1008-973X.2021.09.019

## 1. 相关工作

GAN是无监督的深度学习模型，由生成器G（generator）和鉴别器D（discriminator）2个网络构成. GAN的大概流程：G以噪声z作为输入，生成图像Gz），然后将真实图像xGz）一起输入D，D对xGz）做二分类，检测图像的真假，D的输出为0或1，0代表假图像，1代表真图像. 鉴别器的目标是尽可能区分真假样本，生成器的目标是生成让鉴别器无法区分的假样本，二者相互博弈，生成对抗网络的对抗目标函数为

${\mathop {{\rm{min}}}\limits_G \mathop {{\rm{max}}}\limits_D V\left( {G,D} \right) = {E_{{\boldsymbol{x}} \sim {P_{{\rm{data}}}}}}\log D\left( {\boldsymbol{x}} \right)+{E_{{\boldsymbol{z}} \sim {P_{\boldsymbol{z}}}}}\log \left( {1 - D\left( {G\left( {\boldsymbol{z}} \right)} \right)} \right).}$

pix2pix在原始GAN的基础上，将噪声替换为条件信息y输入生成器，实现图像到图像翻译的过程. 该过程是输入带图像位置掩膜的二元极小极大值的博弈问题，pix2pix生成器G和鉴别器D中都加入条件变量指导数据的生成. pix2pix网络的目标函数为

$\mathop {{\rm{min}}}\limits_G \mathop {{\rm{max}}}\limits_D V\left( {G,D} \right) = {L_{{\rm{CGAN}}}}\left( {G,D} \right)+\lambda {L_{{\rm{L1}}}}\left( G \right).$

$\begin{split} {L_{{\rm{CGAN}}}}\left( {G,D} \right) =& {E_{{\boldsymbol{x}} \sim {P_{{\rm{data}}}}}}\log \left( {D\left( {{\boldsymbol{x}},{\boldsymbol{y}}\left( {\boldsymbol{x}} \right)} \right)} \right)+\\ &{E_{{\boldsymbol{x}} \sim {P_{{\rm{data}}}}{,{{\boldsymbol{y}} \sim {P_{{\rm{data}}}}}}}}\log \left( {1 - D\left( {{\boldsymbol{y}}\left( {\boldsymbol{x}} \right),G\left( {{\boldsymbol{y}}\left( {\boldsymbol{x}} \right)} \right)} \right)} \right), \end{split}$

${L_{{\rm{L1}}}}\left( G \right) = {E_{{\boldsymbol{x}}{\boldsymbol{,}}{\boldsymbol{y}}}}\left( {\parallel {\boldsymbol{x}} - G\left( {{\boldsymbol{y}}\left( {\boldsymbol{x}} \right)} \right)\parallel } \right).$

### 2. 基于MCGAN的血细胞检测方法

MCGAN整体结构如图1所示. 该网络包括细胞图像生成器、MD、细胞分类检测器. 细胞图像生成器输入细胞的位置掩膜和梯度掩膜，输出生成的细胞图有2种形式：1）以随机数为圆心坐标，指定半径与颜色，随机生成的细胞位置与梯度掩膜；2）细胞原图中提取的位置与梯度掩膜. MD输入细胞原图与生成细胞图、位置掩膜与生成细胞图像二元组，首先经过多尺度卷积操作与多尺度池化模块得到全局多尺度特征，最后经过全连接层，激活函数用Sigmoid函数，输出鉴别结果0或1，以此实现对生成细胞图像的真假鉴别与位置匹配鉴别；检测器输入生成的细胞图像和真实细胞图像，输出检测彩色密度图后再经过聚类算法得到每类细胞的准确率，以此实现对细胞图像的分类检测.

### 图 1

Fig.1   Diagram of MCGAN network structure

### 2.1. MCGAN目标函数

${G_{\boldsymbol{x}}}\left( {{{i}},{{j}}} \right) = \frac{{\partial {\boldsymbol{x}}\left( {{{i}},{{j}}} \right)}}{{\partial {{i}}}}+\frac{{\partial {{x}}\left( {{{i}},{{j}}} \right)}}{{\partial {{j}}}}.$

$V\left( {{\boldsymbol{x}},{\boldsymbol{g}}} \right) = {\left[ {l\left( {{\boldsymbol{x}},{\boldsymbol{g}}} \right)} \right]^\alpha }{\left[ {c\left( {{\boldsymbol{x}},{\boldsymbol{g}}} \right)} \right]^\beta }{\left[ {e\left( {{\boldsymbol{x}},{\boldsymbol{g}}} \right)} \right]^\gamma }.$

$l\left( {{\boldsymbol{x}},{\boldsymbol{g}}} \right) = \frac{{2{u_{\boldsymbol{x}}}+{u_{\boldsymbol{g}}}+{c_1}}}{{u_{\boldsymbol{x}}^2+u_{\boldsymbol{g}}^2+{c_1}}},$

$c\left( {{\boldsymbol{x}},{\boldsymbol{g}}} \right) = \frac{{2{\sigma _{\boldsymbol{x}}}\sigma _{\boldsymbol{g}}+{c_2}}}{{{\sigma _{\boldsymbol{x}}}^2+\sigma _{\boldsymbol{g}}^2+{c_2}}},$

$e\left( {{\boldsymbol{x}},{\boldsymbol{g}}} \right) = \frac{{{2\displaystyle\sum\limits_{{j}} {\displaystyle\sum\limits_{{i}}} {G_{\boldsymbol{x}}}\left( {{{i}},{{j}}} \right)} {G_{\boldsymbol{g}}}\left( {{{i}},{{j}}} \right)+{c_3}}}{{{\displaystyle\sum\limits_{{j}}{\displaystyle\sum\limits _{{i}}} {{\left[ {{G_{\boldsymbol{x}}}\left( {{{i}},{{j}}} \right)} \right]}^2}+{\displaystyle\sum\limits_{j}{\displaystyle\sum\limits_{{i}}} {{\left[ {{G_{\boldsymbol{g}}}\left( {{{i}},{{j}}} \right)} \right]}^2}} +{c_3}} }}.$

${L_{{\rm{GS}}}}\left( H \right){\rm{ = 1}} - \frac{{\rm{1}}}{N}\sum\limits_{H{\rm{ = 1}}}^H {V\left( H \right)} .$

${L_{{\rm{class}}}} = \frac{1}{W}\sum\limits_i {{L_i}} - \sum\limits_{c = 1}^K {{y_{ic}}\log \left( {{p_{ic}}} \right)} .$

$\begin{split} \mathop {\min }\limits_G \mathop {{\rm{max}}}\limits_D V\left( {G,D} \right) =& {L_{{\rm{CGAN}}}}\left( {G,D} \right)+{\lambda _{\rm{1}}}{L_{{\rm{L1}}}}\left( G \right)+\\ & {\lambda _{\rm{2}}}{L_{{\rm{GS}}}}\left( G \right)+{L_{{\rm{class}}}}. \end{split}$

$F\left( {{\boldsymbol{x}},{\boldsymbol{g}}} \right) = \parallel {{\boldsymbol{Z}} _{\boldsymbol{x}}} - {{\boldsymbol{Z}} _{\boldsymbol{g}}}{\parallel ^2}+{{\rm{t}}{\rm{r}}}\left( {{{B}}+{{O}} - 2{{\left( {{{BO}}} \right)}^{0.5}}} \right).$

Tab.2  Comparison of PNSR, SSIM and FID values for different models

 模型 R/dB S F CGAN 13.63 0.56 84.13 pix2pix 18.41 0.69 54.37 pix2pix+MD 22.83 0.83 37.34 MCGAN 23.68 0.87 34.45

### 图 6

Fig.6   Comparison of cell classification and detection experimental results.

##### 4.2.2. 不同网络检测准确率对比

Tab.3  Comparison of detection results at different models

 模型 P/% T M e SSD 82.7 84 202 11 883 1 254 YOLOV3 86.5 84 202 10 367 824 FCRN 89.3 84 202 8 910 314 Res-Net 89.7 84 202 8 573 291 U-Net 90.4 84 202 7 903 236 U-Net+pix2pix 92.7 84 202 6 138 172 U-Net+pix2pix+MD 94.1 84 202 5 115 121 MCGAN 94.7 84 202 4 624 76

Tab.4  Comparison of all kinds of cells detection results in U-Net under different database training　　　　　　　　　　 %

 训练集 P1 P2 P3 P4 P5 P6 RD 90.5 91.3 88.6 87.6 90.9 88.1 RD+CGAN 91.0 91.7 90.7 88.7 91.9 89.1 RD+pix2pix 92.2 93.1 91.7 89.9 93.5 91.0 RD+pix2pix+MD 94.1 96.9 96.9 95.5 96.7 96.0 RD+MCGAN 94.7 99.1 98.9 98.9 98.9 99.0

## 参考文献 原文顺序 文献年度倒序 文中引用次数倒序 被引期刊影响因子

[J]. 光电子·激光, 2019, 30 (5): 546- 555

WANG Ya-pin, CAO Yi-ping, FU Guang-kai, et al

Human peripheral blood leukocyte microscopic image classification based on deep convolutional neural network

[J]. Journal of Optoelectronics·Laser, 2019, 30 (5): 546- 555

REN S, HE K, GIRSHICK R, et al

Faster R-CNN: towards real-time object detection with region proposal networks

[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39 (6): 1137- 1149

LIU W, ANGUELOY D, ERHAN D, et al. SSD: single shot multibox detector[C]// European Conference on Computer Vision. Amsterdam: Springer, 2016: 21-37.

REDMON J, DIVVALA S, GIRSHICK R, et al. You only look once: unified, real-time object detection[C]// Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 779-788.

REDMON J, FARHADI A. YOLO9000: better, faster, stronger[C]// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Washington: IEEE, 2017: 6517-6525.

REDMON J, FARHADI A. YOLOV3: an incremental improvement [EB/OL]. (2018-4-10) [2020-8-18]. https://arxiv.org/pdf/1804.02767.pdf.

[J]. 计算机工程与应用, 2020, 56 (14): 98- 103

XU Xiao-tao, SUN Ya-dong, ZHANG Jun

Automated counting of blood cells based on YOLO framework

[J]. Computer Engineering and Applications, 2020, 56 (14): 98- 103

LIU Shu-jie. Red blood cell detection and counting based on convolutional neural network[D]. Guangzhou: South China University of Technology, 2017: 40-57.

HILAL T, KIM G S, KIL T C, et al

Vehicle detection and counting in high-resolution aerial images using convolutional regression neural network

[J]. IEEE Access, 2017, 11 (6): 2220- 2230

CHEN X, LIN J, XIANG S, et al

Detecting maneuvering target accurately based on a two-phase approach from remote sensing imagery

[J]. IEEE Geoscience and Remote Sensing Letters, 2020, 17 (5): 849- 853

GOODFELLOW I J, POUGET A J, MIRZA M, et al

[J]. Advances in Neural Information Processing Systems, 2014, 3: 2672- 2680

CUI Y R, LIU Q, GAO C Y, et al

[J]. Computer Graphics Forum, 2018, 37 (7): 345- 359

ISOLA P, ZHU J Y, ZHOU T, et al. Image-to-image translation with conditional adversarial networks[C]// Proceedings of the IEEE conference on computer vision and pattern recognition. Hawaii: IEEE, 2017: 1125-1134.

RONNEBRGER O, FISCHER P, BROX T. U-Net: convolutional networks for biomedical image segmentation[C]// International Conference on Medical Image Computing and Computer-Assisted Intervention. Munich: Springer, 2015: 234-241.

PRATT H, WILLIAMS B, COENEN F, et al. FCNN: Fourier convolutional neural networks[C]// Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Berlin: Springer, 2017: 786-798.

[J]. 电子学报, 2019, 47 (8): 1738- 1747

LAI Xiao-bo, XU Mao-sheng, XU Xiao-mei

Glioblastoma multiforme multi-modal MR image segmentation using multi-class CNN

[J]. Acta Electronica Sinica, 2019, 47 (8): 1738- 1747

[J]. 浙江大学学报: 工学版, 2019, 53 (7): 1331- 1339

DUAN Ran, ZHOU Deng-wen, ZHAO Li-juan, et al

Image super-resolution reconstruction based on multi-scale feature mapping network

[J]. Journal of Zhejiang University: Engineering Science, 2019, 53 (7): 1331- 1339

[J]. 太赫兹科学与电子信息学报, 2020, 18 (4): 665- 671

WANG Cheng-kai, YANG Xiao-min, YAN Bin-yu

Infrared image super-resolution algorithm based on random forest

[J]. Journal of Terahertz Science and Electronic Information Technology, 2020, 18 (4): 665- 671

[J]. 光学学报, 2020, 40 (5): 75- 84

HU Lin-miao, ZHANG Yong

Facial image translation in short-wavelength infrared and visible light based on generative adversarial network

[J]. Acta Optica Sinica, 2020, 40 (5): 75- 84

/

 〈 〉