Please wait a minute...
Journal of ZheJiang University (Engineering Science)  2021, Vol. 55 Issue (2): 377-385    DOI: 10.3785/j.issn.1008-973X.2021.02.018
    
Detection of small fruit target based on improved DenseNet
Li-feng XU(),Hai-fan HUANG,Wei-long DING,Yu-lei FAN
College of Computer Science and Technology, Zhejiang University of Technology, Hangzhou 310023, China
Download: HTML     PDF(889KB) HTML
Export: BibTeX | EndNote (RIS)      

Abstract  

An improved fruit detection framework based on DenseNet was proposed aiming at the problem that small fruit target detection always obtains low accuracy in natrual environment. A multi-scale feature extraction module was built with DenseNet. A feature pyramid structure was used in dense blocks at different scales of DenseNet in order to strength the network layer feature reuse. Low-level features with high resolution and high-level features with high semantics were combined to achieve accurate localization and prediction of the existence of small fruits. Soft non-maximum suppression (Soft-NMS) algorithm was introduced to avoid the case that detection boxes were mistakenly removed in the clustered fruit structure. In three datasets of apple, mango and almond, the detection speed came up to 40 FPS, and the F1 score reached 0.920, 0.928 and 0.831 with the proposed framework. The detection efficiency and accuracy were improved compared with the commonly used Faster R-CNN network.



Key wordsDenseNet      deep learning      small fruit target detection      feature pyramid network (FPN)      soft non-maximum suppression (Soft-NMS)     
Received: 02 September 2020      Published: 09 March 2021
CLC:  TP 399  
Fund:  国家自然科学基金资助项目(61571400,61702456);浙江省自然科学基金资助项目(LY18C130012)
Cite this article:

Li-feng XU,Hai-fan HUANG,Wei-long DING,Yu-lei FAN. Detection of small fruit target based on improved DenseNet. Journal of ZheJiang University (Engineering Science), 2021, 55(2): 377-385.

URL:

http://www.zjujournals.com/eng/10.3785/j.issn.1008-973X.2021.02.018     OR     http://www.zjujournals.com/eng/Y2021/V55/I2/377


基于改进DenseNet的水果小目标检测

针对自然环境中小目标水果的检测精度普遍较低的问题,提出基于DenseNet改进的水果目标检测框架. 构建以DenseNet为核心的多尺度特征提取模块,在DenseNet不同层级的稠密块中建立特征金字塔结构,加强网络层特征复用. 结合低层特征的高分辨率和高层特征的高语义性,实现准确定位和预测小目标水果存在的目的. 引入软阈值非极大值抑制(Soft-NMS)算法,改善簇状果实结构中检测框被误剔除的情况. 与常用的Faster R-CNN网络相比,所提出的框架在苹果、芒果和杏3个数据集中的平均检测速度大于40 FPS,F1值分别为0.920、0.928、0.831,实现了检测效率及精度的提升.


关键词: DenseNet,  深度学习,  水果小目标检测,  特征金字塔网络 (FPN),  软阈值非极大值抑制 (Soft-NMS) 
Fig.1 Structure of DenseNet
Fig.2 Structure of FPN
Fig.3 Overlapped boxes in fruit detection
Fig.4 Diagram representing structure of improved network
Fig.5 Diagram representing structure of improved Dense Block
Dense 网络层 通道数 参数 输出尺寸/像素
阶段1 Conv 16 3×3/1 512×512
Pool 16 2×2/2 256×256
Conv 32 3×3/1 256×256
Pool 32 2×2/2 128×128
Dense Block 32 1×1/1 Conv 128×128
16 3×3/1 Conv 128×128
阶段2 Conv 64 3×3/1 128×128
Pool 64 2×2/2 64×64
Dense Block 64 1×1/1 Conv 64×64
32 3×3/1 Conv 64×64
阶段3 Conv 128 3×3/1 64×64
Pool 128 2×2/2 32×32
Dense Block 128 1×1/1 Conv 32×32
64 3×3/1 Conv 32×32
阶段4 Conv 256 3×3/1 32×32
Pool 256 2×2/2 16×16
Dense Block 256 1×1/1 Conv 16×16
128 3×3/1 Conv 16×16
Conv 512 3×3/1 16×16
Pool 512 2×2/1 16×16
Conv 1024 3×3/1 16×16
Conv 30 1×1/1 16×16
Tab.1 Parameters for object detection network
Fig.6 Structure of object detection network
水果种类 分辨率/像素 训练集个数 验证集个数/测试集个数
苹果 200×308 729 112/112
芒果 500×500 1154 270/270
300×300 385 100/100
Tab.2 Parameters of fruit dataset
水果 P(Faster R-CNN) P(本研究) R(本研究) F1(本研究)
苹果 90.3 93.19 91.14 0.920
芒果 90.8 93.59 92.03 0.928
77.5 84.31 81.95 0.831
Tab.3 Comparison of proposed method with Faster R-CNN
Fig.7 Visualization of fruit detection
Fig.8 Apple and mango detection in occlusion scenes
水果 O /% C /%
本研究 文献[8] 本研究 文献[8]
苹果 90.37 86.48 92.46 90.75
芒果 88.14 85.91 90.83 89.10
Tab.4 Comparison of detection capabilities between proposed method and literature [8] method in occlusion scenes
水果 F1
VGG16(文献[8]) VGG16(本研究) ResNet
苹果 0.904 0.879 0.887
芒果 0.908 0.880 0.892
0.775 0.739 0.748
Tab.5 Experiments recurrence of original dataset and F1 score calculation
算法工况 P R F1
Faster R-CNN ? ? 0.775
无FPN结构 79.77 78.40 0.791
有FPN结构 83.53 81.94 0.828
Tab.6 Comparison of detection results with or without FPN structure
水果 P
NMS Soft-NMS
苹果 92.96 93.17
芒果 93.18 93.35
84.07 84.42
Tab.7 Comparison of detection accuracy between NMS and Soft-NMS
水果 Ru /%
实验1 实验2 实验3 实验4 实验5
苹果 0.18 0.09 0.2 0.14 0.14
芒果 0.37 0.32 0.41 0.34 0.35
0.63 0.55 0.65 0.61 0.60
Tab.8 Upgrade rate of Soft-NMS under different parameters
[1]   NUSKE S, ACHAR S, BATES T, et al. Yield estimation in vineyards by visual grape detection [C]// 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems. San Francisco: IEEE, 2011: 2352-2358.
[2]   PAYNE A B, WALSH K B, SUBEDI P P, et al Estimation of mango crop yield using image analysis: segmentation method[J]. Computers and Electronics in Agriculture, 2013, 91: 57- 64
doi: 10.1016/j.compag.2012.11.009
[3]   INKYU S, ZONGYUAN G, FERAS D, et al Deep fruits: a fruit detection system using deep neural networks[J]. Sensors, 2016, 16 (8): 1222
doi: 10.3390/s16081222
[4]   GIRSHICK R, DONAHUE J, DARRELL T, et al Region-based convolutional networks for accurate object detection and segmentation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015, 38 (1): 142- 158
[5]   EVERINGHAM M, GOOL L V, WILLIAMS C K I, et al The pascal visual object classes (VOC) challenge[J]. International Journal of Computer Vision, 2010, 88 (2): 303- 338
doi: 10.1007/s11263-009-0275-4
[6]   UIJLINGS J R R, SANDE K E A V D, GEVERS T, et al Selective search for object recognition[J]. International Journal of Computer Vision, 2013, 104 (2): 154- 171
doi: 10.1007/s11263-013-0620-5
[7]   REN S, HE K, GIRSHICK R, et al Faster R-CNN: towards real-time object detection with region proposal networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39 (6): 1137- 1149
[8]   BARGOTI S, UNDERWOOD J. Deep fruit detection in orchards [C]// 2017 IEEE International Conference on Robotics and Automation. Singapore: IEEE, 2017: 3626-3633.
[9]   REDMON J, DIVVALA S, GIRSHICK R, et al. You only look once: unified, real-time object detection [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 779-788.
[10]   SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition [J/OL]. [2020-08-16]. https://arxiv.org/abs/1409.1556.
[11]   HE K, ZHANG X, REN S, et al. Deep residual learning for image recognition [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 770-778.
[12]   曾平平, 李林升 基于卷积神经网络的水果图像分类识别研究[J]. 机械设计与研究, 2019, 35 (1): 23- 26
ZENG Ping-ping, LI Lin-sheng Classification and recognition of common fruit images based on convolutional neural network[J]. Machine Design and Research, 2019, 35 (1): 23- 26
[13]   薛月菊, 黄宁, 涂淑琴, 等 未成熟芒果的改进YOLOv2识别方法[J]. 农业工程学报, 2018, 34 (7): 173- 179
XUE Yue-ju, HUANG Ning, TU Shu-qin, et al Immature mango detection based on improved YOLOv2[J]. Transactions of the Chinese Society of Agricultural Engineering, 2018, 34 (7): 173- 179
doi: 10.11975/j.issn.1002-6819.2018.07.022
[14]   REDMON J, FARHADI A. YOLO9000: better, faster, stronger [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017: 7263-7271.
[15]   WAN S, GOUDOS S Faster R-CNN for multi-class fruit detection using a robotic vision system[J]. Computer Networks, 2020, 168: 107036
doi: 10.1016/j.comnet.2019.107036
[16]   MAI X, ZHANG H, JIA X, et al Faster R-CNN with classifier fusion for automatic detection of small fruits[J]. IEEE Transactions on Automation Science and Engineering, 2020, PP (99): 1- 15
[17]   SRIVASTAVA R K, GREFF K, SCHMIDHUBER J. Training very deep networks [C]// Advances in Neural Information Processing Systems. Montreal: Curran Associates, 2015: 2377-2385.
[18]   LARSSON G, MAIRE M, SHAKHNAROVICH G. Fractalnet: ultra-deep neural networks without residuals [J/OL]. [2020-08-16]. https://arxiv.org/abs/1605.07648.
[19]   HUANG G, LIU Z, VAN DER MAATEN L, et al. Densely connected convolutional networks [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017: 4700-4708.
[20]   LIN T Y, DOLLáR P, GIRSHICK R, et al. Feature pyramid networks for object detection [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017: 2117-2125.
[21]   NEUBECK A, VAN GOOL L. Efficient non-maximum suppression [C]// 18th International Conference on Pattern Recognition. Hong Kong: IEEE, 2006, 3: 850-855.
[22]   BODLA N, SINGH B, CHELLAPPA R, et al. Soft-NMS: improving object detection with one line of code [C]// Proceedings of the IEEE International Conference on Computer Vision. Venice: IEEE, 2017: 5561-5569.
[23]   MATHIAS M, BENENSON R, TIMOFTE R, et al. Handling occlusions with franken-classifiers [C]// Proceedings of the IEEE International Conference on Computer Vision. Sydeny: IEEE, 2013: 1505-1512.
[24]   NING C, MENGLU L, HAO Y, et al. Survey of pedestrian detection with occlusion [J/OL]. Complex and Intelligent Systems. https://doi.org/10.1007/s40747-020-00206-8
[1] Jia-hui XU,Jing-chang WANG,Ling CHEN,Yong WU. Surface water quality prediction model based on graph neural network[J]. Journal of ZheJiang University (Engineering Science), 2021, 55(4): 601-607.
[2] Hong-li WANG,Bin GUO,Si-cong LIU,Jia-qi LIU,Yun-gang WU,Zhi-wen YU. End context-adaptative deep sensing model with edge-end collaboration[J]. Journal of ZheJiang University (Engineering Science), 2021, 55(4): 626-638.
[3] Teng ZHANG,Xin-long JIANG,Yi-qiang CHEN,Qian CHEN,Tao-mian MI,Piu CHAN. Wrist attitude-based Parkinson's disease ON/OFF state assessment after medication[J]. Journal of ZheJiang University (Engineering Science), 2021, 55(4): 639-647.
[4] Hao-can XU,Ji-tuo LI,Guo-dong LU. Reconstruction of three-dimensional human bodies from single image by LeNet-5[J]. Journal of ZheJiang University (Engineering Science), 2021, 55(1): 153-161.
[5] Yi-peng HUANG,Ji-su HU,Xu-sheng QIAN,Zhi-yong ZHOU,Wen-lu ZHAO,Qi MA,Jun-kang SHEN,Ya-kang DAI. SE-Mask-RCNN: segmentation method for prostate cancer on multi-parametric MRI[J]. Journal of ZheJiang University (Engineering Science), 2021, 55(1): 203-212.
[6] Pu ZHENG,Hong-yang BAI,Wei LI,Hong-wei GUO. Small target detection algorithm in complex background[J]. Journal of ZheJiang University (Engineering Science), 2020, 54(9): 1777-1784.
[7] Qiao-hong CHEN,YI CHEN,Wen-shu Li,Yu-bo JIA. Clothing image classification based on multi-scale SE-Xception[J]. Journal of ZheJiang University (Engineering Science), 2020, 54(9): 1727-1735.
[8] Deng-wen ZHOU,Jin-yue TIAN,Lu-yao MA,Xiu-xiu SUN. Lightweight image semantic segmentation based on multi-level feature cascaded network[J]. Journal of ZheJiang University (Engineering Science), 2020, 54(8): 1516-1524.
[9] Tao MING,Dan WANG,Ji-chang GUO,Qiang LI. Breast cancer histopathological image classification using multi-scale channel squeeze-and-excitation model[J]. Journal of ZheJiang University (Engineering Science), 2020, 54(7): 1289-1297.
[10] Xu YAN,Xiao-liang FAN,Chuan-pan ZHENG,Yu ZANG,Cheng WANG,Ming CHENG,Long-biao CHEN. Urban traffic flow prediction algorithm based on graph convolutional neural networks[J]. Journal of ZheJiang University (Engineering Science), 2020, 54(6): 1147-1155.
[11] Zhou-fei WANG,Wei-na YUAN. Channel estimation and detection method for multicarrier system based on deep learning[J]. Journal of ZheJiang University (Engineering Science), 2020, 54(4): 732-738.
[12] Bing YANG,Wen-bo MO,Jin-liang YAO. 3D palmprint recognition by using local features and deep learning[J]. Journal of ZheJiang University (Engineering Science), 2020, 54(3): 540-545.
[13] Yan-jia HONG,Tie-bao MENG,Hao-jiang LI,Li-zhi LIU,Li LI,Shuo-yu XU,Sheng-wen GUO. Deep segmentation method of tumor boundaries from MR images of patients with nasopharyngeal carcinoma using multi-modality and multi-dimension fusion[J]. Journal of ZheJiang University (Engineering Science), 2020, 54(3): 566-573.
[14] Zi-yu JIA,You-fang LIN,Hong-jun ZHANG,Jing WANG. Sleep stage classification model based ondeep convolutional neural network[J]. Journal of ZheJiang University (Engineering Science), 2020, 54(10): 1899-1905.
[15] Wan-liang WANG,Xiao-han YANG,Yan-wei ZHAO,Nan GAO,Chuang LV,Zhao-juan ZHANG. Image enhancement algorithm with convolutional auto-encoder network[J]. Journal of ZheJiang University (Engineering Science), 2019, 53(9): 1728-1740.