Please wait a minute...
浙江大学学报(工学版)
Journal of ZheJiang University (Engineering Science)  2021, Vol. 55 Issue (12): 2334-2341    DOI: 10.3785/j.issn.1008-973X.2021.12.013
    
Real-time smoke segmentation algorithm fused with multi-resolution representation
Hao-yuan WANG(),Yu LIANG,Wei ZHANG*()
School of Microelectronics, Tianjin University, Tianjin 300072, China
Download: HTML     PDF(1015KB) HTML
Export: BibTeX | EndNote (RIS)      

Abstract  

A high-accuracy real-time smoke segmentation algorithm was proposed, aiming at the lack of a real-time smoke segmentation algorithm applied to the actual monitoring systems in the field of smoke segmentation. A lightweight multi-resolution convolution module to extract feature maps in parallel was used in the algorithm, which met the needs of real-time segmentation while obtaining rich semantic information. A smoke foreground enhancement module was proposed to enable smoke pixels to be merged with their corresponding foreground enhancement representations, while avoiding the interference of background information, thereby improving the accuracy of segmentation. A residual attention module was proposed to enhance important feature information from the two dimensions of channel and space, and suppress invalid information. The algorithm had a mean intersection over union of 91.27% on the self-built data set, the prediction time of each picture was 39.06 ms, and the network weight was 74.66 MB. Comparison results on the public data set show that the comprehensive detection performance of this algorithm is better than that of other smoke detection algorithms. The algorithm has high segmentation accuracy, fast detection speed and the model is lightweight, which can be applied to actual video surveillance systems.

Key words： computer vision      smoke segmentation      multi-resolution module      smoke foreground enhancement module      residual attention module     
Received: 15 January 2021      Published: 31 December 2021
CLC:  TP 391.4  
Fund:  国家重点研发计划课题（2020YFC1522405）；科技重大专项与工程（19ZXZNGX00030）；应急管理部消防救援局科研计划重点攻关项目（2019XFGG20）
Corresponding Authors: Wei ZHANG     E-mail: csxueqian@tju.edu.cn;tjuzhangwei@tju.edu.cn
Service
E-mail this article
Add to my bookshelf
Add to citation manager
E-mail Alert
RSS
Articles by authors
Hao-yuan WANG
Yu LIANG
Wei ZHANG
Cite this article:

Hao-yuan WANG,Yu LIANG,Wei ZHANG. Real-time smoke segmentation algorithm fused with multi-resolution representation. Journal of ZheJiang University (Engineering Science), 2021, 55(12): 2334-2341.

URL:

https://www.zjujournals.com/eng/10.3785/j.issn.1008-973X.2021.12.013     OR     https://www.zjujournals.com/eng/Y2021/V55/I12/2334


融合多分辨率表征的实时烟雾分割算法

针对烟雾分割领域缺乏应用于实际监控系统的实时烟雾分割算法的现况，提出高准确率的实时烟雾分割算法. 该算法利用轻量化的多分辨率卷积模块并行提取特征图，在获得丰富语义信息的同时满足实时分割的需求. 提出烟雾前景增强模块，使得烟雾像素点融合前景增强表征、避免背景信息干扰，分割准确率得以提高. 提出残差注意力模块，从通道、空间维度增强重要特征信息，抑制无效信息. 该算法在自建数据集上平均交并比为91.27%，每张图片预测时间为39.06 ms，网络权重为74.66 MB；在公开数据集上的对比结果表明，该算法综合检测性能优于其他烟雾检测算法. 该算法分割准确率高、检测速度快且模型轻量化，可以应用于实际视频监控系统.


关键词： 计算机视觉,  烟雾分割,  多分辨率模块,  烟雾前景增强模块,  残差注意力模块 
Fig.1 Backbone network architecture fused with multi-resolution representation
阶段 模块数 分支数 各分支卷积单元数 各分支输出尺寸
1 1 1 $\left[ 2 \right]$ $ \left[ {224 \times 224 \times 64} \right] $
2 1 2 $\left[ \begin{gathered} 2 \hfill \\ 3 \hfill \\ \end{gathered} \right]$ $\left[ \begin{gathered} 224 \times 224 \times 16 \\ 112 \times 112 \times 36 \\ \end{gathered} \right]$
3 4 3 $\left[ \begin{gathered} 2 \hfill \\ 3 \hfill \\ 4 \hfill \\ \end{gathered} \right]$ $\left[ \begin{gathered} 224 \times 224 \times 16 \\ 112 \times 112 \times 36 \\ 56 \times 56 \times 72 \\ \end{gathered} \right]$
4 3 4 $\left[ \begin{gathered} 2 \hfill \\ 3 \hfill \\ 4 \hfill \\ 4 \hfill \\ \end{gathered} \right]$ $\left[ \begin{gathered} 224 \times 224 \times 16 \\ 112 \times 112 \times 36 \\ 56 \times 56 \times 72 \\ 28 \times 28 \times 144 \\ \end{gathered} \right]$
Tab.1 Backbone network parameter settings
Fig.2 Smoke foreground enhancement module
Fig.3 Residual attention module
Fig.4 Video data of laboratory
Fig.5 Qualitative comparison between representative semantic segmentation network and proposed algorithm
算法 mIoU/% T/ms P/MB
FCN 88.91 60.61 434.11
PSPNet 89.86 56.82 385.13
DeepLabV3 90.92 86.21 534.14
DeepLabV3+ 91.95 63.69 344.12
本研究 91.27 39.06 74.66
Tab.2 Comparison of segmentation results between representative semantic segmentation network and proposed algorithm
视频名称 视频描述 视频帧数/帧
sBehindtheFence 距离远、场景复杂 630
sBtFence 距离远、场景复杂 1400
sMoky 烟雾稀薄、快速运动 900
sWasteBasket 室内、有干扰（白墙） 900
sWindow 室外、运动缓慢 244
Tab.3 Description of public data set
Fig.6 Part of test results of public data set
视频名称 本研究算法 文献[20] 文献[19] 文献[18]
RTPR RTNR RTPR RTNR RTPR RTNR RTPR RTNR
sBehindtheFence 98.64 100.00 98.26 94.60 97.20 96.27 94.72 100.00
sBtFence 98.81 100.00 98.20 100.00 98.17 100.00 99.08 100.00
sMoky 98.27 100.00 98.85 100.00 99.68 100.00 86.23 100.00
sWasteBasket 99.50 96.84 99.41 100.00 97.18 98.36 99.89 92.60
sWindow 98.46 97.87 98.40 100.00 98.10 100.00 94.30 100.00
Tab.4 Comparison of detection results between existing smoke detection algorithms and proposed algorithm %
算法 烟雾前景增强模块 残差注意力模块 mIoU/% T/ms P/MB
文献[8] √ √ 91.84 47.63 93.24
Ours1 ? ? 89.45 34.80 60.00
Ours2 √ ? 90.83 38.52 73.24
本研究 √ √ 91.27 39.06 74.66
Tab.5 Ablation experiment results
Fig.7 Qualitative comparison of ablation experiment results
[1]   TAO C, ZHANG J, WANG P. Smoke detection based on deep convolutional neural networks [C]// 2016 International Conference on Industrial Informatics-Computing Technology, Intelligent Technology, Industrial Information Integration. Wuhan: IEEE, 2016: 150-153.
[2]   SHRIVASTAVA M, MATLANI P. A smoke detection algorithm based on K-means segmentation [C]// 2016 International Conference on Audio, Language and Image Processing. Shanghai: IEEE, 2016: 301-305.
[3]   FILONENKO A, HERNÁNDEZ D C, JO K H Fast smoke detection for video surveillance using CUDA[J]. IEEE Transactions on Industrial Informatics, 2017, 14 (2): 725- 733
[4]   LONG J, SHELHAMER E, DARRELL T. Fully convolutional networks for semantic segmentation [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Boston: IEEE, 2015: 3431-3440.
[5]   YUAN F, ZHANG L, XIA X, et al Deep smoke segmentation[J]. Neurocomputing, 2019, 357: 248- 260
doi: 10.1016/j.neucom.2019.05.011
[6]   XU G, ZHANG Y, ZHANG Q, et al Video smoke detection based on deep saliency network[J]. Fire Safety Journal, 2019, 105: 277- 285
doi: 10.1016/j.firesaf.2019.03.004
[7]   HE K, ZHANG X, REN S, et al. Deep residual learning for image recognition [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 770-778.
[8]   SUN K, XIAO B, LIU D, et al. Deep high-resolution representation learning for human pose estimation [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2019: 5693-5703.
[9]   HOWARD A G, ZHU M, CHEN B, et al. MobileNets: efficient convolutional neural networks for mobile vision applications [EB/OL]. [2020-11-16]. https://arxiv.org/pdf/1704.04861.pdf.
[10]   ZHAO H, SHI J, QI X, et al. Pyramid scene parsing network [C]// Proceedings of the IEEE conference on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017: 2881-2890.
[11]   CHEN L C, PAPANDREOU G, SCHROFF F, et al. Rethinking atrous convolution for semantic image segmentation [EB/OL]. [2020-12-10]. https://arxiv.org/pdf/1706.05587.pdf.
[12]   CHEN L C, PAPANDREOU G, KOKKINOS I, et al DeepLab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 40 (4): 834- 848
doi: 10.1109/TPAMI.2017.2699184
[13]   HE K, ZHANG X, REN S, et al Spatial pyramid pooling in deep convolutional networks for visual recognition[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015, 37 (9): 1904- 1916
doi: 10.1109/TPAMI.2015.2389824
[14]   YUAN Y, CHEN X, WANG J, et. Object-contextual representations for semantic segmentation[C]// Computer Vision-ECCV 2020. [S.l.]: Springer, 2020: 173-190.
[15]   WOO S, PARK J, LEE J Y, et al. CBAM: convolutional block attention module [C]// Proceedings of the European Conference on Computer Vision. Munich: Springer, 2018: 3-19.
[16]   HU J, SHEN L, SUN G. Squeeze-and-excitation networks [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 7132-7141.
[17]   CHEN L C, ZHU Y, PAPANDREOU G, et al. Encoder-decoder with atrous separable convolution for semantic image segmentation [C]// Proceedings of the European Conference on Computer Vision. Munich: Springer, 2018: 801-818.
[18]   BESBES O, BENAZZA-BENYAHIA A. A Novel video-based smoke detection method based on color invariants [C]// 2016 IEEE International Conference on Acoustics, Speech and Signal Processing. Shanghai: IEEE, 2016: 1911-1915.
[19]   赵敏, 张为, 王鑫, 等 时空背景模型下结合多种纹理特征的烟雾检测[J]. 西安交通大学学报, 2018, 52 (8): 67- 73
ZHAO Min, ZHANG Wei, Wang Xin, et al A smoke detection algorithm with multi-texture feature exploration under a spatio-temporal background model[J]. Journal of Xi’an Jiaotong University, 2018, 52 (8): 67- 73
[20]   汪梓艺, 苏育挺, 刘艳艳, 等 一种改进DeeplabV3网络的烟雾分割算法[J]. 西安电子科技大学学报, 2019, 46 (6): 52- 59
WANG Zi-yi, SU Yu-ting, LIU Yan-yan, et al Algorithm for segmentation of smoke using the improved DeeplabV3 network[J]. Journal of Xidian University, 2019, 46 (6): 52- 59
[1] Ying LI,Fang CHENG,Zhi-lin ZHAO. Machining precision online measurement of large span pin hole using structured light[J]. Journal of ZheJiang University (Engineering Science), 2020, 54(3): 557-565.
[2] Yao JIN,Wei ZHANG. Real-time fire detection algorithm with Anchor-Free network architecture[J]. Journal of ZheJiang University (Engineering Science), 2020, 54(12): 2430-2436.
[3] WANG Zhi, ZHU Shi qiang, BU Yan, GUO Zhen min. Stereo matching algorithm using improved guided filtering[J]. Journal of ZheJiang University (Engineering Science), 2016, 50(12): 2262-2269.