Visual perception system for complex environment of coal mine comprehensive excavation working face

doi:10.3785/j.issn.1008-973X.2025.05.013

Journal of ZheJiang University (Engineering Science)

2025, Vol. 59

Issue (5): 995-1006 DOI: 10.3785/j.issn.1008-973X.2025.05.013

Visual perception system for complex environment of coal mine comprehensive excavation working face

Guoyong SU1,2,3(

),Kun HU1,2,3,*(

),Pengyu WANG2,3,Dongyang ZHAO3,Hui ZHANG3

1. State Key Laboratory of Deep Coal Mining Response and Disaster Prevention and Control, Anhui University of Science and Technology, Huainan 232001, China
2. Collaborative Innovation Center for Mining Intelligent Technology and Equipment, Anhui University of Science and Technology, Huainan 232001, China
3. School of Mechatronics Engineering, Anhui University of Science and Technology, Huainan 232001, China

Download:

HTML

PDF(4637KB) HTML
Export: BibTeX | EndNote (RIS)

Abstract

A visual perception system for the complex environment of the coal mine comprehensive excavation working face was proposed aiming at the problem of insufficient robustness of visual detection algorithms in the harsh environment of coal mines. ELAN-DS feature extraction module, SimAM attention module with decoupled detection head were used to optimize the YOLOv7-tiny algorithm in order to construct the coal mining comprehensive excavation face visual inspection network (CMCE-Net). CMCE-Net was migrated and deployed into the visual perception terminal platform in order to test the detection performance of CMCE-Net under the actual working conditions in coal mines. Validation experiments were conducted based on the data set of coal mine comprehensive excavation working face. The experimental results showed that the detection accuracy of CMCE-Net reached 89.5%, which was a 5.2% improvement compared with the YOLOv7-tiny algorithm. The combined detection performance was the best and the model complexity was at a lower level compared with eight algorithms such as Faster RCNN, YOLOv7-tiny and YOLOv8s. The detection speed of CMCE-Net on the test video reached up to 33.4 frames/s within the visual perception terminal platform, and the detection accuracy of CMCE-Net on the equipment and personnel was more than 90.0% under the human-machine multi-target mixing working condition.

Key words： comprehensive excavation working face critical target visual perception detection network

Received: 19 March 2024 Published: 25 April 2025

CLC:	TD 76
	TP 18

Fund: 安徽省高等学校科学研究资助项目(2022AH050834)；国家自然科学基金资助项目(52304166，52274153)；深部煤矿采动响应与灾害防控国家重点实验室开放基金资助项目(SKLMRDPC22KF24)；安徽理工大学矿山智能技术与装备省部共建协同创新中心开放基金资助项目(CICJMITE202206)；安徽理工大学引进人才科研启动基金资助项目(2022yjrc61).

Corresponding Authors: Kun HU E-mail: guoyongs005@sina.cn;hk924@126.com

	Service
	E-mail this article
	Add to my bookshelf
	Add to citation manager
	E-mail Alert
	RSS
	Articles by authors
	Guoyong SU
	Kun HU
	Pengyu WANG
	Dongyang ZHAO
	Hui ZHANG

Cite this article:

Guoyong SU,Kun HU,Pengyu WANG,Dongyang ZHAO,Hui ZHANG. Visual perception system for complex environment of coal mine comprehensive excavation working face. Journal of ZheJiang University (Engineering Science), 2025, 59(5): 995-1006.

URL:

https://www.zjujournals.com/eng/10.3785/j.issn.1008-973X.2025.05.013 OR https://www.zjujournals.com/eng/Y2025/V59/I5/995

面向煤矿综掘工作面复杂环境的视觉感知系统

针对煤矿恶劣环境下视觉检测算法鲁棒性不足的难题，提出面向煤矿综掘工作面复杂环境的视觉感知系统. 该系统采用ELAN-DS特征提取模块、SimAM注意力模块与解耦检测头对YOLOv7-tiny算法进行优化，构建煤矿综掘工作面视觉检测网络(CMCE-Net). 将CMCE-Net迁移部署到视觉感知终端平台内，测试CMCE-Net在煤矿实际作业工况下的检测性能，基于煤矿综掘工作面数据集开展验证实验. 实验结果表明，CMCE-Net的检测精度达到89.5%，相较于YOLOv7-tiny算法提升了5.2%. 与Faster RCNN、YOLOv7-tiny、YOLOv8s等8种算法相比，综合检测性能最佳，模型复杂度处于较低水平. 在视觉感知终端平台内，CMCE-Net对测试视频的检测速度最高达到33.4 帧/s，在人机多目标混杂工况下，CMCE-Net对装备与人员的检测精度均大于90.0%.

关键词： 综掘工作面, 关键目标, 视觉感知, 检测网络

Fig.1 Visual perception system architecture for coal mine comprehensive excavation working face

Fig.2 Overall structure of visual detection network for coal mine comprehensive excavation working face

Fig.3 Schematic diagram of ELAN-DS module and DSConv structure decomposition

Fig.4 Structure of SimAM module

Fig.5 Schematic diagram of coupled and decoupled detection head structure

Fig.6 Excavating, anchor support and airborne anchor support operation scenes in coal mine

Fig.7 Annotation of coal mine comprehensive excavation working face data set

Fig.8 mAP_0.5 curve of ablation experiment

Fig.9 Output feature map of feature extraction module for model A and model B

Fig.10 Heat map of model B and model C

Fig.11 Classification and regression loss curve

Tab.1 Result of ablation experiment

Tab.2 Result of comparison experiment on coal mine comprehensive excavation working face data set

Fig.12 mAP_0.5 curve of comparison experiment for target detection algorithm

Tab.3 Result of comparison experiment on public data set

Tab.4 Performance test result of CMCE-Net

Fig.13 Comparison test result of four YOLO series detection algorithms and CMCE-Net


[1]	程德强, 寇旗旗, 江鹤, 等全矿井智能视频分析关键技术综述[J]. 工矿自动化, 2023, 49 (11): 1- 21 CHENG Deqiang, KOU Qiqi, JIANG He, et al Overview of key technologies for mine-wide intelligent video analysis[J]. Journal of Mine Automation, 2023, 49 (11): 1- 21

[2]	程健, 李昊, 马昆, 等矿井视觉计算体系架构与关键技术[J]. 煤炭科学技术, 2023, 51 (9): 202- 218 CHENG Jian, LI Hao, MA Kun, et al Architecture and key technologies of coalmine underground vision computing[J]. Coal Science and Technology, 2023, 51 (9): 202- 218

[3]	巩师鑫, 赵国瑞, 王飞机器视觉感知理论与技术在煤炭工业领域应用进展综述[J]. 工矿自动化, 2023, 49 (5): 7- 21 GONG Shixin, ZHAO Guorui, WANG Fei Review on the application of machine vision perception theory and technology in coal industry[J]. Journal of Mine Automation, 2023, 49 (5): 7- 21

[4]	WANG Gongxian, ZHANG Libin, SUN Hui, et al Longitudinal tear detection of conveyor belt under uneven light based on Haar-AdaBoost and Cascade algorithm[J]. Measurement, 2021, 16 (8): 108341

[5]	李曼, 段雍, 曹现刚, 等煤矸分选机器人图像识别方法和系统[J]. 煤炭学报, 2020, 45 (10): 3636- 3644 LI Man, DUAN Yong, CAO Xiangang, et al Image identification method and system for coal and gangue sorting robot[J]. Journal of China Coal Society, 2020, 45 (10): 3636- 3644

[6]	王志勇, 李路, 王建, 等基于HOG特征的InSAR矿区开采沉陷盆地检测方法[J]. 中国矿业大学学报, 2021, 50 (2): 404- 410 WANG Zhiyong, LI Lu, WANG Jian, et al A method of detecting the subsidence basin from InSAR interferogram in mining area based on HOG features[J]. Journal of China University of Mining and Technology, 2021, 50 (2): 404- 410

[7]	WANG Yuanbin, WANG Yujing, DANG Langfei Video detection of foreign objects on the surface of belt conveyor underground coal mine based on improved SSD[J]. Journal of Ambient Intelligence and Humanized Computing, 2023, 14 (5): 5507- 5516 doi: 10.1007/s12652-020-02495-w

[8]	邹盛, 周李兵, 季亮, 等基于图像融合和改进CornerNet-Squeeze的煤矿井下行人检测方法[J]. 工矿自动化, 2023, 49 (2): 77- 84 ZOU Sheng, ZHOU Libing, JI Liang, et al A pedestrian target detection method for underground coal mine based on image fusion and improved CornerNet-Squeeze[J]. Journal of Mine Automation, 2023, 49 (2): 77- 84

[9]	ZHENG J H, WANG D Y, GENG Z X Real-time detection of safety hazards in coal mines utilizing an enhanced YOLOv3 algorithm[J]. Traitement du Signal, 2023, 40 (4): 1565- 1572 doi: 10.18280/ts.400424

[10]	郭永存, 杨豚, 王爽基于改进YOLOv4–Tiny的矿井电机车多目标实时检测[J]. 工程科学与技术, 2023, 55 (5): 232- 241 GUO Yongcun, YANG Tun, WANG Shuang Multi-object real-time detection of mine electric locomotive based on improved YOLOv4-Tiny[J]. Advanced Engineering Sciences, 2023, 55 (5): 232- 241

[11]	WANG X, WANG S, GUO Y C, et al Multi-scale coal and gangue detection in dense state based on improved Mask RCNN[J]. Measurement, 2023, 22 (1): 113467

[12]	杨洋 AI智能视频识别分析技术在智能化掘进的研究与应用[J]. 工矿自动化, 2023, 49 (Suppl.1): 26- 28 YANG Yang Research and application of AI intelligent video recognition analysis technology in intelligent excavation[J]. Journal of Mine Automation, 2023, 49 (Suppl.1): 26- 28

[13]	杜雨馨, 张贺, 王树臣, 等综掘系统视觉处理技术研究现状及发展趋势[J]. 工矿自动化, 2023, 49 (11): 22- 38 DU Yuxin, ZHANG He, WANG Shuchen, et al Research status and development trend of visual processing technology for fully mechanized excavation systems[J]. Journal of Mine Automation, 2023, 49 (11): 22- 38

[14]	张农, 袁钰鑫, 韩昌良, 等基于Mask R-CNN的煤矿巷道掘进迎头裂隙检测与定位算法[J]. 采矿与安全工程学报, 2023, 40 (5): 925- 932 ZHANG Nong, YUAN Yuxin, HAN Changliang, et al Research on crack detection and localization for advancing face in coal mine roadways based on Mask R-CNN[J]. Journal of Mining and Safety Engineering, 2023, 40 (5): 925- 932

[15]	SUN S, MA H, WANG K, et al Research on the detection method of coal mine roadway bolt mesh based on improved YOLOv7[J]. Electronics, 2023, 12 (14): 3050 doi: 10.3390/electronics12143050

[16]	张夫净, 王宏伟, 王浩然, 等煤矿巷道支护钢带锚孔智能识别与定位[J]. 工矿自动化, 2022, 48 (10): 76- 81 ZHANG Fujing, WANG Hongwei, WANG Haoran, et al Intelligent identification and positioning of steel belt anchor hole in coal mine roadway support[J]. Journal of Mine Automation, 2022, 48 (10): 76- 81

[17]	ZHAO D Y, SU G Y, CHENG G, et al Research on real-time perception method of key targets in the comprehensive excavation working face of coal mine[J]. Measurement Science and Technology, 2023, 35 (1): 015410

[18]	WANG C Y, BOCHKOVSKIY A, LIAO H Y M. YOLOv7: trainable bag-of-freebies sets new state-of-the-art for real-time object detectors [C]// IEEE/CVF Conference on Computer Vision and Pattern Recognition . Vancouver: IEEE, 2023: 7464-7475.

[19]	KRIZHEVSKY A, SUTSKEVER I, HINTON G E ImageNet classification with deep convolutional neural networks[J]. Communications of the ACM, 2017, 60 (6): 84- 90 doi: 10.1145/3065386

[20]	HE K, ZHANG X, REN S, et al. Deep residual learning for image recognition [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . Las Vegas: IEEE, 2016: 770-778.

[21]	HOWARD A G, ZHU M, CHEN B, et al. Mobilenets: efficient convolutional neural networks for mobile vision applications [EB/OL]. (2017-04-17)[2024-05-22]. https://doi.org/10.48550/arXiv.1704.04861.

[22]	ZHANG X, ZHOU X, LIN M, et al. Shufflenet: an extremely efficient convolutional neural network for mobile devices [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . Salt Lake City: IEEE, 2018: 6848-6856.

[23]	NASCIMENTO M G, FAWCETT R, PRISACARIU V A. DSConv: efficient convolution operator [C]// Proceedings of the IEEE/CVF International Conference on Computer Vision . Seoul: IEEE, 2019: 5148-5157.

[24]	HU J, SHEN L, SUN G. Squeeze-and-excitation networks [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . Salt Lake City: IEEE, 2018: 7132-7141.

[25]	WOO S, PARK J, LEE J Y, et al. CBAM: convolutional block attention module [C]// Proceedings of the European Conference on Computer Vision . Munich: Springer, 2018: 3-19.

[26]	LIU Y, SHAO Z. Global attention mechanism: retain information to enhance channel-spatial interactions [EB/OL]. (2021-12-10)[2024-05-22]. https://doi.org/10.48550/arXiv.2112.05561.

[27]	PAN X, GE C, LU R, et al. On the integration of self-attention and convolution [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . Louisiana: IEEE, 2022: 815-825.

[28]	YANG L X, ZHANG R Y, LI L D, et al. SimAM: a simple, parameter-free attention module for convolutional neural networks [C]// Proceedings of the 38th International Conference on Machine Learning . New York: PMLR, 2021: 11863-11874.

[29]	BOCHKOVSKIY A, WANG C Y, LIAO H Y M. YOLOv4: Optimal speed and accuracy of object detection [EB/OL]. (2020-04-23)[2024-05-22]. https://doi.org/10.48550/arXiv.2004.10934.

[30]	REDMON J, DIVVALA S, GIRSHICK R, et al. You only look once: unified, real-time object detection [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . Las Vegas: IEEE, 2016: 779-788.

[31]	REDMON J, FARHADI A. YOLO9000: better, faster, stronger [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . Hawaii State: IEEE, 2017: 7263-7271.

[32]	FARHADI A, RENMON J. YOLOv3: an incremental improvement [C]// Computer Vision and Pattern Recognition. Berlin: Springer, 2018: 1-6.

[33]	REN S, HE K, GIRSHICK R, et al Faster R-CNN: towards real-time object detection with region proposal networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39 (6): 1137- 1149 doi: 10.1109/TPAMI.2016.2577031

[34]	LIU W, ANGUELOV D, ERHAN D, et al. SSD: single shot multibox detector [C]// Proceedings of the European Conference on Computer Vision . Amsterdam: Springer, 2016: 21-37.

[35]	CARION N, MASSA F, SYNNAEVE G, et al. End-to-end object detection with transformers [C]// Proceedings of the European Conference on Computer Vision . Glasgow: Springer, 2020: 213-229.

[36]	ZHOU X, WANG D, KRAHENBUHL P. Objects as points [EB/OL]. (2019-04-25)[2024-05-22]. https://doi.org/10.48550/arXiv.1904.07850.

[1]	Yao ZENG,Fa-qin GAO. Surface defect detection algorithm of electronic components based on improved YOLOv5[J]. Journal of ZheJiang University (Engineering Science), 2023, 57(3): 455-465.

[2]	Ming-kun FENG,Xiang SHI. Image quality assessment with deep pooling of visual feature[J]. Journal of ZheJiang University (Engineering Science), 2019, 53(3): 512-521.

Viewed

Full text

Abstract

Cited

Shared

Discussed