面向煤矿综掘工作面复杂环境的视觉感知系统

doi:10.3785/j.issn.1008-973X.2025.05.013

浙江大学学报(工学版)

2025, Vol. 59

Issue (5): 995-1006 DOI: 10.3785/j.issn.1008-973X.2025.05.013

机械工程

面向煤矿综掘工作面复杂环境的视觉感知系统

苏国用1,2,3(

),胡坤1,2,3,*(

),王鹏彧2,3,赵东洋3,张辉3

1. 安徽理工大学深部煤矿采动响应与灾害防控国家重点实验室，安徽淮南 232001
2. 安徽理工大学矿山智能技术与装备省部共建协同创新中心，安徽淮南 232001
3. 安徽理工大学机电工程学院，安徽淮南 232001

Visual perception system for complex environment of coal mine comprehensive excavation working face

Guoyong SU1,2,3(

),Kun HU1,2,3,*(

),Pengyu WANG2,3,Dongyang ZHAO3,Hui ZHANG3

1. State Key Laboratory of Deep Coal Mining Response and Disaster Prevention and Control, Anhui University of Science and Technology, Huainan 232001, China
2. Collaborative Innovation Center for Mining Intelligent Technology and Equipment, Anhui University of Science and Technology, Huainan 232001, China
3. School of Mechatronics Engineering, Anhui University of Science and Technology, Huainan 232001, China

全文: PDF(4637 KB) HTML

摘要：

针对煤矿恶劣环境下视觉检测算法鲁棒性不足的难题，提出面向煤矿综掘工作面复杂环境的视觉感知系统. 该系统采用ELAN-DS特征提取模块、SimAM注意力模块与解耦检测头对YOLOv7-tiny算法进行优化，构建煤矿综掘工作面视觉检测网络(CMCE-Net). 将CMCE-Net迁移部署到视觉感知终端平台内，测试CMCE-Net在煤矿实际作业工况下的检测性能，基于煤矿综掘工作面数据集开展验证实验. 实验结果表明，CMCE-Net的检测精度达到89.5%，相较于YOLOv7-tiny算法提升了5.2%. 与Faster RCNN、YOLOv7-tiny、YOLOv8s等8种算法相比，综合检测性能最佳，模型复杂度处于较低水平. 在视觉感知终端平台内，CMCE-Net对测试视频的检测速度最高达到33.4 帧/s，在人机多目标混杂工况下，CMCE-Net对装备与人员的检测精度均大于90.0%.

关键词： 综掘工作面; 关键目标; 视觉感知; 检测网络

Abstract:

A visual perception system for the complex environment of the coal mine comprehensive excavation working face was proposed aiming at the problem of insufficient robustness of visual detection algorithms in the harsh environment of coal mines. ELAN-DS feature extraction module, SimAM attention module with decoupled detection head were used to optimize the YOLOv7-tiny algorithm in order to construct the coal mining comprehensive excavation face visual inspection network (CMCE-Net). CMCE-Net was migrated and deployed into the visual perception terminal platform in order to test the detection performance of CMCE-Net under the actual working conditions in coal mines. Validation experiments were conducted based on the data set of coal mine comprehensive excavation working face. The experimental results showed that the detection accuracy of CMCE-Net reached 89.5%, which was a 5.2% improvement compared with the YOLOv7-tiny algorithm. The combined detection performance was the best and the model complexity was at a lower level compared with eight algorithms such as Faster RCNN, YOLOv7-tiny and YOLOv8s. The detection speed of CMCE-Net on the test video reached up to 33.4 frames/s within the visual perception terminal platform, and the detection accuracy of CMCE-Net on the equipment and personnel was more than 90.0% under the human-machine multi-target mixing working condition.

Key words: comprehensive excavation working face critical target visual perception detection network

收稿日期: 2024-03-19 出版日期: 2025-04-25

CLC:

TD 76

基金资助: 安徽省高等学校科学研究资助项目(2022AH050834)；国家自然科学基金资助项目(52304166，52274153)；深部煤矿采动响应与灾害防控国家重点实验室开放基金资助项目(SKLMRDPC22KF24)；安徽理工大学矿山智能技术与装备省部共建协同创新中心开放基金资助项目(CICJMITE202206)；安徽理工大学引进人才科研启动基金资助项目(2022yjrc61).

通讯作者: 胡坤 E-mail: guoyongs005@sina.cn;hk924@126.com

作者简介: 苏国用（1990—），男，讲师，博士，从事煤矿智能感知与控制系统的研究. orcid.org/0000-0001-7202-0922. E-mail：guoyongs005@sina.cn

	服务
	把本文推荐给朋友
	加入引用管理器
	E-mail Alert
	作者相关文章
	苏国用
	胡坤
	王鹏彧
	赵东洋
	张辉

引用本文:

苏国用,胡坤,王鹏彧,赵东洋,张辉. 面向煤矿综掘工作面复杂环境的视觉感知系统[J]. 浙江大学学报(工学版), 2025, 59(5): 995-1006.

Guoyong SU,Kun HU,Pengyu WANG,Dongyang ZHAO,Hui ZHANG. Visual perception system for complex environment of coal mine comprehensive excavation working face. Journal of ZheJiang University (Engineering Science), 2025, 59(5): 995-1006.

链接本文:

https://www.zjujournals.com/eng/CN/10.3785/j.issn.1008-973X.2025.05.013 或 https://www.zjujournals.com/eng/CN/Y2025/V59/I5/995

图 1 煤矿综掘工作面的视觉感知系统架构

图 2 煤矿综掘工作面视觉检测网络的整体结构

图 3 ELAN-DS模块与DSConv结构分解的示意图

图 4 SimAM模块的结构

图 5 耦合与解耦检测头结构的示意图

图 6 煤矿掘进、人工支锚与机载支锚作业场景

图 7 煤矿综掘工作面数据集的标注

图 8 消融实验的mAP0.5曲线图

图 9 模型A与模型B的特征提取模块的输出特征图

图 10 模型B与模型C的热力图

图 11 分类与回归损失曲线

表 1 消融实验结果

表 2 煤矿综掘工作面数据集的对比实验结果

图 12 目标检测算法对比实验的mAP0.5曲线图

表 3 公共数据集的对比实验结果

表 4 CMCE-Net的性能测试结果

图 13 4种YOLO系列检测算法与CMCE-Net的对比测试结果

1	程德强, 寇旗旗, 江鹤, 等全矿井智能视频分析关键技术综述[J]. 工矿自动化, 2023, 49 (11): 1- 21 CHENG Deqiang, KOU Qiqi, JIANG He, et al Overview of key technologies for mine-wide intelligent video analysis[J]. Journal of Mine Automation, 2023, 49 (11): 1- 21
2	程健, 李昊, 马昆, 等矿井视觉计算体系架构与关键技术[J]. 煤炭科学技术, 2023, 51 (9): 202- 218 CHENG Jian, LI Hao, MA Kun, et al Architecture and key technologies of coalmine underground vision computing[J]. Coal Science and Technology, 2023, 51 (9): 202- 218
3	巩师鑫, 赵国瑞, 王飞机器视觉感知理论与技术在煤炭工业领域应用进展综述[J]. 工矿自动化, 2023, 49 (5): 7- 21 GONG Shixin, ZHAO Guorui, WANG Fei Review on the application of machine vision perception theory and technology in coal industry[J]. Journal of Mine Automation, 2023, 49 (5): 7- 21
4	WANG Gongxian, ZHANG Libin, SUN Hui, et al Longitudinal tear detection of conveyor belt under uneven light based on Haar-AdaBoost and Cascade algorithm[J]. Measurement, 2021, 16 (8): 108341
5	李曼, 段雍, 曹现刚, 等煤矸分选机器人图像识别方法和系统[J]. 煤炭学报, 2020, 45 (10): 3636- 3644 LI Man, DUAN Yong, CAO Xiangang, et al Image identification method and system for coal and gangue sorting robot[J]. Journal of China Coal Society, 2020, 45 (10): 3636- 3644
6	王志勇, 李路, 王建, 等基于HOG特征的InSAR矿区开采沉陷盆地检测方法[J]. 中国矿业大学学报, 2021, 50 (2): 404- 410 WANG Zhiyong, LI Lu, WANG Jian, et al A method of detecting the subsidence basin from InSAR interferogram in mining area based on HOG features[J]. Journal of China University of Mining and Technology, 2021, 50 (2): 404- 410
7	WANG Yuanbin, WANG Yujing, DANG Langfei Video detection of foreign objects on the surface of belt conveyor underground coal mine based on improved SSD[J]. Journal of Ambient Intelligence and Humanized Computing, 2023, 14 (5): 5507- 5516 doi: 10.1007/s12652-020-02495-w
8	邹盛, 周李兵, 季亮, 等基于图像融合和改进CornerNet-Squeeze的煤矿井下行人检测方法[J]. 工矿自动化, 2023, 49 (2): 77- 84 ZOU Sheng, ZHOU Libing, JI Liang, et al A pedestrian target detection method for underground coal mine based on image fusion and improved CornerNet-Squeeze[J]. Journal of Mine Automation, 2023, 49 (2): 77- 84
9	ZHENG J H, WANG D Y, GENG Z X Real-time detection of safety hazards in coal mines utilizing an enhanced YOLOv3 algorithm[J]. Traitement du Signal, 2023, 40 (4): 1565- 1572 doi: 10.18280/ts.400424
10	郭永存, 杨豚, 王爽基于改进YOLOv4–Tiny的矿井电机车多目标实时检测[J]. 工程科学与技术, 2023, 55 (5): 232- 241 GUO Yongcun, YANG Tun, WANG Shuang Multi-object real-time detection of mine electric locomotive based on improved YOLOv4-Tiny[J]. Advanced Engineering Sciences, 2023, 55 (5): 232- 241
11	WANG X, WANG S, GUO Y C, et al Multi-scale coal and gangue detection in dense state based on improved Mask RCNN[J]. Measurement, 2023, 22 (1): 113467
12	杨洋 AI智能视频识别分析技术在智能化掘进的研究与应用[J]. 工矿自动化, 2023, 49 (Suppl.1): 26- 28 YANG Yang Research and application of AI intelligent video recognition analysis technology in intelligent excavation[J]. Journal of Mine Automation, 2023, 49 (Suppl.1): 26- 28
13	杜雨馨, 张贺, 王树臣, 等综掘系统视觉处理技术研究现状及发展趋势[J]. 工矿自动化, 2023, 49 (11): 22- 38 DU Yuxin, ZHANG He, WANG Shuchen, et al Research status and development trend of visual processing technology for fully mechanized excavation systems[J]. Journal of Mine Automation, 2023, 49 (11): 22- 38
14	张农, 袁钰鑫, 韩昌良, 等基于Mask R-CNN的煤矿巷道掘进迎头裂隙检测与定位算法[J]. 采矿与安全工程学报, 2023, 40 (5): 925- 932 ZHANG Nong, YUAN Yuxin, HAN Changliang, et al Research on crack detection and localization for advancing face in coal mine roadways based on Mask R-CNN[J]. Journal of Mining and Safety Engineering, 2023, 40 (5): 925- 932
15	SUN S, MA H, WANG K, et al Research on the detection method of coal mine roadway bolt mesh based on improved YOLOv7[J]. Electronics, 2023, 12 (14): 3050 doi: 10.3390/electronics12143050
16	张夫净, 王宏伟, 王浩然, 等煤矿巷道支护钢带锚孔智能识别与定位[J]. 工矿自动化, 2022, 48 (10): 76- 81 ZHANG Fujing, WANG Hongwei, WANG Haoran, et al Intelligent identification and positioning of steel belt anchor hole in coal mine roadway support[J]. Journal of Mine Automation, 2022, 48 (10): 76- 81
17	ZHAO D Y, SU G Y, CHENG G, et al Research on real-time perception method of key targets in the comprehensive excavation working face of coal mine[J]. Measurement Science and Technology, 2023, 35 (1): 015410
18	WANG C Y, BOCHKOVSKIY A, LIAO H Y M. YOLOv7: trainable bag-of-freebies sets new state-of-the-art for real-time object detectors [C]// IEEE/CVF Conference on Computer Vision and Pattern Recognition . Vancouver: IEEE, 2023: 7464-7475.
19	KRIZHEVSKY A, SUTSKEVER I, HINTON G E ImageNet classification with deep convolutional neural networks[J]. Communications of the ACM, 2017, 60 (6): 84- 90 doi: 10.1145/3065386
20	HE K, ZHANG X, REN S, et al. Deep residual learning for image recognition [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . Las Vegas: IEEE, 2016: 770-778.
21	HOWARD A G, ZHU M, CHEN B, et al. Mobilenets: efficient convolutional neural networks for mobile vision applications [EB/OL]. (2017-04-17)[2024-05-22]. https://doi.org/10.48550/arXiv.1704.04861.
22	ZHANG X, ZHOU X, LIN M, et al. Shufflenet: an extremely efficient convolutional neural network for mobile devices [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . Salt Lake City: IEEE, 2018: 6848-6856.
23	NASCIMENTO M G, FAWCETT R, PRISACARIU V A. DSConv: efficient convolution operator [C]// Proceedings of the IEEE/CVF International Conference on Computer Vision . Seoul: IEEE, 2019: 5148-5157.
24	HU J, SHEN L, SUN G. Squeeze-and-excitation networks [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . Salt Lake City: IEEE, 2018: 7132-7141.
25	WOO S, PARK J, LEE J Y, et al. CBAM: convolutional block attention module [C]// Proceedings of the European Conference on Computer Vision . Munich: Springer, 2018: 3-19.
26	LIU Y, SHAO Z. Global attention mechanism: retain information to enhance channel-spatial interactions [EB/OL]. (2021-12-10)[2024-05-22]. https://doi.org/10.48550/arXiv.2112.05561.
27	PAN X, GE C, LU R, et al. On the integration of self-attention and convolution [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . Louisiana: IEEE, 2022: 815-825.
28	YANG L X, ZHANG R Y, LI L D, et al. SimAM: a simple, parameter-free attention module for convolutional neural networks [C]// Proceedings of the 38th International Conference on Machine Learning . New York: PMLR, 2021: 11863-11874.
29	BOCHKOVSKIY A, WANG C Y, LIAO H Y M. YOLOv4: Optimal speed and accuracy of object detection [EB/OL]. (2020-04-23)[2024-05-22]. https://doi.org/10.48550/arXiv.2004.10934.
30	REDMON J, DIVVALA S, GIRSHICK R, et al. You only look once: unified, real-time object detection [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . Las Vegas: IEEE, 2016: 779-788.
31	REDMON J, FARHADI A. YOLO9000: better, faster, stronger [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . Hawaii State: IEEE, 2017: 7263-7271.
32	FARHADI A, RENMON J. YOLOv3: an incremental improvement [C]// Computer Vision and Pattern Recognition. Berlin: Springer, 2018: 1-6.
33	REN S, HE K, GIRSHICK R, et al Faster R-CNN: towards real-time object detection with region proposal networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39 (6): 1137- 1149 doi: 10.1109/TPAMI.2016.2577031
34	LIU W, ANGUELOV D, ERHAN D, et al. SSD: single shot multibox detector [C]// Proceedings of the European Conference on Computer Vision . Amsterdam: Springer, 2016: 21-37.
35	CARION N, MASSA F, SYNNAEVE G, et al. End-to-end object detection with transformers [C]// Proceedings of the European Conference on Computer Vision . Glasgow: Springer, 2020: 213-229.
36	ZHOU X, WANG D, KRAHENBUHL P. Objects as points [EB/OL]. (2019-04-25)[2024-05-22]. https://doi.org/10.48550/arXiv.1904.07850.

[1]	曾耀,高法钦. 基于改进YOLOv5的电子元件表面缺陷检测算法[J]. 浙江大学学报(工学版), 2023, 57(3): 455-465.
[2]	丰明坤,施祥. 视觉特征深度融合的图像质量评价[J]. 浙江大学学报(工学版), 2019, 53(3): 512-521.

Viewed

Full text

Abstract

Cited

Shared

Discussed