基于改进Mask R-CNN与双目视觉的智能配筋检测

doi:10.3785/j.issn.1008-973X.2024.05.014

浙江大学学报(工学版)

2024, Vol. 58

Issue (5): 1009-1019 DOI: 10.3785/j.issn.1008-973X.2024.05.014

交通工程、土木工程

基于改进Mask R-CNN与双目视觉的智能配筋检测

魏翠婷1(

),赵唯坚1,2,孙博超1,2,*(

),刘芸怡1

1. 浙江大学建筑工程学院，浙江杭州 310058
2. 浙江大学平衡建筑研究中心，浙江杭州 310028

Intelligent rebar inspection based on improved Mask R-CNN and stereo vision

Cuiting WEI1(

),Weijian ZHAO1,2,Bochao SUN1,2,*(

),Yunyi LIU1

1. College of Civil Engineering and Architecture, Zhejiang University, Hangzhou 310058, China
2. Center for Balance Architecture, Zhejiang University, Hangzhou 310028, China

全文: PDF(6948 KB) HTML

摘要：

为了提高配筋检测的智能化水平，提出基于改进掩膜区域卷积神经网络（Mask R-CNN）模型与双目视觉技术的配筋检测方法. 通过在Mask R-CNN中加入自下而上的注意力机制路径，形成了带通道注意力和空间注意力的掩膜区域卷积神经网络（Mask R-CNN+CA-SA）改进模型. 结合双目视觉技术进行坐标转换，获取钢筋直径与间距，实现智能配筋检测. 在自建的包含3 450 张钢筋图片的数据集上进行训练，结果表明，改进模型的F1分数和全类平均精确率（mAP）相较于Mask R-CNN基础网络分别提高了2.54%和2.47%. 通过钢筋网验证试验和复杂背景测试，钢筋直径的绝对误差和相对误差基本小于1.7 mm和10%，钢筋间距的绝对误差和相对误差分别小于4 mm和3.2%，所提方法在实际应用中具有较强的可操作性. 智能配筋检测技术在保证足够的检测精度的同时，能够大大提升工效，降低人工成本.

关键词： 配筋质量检测; Mask R-CNN; 注意力机制; 深度学习; 双目视觉技术

Abstract:

A rebar inspection method based on improved mask region with convolutional neural network (Mask R-CNN) model and stereo vision technology was proposed in order to promote the transformation of reinforcement inspection to intelligence. The improved model Mask R-CNN with channel attention and spatial attention (Mask R-CNN+CA-SA) was formed by adding a bottom-up path with attention mechanism in Mask R-CNN. The diameter and spacing of rebar can be obtained by combining stereo vision technology for coordinate transformation, thereby achieving intelligent rebar inspection. The training was conducted on a self-built dataset containing 3450 rebar pictures. Results showed that the Mask R-CNN+CA-SA model increased the F1 score and mean average precision (mAP) by 2.54% and 2.47% compared with the basic network of Mask R-CNN, respectively. The rebar mesh verification test and complex background test showed that the absolute error and relative error of rebar diameter were basically controlled within 1.7 mm and 10%, and the absolute error and relative error of rebar spacing were controlled within 4 mm and 3.2% respectively. The proposed method is highly operable in practical applications. The intelligent rebar inspection technology can greatly improve work efficiency and reduce labor costs while ensuring sufficient inspection accuracy.

Key words: rebar quality inspection Mask R-CNN attention mechanism deep learning stereo vision technology

收稿日期: 2023-08-09 出版日期: 2024-04-26

CLC:

TP 391

基金资助: 国家自然科学基金资助项目（52108254, 52208215）.

通讯作者: 孙博超 E-mail: 22112262@zju.edu.cn;sunbochao@zju.edu.cn

作者简介: 魏翠婷（1999—），女，硕士生，从事智能建造的研究. orcid.org/0009-0003-6749-7509. E-mail：22112262@zju.edu.cn

	服务
	把本文推荐给朋友
	加入引用管理器
	E-mail Alert
	作者相关文章
	魏翠婷
	赵唯坚
	孙博超
	刘芸怡

引用本文:

魏翠婷,赵唯坚,孙博超,刘芸怡. 基于改进Mask R-CNN与双目视觉的智能配筋检测[J]. 浙江大学学报(工学版), 2024, 58(5): 1009-1019.

Cuiting WEI,Weijian ZHAO,Bochao SUN,Yunyi LIU. Intelligent rebar inspection based on improved Mask R-CNN and stereo vision. Journal of ZheJiang University (Engineering Science), 2024, 58(5): 1009-1019.

链接本文:

https://www.zjujournals.com/eng/CN/10.3785/j.issn.1008-973X.2024.05.014 或 https://www.zjujournals.com/eng/CN/Y2024/V58/I5/1009

图 1 Mask R-CNN网络结构

图 2 通道注意力机制

图 3 空间注意力机制

图 4 自下而上的注意力机制路径

图 5 RealSense相机的深度测量原理图

图 6 智能配筋检测方法的实现流程

图 7 部分数据集图片

图 8 数据增强示意图

表 1 不同模型的评价指标对比

图 9 不同模型的F1和mAP曲线

图 10 基于Mask R-CNN+CA-SA模型的钢筋检测结果

表 2 内、外参数的标定结果

图 11 配筋检测的输入与输出数据

图 12 配筋检测可视化结果

表 3 钢筋直径检测结果

表 4 钢筋间距检测结果

图 13 复杂背景配筋检测结果

1	孙昊. 基于数字图像的钢筋工程快速检查方法研究[D]. 镇江: 江苏大学, 2019. SUN Hao. Research on fast inspection method of reinforcement based on digital image [D]. Zhenjiang: Jiangsu University, 2019.
2	KIM M K, THEDJA J P P, WANG Q Automated dimensional quality assessment for formwork and rebar of reinforced concrete components using 3D point cloud data[J]. Automation in Construction, 2020, 112: 103077 doi: 10.1016/j.autcon.2020.103077
3	CHI H L, KIM M K, LIU K Z, et al Rebar inspection integrating augmented reality and laser scanning[J]. Automation in Construction, 2022, 136: 104183 doi: 10.1016/j.autcon.2022.104183
4	YANG X, LI X, YE Y, et al Road detection and centerline extraction via deep recurrent convolutional neural network U-Net[J]. IEEE Transactions on Geoscience and Remote Sensing, 2019, 57 (9): 7209- 7220 doi: 10.1109/TGRS.2019.2912301
5	PAN Y, ZHANG L Dual attention deep learning network for automatic steel surface defect segmentation[J]. Computer ‐Aided Civil and Infrastructure Engineering, 2022, 37 (11): 1468- 1487 doi: 10.1111/mice.12792
6	LI S, ZHAO X A performance improvement strategy for concrete damage detection using stacking ensemble learning of multiple semantic segmentation networks[J]. Sensors, 2022, 22 (9): 3341 doi: 10.3390/s22093341
7	姚刚, 廖港, 杨阳, 等. 基于卷积神经网络的预制叠合板多目标智能化检测方法[J/OL]. 土木与环境工程学报(中英文) , 2024, 46(1): 93-101 [2023-06-30]. http://kns.cnki.net/kcms/detail/50.1218.TU.20220423.2103.002.html. YAO Gang, LIAO Gang, YANG Yang, et al. Multi-target intelligent detection method of prefabricated laminated board based on convolutional neural network [J/OL]. Journal of Civil and Environmental Engineering , 2024, 46(1): 93-101[2023-06-30]. http://kns.cnki.net/kcms/detail/50.1218.TU.20220423.2103.002.html.
8	董国梁, 张雷, 辛山基于深度学习的钢筋绑扎机器人目标识别定位[J]. 电子测量技术, 2022, 45 (11): 35- 44 DONG Guoliang, ZHANG Lei, XIN Shan Target recognition and location of steel bar binding robot based on deep learning[J]. Electronic Measurement Technology, 2022, 45 (11): 35- 44
9	JIN J, ZHANG W, LI F, et al Robotic binding of rebar based on active perception and planning[J]. Automation in Construction, 2021, 132: 103939 doi: 10.1016/j.autcon.2021.103939
10	LI Y, LU Y, CHEN J A deep learning approach for real-time rebar counting on the construction site based on YOLOv3 detector[J]. Automation in Construction, 2021, 124: 103602 doi: 10.1016/j.autcon.2021.103602
11	SHIN Y, HEO S, HAN S, et al An image-based steel rebar size estimation and counting method using a convolutional neural network combined with homography[J]. Buildings, 2021, 11 (10): 463 doi: 10.3390/buildings11100463
12	REN S, HE K, GIRSHICK R, et al Faster R-CNN: towards real-time object detection with region proposal networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39 (6): 1137- 1149 doi: 10.1109/TPAMI.2016.2577031
13	HE K, GEORGIA G, PIOTR D, et al Mask R-CNN[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 42 (2): 386- 397 doi: 10.1109/TPAMI.2018.2844175
14	ZHAO K, KANG J, JUNG J, et al. Building extraction from satellite images using Mask R-CNN with building boundary regularization [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . Salt Lake City: IEEE, 2018: 247-251.
15	KIM B, CHO S Image-based concrete crack assessment using mask and region-based convolutional neural network[J]. Structural Control and Health Monitoring, 2019, 26 (8): e2381
16	XU Y, LI D, XIE Q, et al Automatic defect detection and segmentation of tunnel surface using modified Mask R-CNN[J]. Measurement, 2021, 178: 109316 doi: 10.1016/j.measurement.2021.109316
17	NIE S, JIANG Z, ZHANG H, et al. Inshore ship detection based on mask R-CNN [C]// IEEE International Geoscience and Remote Sensing Symposium . Valencia: IEEE, 2018: 693-696.
18	LIU S, QI L, QIN H, et al. Path aggregation network for instance segmentation [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . Salt Lake City: IEEE, 2018: 8759-8768.
19	HUANG Z, ZHONG Z, SUN L, et al. Mask R-CNN with pyramid attention network for scene text detection [C]// IEEE Winter Conference on Applications of Computer Vision . Waikoloa: IEEE, 2019: 764-772.
20	LI Z, ZHOU F. FSSD: Feature fusion single shot multibox detector [EB/OL]. [2023-06-30]. https:// arxiv.org/abs/1712.00960.
21	VASWANI A, SHAZEER N, PARMAR N, et al Attention is all you need[J]. Advances in Neural Information Processing Systems, 2017, 30: 5999- 6009
22	WOO S, PARK J, LEE J Y, et al. CBAM: convolutional block attention module [C]// Proceedings of the European Conference on Computer Vision . Switzerland: Springer, 2018: 3-19.
23	YANG J, REN P, ZHANG D, et al. Neural aggregation network for video face recognition [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . Honolulu: IEEE, 2017: 5216-5225.
24	WANG Q, WU T, ZHENG H, et al. Hierarchical pyramid diverse attention networks for face recognition [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway: IEEE, 2020: 8326-8335.
25	CHEN Y, DAI X, LIU M, et al. Dynamic convolution: attention over convolution kernels [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . Seattle: IEEE, 2020: 11030-11039.
26	XIE S, LIU S, CHEN Z, et al. Attentional ShapeContextNet for point cloud recognition [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . Salt Lake City: IEEE, 2018: 4606-4615.
27	ZAARANE A, SLIMANI I, AL OKAISHI W, et al Distance measurement system for autonomous vehicles using stereo camera[J]. Array, 2020, 5: 100016 doi: 10.1016/j.array.2020.100016
28	PEPE M, FREGONESE L, SCAIONI M Planning airborne photogrammetry and remote-sensing missions with modern platforms and sensors[J]. European Journal of Remote Sensing, 2018, 51 (1): 412- 436 doi: 10.1080/22797254.2018.1444945
29	TUOMINEN S, NÄSI R, HONKAVAARA E, et al Assessment of classifiers and remote sensing features of hyperspectral imagery and stereo-photogrammetric point clouds for recognition of tree species in a forest area of high species diversity[J]. Remote Sensing, 2018, 10 (5): 714 doi: 10.3390/rs10050714
30	LI D, XU L, TANG X, et al 3D imaging of greenhouse plants with an inexpensive binocular stereo vision system[J]. Remote Sensing, 2017, 9 (5): 508 doi: 10.3390/rs9050508
31	ZHOU Y, GALLEGO G, REBECQ H, et al. Semi-dense 3D reconstruction with a stereo event camera [C]// Proceedings of the European Conference on Computer Vision . Munich: Springer, 2018: 235-251.

[1]	邢志伟,朱书杰,李彪. 基于改进图卷积神经网络的航空行李特征感知[J]. 浙江大学学报(工学版), 2024, 58(5): 941-950.
[2]	刘毅,陈一丹,高琳,洪姣. 基于多尺度特征融合的轻量化道路提取模型[J]. 浙江大学学报(工学版), 2024, 58(5): 951-959.
[3]	钟博,王鹏飞,王乙乔,王晓玲. 基于深度学习的EEG数据分析技术综述[J]. 浙江大学学报(工学版), 2024, 58(5): 879-890.
[4]	宦海,盛宇,顾晨曦. 基于遥感图像道路提取的全局指导多特征融合网络[J]. 浙江大学学报(工学版), 2024, 58(4): 696-707.
[5]	罗向龙,王亚飞,王彦博,王立新. 基于双向门控式宽度学习系统的监测数据结构变形预测[J]. 浙江大学学报(工学版), 2024, 58(4): 729-736.
[6]	宋明俊,严文,邓益昭,张俊然,涂海燕. 轻量化机器人抓取位姿实时检测算法[J]. 浙江大学学报(工学版), 2024, 58(3): 599-610.
[7]	钱庆杰,余军合,战洪飞,王瑞,胡健. 基于DL-BiGRU多特征融合的注塑件尺寸预测方法[J]. 浙江大学学报(工学版), 2024, 58(3): 646-654.
[8]	姚鑫骅,于涛,封森文,马梓健,栾丛丛,沈洪垚. 基于图神经网络的零件机加工特征识别方法[J]. 浙江大学学报(工学版), 2024, 58(2): 349-359.
[9]	秦思怡,盖绍彦,达飞鹏. 混合采样下多级特征聚合的视频目标检测算法[J]. 浙江大学学报(工学版), 2024, 58(1): 10-19.
[10]	孙雪菲,张瑞峰,关欣,李锵. 强化先验骨架结构的轻量型高效人体姿态估计[J]. 浙江大学学报(工学版), 2024, 58(1): 50-60.
[11]	郑超昊,尹志伟,曾钢锋,许月萍,周鹏,刘莉. 基于时空深度学习模型的数值降水预报后处理[J]. 浙江大学学报(工学版), 2023, 57(9): 1756-1765.
[12]	李海烽,张雪英,段淑斐,贾海蓉,Huizhi Liang . 融合生成对抗网络与时间卷积网络的普通话情感识别[J]. 浙江大学学报(工学版), 2023, 57(9): 1865-1875.
[13]	赵小强,王泽,宋昭漾,蒋红梅. 基于动态注意力网络的图像超分辨率重建[J]. 浙江大学学报(工学版), 2023, 57(8): 1487-1494.
[14]	王慧欣,童向荣. 融合知识图谱的推荐系统研究进展[J]. 浙江大学学报(工学版), 2023, 57(8): 1527-1540.
[15]	宋秀兰,董兆航,单杭冠,陆炜杰. 基于时空融合的多头注意力车辆轨迹预测[J]. 浙江大学学报(工学版), 2023, 57(8): 1636-1643.

Viewed

Full text

Abstract

Cited

Shared

Discussed