Please wait a minute...
浙江大学学报(工学版)  2024, Vol. 58 Issue (5): 1009-1019    DOI: 10.3785/j.issn.1008-973X.2024.05.014
交通工程、土木工程     
基于改进Mask R-CNN与双目视觉的智能配筋检测
魏翠婷1(),赵唯坚1,2,孙博超1,2,*(),刘芸怡1
1. 浙江大学 建筑工程学院,浙江 杭州 310058
2. 浙江大学 平衡建筑研究中心,浙江 杭州 310028
Intelligent rebar inspection based on improved Mask R-CNN and stereo vision
Cuiting WEI1(),Weijian ZHAO1,2,Bochao SUN1,2,*(),Yunyi LIU1
1. College of Civil Engineering and Architecture, Zhejiang University, Hangzhou 310058, China
2. Center for Balance Architecture, Zhejiang University, Hangzhou 310028, China
 全文: PDF(6948 KB)   HTML
摘要:

为了提高配筋检测的智能化水平,提出基于改进掩膜区域卷积神经网络(Mask R-CNN)模型与双目视觉技术的配筋检测方法. 通过在Mask R-CNN中加入自下而上的注意力机制路径,形成了带通道注意力和空间注意力的掩膜区域卷积神经网络(Mask R-CNN+CA-SA)改进模型. 结合双目视觉技术进行坐标转换,获取钢筋直径与间距,实现智能配筋检测. 在自建的包含3 450 张钢筋图片的数据集上进行训练,结果表明,改进模型的F1分数和全类平均精确率(mAP)相较于Mask R-CNN基础网络分别提高了2.54%和2.47%. 通过钢筋网验证试验和复杂背景测试,钢筋直径的绝对误差和相对误差基本小于1.7 mm和10%,钢筋间距的绝对误差和相对误差分别小于4 mm和3.2%,所提方法在实际应用中具有较强的可操作性. 智能配筋检测技术在保证足够的检测精度的同时,能够大大提升工效,降低人工成本.

关键词: 配筋质量检测Mask R-CNN注意力机制深度学习双目视觉技术    
Abstract:

A rebar inspection method based on improved mask region with convolutional neural network (Mask R-CNN) model and stereo vision technology was proposed in order to promote the transformation of reinforcement inspection to intelligence. The improved model Mask R-CNN with channel attention and spatial attention (Mask R-CNN+CA-SA) was formed by adding a bottom-up path with attention mechanism in Mask R-CNN. The diameter and spacing of rebar can be obtained by combining stereo vision technology for coordinate transformation, thereby achieving intelligent rebar inspection. The training was conducted on a self-built dataset containing 3450 rebar pictures. Results showed that the Mask R-CNN+CA-SA model increased the F1 score and mean average precision (mAP) by 2.54% and 2.47% compared with the basic network of Mask R-CNN, respectively. The rebar mesh verification test and complex background test showed that the absolute error and relative error of rebar diameter were basically controlled within 1.7 mm and 10%, and the absolute error and relative error of rebar spacing were controlled within 4 mm and 3.2% respectively. The proposed method is highly operable in practical applications. The intelligent rebar inspection technology can greatly improve work efficiency and reduce labor costs while ensuring sufficient inspection accuracy.

Key words: rebar quality inspection    Mask R-CNN    attention mechanism    deep learning    stereo vision technology
收稿日期: 2023-08-09 出版日期: 2024-04-26
CLC:  TP 391  
基金资助: 国家自然科学基金资助项目(52108254, 52208215).
通讯作者: 孙博超     E-mail: 22112262@zju.edu.cn;sunbochao@zju.edu.cn
作者简介: 魏翠婷(1999—),女,硕士生,从事智能建造的研究. orcid.org/0009-0003-6749-7509. E-mail:22112262@zju.edu.cn
服务  
把本文推荐给朋友
加入引用管理器
E-mail Alert
作者相关文章  
魏翠婷
赵唯坚
孙博超
刘芸怡

引用本文:

魏翠婷,赵唯坚,孙博超,刘芸怡. 基于改进Mask R-CNN与双目视觉的智能配筋检测[J]. 浙江大学学报(工学版), 2024, 58(5): 1009-1019.

Cuiting WEI,Weijian ZHAO,Bochao SUN,Yunyi LIU. Intelligent rebar inspection based on improved Mask R-CNN and stereo vision. Journal of ZheJiang University (Engineering Science), 2024, 58(5): 1009-1019.

链接本文:

https://www.zjujournals.com/eng/CN/10.3785/j.issn.1008-973X.2024.05.014        https://www.zjujournals.com/eng/CN/Y2024/V58/I5/1009

图 1  Mask R-CNN网络结构
图 2  通道注意力机制
图 3  空间注意力机制
图 4  自下而上的注意力机制路径
图 5  RealSense相机的深度测量原理图
图 6  智能配筋检测方法的实现流程
图 7  部分数据集图片
图 8  数据增强示意图
模型PF1mIoUDice损失mAP
Mask R-CNN+CA-SA93.8994.9789.0312.2687.62
Mask R-CNN+Soft-NMS91.3492.1987.4312.9886.02
Mask R-CNN90.3392.4386.4313.3285.15
U-Net89.7391.5086.2214.3484.53
DeepLabV3+91.9591.6986.7513.9685.07
PSPNet87.1288.1381.4616.5579.44
表 1  不同模型的评价指标对比
图 9  不同模型的F1和mAP曲线
图 10  基于Mask R-CNN+CA-SA模型的钢筋检测结果
参数数值
内参矩阵$\left[ {\begin{array}{*{20}{c}} {910.575}&{\text{0}}&{637.795} \\ {\text{0}}&{908.432}&{380.199} \\ 0&{\text{0}}&{\text{1}} \end{array}} \right]$
旋转矩阵$\left[ {\begin{array}{*{20}{c}} {{\text{0}}{\text{.999\;981}}}&{{\text{0}}{\text{.005\;517\;8}}}&{{\text{0}}{\text{.002\;588\;0}}} \\ {{{ - 0}}{\text{.005\;515\;7}}}&{{\text{0}}{\text{.999\;984}}}&{{{ - 0}}{\text{.000\;819\;70}}} \\ {{{ - 0}}{\text{.002\;592\;44}}}&{{\text{0}}{\text{.000\;805\;408}}}&{{\text{0}}{\text{.999\;996}}} \end{array}} \right]$
平移矩阵$\left[ {\begin{array}{*{20}{c}} {{\text{ 0}}{\text{.014\;838\;3}}}&{{\text{0}}{\text{.000\;191\;953}}}&{{\text{0}}{\text{.000\;010\;653\;8}}} \end{array}} \right]$
表 2  内、外参数的标定结果
图 11  配筋检测的输入与输出数据
图 12  配筋检测可视化结果
位置
编号
Dn/mmD/mmEda/mmEdr/%位置
编号
Dn/mmD/mmEda/mmEdr/%位置
编号
Dn/mmD/mmEda/mmEdr/%
x120.020.60.63.0x216.016.70.74.4x310.010.60.66.0
x410.010.70.77.0x516.014.2?1.710.6x620.021.51.57.5
y120.020.60.63.0y216.016.90.95.6y310.011.21.212.0
y410.09.7?0.33.0y516.014.3?1.710.6y620.020.000
表 3  钢筋直径检测结果
位置编号Sr/mmS/mmEsa/mmEsr/%位置编号Sr/mmS/mmEsa/mmEsr/%位置编号Sr/mmS/mmEsa/mmEsr/%
x1-1200.0201.31.30.7x2-1198.0200.82.81.4x3-1200.0200.10.10.1
x1-2199.0200.81.80.9x2-2200.0197.3?2.71.4x3-2198.0198.70.70.4
x1-3152.0152.00.00.0x2-3153.0154.51.51.0x3-3152.0154.52.51.6
x1-4149.0149.60.60.4x2-4147.0149.72.71.8x3-4150.0153.23.22.1
x1-5102.0102.30.30.3x2-5103.0106.33.33.2x3-5100.0102.42.42.4
x4-1200.0200.60.60.3x5-1200.0200.00.00.0x6-1198.0199.11.10.6
x4-2197.0196.7?0.30.2x5-2196.0195.5?0.50.3x6-2198.0198.20.20.1
x4-3153.0155.02.01.3x5-3154.0154.00.00.0x6-3154.0151.2?2.81.8
x4-4150.0150.50.50.3x5-4150.0151.41.40.9x6-4150.0151.41.40.9
x4-5100.0102.82.82.8x5-5101.0101.50.50.5x6-5100.0102.52.52.5
y1-199.099.60.60.6y2-1100.097.7?2.32.3y3-1100.097.1?2.92.9
y1-2146.0146.10.10.1y2-2147.0144.1?2.92.0y3-2152.0154.32.31.5
y1-3151.0152.71.71.1y2-3148.0149.41.40.9y3-3145.0144.1?0.90.6
y1-4203.0206.03.01.5y2-4205.0208.83.81.9y3-4203.0204.71.70.8
y1-5200.0202.42.41.2y2-5200.0201.81.80.9y3-5200.0201.81.80.9
y4-1103.0102.3?0.70.7y5-1102.0100.0?2.02.0y6-1103.0105.72.72.6
y4-2150.0149.2?0.80.5y5-2150.0148.3?1.71.1y6-2148.0144.2?3.82.6
y4-3147.0149.92.92.0y5-3148.0148.70.70.5y6-3150.0150.80.80.5
y4-4200.0199.5?0.50.3y5-4198.0201.53.51.8y6-4198.0199.41.40.7
y4-5203.0204.61.60.8y5-5202.0204.32.31.1y6-5204.0200.1?3.91.9
表 4  钢筋间距检测结果
图 13  复杂背景配筋检测结果
1 孙昊. 基于数字图像的钢筋工程快速检查方法研究[D]. 镇江: 江苏大学, 2019.
SUN Hao. Research on fast inspection method of reinforcement based on digital image [D]. Zhenjiang: Jiangsu University, 2019.
2 KIM M K, THEDJA J P P, WANG Q Automated dimensional quality assessment for formwork and rebar of reinforced concrete components using 3D point cloud data[J]. Automation in Construction, 2020, 112: 103077
doi: 10.1016/j.autcon.2020.103077
3 CHI H L, KIM M K, LIU K Z, et al Rebar inspection integrating augmented reality and laser scanning[J]. Automation in Construction, 2022, 136: 104183
doi: 10.1016/j.autcon.2022.104183
4 YANG X, LI X, YE Y, et al Road detection and centerline extraction via deep recurrent convolutional neural network U-Net[J]. IEEE Transactions on Geoscience and Remote Sensing, 2019, 57 (9): 7209- 7220
doi: 10.1109/TGRS.2019.2912301
5 PAN Y, ZHANG L Dual attention deep learning network for automatic steel surface defect segmentation[J]. Computer ‐Aided Civil and Infrastructure Engineering, 2022, 37 (11): 1468- 1487
doi: 10.1111/mice.12792
6 LI S, ZHAO X A performance improvement strategy for concrete damage detection using stacking ensemble learning of multiple semantic segmentation networks[J]. Sensors, 2022, 22 (9): 3341
doi: 10.3390/s22093341
7 姚刚, 廖港, 杨阳, 等. 基于卷积神经网络的预制叠合板多目标智能化检测方法[J/OL]. 土木与环境工程学报(中英文) , 2024, 46(1): 93-101 [2023-06-30]. http://kns.cnki.net/kcms/detail/50.1218.TU.20220423.2103.002.html.
YAO Gang, LIAO Gang, YANG Yang, et al. Multi-target intelligent detection method of prefabricated laminated board based on convolutional neural network [J/OL]. Journal of Civil and Environmental Engineering , 2024, 46(1): 93-101[2023-06-30]. http://kns.cnki.net/kcms/detail/50.1218.TU.20220423.2103.002.html.
8 董国梁, 张雷, 辛山 基于深度学习的钢筋绑扎机器人目标识别定位[J]. 电子测量技术, 2022, 45 (11): 35- 44
DONG Guoliang, ZHANG Lei, XIN Shan Target recognition and location of steel bar binding robot based on deep learning[J]. Electronic Measurement Technology, 2022, 45 (11): 35- 44
9 JIN J, ZHANG W, LI F, et al Robotic binding of rebar based on active perception and planning[J]. Automation in Construction, 2021, 132: 103939
doi: 10.1016/j.autcon.2021.103939
10 LI Y, LU Y, CHEN J A deep learning approach for real-time rebar counting on the construction site based on YOLOv3 detector[J]. Automation in Construction, 2021, 124: 103602
doi: 10.1016/j.autcon.2021.103602
11 SHIN Y, HEO S, HAN S, et al An image-based steel rebar size estimation and counting method using a convolutional neural network combined with homography[J]. Buildings, 2021, 11 (10): 463
doi: 10.3390/buildings11100463
12 REN S, HE K, GIRSHICK R, et al Faster R-CNN: towards real-time object detection with region proposal networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39 (6): 1137- 1149
doi: 10.1109/TPAMI.2016.2577031
13 HE K, GEORGIA G, PIOTR D, et al Mask R-CNN[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 42 (2): 386- 397
doi: 10.1109/TPAMI.2018.2844175
14 ZHAO K, KANG J, JUNG J, et al. Building extraction from satellite images using Mask R-CNN with building boundary regularization [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . Salt Lake City: IEEE, 2018: 247-251.
15 KIM B, CHO S Image-based concrete crack assessment using mask and region-based convolutional neural network[J]. Structural Control and Health Monitoring, 2019, 26 (8): e2381
16 XU Y, LI D, XIE Q, et al Automatic defect detection and segmentation of tunnel surface using modified Mask R-CNN[J]. Measurement, 2021, 178: 109316
doi: 10.1016/j.measurement.2021.109316
17 NIE S, JIANG Z, ZHANG H, et al. Inshore ship detection based on mask R-CNN [C]// IEEE International Geoscience and Remote Sensing Symposium . Valencia: IEEE, 2018: 693-696.
18 LIU S, QI L, QIN H, et al. Path aggregation network for instance segmentation [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . Salt Lake City: IEEE, 2018: 8759-8768.
19 HUANG Z, ZHONG Z, SUN L, et al. Mask R-CNN with pyramid attention network for scene text detection [C]// IEEE Winter Conference on Applications of Computer Vision . Waikoloa: IEEE, 2019: 764-772.
20 LI Z, ZHOU F. FSSD: Feature fusion single shot multibox detector [EB/OL]. [2023-06-30]. https:// arxiv.org/abs/1712.00960.
21 VASWANI A, SHAZEER N, PARMAR N, et al Attention is all you need[J]. Advances in Neural Information Processing Systems, 2017, 30: 5999- 6009
22 WOO S, PARK J, LEE J Y, et al. CBAM: convolutional block attention module [C]// Proceedings of the European Conference on Computer Vision . Switzerland: Springer, 2018: 3-19.
23 YANG J, REN P, ZHANG D, et al. Neural aggregation network for video face recognition [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . Honolulu: IEEE, 2017: 5216-5225.
24 WANG Q, WU T, ZHENG H, et al. Hierarchical pyramid diverse attention networks for face recognition [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway: IEEE, 2020: 8326-8335.
25 CHEN Y, DAI X, LIU M, et al. Dynamic convolution: attention over convolution kernels [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . Seattle: IEEE, 2020: 11030-11039.
26 XIE S, LIU S, CHEN Z, et al. Attentional ShapeContextNet for point cloud recognition [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . Salt Lake City: IEEE, 2018: 4606-4615.
27 ZAARANE A, SLIMANI I, AL OKAISHI W, et al Distance measurement system for autonomous vehicles using stereo camera[J]. Array, 2020, 5: 100016
doi: 10.1016/j.array.2020.100016
28 PEPE M, FREGONESE L, SCAIONI M Planning airborne photogrammetry and remote-sensing missions with modern platforms and sensors[J]. European Journal of Remote Sensing, 2018, 51 (1): 412- 436
doi: 10.1080/22797254.2018.1444945
29 TUOMINEN S, NÄSI R, HONKAVAARA E, et al Assessment of classifiers and remote sensing features of hyperspectral imagery and stereo-photogrammetric point clouds for recognition of tree species in a forest area of high species diversity[J]. Remote Sensing, 2018, 10 (5): 714
doi: 10.3390/rs10050714
30 LI D, XU L, TANG X, et al 3D imaging of greenhouse plants with an inexpensive binocular stereo vision system[J]. Remote Sensing, 2017, 9 (5): 508
doi: 10.3390/rs9050508
31 ZHOU Y, GALLEGO G, REBECQ H, et al. Semi-dense 3D reconstruction with a stereo event camera [C]// Proceedings of the European Conference on Computer Vision . Munich: Springer, 2018: 235-251.
[1] 邢志伟,朱书杰,李彪. 基于改进图卷积神经网络的航空行李特征感知[J]. 浙江大学学报(工学版), 2024, 58(5): 941-950.
[2] 刘毅,陈一丹,高琳,洪姣. 基于多尺度特征融合的轻量化道路提取模型[J]. 浙江大学学报(工学版), 2024, 58(5): 951-959.
[3] 钟博,王鹏飞,王乙乔,王晓玲. 基于深度学习的EEG数据分析技术综述[J]. 浙江大学学报(工学版), 2024, 58(5): 879-890.
[4] 宦海,盛宇,顾晨曦. 基于遥感图像道路提取的全局指导多特征融合网络[J]. 浙江大学学报(工学版), 2024, 58(4): 696-707.
[5] 罗向龙,王亚飞,王彦博,王立新. 基于双向门控式宽度学习系统的监测数据结构变形预测[J]. 浙江大学学报(工学版), 2024, 58(4): 729-736.
[6] 宋明俊,严文,邓益昭,张俊然,涂海燕. 轻量化机器人抓取位姿实时检测算法[J]. 浙江大学学报(工学版), 2024, 58(3): 599-610.
[7] 钱庆杰,余军合,战洪飞,王瑞,胡健. 基于DL-BiGRU多特征融合的注塑件尺寸预测方法[J]. 浙江大学学报(工学版), 2024, 58(3): 646-654.
[8] 姚鑫骅,于涛,封森文,马梓健,栾丛丛,沈洪垚. 基于图神经网络的零件机加工特征识别方法[J]. 浙江大学学报(工学版), 2024, 58(2): 349-359.
[9] 秦思怡,盖绍彦,达飞鹏. 混合采样下多级特征聚合的视频目标检测算法[J]. 浙江大学学报(工学版), 2024, 58(1): 10-19.
[10] 孙雪菲,张瑞峰,关欣,李锵. 强化先验骨架结构的轻量型高效人体姿态估计[J]. 浙江大学学报(工学版), 2024, 58(1): 50-60.
[11] 郑超昊,尹志伟,曾钢锋,许月萍,周鹏,刘莉. 基于时空深度学习模型的数值降水预报后处理[J]. 浙江大学学报(工学版), 2023, 57(9): 1756-1765.
[12] 李海烽,张雪英,段淑斐,贾海蓉,Huizhi Liang . 融合生成对抗网络与时间卷积网络的普通话情感识别[J]. 浙江大学学报(工学版), 2023, 57(9): 1865-1875.
[13] 赵小强,王泽,宋昭漾,蒋红梅. 基于动态注意力网络的图像超分辨率重建[J]. 浙江大学学报(工学版), 2023, 57(8): 1487-1494.
[14] 王慧欣,童向荣. 融合知识图谱的推荐系统研究进展[J]. 浙江大学学报(工学版), 2023, 57(8): 1527-1540.
[15] 宋秀兰,董兆航,单杭冠,陆炜杰. 基于时空融合的多头注意力车辆轨迹预测[J]. 浙江大学学报(工学版), 2023, 57(8): 1636-1643.