Please wait a minute...
Journal of ZheJiang University (Engineering Science)  2024, Vol. 58 Issue (5): 1009-1019    DOI: 10.3785/j.issn.1008-973X.2024.05.014
    
Intelligent rebar inspection based on improved Mask R-CNN and stereo vision
Cuiting WEI1(),Weijian ZHAO1,2,Bochao SUN1,2,*(),Yunyi LIU1
1. College of Civil Engineering and Architecture, Zhejiang University, Hangzhou 310058, China
2. Center for Balance Architecture, Zhejiang University, Hangzhou 310028, China
Download: HTML     PDF(6948KB) HTML
Export: BibTeX | EndNote (RIS)      

Abstract  

A rebar inspection method based on improved mask region with convolutional neural network (Mask R-CNN) model and stereo vision technology was proposed in order to promote the transformation of reinforcement inspection to intelligence. The improved model Mask R-CNN with channel attention and spatial attention (Mask R-CNN+CA-SA) was formed by adding a bottom-up path with attention mechanism in Mask R-CNN. The diameter and spacing of rebar can be obtained by combining stereo vision technology for coordinate transformation, thereby achieving intelligent rebar inspection. The training was conducted on a self-built dataset containing 3450 rebar pictures. Results showed that the Mask R-CNN+CA-SA model increased the F1 score and mean average precision (mAP) by 2.54% and 2.47% compared with the basic network of Mask R-CNN, respectively. The rebar mesh verification test and complex background test showed that the absolute error and relative error of rebar diameter were basically controlled within 1.7 mm and 10%, and the absolute error and relative error of rebar spacing were controlled within 4 mm and 3.2% respectively. The proposed method is highly operable in practical applications. The intelligent rebar inspection technology can greatly improve work efficiency and reduce labor costs while ensuring sufficient inspection accuracy.



Key wordsrebar quality inspection      Mask R-CNN      attention mechanism      deep learning      stereo vision technology     
Received: 09 August 2023      Published: 26 April 2024
CLC:  TP 391  
Fund:  国家自然科学基金资助项目(52108254, 52208215).
Corresponding Authors: Bochao SUN     E-mail: 22112262@zju.edu.cn;sunbochao@zju.edu.cn
Cite this article:

Cuiting WEI,Weijian ZHAO,Bochao SUN,Yunyi LIU. Intelligent rebar inspection based on improved Mask R-CNN and stereo vision. Journal of ZheJiang University (Engineering Science), 2024, 58(5): 1009-1019.

URL:

https://www.zjujournals.com/eng/10.3785/j.issn.1008-973X.2024.05.014     OR     https://www.zjujournals.com/eng/Y2024/V58/I5/1009


基于改进Mask R-CNN与双目视觉的智能配筋检测

为了提高配筋检测的智能化水平,提出基于改进掩膜区域卷积神经网络(Mask R-CNN)模型与双目视觉技术的配筋检测方法. 通过在Mask R-CNN中加入自下而上的注意力机制路径,形成了带通道注意力和空间注意力的掩膜区域卷积神经网络(Mask R-CNN+CA-SA)改进模型. 结合双目视觉技术进行坐标转换,获取钢筋直径与间距,实现智能配筋检测. 在自建的包含3 450 张钢筋图片的数据集上进行训练,结果表明,改进模型的F1分数和全类平均精确率(mAP)相较于Mask R-CNN基础网络分别提高了2.54%和2.47%. 通过钢筋网验证试验和复杂背景测试,钢筋直径的绝对误差和相对误差基本小于1.7 mm和10%,钢筋间距的绝对误差和相对误差分别小于4 mm和3.2%,所提方法在实际应用中具有较强的可操作性. 智能配筋检测技术在保证足够的检测精度的同时,能够大大提升工效,降低人工成本.


关键词: 配筋质量检测,  Mask R-CNN,  注意力机制,  深度学习,  双目视觉技术 
Fig.1 Network structure of Mask R-CNN
Fig.2 Channel attention mechanism
Fig.3 Spatial attention mechanism
Fig.4 Bottom-up path with attention mechanism module
Fig.5 Schematic diagram for depth measurement by RealSense
Fig.6 Flow chart of intelligent rebar inspection method
Fig.7 Partial pictures in dataset
Fig.8 Schematic diagram of data enhancement
模型PF1mIoUDice损失mAP
Mask R-CNN+CA-SA93.8994.9789.0312.2687.62
Mask R-CNN+Soft-NMS91.3492.1987.4312.9886.02
Mask R-CNN90.3392.4386.4313.3285.15
U-Net89.7391.5086.2214.3484.53
DeepLabV3+91.9591.6986.7513.9685.07
PSPNet87.1288.1381.4616.5579.44
Tab.1 Evaluation index comparison of different models
Fig.9 F1 and mAP curves of different models
Fig.10 Inspection result of rebars based on Mask R-CNN +CA-SA model
参数数值
内参矩阵$\left[ {\begin{array}{*{20}{c}} {910.575}&{\text{0}}&{637.795} \\ {\text{0}}&{908.432}&{380.199} \\ 0&{\text{0}}&{\text{1}} \end{array}} \right]$
旋转矩阵$\left[ {\begin{array}{*{20}{c}} {{\text{0}}{\text{.999\;981}}}&{{\text{0}}{\text{.005\;517\;8}}}&{{\text{0}}{\text{.002\;588\;0}}} \\ {{{ - 0}}{\text{.005\;515\;7}}}&{{\text{0}}{\text{.999\;984}}}&{{{ - 0}}{\text{.000\;819\;70}}} \\ {{{ - 0}}{\text{.002\;592\;44}}}&{{\text{0}}{\text{.000\;805\;408}}}&{{\text{0}}{\text{.999\;996}}} \end{array}} \right]$
平移矩阵$\left[ {\begin{array}{*{20}{c}} {{\text{ 0}}{\text{.014\;838\;3}}}&{{\text{0}}{\text{.000\;191\;953}}}&{{\text{0}}{\text{.000\;010\;653\;8}}} \end{array}} \right]$
Tab.2 Calibration result of intrinsic and extrinsic parameters
Fig.11 Input and output data of rebar inspection
Fig.12 Visualization results of rebar inspection
位置
编号
Dn/mmD/mmEda/mmEdr/%位置
编号
Dn/mmD/mmEda/mmEdr/%位置
编号
Dn/mmD/mmEda/mmEdr/%
x120.020.60.63.0x216.016.70.74.4x310.010.60.66.0
x410.010.70.77.0x516.014.2?1.710.6x620.021.51.57.5
y120.020.60.63.0y216.016.90.95.6y310.011.21.212.0
y410.09.7?0.33.0y516.014.3?1.710.6y620.020.000
Tab.3 Inspection results of rebar diameter
位置编号Sr/mmS/mmEsa/mmEsr/%位置编号Sr/mmS/mmEsa/mmEsr/%位置编号Sr/mmS/mmEsa/mmEsr/%
x1-1200.0201.31.30.7x2-1198.0200.82.81.4x3-1200.0200.10.10.1
x1-2199.0200.81.80.9x2-2200.0197.3?2.71.4x3-2198.0198.70.70.4
x1-3152.0152.00.00.0x2-3153.0154.51.51.0x3-3152.0154.52.51.6
x1-4149.0149.60.60.4x2-4147.0149.72.71.8x3-4150.0153.23.22.1
x1-5102.0102.30.30.3x2-5103.0106.33.33.2x3-5100.0102.42.42.4
x4-1200.0200.60.60.3x5-1200.0200.00.00.0x6-1198.0199.11.10.6
x4-2197.0196.7?0.30.2x5-2196.0195.5?0.50.3x6-2198.0198.20.20.1
x4-3153.0155.02.01.3x5-3154.0154.00.00.0x6-3154.0151.2?2.81.8
x4-4150.0150.50.50.3x5-4150.0151.41.40.9x6-4150.0151.41.40.9
x4-5100.0102.82.82.8x5-5101.0101.50.50.5x6-5100.0102.52.52.5
y1-199.099.60.60.6y2-1100.097.7?2.32.3y3-1100.097.1?2.92.9
y1-2146.0146.10.10.1y2-2147.0144.1?2.92.0y3-2152.0154.32.31.5
y1-3151.0152.71.71.1y2-3148.0149.41.40.9y3-3145.0144.1?0.90.6
y1-4203.0206.03.01.5y2-4205.0208.83.81.9y3-4203.0204.71.70.8
y1-5200.0202.42.41.2y2-5200.0201.81.80.9y3-5200.0201.81.80.9
y4-1103.0102.3?0.70.7y5-1102.0100.0?2.02.0y6-1103.0105.72.72.6
y4-2150.0149.2?0.80.5y5-2150.0148.3?1.71.1y6-2148.0144.2?3.82.6
y4-3147.0149.92.92.0y5-3148.0148.70.70.5y6-3150.0150.80.80.5
y4-4200.0199.5?0.50.3y5-4198.0201.53.51.8y6-4198.0199.41.40.7
y4-5203.0204.61.60.8y5-5202.0204.32.31.1y6-5204.0200.1?3.91.9
Tab.4 Inspection results of rebar spacing
Fig.13 Rebar inspection results in complex background
[1]   孙昊. 基于数字图像的钢筋工程快速检查方法研究[D]. 镇江: 江苏大学, 2019.
SUN Hao. Research on fast inspection method of reinforcement based on digital image [D]. Zhenjiang: Jiangsu University, 2019.
[2]   KIM M K, THEDJA J P P, WANG Q Automated dimensional quality assessment for formwork and rebar of reinforced concrete components using 3D point cloud data[J]. Automation in Construction, 2020, 112: 103077
doi: 10.1016/j.autcon.2020.103077
[3]   CHI H L, KIM M K, LIU K Z, et al Rebar inspection integrating augmented reality and laser scanning[J]. Automation in Construction, 2022, 136: 104183
doi: 10.1016/j.autcon.2022.104183
[4]   YANG X, LI X, YE Y, et al Road detection and centerline extraction via deep recurrent convolutional neural network U-Net[J]. IEEE Transactions on Geoscience and Remote Sensing, 2019, 57 (9): 7209- 7220
doi: 10.1109/TGRS.2019.2912301
[5]   PAN Y, ZHANG L Dual attention deep learning network for automatic steel surface defect segmentation[J]. Computer ‐Aided Civil and Infrastructure Engineering, 2022, 37 (11): 1468- 1487
doi: 10.1111/mice.12792
[6]   LI S, ZHAO X A performance improvement strategy for concrete damage detection using stacking ensemble learning of multiple semantic segmentation networks[J]. Sensors, 2022, 22 (9): 3341
doi: 10.3390/s22093341
[7]   姚刚, 廖港, 杨阳, 等. 基于卷积神经网络的预制叠合板多目标智能化检测方法[J/OL]. 土木与环境工程学报(中英文) , 2024, 46(1): 93-101 [2023-06-30]. http://kns.cnki.net/kcms/detail/50.1218.TU.20220423.2103.002.html.
YAO Gang, LIAO Gang, YANG Yang, et al. Multi-target intelligent detection method of prefabricated laminated board based on convolutional neural network [J/OL]. Journal of Civil and Environmental Engineering , 2024, 46(1): 93-101[2023-06-30]. http://kns.cnki.net/kcms/detail/50.1218.TU.20220423.2103.002.html.
[8]   董国梁, 张雷, 辛山 基于深度学习的钢筋绑扎机器人目标识别定位[J]. 电子测量技术, 2022, 45 (11): 35- 44
DONG Guoliang, ZHANG Lei, XIN Shan Target recognition and location of steel bar binding robot based on deep learning[J]. Electronic Measurement Technology, 2022, 45 (11): 35- 44
[9]   JIN J, ZHANG W, LI F, et al Robotic binding of rebar based on active perception and planning[J]. Automation in Construction, 2021, 132: 103939
doi: 10.1016/j.autcon.2021.103939
[10]   LI Y, LU Y, CHEN J A deep learning approach for real-time rebar counting on the construction site based on YOLOv3 detector[J]. Automation in Construction, 2021, 124: 103602
doi: 10.1016/j.autcon.2021.103602
[11]   SHIN Y, HEO S, HAN S, et al An image-based steel rebar size estimation and counting method using a convolutional neural network combined with homography[J]. Buildings, 2021, 11 (10): 463
doi: 10.3390/buildings11100463
[12]   REN S, HE K, GIRSHICK R, et al Faster R-CNN: towards real-time object detection with region proposal networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39 (6): 1137- 1149
doi: 10.1109/TPAMI.2016.2577031
[13]   HE K, GEORGIA G, PIOTR D, et al Mask R-CNN[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 42 (2): 386- 397
doi: 10.1109/TPAMI.2018.2844175
[14]   ZHAO K, KANG J, JUNG J, et al. Building extraction from satellite images using Mask R-CNN with building boundary regularization [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . Salt Lake City: IEEE, 2018: 247-251.
[15]   KIM B, CHO S Image-based concrete crack assessment using mask and region-based convolutional neural network[J]. Structural Control and Health Monitoring, 2019, 26 (8): e2381
[16]   XU Y, LI D, XIE Q, et al Automatic defect detection and segmentation of tunnel surface using modified Mask R-CNN[J]. Measurement, 2021, 178: 109316
doi: 10.1016/j.measurement.2021.109316
[17]   NIE S, JIANG Z, ZHANG H, et al. Inshore ship detection based on mask R-CNN [C]// IEEE International Geoscience and Remote Sensing Symposium . Valencia: IEEE, 2018: 693-696.
[18]   LIU S, QI L, QIN H, et al. Path aggregation network for instance segmentation [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . Salt Lake City: IEEE, 2018: 8759-8768.
[19]   HUANG Z, ZHONG Z, SUN L, et al. Mask R-CNN with pyramid attention network for scene text detection [C]// IEEE Winter Conference on Applications of Computer Vision . Waikoloa: IEEE, 2019: 764-772.
[20]   LI Z, ZHOU F. FSSD: Feature fusion single shot multibox detector [EB/OL]. [2023-06-30]. https:// arxiv.org/abs/1712.00960.
[21]   VASWANI A, SHAZEER N, PARMAR N, et al Attention is all you need[J]. Advances in Neural Information Processing Systems, 2017, 30: 5999- 6009
[22]   WOO S, PARK J, LEE J Y, et al. CBAM: convolutional block attention module [C]// Proceedings of the European Conference on Computer Vision . Switzerland: Springer, 2018: 3-19.
[23]   YANG J, REN P, ZHANG D, et al. Neural aggregation network for video face recognition [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . Honolulu: IEEE, 2017: 5216-5225.
[24]   WANG Q, WU T, ZHENG H, et al. Hierarchical pyramid diverse attention networks for face recognition [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway: IEEE, 2020: 8326-8335.
[25]   CHEN Y, DAI X, LIU M, et al. Dynamic convolution: attention over convolution kernels [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . Seattle: IEEE, 2020: 11030-11039.
[26]   XIE S, LIU S, CHEN Z, et al. Attentional ShapeContextNet for point cloud recognition [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . Salt Lake City: IEEE, 2018: 4606-4615.
[27]   ZAARANE A, SLIMANI I, AL OKAISHI W, et al Distance measurement system for autonomous vehicles using stereo camera[J]. Array, 2020, 5: 100016
doi: 10.1016/j.array.2020.100016
[28]   PEPE M, FREGONESE L, SCAIONI M Planning airborne photogrammetry and remote-sensing missions with modern platforms and sensors[J]. European Journal of Remote Sensing, 2018, 51 (1): 412- 436
doi: 10.1080/22797254.2018.1444945
[29]   TUOMINEN S, NÄSI R, HONKAVAARA E, et al Assessment of classifiers and remote sensing features of hyperspectral imagery and stereo-photogrammetric point clouds for recognition of tree species in a forest area of high species diversity[J]. Remote Sensing, 2018, 10 (5): 714
doi: 10.3390/rs10050714
[30]   LI D, XU L, TANG X, et al 3D imaging of greenhouse plants with an inexpensive binocular stereo vision system[J]. Remote Sensing, 2017, 9 (5): 508
doi: 10.3390/rs9050508
[31]   ZHOU Y, GALLEGO G, REBECQ H, et al. Semi-dense 3D reconstruction with a stereo event camera [C]// Proceedings of the European Conference on Computer Vision . Munich: Springer, 2018: 235-251.
[1] Zhiwei XING,Shujie ZHU,Biao LI. Airline baggage feature perception based on improved graph convolutional neural network[J]. Journal of ZheJiang University (Engineering Science), 2024, 58(5): 941-950.
[2] Yi LIU,Yidan CHEN,Lin GAO,Jiao HONG. Lightweight road extraction model based on multi-scale feature fusion[J]. Journal of ZheJiang University (Engineering Science), 2024, 58(5): 951-959.
[3] Bo ZHONG,Pengfei WANG,Yiqiao WANG,Xiaoling WANG. Survey of deep learning based EEG data analysis technology[J]. Journal of ZheJiang University (Engineering Science), 2024, 58(5): 879-890.
[4] Hai HUAN,Yu SHENG,Chenxi GU. Global guidance multi-feature fusion network based on remote sensing image road extraction[J]. Journal of ZheJiang University (Engineering Science), 2024, 58(4): 696-707.
[5] Xianglong LUO,Yafei WANG,Yanbo WANG,Lixin WANG. Structural deformation prediction of monitoring data based on bi-directional gate board learning system[J]. Journal of ZheJiang University (Engineering Science), 2024, 58(4): 729-736.
[6] Mingjun SONG,Wen YAN,Yizhao DENG,Junran ZHANG,Haiyan TU. Light-weight algorithm for real-time robotic grasp detection[J]. Journal of ZheJiang University (Engineering Science), 2024, 58(3): 599-610.
[7] Qingjie QIAN,Junhe YU,Hongfei ZHAN,Rui WANG,Jian HU. Dimension prediction method of injection molded parts based on multi-feature fusion of DL-BiGRU[J]. Journal of ZheJiang University (Engineering Science), 2024, 58(3): 646-654.
[8] Xinhua YAO,Tao YU,Senwen FENG,Zijian MA,Congcong LUAN,Hongyao SHEN. Recognition method of parts machining features based on graph neural network[J]. Journal of ZheJiang University (Engineering Science), 2024, 58(2): 349-359.
[9] Siyi QIN,Shaoyan GAI,Feipeng DA. Video object detection algorithm based on multi-level feature aggregation under mixed sampler[J]. Journal of ZheJiang University (Engineering Science), 2024, 58(1): 10-19.
[10] Xuefei SUN,Ruifeng ZHANG,Xin GUAN,Qiang LI. Lightweight and efficient human pose estimation with enhanced priori skeleton structure[J]. Journal of ZheJiang University (Engineering Science), 2024, 58(1): 50-60.
[11] Chao-hao ZHENG,Zhi-wei YIN,Gang-feng ZENG,Yue-ping XU,Peng ZHOU,Li LIU. Post-processing of numerical precipitation forecast based on spatial-temporal deep learning model[J]. Journal of ZheJiang University (Engineering Science), 2023, 57(9): 1756-1765.
[12] Hai-feng LI,Xue-ying ZHANG,Shu-fei DUAN,Hai-rong JIA,Hui-zhi LIANG. Fusing generative adversarial network and temporal convolutional network for Mandarin emotion recognition[J]. Journal of ZheJiang University (Engineering Science), 2023, 57(9): 1865-1875.
[13] Xiao-qiang ZHAO,Ze WANG,Zhao-yang SONG,Hong-mei JIANG. Image super-resolution reconstruction based on dynamic attention network[J]. Journal of ZheJiang University (Engineering Science), 2023, 57(8): 1487-1494.
[14] Hui-xin WANG,Xiang-rong TONG. Research progress of recommendation system based on knowledge graph[J]. Journal of ZheJiang University (Engineering Science), 2023, 57(8): 1527-1540.
[15] Xiu-lan SONG,Zhao-hang DONG,Hang-guan SHAN,Wei-jie LU. Vehicle trajectory prediction based on temporal-spatial multi-head attention mechanism[J]. Journal of ZheJiang University (Engineering Science), 2023, 57(8): 1636-1643.