1. College of Civil Engineering and Architecture, Zhejiang University, Hangzhou 310058, China 2. Center for Balance Architecture, Zhejiang University, Hangzhou 310028, China
A rebar inspection method based on improved mask region with convolutional neural network (Mask R-CNN) model and stereo vision technology was proposed in order to promote the transformation of reinforcement inspection to intelligence. The improved model Mask R-CNN with channel attention and spatial attention (Mask R-CNN+CA-SA) was formed by adding a bottom-up path with attention mechanism in Mask R-CNN. The diameter and spacing of rebar can be obtained by combining stereo vision technology for coordinate transformation, thereby achieving intelligent rebar inspection. The training was conducted on a self-built dataset containing 3450 rebar pictures. Results showed that the Mask R-CNN+CA-SA model increased the F1 score and mean average precision (mAP) by 2.54% and 2.47% compared with the basic network of Mask R-CNN, respectively. The rebar mesh verification test and complex background test showed that the absolute error and relative error of rebar diameter were basically controlled within 1.7 mm and 10%, and the absolute error and relative error of rebar spacing were controlled within 4 mm and 3.2% respectively. The proposed method is highly operable in practical applications. The intelligent rebar inspection technology can greatly improve work efficiency and reduce labor costs while ensuring sufficient inspection accuracy.
Cuiting WEI,Weijian ZHAO,Bochao SUN,Yunyi LIU. Intelligent rebar inspection based on improved Mask R-CNN and stereo vision. Journal of ZheJiang University (Engineering Science), 2024, 58(5): 1009-1019.
Tab.2Calibration result of intrinsic and extrinsic parameters
Fig.11Input and output data of rebar inspection
Fig.12Visualization results of rebar inspection
位置 编号
Dn/mm
D/mm
Eda/mm
Edr/%
位置 编号
Dn/mm
D/mm
Eda/mm
Edr/%
位置 编号
Dn/mm
D/mm
Eda/mm
Edr/%
x1
20.0
20.6
0.6
3.0
x2
16.0
16.7
0.7
4.4
x3
10.0
10.6
0.6
6.0
x4
10.0
10.7
0.7
7.0
x5
16.0
14.2
?1.7
10.6
x6
20.0
21.5
1.5
7.5
y1
20.0
20.6
0.6
3.0
y2
16.0
16.9
0.9
5.6
y3
10.0
11.2
1.2
12.0
y4
10.0
9.7
?0.3
3.0
y5
16.0
14.3
?1.7
10.6
y6
20.0
20.0
0
0
Tab.3Inspection results of rebar diameter
位置编号
Sr/mm
S/mm
Esa/mm
Esr/%
位置编号
Sr/mm
S/mm
Esa/mm
Esr/%
位置编号
Sr/mm
S/mm
Esa/mm
Esr/%
x1-1
200.0
201.3
1.3
0.7
x2-1
198.0
200.8
2.8
1.4
x3-1
200.0
200.1
0.1
0.1
x1-2
199.0
200.8
1.8
0.9
x2-2
200.0
197.3
?2.7
1.4
x3-2
198.0
198.7
0.7
0.4
x1-3
152.0
152.0
0.0
0.0
x2-3
153.0
154.5
1.5
1.0
x3-3
152.0
154.5
2.5
1.6
x1-4
149.0
149.6
0.6
0.4
x2-4
147.0
149.7
2.7
1.8
x3-4
150.0
153.2
3.2
2.1
x1-5
102.0
102.3
0.3
0.3
x2-5
103.0
106.3
3.3
3.2
x3-5
100.0
102.4
2.4
2.4
x4-1
200.0
200.6
0.6
0.3
x5-1
200.0
200.0
0.0
0.0
x6-1
198.0
199.1
1.1
0.6
x4-2
197.0
196.7
?0.3
0.2
x5-2
196.0
195.5
?0.5
0.3
x6-2
198.0
198.2
0.2
0.1
x4-3
153.0
155.0
2.0
1.3
x5-3
154.0
154.0
0.0
0.0
x6-3
154.0
151.2
?2.8
1.8
x4-4
150.0
150.5
0.5
0.3
x5-4
150.0
151.4
1.4
0.9
x6-4
150.0
151.4
1.4
0.9
x4-5
100.0
102.8
2.8
2.8
x5-5
101.0
101.5
0.5
0.5
x6-5
100.0
102.5
2.5
2.5
y1-1
99.0
99.6
0.6
0.6
y2-1
100.0
97.7
?2.3
2.3
y3-1
100.0
97.1
?2.9
2.9
y1-2
146.0
146.1
0.1
0.1
y2-2
147.0
144.1
?2.9
2.0
y3-2
152.0
154.3
2.3
1.5
y1-3
151.0
152.7
1.7
1.1
y2-3
148.0
149.4
1.4
0.9
y3-3
145.0
144.1
?0.9
0.6
y1-4
203.0
206.0
3.0
1.5
y2-4
205.0
208.8
3.8
1.9
y3-4
203.0
204.7
1.7
0.8
y1-5
200.0
202.4
2.4
1.2
y2-5
200.0
201.8
1.8
0.9
y3-5
200.0
201.8
1.8
0.9
y4-1
103.0
102.3
?0.7
0.7
y5-1
102.0
100.0
?2.0
2.0
y6-1
103.0
105.7
2.7
2.6
y4-2
150.0
149.2
?0.8
0.5
y5-2
150.0
148.3
?1.7
1.1
y6-2
148.0
144.2
?3.8
2.6
y4-3
147.0
149.9
2.9
2.0
y5-3
148.0
148.7
0.7
0.5
y6-3
150.0
150.8
0.8
0.5
y4-4
200.0
199.5
?0.5
0.3
y5-4
198.0
201.5
3.5
1.8
y6-4
198.0
199.4
1.4
0.7
y4-5
203.0
204.6
1.6
0.8
y5-5
202.0
204.3
2.3
1.1
y6-5
204.0
200.1
?3.9
1.9
Tab.4Inspection results of rebar spacing
Fig.13Rebar inspection results in complex background
[1]
孙昊. 基于数字图像的钢筋工程快速检查方法研究[D]. 镇江: 江苏大学, 2019. SUN Hao. Research on fast inspection method of reinforcement based on digital image [D]. Zhenjiang: Jiangsu University, 2019.
[2]
KIM M K, THEDJA J P P, WANG Q Automated dimensional quality assessment for formwork and rebar of reinforced concrete components using 3D point cloud data[J]. Automation in Construction, 2020, 112: 103077
doi: 10.1016/j.autcon.2020.103077
[3]
CHI H L, KIM M K, LIU K Z, et al Rebar inspection integrating augmented reality and laser scanning[J]. Automation in Construction, 2022, 136: 104183
doi: 10.1016/j.autcon.2022.104183
[4]
YANG X, LI X, YE Y, et al Road detection and centerline extraction via deep recurrent convolutional neural network U-Net[J]. IEEE Transactions on Geoscience and Remote Sensing, 2019, 57 (9): 7209- 7220
doi: 10.1109/TGRS.2019.2912301
[5]
PAN Y, ZHANG L Dual attention deep learning network for automatic steel surface defect segmentation[J]. Computer ‐Aided Civil and Infrastructure Engineering, 2022, 37 (11): 1468- 1487
doi: 10.1111/mice.12792
[6]
LI S, ZHAO X A performance improvement strategy for concrete damage detection using stacking ensemble learning of multiple semantic segmentation networks[J]. Sensors, 2022, 22 (9): 3341
doi: 10.3390/s22093341
[7]
姚刚, 廖港, 杨阳, 等. 基于卷积神经网络的预制叠合板多目标智能化检测方法[J/OL]. 土木与环境工程学报(中英文) , 2024, 46(1): 93-101 [2023-06-30]. http://kns.cnki.net/kcms/detail/50.1218.TU.20220423.2103.002.html. YAO Gang, LIAO Gang, YANG Yang, et al. Multi-target intelligent detection method of prefabricated laminated board based on convolutional neural network [J/OL]. Journal of Civil and Environmental Engineering , 2024, 46(1): 93-101[2023-06-30]. http://kns.cnki.net/kcms/detail/50.1218.TU.20220423.2103.002.html.
[8]
董国梁, 张雷, 辛山 基于深度学习的钢筋绑扎机器人目标识别定位[J]. 电子测量技术, 2022, 45 (11): 35- 44 DONG Guoliang, ZHANG Lei, XIN Shan Target recognition and location of steel bar binding robot based on deep learning[J]. Electronic Measurement Technology, 2022, 45 (11): 35- 44
[9]
JIN J, ZHANG W, LI F, et al Robotic binding of rebar based on active perception and planning[J]. Automation in Construction, 2021, 132: 103939
doi: 10.1016/j.autcon.2021.103939
[10]
LI Y, LU Y, CHEN J A deep learning approach for real-time rebar counting on the construction site based on YOLOv3 detector[J]. Automation in Construction, 2021, 124: 103602
doi: 10.1016/j.autcon.2021.103602
[11]
SHIN Y, HEO S, HAN S, et al An image-based steel rebar size estimation and counting method using a convolutional neural network combined with homography[J]. Buildings, 2021, 11 (10): 463
doi: 10.3390/buildings11100463
[12]
REN S, HE K, GIRSHICK R, et al Faster R-CNN: towards real-time object detection with region proposal networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39 (6): 1137- 1149
doi: 10.1109/TPAMI.2016.2577031
[13]
HE K, GEORGIA G, PIOTR D, et al Mask R-CNN[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 42 (2): 386- 397
doi: 10.1109/TPAMI.2018.2844175
[14]
ZHAO K, KANG J, JUNG J, et al. Building extraction from satellite images using Mask R-CNN with building boundary regularization [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . Salt Lake City: IEEE, 2018: 247-251.
[15]
KIM B, CHO S Image-based concrete crack assessment using mask and region-based convolutional neural network[J]. Structural Control and Health Monitoring, 2019, 26 (8): e2381
[16]
XU Y, LI D, XIE Q, et al Automatic defect detection and segmentation of tunnel surface using modified Mask R-CNN[J]. Measurement, 2021, 178: 109316
doi: 10.1016/j.measurement.2021.109316
[17]
NIE S, JIANG Z, ZHANG H, et al. Inshore ship detection based on mask R-CNN [C]// IEEE International Geoscience and Remote Sensing Symposium . Valencia: IEEE, 2018: 693-696.
[18]
LIU S, QI L, QIN H, et al. Path aggregation network for instance segmentation [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . Salt Lake City: IEEE, 2018: 8759-8768.
[19]
HUANG Z, ZHONG Z, SUN L, et al. Mask R-CNN with pyramid attention network for scene text detection [C]// IEEE Winter Conference on Applications of Computer Vision . Waikoloa: IEEE, 2019: 764-772.
[20]
LI Z, ZHOU F. FSSD: Feature fusion single shot multibox detector [EB/OL]. [2023-06-30]. https:// arxiv.org/abs/1712.00960.
[21]
VASWANI A, SHAZEER N, PARMAR N, et al Attention is all you need[J]. Advances in Neural Information Processing Systems, 2017, 30: 5999- 6009
[22]
WOO S, PARK J, LEE J Y, et al. CBAM: convolutional block attention module [C]// Proceedings of the European Conference on Computer Vision . Switzerland: Springer, 2018: 3-19.
[23]
YANG J, REN P, ZHANG D, et al. Neural aggregation network for video face recognition [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . Honolulu: IEEE, 2017: 5216-5225.
[24]
WANG Q, WU T, ZHENG H, et al. Hierarchical pyramid diverse attention networks for face recognition [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway: IEEE, 2020: 8326-8335.
[25]
CHEN Y, DAI X, LIU M, et al. Dynamic convolution: attention over convolution kernels [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . Seattle: IEEE, 2020: 11030-11039.
[26]
XIE S, LIU S, CHEN Z, et al. Attentional ShapeContextNet for point cloud recognition [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . Salt Lake City: IEEE, 2018: 4606-4615.
[27]
ZAARANE A, SLIMANI I, AL OKAISHI W, et al Distance measurement system for autonomous vehicles using stereo camera[J]. Array, 2020, 5: 100016
doi: 10.1016/j.array.2020.100016
[28]
PEPE M, FREGONESE L, SCAIONI M Planning airborne photogrammetry and remote-sensing missions with modern platforms and sensors[J]. European Journal of Remote Sensing, 2018, 51 (1): 412- 436
doi: 10.1080/22797254.2018.1444945
[29]
TUOMINEN S, NÄSI R, HONKAVAARA E, et al Assessment of classifiers and remote sensing features of hyperspectral imagery and stereo-photogrammetric point clouds for recognition of tree species in a forest area of high species diversity[J]. Remote Sensing, 2018, 10 (5): 714
doi: 10.3390/rs10050714
[30]
LI D, XU L, TANG X, et al 3D imaging of greenhouse plants with an inexpensive binocular stereo vision system[J]. Remote Sensing, 2017, 9 (5): 508
doi: 10.3390/rs9050508
[31]
ZHOU Y, GALLEGO G, REBECQ H, et al. Semi-dense 3D reconstruction with a stereo event camera [C]// Proceedings of the European Conference on Computer Vision . Munich: Springer, 2018: 235-251.
