Please wait a minute...
浙江大学学报(工学版)
Journal of ZheJiang University (Engineering Science)  2023, Vol. 57 Issue (1): 47-54    DOI: 10.3785/j.issn.1008-973X.2023.01.005
    
Position measurement method for tunnel segment grabbing based on RGB and depth information fusion
Lin-tao WANG(),Qi MAO
School of Mechanical Engineering, Dalian University of Technology, Dalian 116024, China
Download: HTML     PDF(3034KB) HTML
Export: BibTeX | EndNote (RIS)      

Abstract  

A position measurement method which fits automatic segment assembly to grab segment bolt was proposed in order to solve the problem that grabbing segments relied on manual labor. SIFT algorithm was used to match the target segment bolts, and Faster-Rcnn algorithm was used to choose the target segment placed in working area. Convolutional attention block module was implemented to change the structure of feature extracting network, which maintained the recognition accuracy around 94% under stricter IOU (intersection over union) of 0.8 and prevented the affect of other segment bolts. Information from depth camera was fused to get the complete three-dimensional coordination after target segment bolt being recognized. The error of measurement in each axis was less than 3 mm when grabbing facility was in position, which meeted the requirement of grabbing with mechanical hoisting facility. The segment bolt was directly recognized. Then the error caused by segment placement uncertainty need not be considered. The relative error between segment and target object and the manual and time cost of setting target object can be prevented.

Key words： SIFT feature matching      Faster-Rcnn algorithm      attention mechanism      information fusion      segment bolt position measurement     
Received: 26 January 2022      Published: 17 January 2023
CLC:  TP 212  
Service
E-mail this article
Add to my bookshelf
Add to citation manager
E-mail Alert
RSS
Articles by authors
Lin-tao WANG
Qi MAO
Cite this article:

Lin-tao WANG,Qi MAO. Position measurement method for tunnel segment grabbing based on RGB and depth information fusion. Journal of ZheJiang University (Engineering Science), 2023, 57(1): 47-54.

URL:

https://www.zjujournals.com/eng/10.3785/j.issn.1008-973X.2023.01.005     OR     https://www.zjujournals.com/eng/Y2023/V57/I1/47


基于RGB与深度信息融合的管片抓取位置测量方法

针对管片拼装机在管片抓取阶段依赖人工的问题，提出用于自动化拼装的管片螺栓抓取阶段的位置测量方法. 该方案通过SIFT算法匹配管片螺栓，利用Faster-Rcnn算法，筛选出位于工作区的待抓取的管片螺栓. 通过添加注意力机制改变特征提取网络结构，使其在0.8的交并比下保持约94%的准确率并排除其他管片螺栓的识别干扰. 在识别到目标管片螺栓后，结合深度相机的信息获取完整的三维坐标，使得抓取设备位于抓取位置时测量的管片螺栓位置各轴的误差均不超过3 mm，满足机械式抓取装置对抓取的精度要求. 直接对管片螺栓进行识别，无须考虑管片在工作区摆放位置的不确定性造成的误差，避免了使用靶标进行测量时靶标与管片之间的相对位置误差及设置靶标的人力与时间成本.


关键词： SIFT特征匹配,  Faster-Rcnn算法,  注意力机制,  信息融合,  管片螺栓位置测量 
Fig.1 Process of feature matching of segment bolt
Fig.2 Predicting process of Faster-Rcnn algorithm
Fig.3 Mismatching of more than 1 segment bolts under 0.5 IOU
Fig.4 Structure of convolutional block attention module
Fig.5 Structure of Identity Block after adding attention module
Fig.6 Process of position measurement of segment bolt based on information fusion
Fig.7 Registration of RGB camera and depth camera
Fig.8 Strategy of segment images collection for training set
Fig.9 Influence of attention module on accuracy of test set under different IOU conditions
Fig.10 Comparison of Grad-Cam heat map between before and after adding attention module
Fig.11 Segment grabbing and locking facility
序号 坐标实际值/mm 坐标测量值/mm 各轴误差/mm
1 (?150,260,535) (?151.315, 259.504, 533.467) (1.315, 0.496, 1.533)
2 (300,180,710) (299.407, 178.901, 707.661) (0.593, 1.099, 2.339)
3 (?350,340,605) (?348.159, 342.073, 606.517) (?1.841, ?2.073, ?1.517)
4 (550,420,570) (552.542, 419.404, 570.113) (?2.542, 0.596, ?0.113)
5 (?400,220,745) (?399.669, 221.963, 747.673) (?0.331, ?1.963, ?2.673)
6 (200,140,780) (199.867, 139.417, 782.735) (0.133, 0.583, ?2.735)
7 (500,380,675) (497.884, 381.682, 672.139) (2.116, ?1.682, 2.861)
8 (?250,100,500) (?247.947, 98.754, 502.227) (?2.053, 1.246, ?2.227)
9 (450,300,640) (448.682, 300.676, 642.410) (1.318, ?0.676, ?2.410)
Tab.1 Result of position measurement of segment bolt
[1]   WADA M Automatic segment erection system for shield tunnels[J]. Advanced Robotics, 1990, 5 (4): 429- 443
doi: 10.1163/156855391X00304
[2]   张碧, 赵海峰, 杨涛, 等 盾构管片拼装机国内外研究现状[J]. 矿山机械, 2014, (4): 1- 6
ZHANG Bi, ZHAO Hai-feng, YANG Tao, et al Research status of segment erector in shield tunneling machine at home and abroad[J]. Mining and Processing Equipment, 2014, (4): 1- 6
[3]   HIRASAWA Y, HASHIMOTO H, WORI Y, et al. Development of automatic segment assembly robot for shield tunnelling machine [C]// BRILAKIS I. Proceedings of International Symposium on Automation and Robotics in Construction. Tokyo: [s. n. ], 1996: 813−820.
[4]   TANAKA Y Automatic segment assembly robot for shield tunneling machine[J]. Computer-Aided Civil and Infrastructure Engineering, 1995, 10 (5): 325- 337
doi: 10.1111/j.1467-8667.1995.tb00295.x
[5]   刘飞香 管片拼装机抓取和拼装智能化研究[J]. 铁道建筑, 2020, 60 (8): 58- 63
LIU Fei-xiang Study on intelligent grab and assemble of tunnel segment erecting robot[J]. Railway Engineering, 2020, 60 (8): 58- 63
[6]   朱凯源. 针对管片自动拼装的视觉测量技术[D]. 大连: 大连理工大学, 2020.
ZHU Kai-yuan. Vision measurement technology for automatic assembly of segments [D]. Dalian: Dalian University of Technology, 2020.
[7]   张龙, 王海波, 范曙远 盾构机管片拼装自动控制传感检测系统的设计[J]. 机械制造, 2018, 56 (2): 74- 77
ZHANG Long, WANG Hai-bo, FAN Shu-yuan Design of automatic control, sensing and detection system for segment assembly of shield machine[J]. Machinery, 2018, 56 (2): 74- 77
[8]   高翔, 陶建峰, 王明斗, 等 基于线激光传感器的盾构管片位姿检测方法[J]. 中南大学学报: 自然科学版, 2020, 51 (1): 41- 48
GAO Xiang, TAO Jian-feng, WANG Ming-dou, et al Position and posture detection method of shield segments using line laser sensors[J]. Journal of Central South University: Science and Technology, 2020, 51 (1): 41- 48
[9]   MAKOTO S, FUMIO T, TERUYUKI M, et al. Assembly method for erector controller and lining work member: Japan, 2004131979A [P]. 2004−04−30.
[10]   李穗婷. 盾构机管片拼装机器人与视觉系统研究[D]. 广州: 华南理工大学, 2018.
LI Sui-ting. Research on segment robot and vision system for shield machine [D]. Guangzhou: South China University of Technology, 2018.
[11]   REDMON J, DIVVALA S, GIRSHICK R, et al. You only look once: unified, real-time object detection [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. [S. l. ]: IEEE, 2016: 779-788.
[12]   REN S, HE K, GIRSHICK R, et al Faster R-CNN: towards real-time object detection with region proposal networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39 (6): 1137- 1149
doi: 10.1109/TPAMI.2016.2577031
[13]   RONNEBERGER O, FISCHER P, BROX T. U-net: convolutional networks for biomedical image segmentation [C]// Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention. Cham: Springer, 2015: 234-241.
[14]   朵建华, 杨柏钟 基于多传感器融合的轨道识别方法探究[J]. 现代城市轨道交通, 2021, (9): 98- 102
DUO Jian-hua, YANG Bai-zhong Recognition method of railway tracks based on multi-sensor fusion[J]. Modern Urban Transit, 2021, (9): 98- 102
[15]   李国瑞, 罗维, 李雨来, 等 基于传感器融合的障碍物轮廓识别方法研究[J]. 生物医学工程与临床, 2021, 25 (4): 395- 400
LI Guo-rui, LUO Wei, LI Yu-lai, et al Obstacle contour recognition method based on sensor fusion[J]. Biomedical Engineering and Clinical Medicine, 2021, 25 (4): 395- 400
[16]   LOWE D Distinctive image features from scale-invariant keypoints[J]. International Journal of Computer Vision, 2004, 60 (2): 91- 110
doi: 10.1023/B:VISI.0000029664.99615.94
[17]   WOO S, PARK J, LEE J Y, et al. CBAM: convolutional block attention module [C]// European Conference on Computer Vision. Cham: Springer, 2018.
[18]   ZHANG Z A flexible new technique for camera calibration[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2000, 22 (11): 1330- 1334
doi: 10.1109/34.888718
[19]   SELVARAJU R R, COGSWEL M, DAS A, et al Grad-CAM: visual explanations from deep networks via gradient-based localization[J]. International Journal of Computer Vision, 2020, 128 (2): 336- 359
doi: 10.1007/s11263-019-01228-7
[20]   陈艳 盾构机管片吊运系统技术综合分析[J]. 科学技术创新, 2019, (5): 185- 186
CHEN Yan Comprehensive analysis of segment lifting system technology of shield machine[J]. Scientific and Technological Innovation, 2019, (5): 185- 186
[21]   乔万良, 刘子铭. 一种盾构机管片抓取轴锁紧装置: CN207989053U [P]. 2018-10-19.
QIAO Wan-liang, LIU Zi-ming. A shield machine segment grab shaft locking device: CN207989053U [P]. 2018-10-19.
[1] Kun HAO,Kuo WANG,Bei-bei WANG. Lightweight underwater biological detection algorithm based on improved Mobilenet-YOLOv3[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(8): 1622-1632.
[2] Ren-peng MO,Xiao-sheng SI,Tian-mei LI,Xu ZHU. Bearing life prediction based on multi-scale features and attention mechanism[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(7): 1447-1456.
[3] You-wei WANG,Shuang TONG,Li-zhou FENG,Jian-ming ZHU,Yang LI,Fu CHEN. New inductive microblog rumor detection method based on graph convolutional network[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(5): 956-966.
[4] Xiao-chen JU,Xin-xin ZHAO,Sheng-sheng QIAN. Self-attention mechanism based bridge bolt detection algorithm[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(5): 901-908.
[5] Xue-qin ZHANG,Tian-ren LI. Breast cancer pathological image classification based on Cycle-GAN and improved DPN network[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(4): 727-735.
[6] Meng XU,Dan WANG,Zhi-yuan LI,Yuan-fang CHEN. IncepA-EEGNet: P300 signal detection method based on fusion of Inception network and attention mechanism[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(4): 745-753, 782.
[7] Chang-yuan LIU,Xian-ping HE,Xiao-jun BI. Efficient network vehicle recognition combined with attention mechanism[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(4): 775-782.
[8] Qiao-hong CHEN,Hao-lei PEI,Qi SUN. Image caption based on relational reasoning and context gate mechanism[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(3): 542-549.
[9] Yuan-jun NONG,Jun-jie WANG,Hong CHEN,Wen-han SUN,Hui GENG,Shu-yue LI. A image caption method of construction scene based on attention mechanism and encoding-decoding architecture[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(2): 236-244.
[10] Ying-li LIU,Rui-gang WU,Chang-hui YAO,Tao SHEN. Construction method of extraction dataset of Al-Si alloy entity relationship[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(2): 245-253.
[11] Tian-le YUAN,Ju-long YUAN,Yong-jian ZHU,Han-chen ZHENG. Surface defect detection algorithm of thrust ball bearing based on improved YOLOv5[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(12): 2349-2357.
[12] Nan-jing YU,Xiao-biao FAN,Tian-min DENG,Guo-tao MAO. Ship detection algorithm in complex backgrounds via multi-head self-attention[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(12): 2392-2402.
[13] Qiao-hong CHEN,Fei-yu LI,Qi SUN,Yu-bo JIA. Answer selection model based on LSTM and decay self-attention[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(12): 2436-2444.
[14] Fei LI,Kun HU,Yong ZHANG,Wen-shan WANG,Hao JIANG. Multi-dimensional detection of longitudinal tearing of conveyor belt based on YOLOv4 of hybrid domain attention[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(11): 2156-2167.
[15] Kai LI,Yu-shun LIN,Xiao-lin WU,Fei-yu LIAO. Small target vehicle detection based on multi-scale fusion technology and attention mechanism[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(11): 2241-2250.