Please wait a minute...
浙江大学学报(工学版)
浙江大学学报(工学版)  2023, Vol. 57 Issue (1): 47-54    DOI: 10.3785/j.issn.1008-973X.2023.01.005
机械与能源工程     
基于RGB与深度信息融合的管片抓取位置测量方法
王林涛(),毛齐
大连理工大学 机械工程学院，辽宁 大连 116024
Position measurement method for tunnel segment grabbing based on RGB and depth information fusion
Lin-tao WANG(),Qi MAO
School of Mechanical Engineering, Dalian University of Technology, Dalian 116024, China
 全文: PDF(3034 KB)   HTML
摘要：

针对管片拼装机在管片抓取阶段依赖人工的问题，提出用于自动化拼装的管片螺栓抓取阶段的位置测量方法. 该方案通过SIFT算法匹配管片螺栓，利用Faster-Rcnn算法，筛选出位于工作区的待抓取的管片螺栓. 通过添加注意力机制改变特征提取网络结构，使其在0.8的交并比下保持约94%的准确率并排除其他管片螺栓的识别干扰. 在识别到目标管片螺栓后，结合深度相机的信息获取完整的三维坐标，使得抓取设备位于抓取位置时测量的管片螺栓位置各轴的误差均不超过3 mm，满足机械式抓取装置对抓取的精度要求. 直接对管片螺栓进行识别，无须考虑管片在工作区摆放位置的不确定性造成的误差，避免了使用靶标进行测量时靶标与管片之间的相对位置误差及设置靶标的人力与时间成本.
关键词： SIFT特征匹配;  Faster-Rcnn算法;  注意力机制;  信息融合;  管片螺栓位置测量    
Abstract:

A position measurement method which fits automatic segment assembly to grab segment bolt was proposed in order to solve the problem that grabbing segments relied on manual labor. SIFT algorithm was used to match the target segment bolts, and Faster-Rcnn algorithm was used to choose the target segment placed in working area. Convolutional attention block module was implemented to change the structure of feature extracting network, which maintained the recognition accuracy around 94% under stricter IOU (intersection over union) of 0.8 and prevented the affect of other segment bolts. Information from depth camera was fused to get the complete three-dimensional coordination after target segment bolt being recognized. The error of measurement in each axis was less than 3 mm when grabbing facility was in position, which meeted the requirement of grabbing with mechanical hoisting facility. The segment bolt was directly recognized. Then the error caused by segment placement uncertainty need not be considered. The relative error between segment and target object and the manual and time cost of setting target object can be prevented.
Key words: SIFT feature matching    Faster-Rcnn algorithm    attention mechanism    information fusion    segment bolt position measurement
收稿日期: 2022-01-26 出版日期: 2023-01-17
CLC:  TP 212  
作者简介: 王林涛（1987—），男，副教授，从事复杂机电系统多学科建模与优化、重大装备电液控制与智能化技术、流体机械CFD分析与优化研究. orcid.org/0000-0002-7647-8559. E-mail： wlt@dlut.edu.cn
服务　　
把本文推荐给朋友
加入引用管理器
E-mail Alert
作者相关文章　　
王林涛
毛齐

引用本文:

王林涛,毛齐. 基于RGB与深度信息融合的管片抓取位置测量方法[J]. 浙江大学学报(工学版), 2023, 57(1): 47-54.

Lin-tao WANG,Qi MAO. Position measurement method for tunnel segment grabbing based on RGB and depth information fusion. Journal of ZheJiang University (Engineering Science), 2023, 57(1): 47-54.

链接本文:

https://www.zjujournals.com/eng/CN/10.3785/j.issn.1008-973X.2023.01.005    或    https://www.zjujournals.com/eng/CN/Y2023/V57/I1/47

图 1  管片螺栓特征匹配的过程
图 2  Faster-Rcnn算法的预测流程
图 3  0.5交并比下误匹配多个管片螺栓
图 4  混合注意力机制的结构
图 5  添加注意力机制的Identity Block结构
图 6  基于信息融合的管片螺栓位置测量流程
图 7  RGB相机与深度相机的配准
图 8  管片训练集的图像采集策略
图 9  不同交并比条件下注意力机制对测试集准确率的影响变化
图 10  添加注意力机制前、后Grad-Cam可视化热图的对比
图 11  管片抓取及锁紧装置
序号 坐标实际值/mm 坐标测量值/mm 各轴误差/mm
1 (?150,260,535) (?151.315, 259.504, 533.467) (1.315, 0.496, 1.533)
2 (300,180,710) (299.407, 178.901, 707.661) (0.593, 1.099, 2.339)
3 (?350,340,605) (?348.159, 342.073, 606.517) (?1.841, ?2.073, ?1.517)
4 (550,420,570) (552.542, 419.404, 570.113) (?2.542, 0.596, ?0.113)
5 (?400,220,745) (?399.669, 221.963, 747.673) (?0.331, ?1.963, ?2.673)
6 (200,140,780) (199.867, 139.417, 782.735) (0.133, 0.583, ?2.735)
7 (500,380,675) (497.884, 381.682, 672.139) (2.116, ?1.682, 2.861)
8 (?250,100,500) (?247.947, 98.754, 502.227) (?2.053, 1.246, ?2.227)
9 (450,300,640) (448.682, 300.676, 642.410) (1.318, ?0.676, ?2.410)
表 1  管片螺栓的位置测量结果
1 WADA M Automatic segment erection system for shield tunnels[J]. Advanced Robotics, 1990, 5 (4): 429- 443
doi: 10.1163/156855391X00304
2 张碧, 赵海峰, 杨涛, 等 盾构管片拼装机国内外研究现状[J]. 矿山机械, 2014, (4): 1- 6
ZHANG Bi, ZHAO Hai-feng, YANG Tao, et al Research status of segment erector in shield tunneling machine at home and abroad[J]. Mining and Processing Equipment, 2014, (4): 1- 6
3 HIRASAWA Y, HASHIMOTO H, WORI Y, et al. Development of automatic segment assembly robot for shield tunnelling machine [C]// BRILAKIS I. Proceedings of International Symposium on Automation and Robotics in Construction. Tokyo: [s. n. ], 1996: 813−820.
4 TANAKA Y Automatic segment assembly robot for shield tunneling machine[J]. Computer-Aided Civil and Infrastructure Engineering, 1995, 10 (5): 325- 337
doi: 10.1111/j.1467-8667.1995.tb00295.x
5 刘飞香 管片拼装机抓取和拼装智能化研究[J]. 铁道建筑, 2020, 60 (8): 58- 63
LIU Fei-xiang Study on intelligent grab and assemble of tunnel segment erecting robot[J]. Railway Engineering, 2020, 60 (8): 58- 63
6 朱凯源. 针对管片自动拼装的视觉测量技术[D]. 大连: 大连理工大学, 2020.
ZHU Kai-yuan. Vision measurement technology for automatic assembly of segments [D]. Dalian: Dalian University of Technology, 2020.
7 张龙, 王海波, 范曙远 盾构机管片拼装自动控制传感检测系统的设计[J]. 机械制造, 2018, 56 (2): 74- 77
ZHANG Long, WANG Hai-bo, FAN Shu-yuan Design of automatic control, sensing and detection system for segment assembly of shield machine[J]. Machinery, 2018, 56 (2): 74- 77
8 高翔, 陶建峰, 王明斗, 等 基于线激光传感器的盾构管片位姿检测方法[J]. 中南大学学报: 自然科学版, 2020, 51 (1): 41- 48
GAO Xiang, TAO Jian-feng, WANG Ming-dou, et al Position and posture detection method of shield segments using line laser sensors[J]. Journal of Central South University: Science and Technology, 2020, 51 (1): 41- 48
9 MAKOTO S, FUMIO T, TERUYUKI M, et al. Assembly method for erector controller and lining work member: Japan, 2004131979A [P]. 2004−04−30.
10 李穗婷. 盾构机管片拼装机器人与视觉系统研究[D]. 广州: 华南理工大学, 2018.
LI Sui-ting. Research on segment robot and vision system for shield machine [D]. Guangzhou: South China University of Technology, 2018.
11 REDMON J, DIVVALA S, GIRSHICK R, et al. You only look once: unified, real-time object detection [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. [S. l. ]: IEEE, 2016: 779-788.
12 REN S, HE K, GIRSHICK R, et al Faster R-CNN: towards real-time object detection with region proposal networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39 (6): 1137- 1149
doi: 10.1109/TPAMI.2016.2577031
13 RONNEBERGER O, FISCHER P, BROX T. U-net: convolutional networks for biomedical image segmentation [C]// Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention. Cham: Springer, 2015: 234-241.
14 朵建华, 杨柏钟 基于多传感器融合的轨道识别方法探究[J]. 现代城市轨道交通, 2021, (9): 98- 102
DUO Jian-hua, YANG Bai-zhong Recognition method of railway tracks based on multi-sensor fusion[J]. Modern Urban Transit, 2021, (9): 98- 102
15 李国瑞, 罗维, 李雨来, 等 基于传感器融合的障碍物轮廓识别方法研究[J]. 生物医学工程与临床, 2021, 25 (4): 395- 400
LI Guo-rui, LUO Wei, LI Yu-lai, et al Obstacle contour recognition method based on sensor fusion[J]. Biomedical Engineering and Clinical Medicine, 2021, 25 (4): 395- 400
16 LOWE D Distinctive image features from scale-invariant keypoints[J]. International Journal of Computer Vision, 2004, 60 (2): 91- 110
doi: 10.1023/B:VISI.0000029664.99615.94
17 WOO S, PARK J, LEE J Y, et al. CBAM: convolutional block attention module [C]// European Conference on Computer Vision. Cham: Springer, 2018.
18 ZHANG Z A flexible new technique for camera calibration[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2000, 22 (11): 1330- 1334
doi: 10.1109/34.888718
19 SELVARAJU R R, COGSWEL M, DAS A, et al Grad-CAM: visual explanations from deep networks via gradient-based localization[J]. International Journal of Computer Vision, 2020, 128 (2): 336- 359
doi: 10.1007/s11263-019-01228-7
20 陈艳 盾构机管片吊运系统技术综合分析[J]. 科学技术创新, 2019, (5): 185- 186
CHEN Yan Comprehensive analysis of segment lifting system technology of shield machine[J]. Scientific and Technological Innovation, 2019, (5): 185- 186
21 乔万良, 刘子铭. 一种盾构机管片抓取轴锁紧装置: CN207989053U [P]. 2018-10-19.
QIAO Wan-liang, LIU Zi-ming. A shield machine segment grab shaft locking device: CN207989053U [P]. 2018-10-19.
[1] 郝琨,王阔,王贝贝. 基于改进Mobilenet-YOLOv3的轻量级水下生物检测算法[J]. 浙江大学学报(工学版), 2022, 56(8): 1622-1632.
[2] 莫仁鹏,司小胜,李天梅,朱旭. 基于多尺度特征与注意力机制的轴承寿命预测[J]. 浙江大学学报(工学版), 2022, 56(7): 1447-1456.
[3] 王友卫,童爽,凤丽洲,朱建明,李洋,陈福. 基于图卷积网络的归纳式微博谣言检测新方法[J]. 浙江大学学报(工学版), 2022, 56(5): 956-966.
[4] 鞠晓臣,赵欣欣,钱胜胜. 基于自注意力机制的桥梁螺栓检测算法[J]. 浙江大学学报(工学版), 2022, 56(5): 901-908.
[5] 张雪芹,李天任. 基于Cycle-GAN和改进DPN网络的乳腺癌病理图像分类[J]. 浙江大学学报(工学版), 2022, 56(4): 727-735.
[6] 许萌,王丹,李致远,陈远方. IncepA-EEGNet: 融合Inception网络和注意力机制的P300信号检测方法[J]. 浙江大学学报(工学版), 2022, 56(4): 745-753, 782.
[7] 柳长源,何先平,毕晓君. 融合注意力机制的高效率网络车型识别[J]. 浙江大学学报(工学版), 2022, 56(4): 775-782.
[8] 陈巧红,裴皓磊,孙麒. 基于视觉关系推理与上下文门控机制的图像描述[J]. 浙江大学学报(工学版), 2022, 56(3): 542-549.
[9] 农元君,王俊杰,陈红,孙文涵,耿慧,李书悦. 基于注意力机制和编码-解码架构的施工场景图像描述方法[J]. 浙江大学学报(工学版), 2022, 56(2): 236-244.
[10] 刘英莉,吴瑞刚,么长慧,沈韬. 铝硅合金实体关系抽取数据集的构建方法[J]. 浙江大学学报(工学版), 2022, 56(2): 245-253.
[11] 袁天乐,袁巨龙,朱勇建,郑翰辰. 基于改进YOLOv5的推力球轴承表面缺陷检测算法[J]. 浙江大学学报(工学版), 2022, 56(12): 2349-2357.
[12] 于楠晶,范晓飚,邓天民,冒国韬. 基于多头自注意力的复杂背景船舶检测算法[J]. 浙江大学学报(工学版), 2022, 56(12): 2392-2402.
[13] 陈巧红,李妃玉,孙麒,贾宇波. 基于LSTM与衰减自注意力的答案选择模型[J]. 浙江大学学报(工学版), 2022, 56(12): 2436-2444.
[14] 李凯,林宇舜,吴晓琳,廖飞宇. 基于多尺度融合与注意力机制的小目标车辆检测[J]. 浙江大学学报(工学版), 2022, 56(11): 2241-2250.
[15] 李飞,胡坤,张勇,王文善,蒋浩. 基于混合域注意力YOLOv4的输送带纵向撕裂多维度检测[J]. 浙江大学学报(工学版), 2022, 56(11): 2156-2167.