Please wait a minute...
Journal of ZheJiang University (Engineering Science)  2024, Vol. 58 Issue (9): 1956-1969    DOI: 10.3785/j.issn.1008-973X.2024.09.020
    
Digital twin system of legged robot for mobile operation
Junjie LIN1(),Yaguang ZHU1,2,*(),Chunchao LIU1,Haoyang LIU1
1. The Key Laboratory of Road Construction Technology and Equipment of MOE, Chang’an University, Xi’an 710064, China
2. State Key Laboratory of Robotics and Systems, Harbin Institute of Technology, Harbin 150001, China
Download: HTML     PDF(2451KB) HTML
Export: BibTeX | EndNote (RIS)      

Abstract  

A digital twin system for the mobile operation of legged robots was proposed, encompassing the design of architecture, module structure, hardware framework, and software framework. The system enabled reliable and accurate acquisition of environmental states and robot states in mobile scenarios by integrating multiple sensor inputs and data sources. The point and line feature matching theory was used to optimize the autonomous positioning accuracy and the robustness of the legged robot, and the odometer functionality and the real-time mobile mapping were effectively achieved through integration with the environmental modeling data. A general modeling method was introduced to establish a digital twin model that ensured high consistency between the simulated robot motion state and the real robot motion state through error compensation techniques. Experimental results on both datasets and real robots demonstrated that the proposed digital twin system not only operated stably and efficiently across various legged robot platforms but also ensured the real-time state feedback and the odometer positioning accuracy. Compared with ORB-SLAM3, the memory overhead was reduced by about 68.7%, and the CPU usage was reduced by about 17.8%. The hardware experiments showed that the communication delay was basically consistent with the network delay of about 30 ms, which helped to improve the efficiency of task execution.



Key wordsdigital twin      control system      legged robot      autonomous positioning      point and line feature extraction     
Received: 01 January 2024      Published: 30 August 2024
CLC:  TP 24  
Fund:  国家自然科学基金资助项目(62373064);机器人技术与系统国家重点实验室开放基金资助项目(SKLRS-2023-KF-05);中央高校基本科研业务费专项资金资助项目(300102259308, 300102259401).
Corresponding Authors: Yaguang ZHU     E-mail: linjunjie@chd.edu.cn;zhuyaguang@chd.edu.cn
Cite this article:

Junjie LIN,Yaguang ZHU,Chunchao LIU,Haoyang LIU. Digital twin system of legged robot for mobile operation. Journal of ZheJiang University (Engineering Science), 2024, 58(9): 1956-1969.

URL:

https://www.zjujournals.com/eng/10.3785/j.issn.1008-973X.2024.09.020     OR     https://www.zjujournals.com/eng/Y2024/V58/I9/1956


面向移动作业的腿足机器人数字孪生系统

针对移动作业中的腿足机器人,结合数字孪生技术,设计整机系统的体系结构、模块结构、硬件框架和软件框架,通过集成多个传感器输入和数据源,可在移动场景下获得可靠准确的环境状态和机器人状态. 基于点线特征匹配理论,优化腿足机器人的自主定位精度和鲁棒性,结合环境建模数据实现高效里程计和实时移动建图. 提出建立数字孪生模型的通用方法,并通过误差补偿保证机器人数字孪生体的运动状态与实际机器人的运动状态高度一致. 在数据集和真实机器人上进行实验,结果表明所提出的数字孪生系统不仅能够在不同的腿足机器人平台上稳定高效运行,而且能够保证实时状态反馈和里程计定位精度. 与ORB-SLAM3相比,内存开销降低约68.7%,CPU使用率降低约17.8%. 硬件实验表明,通信延迟与网络延迟基本一致,约为30 ms,有助于提高任务执行效率.


关键词: 数字孪生,  控制系统,  腿足机器人,  自主定位,  点线特征提取 
Fig.1 Architecture of digital twin system for legged robot
Fig.2 Modular structure of digital twin system for legged robot
Fig.3 Hardware framework of digital twin system for legged robot
Fig.4 Software framework of digital twin system for legged robot
Fig.5 Robot inertial coordinate system and center of mass coordinate system
Fig.6 Control frame of legged robot
Fig.7 Central neural network of hexapod robot
Fig.8 Motions of physical and twin hexapod robots with different gaits
Fig.9 Physical motion, simulation, and digital twin models of quadruped robot
Fig.10 Simulation and physical joint fitting curves under no-load conditions
Fig.11 Error between physical entity and digital twin after adding dynamics and uncertainty compensation under ground touch condition
Fig.12 Average CPU usage and memory consumption of three algorithms
方法算法APERMSE
1VINS-Fusion0.4194680.446491
2ORB-SLAM30.0498370.049837
3本研究算法0.2613310.281762
Tab.1 Absolute pose error and root-mean-square error of three algorithms
Fig.13 Experimental platform for hexapod robot
Fig.14 Robot odometer, environment model and real-time screen displayed on UI interface
通信方式发送节点接收节点时差/ms
USBIMUSLAM18
LAN激光雷达SLAM16
USB深度相机SLAM23
WIFISLAMUI90
LANSLAM运动控制16
WIFIUI运动控制30
Tab.2 Communication delay between nodes in different communication modes
[1]   LU Y Industry 4.0: a survey on technologies, applications and open research issues[J]. Journal of Industrial Information Integration, 2017, 6: 1- 10
doi: 10.1016/j.jii.2017.04.005
[2]   ZEID I, STEIGER-ESCOBAR S, BOGRAD M, et al. Industry partnership to help transform liberal arts graduates to advanced manufacturing careers [C]// ASME International Mechanical Engineering Congress and Exposition . Houston: American Society of Mechanical Engineers, 2015.
[3]   GRANGEL-GONZÁLEZ I, HALILAJ L, COSKUN G, et al. Towards a semantic administrative shell for industry 4.0 components [C]// 2016 IEEE 10th International Conference on Semantic Computing . Laguna Hills: IEEE, 2016: 230−237.
[4]   LI L China's manufacturing locus in 2025: with a comparison of "Made-in-China 2025" and "Industry 4.0"[J]. Technological Forecasting and Social Change, 2018, 135: 66- 74
doi: 10.1016/j.techfore.2017.05.028
[5]   MALIK A A, BREM A Digital twins for collaborative robots: a case study in human-robot interaction[J]. Robotics and Computer-Integrated Manufacturing, 2021, 68: 102092
doi: 10.1016/j.rcim.2020.102092
[6]   TUEGEL E J, INGRAFFEA A R, EASON T G, et al Reengineering aircraft structural life prediction using a digital twin[J]. International Journal of Aerospace Engineering, 2011, (1): 154798
[7]   CERRONE A, HOCHHALTER J, HEBER G, et al On the effects of modeling as-manufactured geometry: toward digital twin[J]. International Journal of Aerospace Engineering, 2014, (1): 439278
[8]   HUANG Z Q, SHEN Y, LI J Y, et al A survey on AI-driven digital twins in industry 4.0: smart manufacturing and advanced robotics[J]. Sensors, 2021, 21 (19): 6340
doi: 10.3390/s21196340
[9]   杨艳芳, 贺焕, 舒亮, 等 断路器柔性装配数字孪生机器人及其运动控制[J]. 计算机集成制造系统, 2020, 26 (11): 2915- 2926
YANG Yanfang, HE Huan, SHU Liang, et al Digital twin robot and its motion control for flexible assembly of circuit breaker[J]. Computer Integrated Manufacturing Systems, 2020, 26 (11): 2915- 2926
[10]   LUMER-KLABBERS G, HAUSTED J O, KVISTGAARD J L, et al. Towards a digital twin framework for autonomous robots [C]// 2021 IEEE 45th Annual Computers, Software, and Applications Conference . Madrid: IEEE, 2021: 1254−1259.
[11]   LIU H, ZHAO W, LI S, et al Construction method of virtual-real drive systems for robots in digital twin workshops[J]. China Mechanical Engineering, 2022, 33 (21): 2623
[12]   HOU Z, HE W Modeling and control of digital twin-based aircraft assembly state inspection robot[J]. Computer Integrated Manufacturing Systems, 2021, 27 (4): 981- 989
[13]   YAMADA T, ABE H, KAWABATA K. Development of testing method considering tasks with remotely controlled robots in Fukushima Daiichi nuclear power station [C]// 2 021 IEEE International Conference on Intelligence and Safety for Robotics . Tokoname: IEEE, 2021: 131−134.
[14]   GARG G, KUTS V, ANBARJAFARI G Digital twin for fanuc robots: industrial robot programming and simulation using virtual reality[J]. Sustainability, 2021, 13 (18): 10336
doi: 10.3390/su131810336
[15]   SWEE S K, AL-QUDAH A. Wireless control system for six-legged autonomous insect robot [C]// MATEC Web of Conferences . Cape Town: EDP Sciences, 2016.
[16]   LIANG S N, TAN K O, CLEMENT T H L, et al. Open source hardware and software platform for robotics and artificial intelligence applications [C]// IOP Conference Series: Materials Science and Engineering . Kuala Lumpur: IOP Publishing, 2016.
[17]   LIN R, GUO W, LI M, et al. Novel design of a legged mobile lander for extraterrestrial planet exploration [J]. International Journal of Advanced Robotic Systems , 2017, 14(6): 1729881417746120.
[18]   WANG Y, MA H mvil-fusion: monocular visual-inertial-lidar simultaneous localization and mapping in challenging environments[J]. IEEE Robotics and Automation Letters, 2022, 8 (2): 504- 511
[19]   LIU Y, LI Z, XIAO L, et al FDO-Calibr: visual-aided IMU calibration based on frequency-domain optimization[J]. Measurement Science and Technology, 2023, 34 (4): 045108
doi: 10.1088/1361-6501/acadfb
[20]   GOMEZ-OJEDA R, MORENO F A, ZUNIGA-NOËL D, et al PL-SLAM: a stereo SLAM system through the combination of points and line segments[J]. IEEE Transactions on Robotics, 2019, 35 (3): 734- 746
doi: 10.1109/TRO.2019.2899783
[21]   LU Y, LIU C, KEVIN I, et al Digital twin-driven smart manufacturing: connotation, reference model, applications and research issues[J]. Robotics and Computer-Integrated Manufacturing, 2020, 61: 101837
doi: 10.1016/j.rcim.2019.101837
[22]   REZAEE H, ABDOLLAHI F A decentralized cooperative control scheme with obstacle avoidance for a team of mobile robots[J]. IEEE Transactions on Industrial Electronics, 2013, 61 (1): 347- 354
[23]   ZHU Y, ZHANG L, MANOONPONG P Generic mechanism for waveform regulation and synchronization of oscillators: an application for robot behavior diversity generation[J]. IEEE Transactions on Cybernetics, 2020, 52 (6): 4495- 4507
[24]   ZHU Y, ZHOU S, GAO D, et al Synchronization of non-linear oscillators for neurobiologically inspired control on a bionic parallel waist of legged robot[J]. Frontiers in Neurorobotics, 2019, 13: 59
[25]   DI Carlo J, WENSING P M, KATZ B, et al. Dynamic locomotion in the mit cheetah 3 through convex model-predictive control [C]// IEEE/RSJ International Conference on Intelligent Robots and Systems . Madrid: IEEE, 2018: 1−9.
[26]   VILLARREAL O, BARASUOL V, WENSING P M, et al. MPC-based controller with terrain insight for dynamic legged locomotion [C]// IEEE International Conference on Robotics and Automation . Paris: IEEE, 2020: 2436−2442.
[27]   RIGHETTI L, IJSPEERT A J. Pattern generators with sensory feedback for the control of quadruped locomotion [C]// IEEE International Conference on Robotics and Automation . Pasadena: IEEE, 2008: 819−824.
[28]   QIN H, ZHU Y, ZHANG Y, et al. Terrain estimation with least squares and virtual model control for quadruped robots [C]// Journal of Physics: Conference Series . Wuhan: IOP Publishing, 2022.
[29]   QIN T, LI P, SHEN S VINS-Mono: a robust and versatile monocular visual-inertial state estimator[J]. IEEE Transactions on Robotics, 2018, 34 (4): 1004- 1020
doi: 10.1109/TRO.2018.2853729
[30]   AKINLAR C, TOPAL C. Edlines: real-time line segment detection by edge drawing (ed) [C]// 18th IEEE International Conference on Image Processing . Brussels: IEEE, 2011: 2837−2840.
[31]   ZHANG L, KOCH R An efficient and robust line segment matching approach based on LBD descriptor and pairwise geometric consistency[J]. Journal of Visual Communication and Image Representation, 2013, 24 (7): 794- 805
doi: 10.1016/j.jvcir.2013.05.006
[32]   QIN T, SHEN S. Online temporal calibration for monocular visual-inertial systems [C]// IEEE/RSJ International Conference on Intelligent Robots and Systems . Madrid: IEEE, 2018: 3662−3669.
[33]   CAMPOS C, ELVIRA R, RODRÍGUEZ J J G, et al Orb-slam3: an accurate open-source library for visual, visual-inertial, and multimap slam[J]. IEEE Transactions on Robotics, 2021, 37 (6): 1874- 1890
doi: 10.1109/TRO.2021.3075644
[1] Xiaoyong JIANG,Kaijian YING,Qiwei WU,Xuan WEI. Research overview on touchdown detection methods for footed robots[J]. Journal of ZheJiang University (Engineering Science), 2024, 58(2): 334-348.
[2] Xiao-yan CAO,Min YU,Jin ZHOU,Yun-zhi WANG. Multi-objective optimization design of adjustable rotary fluid damper parameter[J]. Journal of ZheJiang University (Engineering Science), 2023, 57(7): 1439-1449.
[3] Yong-sheng ZHAO,Zhi-yong ZHAO,Ying LI,Tao ZHANG. Optimal control method of fuselage docking accuracy based on digital twin[J]. Journal of ZheJiang University (Engineering Science), 2023, 57(5): 883-891.
[4] Yong-sheng ZHAO,Rui-xiang LI,Na-na NIU,Zhi-yong ZHAO. Shape control method of fuselage driven by digital twin[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(7): 1457-1463.
[5] Bo-han LENG,Tang-bin XIA,He SUN,Hao WANG,Li-feng XI. Digital twin mapping modeling and method of monitoring and simulation for reconfigurable manufacturing system[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(5): 843-855.
[6] Lin-li LI,Fu GU,Hao LI,Xin-jian GU,Guo-fu LUO,Zhi-qiang WU,Yi-jin GANG. Framework and key technologies of digital twin system cyber security under perspective of bionics[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(3): 419-435.
[7] Zhe-wu CHENG,Shui-guang TONG,Zhe-ming TONG,Qin-guo ZHANG. Review of digital design and digital twin of industrial boiler[J]. Journal of ZheJiang University (Engineering Science), 2021, 55(8): 1518-1528.
[8] Shou-guo ZHENG,Yong-de ZHANG,Wen-tian XIE,Hu FAN,Qing WANG. Aircraft final assembly line modeling based on digital twin[J]. Journal of ZheJiang University (Engineering Science), 2021, 55(5): 843-854.
[9] Xiang-fei MENG,Ren-guang WANG,Yuan-li XU. Torque distribution strategy of pure electric driving mode for dual planetary vehicle[J]. Journal of ZheJiang University (Engineering Science), 2020, 54(11): 2214-2223.
[10] SHI Jian-peng, QUAN Long, ZHANG Xiao-gang, XIONG Xiao-yan. Velocity and position characteristics of excavator's boom by separate meter-in and meter-out compound control[J]. Journal of ZheJiang University (Engineering Science), 2017, 51(9): 1797-1807.
[11] DING Ru-qi, XU Bing, ZHANG Jun-hui. Coupling property of pressure and velocity compound control in individual metering systems[J]. Journal of ZheJiang University (Engineering Science), 2017, 51(6): 1126-1134.
[12] SUN Wei, DU Jia-nan, WANG Lin-tao, MA Hong-hui. High speed and low impact control method for electro-hydraulic system of segment erector in tunnel boring machine[J]. Journal of ZheJiang University (Engineering Science), 2017, 51(10): 1948-1958.
[13] JIA Chi qian, FENG Dong qin. Security assessment for industrial control systems based on fuzzy analytic hierarchy process[J]. Journal of ZheJiang University (Engineering Science), 2016, 50(4): 759-765.
[14] LIANG Yao, FENG Dong qin. Security risks assessment for physical layer of industrial control system based on attack gain[J]. Journal of ZheJiang University (Engineering Science), 2016, 50(3): 589-.
[15] LIU Gang, JIN Li qiang, WANG Yi. Vehicle stability control system based on active disturbance-rejection control strategy[J]. Journal of ZheJiang University (Engineering Science), 2016, 50(12): 2289-2296.