Real-time positioning and control of soft robot based on three-dimensional vision

doi:10.3785/j.issn.1008-973X.2025.08.003

Journal of ZheJiang University (Engineering Science)

2025, Vol. 59

Issue (8): 1574-1582 DOI: 10.3785/j.issn.1008-973X.2025.08.003

Real-time positioning and control of soft robot based on three-dimensional vision

Hong ZHANG1,2(

),Xuecheng ZHANG1,Guoqiang WANG1,Panlong GU2,Nan JIANG1

1. Center for Soft Machines and Smart Devices, Huanjiang Laboratory, Zhuji 311800, China
2. School of Aeronautics and Astronautics, Zhejiang University, Hangzhou 310058, China

Download:

HTML

PDF(2000KB) HTML
Export: BibTeX | EndNote (RIS)

Abstract

A platform for modular pneumatic soft robotic arm was designed, and a forward position model from the driving space, virtual joint space to the workspace was derived aiming at the problems of difficult modeling, poor control stability and low accuracy of soft robotic. The workspace data set was generated according to the actual structural parameter constraints. The position inverse solution was effectively completed by constructing the three-dimensional spatial index structure of KD tree. Then the solution speed was greatly improved. The proposed inverse solution selection principle was used to select the optimal solution from multiple solutions, which enhanced the control stability and accuracy of the pneumatic soft robotic arm. A real-time 3D visual positioning algorithm for the end of robotic arm combining deep learning and RANSAC algorithm was developed in order to improve the control accuracy of the end of the robot arm. Then closed-loop control of the end position and orientation of robotic arm was achieved, and about four times more precision improvement was achieved compared with open-loop control.

Key words： soft robotic arm KD tree three-dimensional vision deep learning kinematics

Received: 13 June 2024 Published: 28 July 2025

CLC:

TP 393

Fund: 浣江实验室专项资助项目（128102-E52201/031）；浙江省自然科学基金资助项目（LGG21F030003, LGG20E050011）.

	Service
	E-mail this article
	Add to my bookshelf
	Add to citation manager
	E-mail Alert
	RSS
	Articles by authors
	Hong ZHANG
	Xuecheng ZHANG
	Guoqiang WANG
	Panlong GU
	Nan JIANG

Cite this article:

Hong ZHANG,Xuecheng ZHANG,Guoqiang WANG,Panlong GU,Nan JIANG. Real-time positioning and control of soft robot based on three-dimensional vision. Journal of ZheJiang University (Engineering Science), 2025, 59(8): 1574-1582.

URL:

https://www.zjujournals.com/eng/10.3785/j.issn.1008-973X.2025.08.003 OR https://www.zjujournals.com/eng/Y2025/V59/I8/1574

基于三维视觉的软体机器人实时定位与控制

针对软体机器人建模困难、控制稳定性差和精度低等问题，设计模块化气动软体机械臂平台，推导得到从驱动空间、虚拟关节空间到工作空间的位置正解模型. 根据实际结构参数的约束生成工作空间数据集，通过构建KD树的三维空间索引结构，有效地完成了位置逆解求解，极大地提高了求解速度. 采用提出的逆解筛选原则，在多个解中选取最优解，增强了气动软体机械臂的控制稳定性及精度. 为了提升机械臂末端的控制精度，开发结合深度学习与RANSAC算法的机械臂末端三维视觉实时定位算法，实现了机械臂末端位姿的闭环控制，与开环控制相比，实现了约4倍的精度提升.

关键词： 软体机械臂, KD树, 三维视觉, 深度学习, 运动学

Fig.1 Experimental platform of pneumatic soft robotic arm motion control

Tab.1 Main hardware of robotic experimental platform

Tab.2 Development environment configuration of deep learning-based algorithm

Fig.2 Structural diagram of soft robotic arm

Fig.3 Schematic of deformation of soft driver

Fig.4 Electrical logic diagram of control system

Tab.3 Structure parameter of soft robotic arm

Fig.5 Analytical schematic of forward kinematics for end position of soft robotic arm

Tab.4 D-H parameter table of soft robotic arm

Fig.6 Position inverse flow chart of soft robotic arm

Tab.5 Performance comparison of inverse solving algorithm by different data volume

Fig.7 Schematic of KD tree position inverse solution

Fig.8 Schematic diagram of visual recognition of positioning ring

Fig.9 Preprocessing diagram of point cloud of positioning ring

Tab.6 Comparison of fitted coordinates and true coordinates

Fig.10 Closed-loop control flowchart of soft robotic arm

Fig.11 Schematic diagram of bending motion of soft robotic arm

Fig.12 Trajectory of bending movement of robotic end in open-loop

Fig.13 Trajectory of bending movement of robotic end in closed-loop

Tab.7 End velocity of robotic arm during experiment of bending motion

Fig.14 Schematic diagram of deflection of soft robotic arm

Fig.15 Trajectory of robotic end of deflection in open-loop

Fig.16 Trajectory of robotic end of deflection in closed-loop

Fig.17 End velocity of robotic arm during experiment of deflection


[1]	LUAN H, WANG M, ZHANG Q, et al Variable stiffness fibers enabled universal and programmable re-foldability strategy for modular soft robotics[J]. Advanced Science, 2024, 11 (10): 2307350 doi: 10.1002/advs.202307350

[2]	郝天泽, 肖华平, 刘书海, 等集成化智能软体机器人研究进展[J]. 浙江大学学报: 工学版, 2021, 55 (2): 229- 243 HAO Tianze, XIAO Huaping, LIU Shuhai, et al Research status of integrated intelligent soft robots[J]. Journal of Zhejiang University: Engineering Science, 2021, 55 (2): 229- 243

[3]	ARACHCHIGE D, PERERA D, MALLIKARACHCHI S, et al Soft steps: exploring quadrupedal locomotion with modular soft robots[J]. IEEE Access, 2023, 11: 63136- 63148 doi: 10.1109/ACCESS.2023.3289156

[4]	HUANG J D, ZHOU J S, WANG Z Y, et al Modular origami soft robot with the perception of interaction force and body configuration[J]. Advanced Intelligent Systems, 2022, 4 (9): 2200081 doi: 10.1002/aisy.202200081

[5]	THURUTHEL T G, ANSARI Y, FALOTICO E, et al Control strategies for soft robotic manipulators: a survey[J]. Soft Robotics, 2018, 5 (2): 149- 163 doi: 10.1089/soro.2017.0007

[6]	QU J T, XU Y N, LI Z K, et al Recent advances on underwater soft robots[J]. Advanced Intelligent Systems, 2024, 6 (2): 2300299 doi: 10.1002/aisy.202300299

[7]	WANG S, SUN Z Hydrogel and machine learning for soft robots' sensing and signal processing: a review[J]. Journal of Bionic Engineering, 2023, 20 (3): 845- 857 doi: 10.1007/s42235-022-00320-y

[8]	YIMING B, HAN Y, HAN Z, et al A mechanically robust and versatile liquid-free ionic conductive elastomer[J]. Advanced Materials, 2021, 33 (11): 2006111 doi: 10.1002/adma.202006111

[9]	SHAO X, CAI Y, YIN S, et al Mechanics of interfacial delamination in deep-sea soft robots under hydrostatic pressure[J]. Journal of Applied Mechanics, 2023, 90 (2): 021009 doi: 10.1115/1.4056214

[10]	TANG W, ZHANG Y, XU H, et al Self-protection soft fluidic robots with rapid large-area self-healing capabilities[J]. Nature Communications, 2023, 14 (1): 6430 doi: 10.1038/s41467-023-42214-5

[11]	LI G, CHEN X, ZHOU F, et al Self-powered soft robot in the Mariana Trench[J]. Nature, 2021, 591 (7848): 66- 71 doi: 10.1038/s41586-020-03153-z

[12]	XU F, WANG H Soft Robotics: morphology and morphology-inspired motion strategy[J]. IEEE-CAA Journal of Automatica Sinica, 2021, 8 (9): 1500- 1522 doi: 10.1109/JAS.2021.1004105

[13]	VINOCOUR P E T, KOKUBU S, MATSUNAGA F, et al Development of a dual function joint modular soft actuator and its evaluation using a novel dummy finger joint-soft actuator complex model[J]. IEEE Robotics and Automation Letters, 2024, 9 (5): 4210- 4217 doi: 10.1109/LRA.2024.3376975

[14]	SU M, XIE R, QIU Y, et al Design, mobility analysis and gait planning of a leech-like soft crawling robot with stretching and bending deformation[J]. Journal of Bionic Engineering, 2023, 20 (1): 69- 80 doi: 10.1007/s42235-022-00256-3

[15]	DAS R, BABU S P M, VISENTIN F, et al An earthworm-like modular soft robot for locomotion in multi-terrain environments[J]. Scientific Reports, 2023, 13 (1): 1571 doi: 10.1038/s41598-023-28873-w

[16]	ZHANG K, FAN Y, SHEN S Tunable folding assembly strategy for soft pneumatic actuators[J]. Soft Robotics, 2023, 10 (6): 1099- 1114 doi: 10.1089/soro.2022.0166

[17]	LI G, WONG T W, SHIH B, et al Bioinspired soft robots for deep-sea exploration[J]. Nature Communications, 2023, 14 (1): 7097 doi: 10.1038/s41467-023-42882-3

[18]	GONG Z, FANG X, CHEN X, et al A soft manipulator for efficient delicate grasping in shallow water: modeling, control, and real-world experiments[J]. International Journal of Robotics Research, 2021, 40 (1): 449- 469 doi: 10.1177/0278364920917203

[19]	WAN Z, SUN Y, QIN Y, et al Design, analysis, and real-time simulation of a 3D soft robotic snake[J]. Soft Robotics, 2023, 10 (2): 258- 268 doi: 10.1089/soro.2021.0144

[20]	FOKOU M R T, XIA Q, JIN H, et al A soft robotic fish actuated by artificial muscle modules (SoRoFAAM-1)[J]. Journal of Bionic Engineering, 2023, 20 (5): 2030- 2043 doi: 10.1007/s42235-023-00390-6

[21]	HABICH T L, HAACK J, BELHADJ M, et al SPONGE: open-source designs of modular articulated soft robots[J]. IEEE Robotics and Automation Letters, 2024, 9 (6): 5346- 5353 doi: 10.1109/LRA.2024.3388855

[22]	SHAN Y, ZHAO Y, YU H, et al Design and grasping force modeling for a soft robotic gripper with multi-stem twining[J]. Journal of Bionic Engineering, 2023, 20 (5): 2123- 2134 doi: 10.1007/s42235-023-00371-9

[23]	ZHANG J, LI Y, KAN Z, et al A preprogrammable continuum robot inspired by elephant trunk for dexterous manipulation[J]. Soft Robotics, 2023, 10 (3): 636- 646 doi: 10.1089/soro.2022.0048

[24]	CHEN G, LIN T, LODEWIJKS G, et al Design of an active flexible spine for wall climbing robot using pneumatic soft actuators[J]. Journal of Bionic Engineering, 2023, 20 (2): 530- 542 doi: 10.1007/s42235-022-00273-2

[25]	张志远, 王松涛, 王学谦, 等螺线型气动软体致动器设计与建模[J]. 机器人, 2020, 42 (1): 10- 20 ZHANG Zhiyuan, WANG Songtao, WANG Xueqian, et al Design and modeling of a spiral pneumatic soft actuator[J]. Robot, 2020, 42 (1): 10- 20

[26]	夏泽洋, 陈君, 甘阳洲, 等一种用于刚软混杂机器人仿真的耦合模型[J]. 机器人, 2021, 43 (1): 29- 35 XIA Zeyang, CHEN Jun, GAN Yangzhou, et al A coupled model for the simulation of rigid-soft hybrid robot[J]. Robot, 2021, 43 (1): 29- 35

[27]	王昱欣, 王贺升, 陈卫东软体机器人手眼视觉/形状混合控制[J]. 机器人, 2018, 40 (5): 619- 625 WANG Yuxin, WANG Hesheng, CHEN Weidong The hybrid eye-in-hand visual/shape control for soft robots[J]. Robot, 2018, 40 (5): 619- 625

[28]	MULLER D, SAWODNY O Modeling the soft bellows of the bionic soft arm[J]. Ifac Papersonline, 2022, 55 (20): 229- 234 doi: 10.1016/j.ifacol.2022.09.100

[29]	徐璠, 王贺升软体机械臂水下自适应鲁棒视觉伺服[J]. 自动化学报, 2023, 49 (4): 744- 753 XU Fan, WANG Hesheng Underwater adaptive robust visual servo for soft manipulator[J]. Acta Automatica Sinica, 2023, 49 (4): 744- 753

[30]	CHENG D, XU R, ZHANG B, et al Fast density estimation for density-based clustering methods[J]. Neurocomputing, 2023, 532 (1): 170- 182

[31]	徐志博, 刘永生, 户盼茹结合离群因子和K-means++聚类改进的点云去噪算法[J]. 信息技术与信息化, 2023, (3): 21- 24 XU Zhibo, LIU Yongsheng, HU Panru Improved denoising algorithm for point cloud based on outlier factor and K-means++ clustering[J]. Information Technology and Informatization, 2023, (3): 21- 24

[32]	唐佳林, 林寿南, 周壮, 等基于特征变换结合KD树改进ICP的快速点云配准方法[J]. 计算机科学, 2023, 50 (Suppl.2): 248- 252 TANG Jialin, LIN Shounan, ZHOU Zhuang, et al Fast point cloud registration method for improving ICP based on feature transformation combined with KD tree[J]. Computer Science, 2023, 50 (Suppl.2): 248- 252

[1]	Shengju WANG,Zan ZHANG. Missing value imputation algorithm based on accelerated diffusion model[J]. Journal of ZheJiang University (Engineering Science), 2025, 59(7): 1471-1480.

[2]	Dongping ZHANG,Dawei WANG,Shuji HE,Siliang TANG,Zhiyong LIU,Zhongqiu LIU. Remaining useful life prediction of aircraft engines based on cross-dimensional feature fusion[J]. Journal of ZheJiang University (Engineering Science), 2025, 59(7): 1504-1513.

[3]	Yongqing CAI,Cheng HAN,Wei QUAN,Wudi CHEN. Visual induced motion sickness estimation model based on attention mechanism[J]. Journal of ZheJiang University (Engineering Science), 2025, 59(6): 1110-1118.

[4]	Lihong WANG,Xinqian LIU,Jing LI,Zhiquan FENG. Network intrusion detection method based on federated learning and spatiotemporal feature fusion[J]. Journal of ZheJiang University (Engineering Science), 2025, 59(6): 1201-1210.

[5]	Huizhi XU,Xiuqing WANG. Perception of distance and speed of front vehicle based on vehicle image features[J]. Journal of ZheJiang University (Engineering Science), 2025, 59(6): 1219-1232.

[6]	Zan CHEN,Ran LI,Yuanjing FENG,Yongqiang LI. Video snapshot compressive imaging reconstruction based on temporal super-resolution[J]. Journal of ZheJiang University (Engineering Science), 2025, 59(5): 956-963.

[7]	Li MA,Yongshun WANG,Yao HU,Lei FAN. Pre-trained long-short spatiotemporal interleaved Transformer for traffic flow prediction applications[J]. Journal of ZheJiang University (Engineering Science), 2025, 59(4): 669-678.

[8]	Qiaohong CHEN,Menghao GUO,Xian FANG,Qi SUN. Image captioning based on cross-modal cascaded diffusion model[J]. Journal of ZheJiang University (Engineering Science), 2025, 59(4): 787-794.

[9]	Zhengyu GU,Feifei LAI,Chen GENG,Ximing WANG,Yakang DAI. Knowledge-guided infarct segmentation of ischemic stroke[J]. Journal of ZheJiang University (Engineering Science), 2025, 59(4): 814-820.

[10]	Minghui YAO,Yueyan WANG,Qiliang WU,Yan NIU,Cong WANG. Siamese networks algorithm based on small human motion behavior recognition[J]. Journal of ZheJiang University (Engineering Science), 2025, 59(3): 504-511.

[11]	Liming LIANG,Pengwei LONG,Jiaxin JIN,Renjie LI,Lu ZENG. Steel surface defect detection algorithm based on improved YOLOv8s[J]. Journal of ZheJiang University (Engineering Science), 2025, 59(3): 512-522.

[12]	Kaibo YANG,Mingen ZHONG,Jiawei TAN,Zhiying DENG,Mengli ZHOU,Ziji XIAO. Small-scale sparse smoke detection in multiple fire scenarios based on semi-supervised learning[J]. Journal of ZheJiang University (Engineering Science), 2025, 59(3): 546-556.

[13]	Zhichao CHEN,Jie YANG,Fan LI,Zhicheng FENG. Review on deep learning-based key algorithm for train running environment perception[J]. Journal of ZheJiang University (Engineering Science), 2025, 59(1): 1-17.

[14]	Dengfeng LIU,Shihai CHEN,Wenjing GUO,Zhilei CHAI. Efficient halftone algorithm based on lightweight residual networks[J]. Journal of ZheJiang University (Engineering Science), 2025, 59(1): 62-69.

[15]	Yi ZHAO,Chun AN,Minghao LI,Jianxiao MA,Shuo HUAI. Selection of lane-changing distance for vehicles in urban expressway interchange weaving section[J]. Journal of ZheJiang University (Engineering Science), 2025, 59(1): 205-212.

Viewed

Full text

Abstract

Cited

Shared

Discussed