A novel curved palm visuotactile sensor

doi:10.3785/j.issn.1008-973X.2025.06.001

Journal of ZheJiang University (Engineering Science)

2025, Vol. 59

Issue (6): 1103-1109 DOI: 10.3785/j.issn.1008-973X.2025.06.001

A novel curved palm visuotactile sensor

Jingyi HU1,2(

),Shaowei CUI1,*(

),Shaolin ZHANG1,Shuo WANG1,3

1. State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China
2. School of Future Technology, University of Chinese Academy of Sciences, Beijing 100049, China
3. School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China

Download:

HTML

PDF(1967KB) HTML
Export: BibTeX | EndNote (RIS)

Abstract

A curved palm visuotactile sensor was proposed to address the challenges faced by existing robotic visuotactile sensors, as most of the sensors employ planar sensing structures and have the limitations of incomplete contact information acquisition and insufficient depth perception in complex 3D environments and delicate manipulation tasks. The innovative curved structural design that extended the effective sensing area into the 3D space significantly enhanced the tactile perception depth and contact information integrity. A point cloud reconstruction framework using the binocular stereo vision-based refractive ray tracing model was established to generate high-precision 3D tactile point cloud data. An efficient point cloud saliency detection algorithm that enabled intelligent recognition and real-time extraction of key contact regions was proposed to improve the efficiency and accuracy of data processing. Experimental results demonstrated that the structural design and the point cloud reconstruction framework of the proposed sensor exhibited strong robustness, achieving a 3D tactile point cloud perception accuracy of 0.20 mm, a point cloud reconstruction frame rate of 30 frames per second, and a maximum pressing depth of 10 mm. The sensor can precisely extract the salient contact point clouds in real time, providing robust technical support for robotic fine manipulation in complex environments.

Key words： robotic tactile sensing tactile sensor visuotactile sensor 3D tactile point cloud point cloud saliency detection

Received: 20 November 2024 Published: 30 May 2025

CLC:

TP 242.6

Fund: 国家重点研发计划资助项目（2023YFB4705000）；国家自然科学基金资助项目（62303455, 62273342, 62122087）；北京自然科学基金资助项目（L233006）；多模态人工智能系统国家重点实验室青年计划资助项目.

Corresponding Authors: Shaowei CUI E-mail: hujingyi2019@ia.ac.cn;shaowei.cui@ia.ac.cn

	Service
	E-mail this article
	Add to my bookshelf
	Add to citation manager
	E-mail Alert
	RSS
	Articles by authors
	Jingyi HU
	Shaowei CUI
	Shaolin ZHANG
	Shuo WANG

Cite this article:

Jingyi HU,Shaowei CUI,Shaolin ZHANG,Shuo WANG. A novel curved palm visuotactile sensor. Journal of ZheJiang University (Engineering Science), 2025, 59(6): 1103-1109.

URL:

https://www.zjujournals.com/eng/10.3785/j.issn.1008-973X.2025.06.001 OR https://www.zjujournals.com/eng/Y2025/V59/I6/1103

新颖的曲面手掌视触觉传感器

大多数现有的机器人视触觉传感器由于采用平面式感知结构，在复杂三维环境和精细操作任务中面临接触信息缺失、深度感知不足的局限性. 为此，提出曲面手掌视触觉传感器，其创新性曲面结构设计将有效感知区域扩展至三维空间，显著地提升触觉感知深度和接触信息完整性. 建立基于双目立体视觉折射光线追踪模型的点云重建框架，用于生成高精度的三维触觉点云数据. 为了提高数据处理效率和准确性，提出高效的点云显著性检测算法，以实现关键接触区域的智能识别与实时点云提取. 实验结果表明，所提传感器的结构设计和点云重建框架具有良好的鲁棒性，三维触觉点云感知精度达到0.20 mm，点云重建帧率达到30 帧/s，最大按压深度为10 mm. 该传感器能够精准实时地提取显著接触点云，为机器人在复杂环境中的精细操作提供了有力支持.

关键词： 机器人触觉感知, 触觉传感器, 视触觉传感器, 三维触觉点云, 点云显著性检测

Fig.1 Conceptual design and physical diagram of sensor hardware structure

Fig.2 Marker pattern on surface of sensor gel layer

Fig.3 Schematic of gel deformation range before and after improvement

Fig.4 Flowchart of 3D tactile point cloud reconstruction

Fig.5 Marker detection, tracking and 3D reconstruction results

Fig.6 Schematic of light propagation in RSRT model

Fig.7 Schematic diagram of parameter calibration for 3D tactile point cloud reconstruction

Fig.8 Flowchart of tactile point cloud saliency detection algorithm

Fig.9 Quantitative experimental platform for 3D tactile point reconstruction

Tab.1 Mean error of 3D reconstruction per circle

Tab.2 Performance comparison of marker-based curved visuotactile sensors

Fig.10 Salient point clouds extracted by tactile sensing algorithm using different objects pressing on sensor

Tab.3 Salient point cloud errors for different objects at different press depths


[1]	LUO S, BIMBO J, DAHIYA R, et al Robotic tactile perception of object properties: a review[J]. Mechatronics, 2017, 48 (7): 54- 67

[2]	SUN H, KUCHENBECKER K J, MARTIUS G A soft thumb-sized vision-based sensor with accurate all-round force perception[J]. Nature Machine Intelligence, 2022, 4 (2): 135- 145 doi: 10.1038/s42256-021-00439-3

[3]	ZOU L, GE C, WANG Z, et al Novel tactile sensor technology and smart tactile sensing systems: a review[J]. Sensors, 2017, 17 (11): 2653 doi: 10.3390/s17112653

[4]	王云灏, 孙铭会, 辛毅, 等基于压电薄膜传感器的机器人触觉识别系统[J]. 浙江大学学报: 工学版, 2022, 56 (4): 702- 710 WANG Yunhao, SUN Minghui, XIN Yi, et al Robot tactile recognition system based on piezoelectric film sensor[J]. Journal of Zhejiang University: Engineering Science, 2022, 56 (4): 702- 710

[5]	HARADA S, KANAO K, YAMAMOTO Y, et al Fully printed flexible fingerprint-like three-axis tactile and slip force and temperature sensors for artificial skin[J]. ACS Nano, 2014, 8 (12): 12851- 12857 doi: 10.1021/nn506293y

[6]	SHIMONOMURA K Tactile image sensors employing camera: a review[J]. Sensors, 2019, 19 (18): 3933 doi: 10.3390/s19183933

[7]	崔少伟, 王硕, 胡静怡, 等面向机器人操作任务的视触觉传感技术综述[J]. 智能科学与技术学报, 2022, 4 (2): 186- 199 CUI Shaowei, WANG Shuo, HU Jingyi, et al A survey of visuotactile sensing technologies for robotic manipulation[J]. Chinese Journal of Intelligent Science and Technology, 2022, 4 (2): 186- 199 doi: 10.11959/j.issn.2096-6652.202222

[8]	ZHANG S, CHEN Z, GAO Y, et al Hardware technology of vision-based tactile sensor: a review[J]. IEEE Sensors Journal, 2022, 22 (22): 21410- 21427 doi: 10.1109/JSEN.2022.3210210

[9]	WARD-CHERRIER B, PESTELL N, CRAMPHORN L, et al The TacTip family: soft optical tactile sensors with 3D-printed biomimetic morphologies[J]. Soft Robotics, 2018, 5 (2): 216- 227 doi: 10.1089/soro.2017.0052

[10]	KAMIYAMA K, VLACK K, MIZOTA T, et al Vision-based sensor for real-time measuring of surface traction fields[J]. IEEE Computer Graphics and Applications, 2005, 25 (1): 68- 75 doi: 10.1109/MCG.2005.27

[11]	YAMAGUCHI A, ATKESON C G. Combining finger vision and optical tactile sensing: reducing and handling errors while cutting vegetables [C]// IEEE-RAS 16th International Conference on Humanoid Robots. Cancun: IEEE, 2016: 1045–1051.

[12]	YUAN W, DONG S, ADELSON E GelSight: high-resolution robot tactile sensors for estimating geometry and force[J]. Sensors, 2017, 17 (12): 2762 doi: 10.3390/s17122762

[13]	ABAD A C, RANASINGHE A Visuotactile sensors with emphasis on GelSight sensor: a review[J]. IEEE Sensors Journal, 2020, 20 (14): 7628- 7638 doi: 10.1109/JSEN.2020.2979662

[14]	SHE Y, WANG S, DONG S, et al Cable manipulation with a tactile-reactive gripper[J]. The International Journal of Robotics Research, 2021, 40 (12/13/14): 1385- 1401 doi: 10.1177/02783649211027233

[15]	DO W K, KENNEDY M. DenseTact: optical tactile sensor for dense shape reconstruction [C]// International Conference on Robotics and Automation. Philadelphia: IEEE, 2022: 6188–6194.

[16]	LAMBETA M, CHOU P W, TIAN S, et al DIGIT: a novel design for a low-cost compact high-resolution tactile sensor with application to in-hand manipulation[J]. IEEE Robotics and Automation Letters, 2020, 5 (3): 3838- 3845 doi: 10.1109/LRA.2020.2977257

[17]	ROMERO B, VEIGA F, ADELSON E. Soft, round, high resolution tactile fingertip sensors for dexterous robotic manipulation [C]// IEEE International Conference on Robotics and Automation. Paris: IEEE, 2020: 4796–4802.

[18]	ALSPACH A, HASHIMOTO K, KUPPUSWAMY N, et al. Soft-bubble: a highly compliant dense geometry tactile sensor for robot manipulation [C]// 2nd IEEE International Conference on Soft Robotics. Seoul: IEEE, 2019: 597–604.

[19]	CUI S, WANG R, HU J, et al In-hand object localization using a novel high-resolution visuotactile sensor[J]. IEEE Transactions on Industrial Electronics, 2022, 69 (6): 6015- 6025 doi: 10.1109/TIE.2021.3090697

[20]	HU J, CUI S, WANG S, et al GelStereo Palm: a novel curved visuotactile sensor for 3-D geometry sensing[J]. IEEE Transactions on Industrial Informatics, 2023, 19 (11): 10853- 10863 doi: 10.1109/TII.2023.3241685

[21]	SORENSEN S, KOLAGUNDA A, SAPONARO P, et al. Refractive stereo ray tracing for reconstructing underwater structures [C]// IEEE International Conference on Image Processing. Quebec City: IEEE, 2015: 1712–1716.

[22]	GU C, CONG Y, SUN G, et al MedUCC: medium-driven underwater camera calibration for refractive 3-D reconstruction[J]. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 2022, 52 (9): 5937- 5948 doi: 10.1109/TSMC.2021.3132146

[23]	KUNZ C, SINGH H. Hemispherical refraction and camera calibration in underwater vision [C]// OCEANS. Quebec City: IEEE, 2008: 1–7.

[24]	YOON S, CHOI T, SULL S Depth estimation from stereo cameras through a curved transparent medium[J]. Pattern Recognition Letters, 2020, 129: 101- 107 doi: 10.1016/j.patrec.2019.11.012

[25]	SHE M, NAKATH D, SONG Y, et al Refractive geometry for underwater domes[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2022, 183 (3): 525- 540

[26]	LEPORA N F Soft biomimetic optical tactile sensing with the TacTip: a review[J]. IEEE Sensors Journal, 2021, 21 (19): 21131- 21143 doi: 10.1109/JSEN.2021.3100645

[27]	SATO K, KAMIYAMA K, KAWAKAMI N, et al Finger-shaped GelForce: sensor for measuring surface traction fields for robotic hand[J]. IEEE Transactions on Haptics, 2010, 3 (1): 37- 47

[28]	LIN X, WILLEMET L, BAILLEUL A, et al. Curvature sensing with a spherical tactile sensor using the color-interference of a marker array [C]// IEEE International Conference on Robotics and Automation. Paris: IEEE, 2020: 603–609.

[1]	Xiao-hui GUO,Wei-qiang HONG,Guo-qing ZHENG,Jing-yi WANG,Guo-peng TANG,Jin-yang YANG,Chao-qiang ZHUO,Yao-hua XU,Yu-nong ZHAO,Hong-wei ZHANG. High sensitivity flexible tactile sensor with hierarchical tilted micro-pillar structure[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(6): 1079-1087, 1126.

Viewed

Full text

Abstract

Cited

Shared

Discussed