NeuralTouch: Neural Descriptors for Precise Sim-to-Real Tactile Robot Control

Abstract

Grasping accuracy is a critical prerequisite for precise object manipulation that may require careful alignment between the robot hand and an object. Neural Descriptor Fields (NDFs) offer a promising vision-based method to generate grasping poses that generalize across object categories. However, NDFs alone can produce inaccurate poses due to imperfect camera calibration, incomplete point clouds, and object variability. Alternatively, tactile sensing enables more precise contact, but existing approaches limit policies to simple predefined contact geometries.

In this work, we introduce NeuralTouch, a multimodal framework that integrates NDFs and tactile sensing to enable accurate generalizable grasping through gentle physical interaction. Our approach leverages NDFs to implicitly represent the target contact geometry, from which a deep reinforcement learning policy is trained to refine the grasp using tactile feedback. This policy is conditioned on the neural descriptors and does not require explicit specification of contact types. We validate NeuralTouch through ablation studies in simulation and with zero-shot transfer to real-world manipulation tasks, such as peg-out/in-hole and bottle lid opening. Our results show that NeuralTouch significantly improves grasping accuracy and robustness over baseline methods.

Method Overview

Overview of the NeuralTouch: In simulation, we first pre-train an occupancy network which is the core component of the Neural Pose Descriptor Fields. Secondly, we collect human demonstrations along with object point clouds and robot target grasping pose descriptors depending on the manipulation tasks. Thirdly, we train an RL policy with tactile and proprioceptive feedback, to achieve fine grasping poses implicitly specified by these collected descriptors. After obtaining the NPDF and a well-trained policy, our system is directly deployed in the real world with a real-to-sim tactile transfer to accurately grasp unseen objects, executing manipulation tasks such as unplugging a bolt-like USB and inserting it into a socket.

Simulation Overview

Simulation-only overview of NeuralTouch across the paper's object categories and target features, illustrating how the descriptor-conditioned tactile policy operates before transferring to real-world tasks.

Real-world Experiments with Different Objects

Real-world experiments with different objects, including ketchup bottle, green bolt, and adaptor plug, demonstrating that NeuralTouch can maintain accurate grasp acquisition and robust manipulation across varied geometries.

Real-world Experiments with Bottle Lids of Various Shapes

A diverse set of bottle lids with varying geometries and surface textures used to evaluate the generalization ability of NeuralTouch in real-world bottle-cap opening.

Real-world NeuralTouch demos on varied bottle-cap geometries, demonstrating generalization across cap shapes and surface textures with robust contact correction and accurate lid grasping. Sample numbers refer to the lid numbers listed in the top figure in this session.

Comparison videos for various-shape bottle lids showing the performance of NeuralTouch (ours), NDF, NDF+TOUCH, C2FIL, and C2FIL+TOUCH on the same real-world task setup.

Recovery against Perturbation with NeuralTouch

The real-time performance and instant recovery of the tactile robot under perturbations demonstrate the robustness of NeuralTouch and its applicability to tasks with consistent disturbances. The real tactile image streams are shown in the first and third columns on the PC, while the simulated tactile image streams are shown in the second and fourth columns.

BibTeX

@article{lin2026neuraltouch,
  title   = {NeuralTouch: Neural Descriptors for Precise Sim-to-Real Tactile Robot Control},
  author  = {Lin, Yijiong and Deng, Bowen and Pu, Keju and Lu, Chenghua and Yang, Max and Psomopoulou, Efi and Lepora, Nathan F.},
  journal = {To be added},
  year    = {2026}
}