TactileLab: Efficient Multimodal Tactile Simulation for Shear-Sensitive Dexterous Sim2Real Robotic Manipulation

Abstract

Robotic simulators with tactile sensing are increasingly important for contact-rich manipulation, but high-fidelity deformable or soft-body simulation often sacrifices efficiency, limiting scalability for large-scale reinforcement learning. We introduce TactileLab, a scalable multimodal tactile simulation framework built on IsaacLab, with a particular focus on a new optical-flow-based tactile representation, termed simulated tactile flow. Instead of relying on expensive soft-body physics to model shear deformation, Tactile Lab uses rigid-body simulation to efficiently estimate dense tangential motion at the contact interface, producing tactile flow fields that encode shear-relevant contact dynamics. This enables policies to perceive not only where contact occurs through contact depth, but also how contact evolves through local motion cues, which is critical for shear-dependent manipulation. Tactile Lab supports single-arm, dual-arm, and dexterous in-hand manipulation tasks, with both low-dimensional tactile signals and high-dimensional tactile observations. We further introduce PETS-Net (Positional-Encoding Temporal-Spatial Network), a policy architecture for jointly processing long-horizon proprioceptive signals and dense tactile observations. Across multiple contact-rich tasks, policies trained with simulated tactile flow learn efficiently at scale and exhibit emergent contact-aware behaviours. Finally, we propose a real-to-sim tactile translation method that converts real tactile image streams into simulated tactile flow, reducing the real-sim observation gap and enabling real-world deployment of policies trained with optical-flow-based tactile observations.

TactileLab is built on top of IsaacLab and supports GPU-parallel simulation for efficient reinforcement learning. It provides multiple tactile sensing modalities, including contact forces, contact positions, contact depth images, and simulated tactile flow, allowing users to study both low-dimensional and high-dimensional tactile feedback within a unified framework.

Methodology

Overview of multimodal tactile policy learning for dexterous in-hand manipulation. To evaluate the benefit of our rich tactile observations, unknown random force and torque perturbations are applied to the manipulated object, making the task more challenging and requiring robust and informative tactile feedback.

Main Results

(More experimental results will be released in the paper.) Training performance with different policy input modalities. The curves report the mean reward averaged over five random seeds. We compare our high-dimensional tactile modalities against representative state-of-the-art baselines from prior in-hand manipulation work, including proprioception-only input following HORA and proprioception with low-dimensional contact information following AnyRotate. Policies using the proposed high-dimensional tactile modalities converge faster and achieve higher final reward, demonstrating the benefit of richer tactile observations for policy training.

In-Hand Manipulation in TactileLab

Tactile Allegro Hand

In-hand manipulation with rich tactile observations (from left to right: contact depth map, pixel-level tactile flow, sparse tactile flow) using Allegro Hand with TacTips.

Tactile Leap Hand

TactileLab supports multiple tactile hands, such as Leap Hand (observation frequency 10 FPS).

TactileLab Tasks Gallery

Multi-Environment Object Following

Single-Arm Pushing

Single-Arm Edge Following

Bi-Touch Bi-Push

Palm-Supported Object Repose

In-Hand Rotation

TactileLab Sim2Real Deployment

Dual-Arm Object-Tracking Tasks

Cube Tracking

A policy trained in TactileLab on a rigid cube transfers to the real system for straight-line object following.

Soft Blue Ball Tracking

The blue ball introduces curved-surface contact with a soft object, testing whether tactile flow observations remain useful beyond the rigid training object.

Soft Brain Tracking

The brain object is soft with an irregular contact surface, providing a more challenging sim-to-real deployment case with changing local contact geometry.

In-Hand Maniulation

Video Coming Soon

BibTeX

@article{lin2026tactilelab,
  author    = {Lin,Yijiong and Lepora, Nathan F.},
  title     = {TactileLab: Efficient Multimodal Tactile Simulation for Shear-Sensitive Dexterous Sim2Real Robotic Manipulation},
  journal   = {arxiv},
  year      = {2026},
}