Naïve sensor fusion concatenating the latent representation of the two data streams often helped little or even hurt. Common fusion methods like VTFSA, ConViTac, and AdapTac have tried learned weights and attention to fix this. They treat the task as a pure data problem: let the AI model sort out what matters. None bake in the physics of how gripping and insertion actually work.

The core issue is a data mismatch. A wrist camera yields a 64 × 64 color image of the whole scene. Each tactile sensor yields a 32 × 32 force map of one fingertip. These inputs differ in scale, units, and meaning. When a model must learn from scratch how they relate, it often fails, especially early in training when tactile signals are noisy and sparse.

Past work used gating or attention to re-weight each stream. These help, but they still rely on the model to find the right blend. A stronger fix is to tell the model what to look for in the tactile data, using known physics.

A new paper from Analog Devices, Inc. (ADI), presented at ICRA, does exactly this. Authors Wonju Lee, Matteo Grimaldi, and Tao Yu propose a Cross-Modal Transformer (CMT) framework that embeds a physical constraint directly into visuo-tactile fusion using balanced grip force, a concept from human motor control.

When you push a plug into a socket, you keep your finger forces roughly even on both sides to succeed in this task. Studies by Morasso and Bizzi et al. showed this balance is key to stable grip. The team turns this insight into a model architecture design element with an auxiliary loss: it pays attention to the tactile force maps between the two fingers in latent space and penalizes gaps between left-finger and right-finger force patterns during training. This nudges the robot to keep its grip balanced during the whole insertion.

The CMT model architecture contains two components:

1. Self-attention links the left and right tactile signals, so the model sees them as a pair.
2. Cross-attention lets the camera stream query the tactile stream to ask "what correction do I need?" Tactile responds with the force patterns that reveal misalignment.

The full system trains end-to-end with proximal policy optimization (PPO), blending the task reward with the force-balance penalty, an approach rooted in the broader field of physics-informed machine learning.

The team evaluated their work using the TacSL benchmark, run in IsaacGymEnvs, NVIDIA's GPU-accelerated reinforcement learning environment. The assessment found that CMT with its force-balance penalty hits a 96.59% insertion success rate. That is 3.62 percent above naive fusion (92.97%) and beats gated fusion (94.53%). It also nearly matches a baseline that gets to cheat and uses ground-truth contact forces from the simulator (96.09%). Real robots do not have access to those forces, so closing this gap matters..

The gains show up in behavior, not just numbers. With naive fusion, contact with the socket wall, where a peg is inserted, throws the grip off balance. The robot wobbles, re-grips, and wastes steps. With the CMT, forces stay even and the plug goes in straight. On average, the CMT needs 2.83% fewer steps per insertion. In a side test on a screw task, a tactile-only policy hits 100% success, beating even the ground-truth-force baseline. This shows that high-resolution tactile alone can drive precise assembly when contact is the whole task.

Speed is also strong. The CMT runs at 153 frames per second with 6.52ms per forward pass, well above the 60Hz bar for real-time robot control.

For teams building automated assembly, three points stand out.

• Tactile sensing is not a nice-to-have for tight-fit tasks: it is a must.
• How you merge sensor data matters as much as the sensors themselves. Buying good tactile sensors is not enough; you need a model that can use them well.
• The physics-based approach cuts the need for task-specific tuning. The balanced grip force works because most gripping tasks share the same physics. And at 153fps, the CMT runs on standard hardware without requiring any special hardware.

All tests ran in simulation. No real-robot results exist yet. The balanced grip force approach was only tested on symmetric plugs. The paper describes how to extend it to lopsided objects via a calibration step, but this remains untested. The gain from adding the balanced grip force to the CMT alone is small (96.22% to 96.59%), and error bars overlap between some methods. The work also skips recent language-guided fusion methods like VTLA, which take a different angle.