What Matters, When? Diagnosing and Improving Conditional Visual Grounding in Visuomotor Imitation Policies

The paper investigates how visuomotor imitation policies fail when visually similar distractors are introduced, framing the issue as conditional visual grounding where the target changes with manipulation phase and task state. Using Action Chunking with Transformers (ACT), the authors systematically vary color and shape similarity of distractors, pinpointing failures to picking and placement stages. They then test distractor augmentation, phase‑dependent attention regularization, and appearance‑based visual prompting, which together significantly improve robustness in simulation and on a physical UR3e robot, and demonstrate similar improvements on a pretrained vision‑language‑action policy for instrument handling.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

• llms
• rag
• robotics
• multimodal