VLMs Trace Without Tracking: Diagnosing Failures in Visual Path Following

Vision‑language models (VLMs) are evaluated on their ability to follow a continuous visual path, a fundamental operation that requires maintaining a selected trajectory through successive visual inputs. In controlled experiments that eliminate confounding factors such as crossings or overlaps, VLMs exhibit fragile tracing behavior, failing to preserve the correct path when other nearby paths are present. Internal representation analysis reveals that the necessary information for correct path continuation weakens already at the visual encoding stage, and common mitigation strategies—model scaling, additional reasoning, or explicit tracing instructions—do not reliably address this vulnerability, even in more realistic visual settings.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

• llms
• multimodal
• benchmarks