World Modeling in Transformers

The paper investigates how transformers can possess a world model despite exhibiting behavioral failures. Using TaxiGPT, a transformer trained on random Manhattan walks, the authors show that the model internally represents intersections, streets, and its position, and uses a goal compass for navigation. They attribute failures to interference between overlapping intersection features and demonstrate that affordance packing mitigates these errors, concluding that world‑modeling abilities emerge at distinct training stages and should be studied mechanistically rather than merely observed behaviorally.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

• llms