arXiv Computation and Language
Sep 14

LLM-BabyBench: Can Language Models Plan in Worlds They Can Simulate?

LLM‑BabyBench transforms the BabyAI gridworld into a fully observable, purely textual setting that isolates planning as the sole source of failure. By serialising the entire grid, providing formal instructions, and validating actions deterministically, the benchmark introduces the PPD suite—Predict, Plan, and Decompose tasks—each scored with metrics that separate mission understanding from sequencing. Across a range of large language models, simulation accuracy is high while planning success drops sharply beyond a model‑specific horizon, revealing that plan length—not grid size—drives failure and that models often commit to a single corridor‑shaped route without backtracking.

By Idriss Malek, Omar Choukrani, Daniil Orel, Anh Duy Le Dinh, Zhuohan Xie, Zangir Iklassov, Martin Tak\'a\v{c}, Salem Lahlou
arXiv AI
Oct 5

Keeping JEPA World Models Plannable When Little of the Frame Moves

The paper introduces SLIM, a pushing benchmark that tests language‑based goal specification for latent world models. It shows that a vanilla LeWM fails almost entirely due to an action‑insensitive encoder, but adding an inverse‑dynamics auxiliary loss restores encoder sensitivity and dramatically improves success rates. With the repaired latent, a lightweight language‑goal head can plan from natural language sentences, achieving high performance on navigation and pushing tasks without retraining the world model.

By Florian Strohm, Patrick Wagner, Jannik Schwab, Marco Huber