arXiv:2602. 16902v4 Announce Type: replace Abstract: We introduce LLM-Wikirace, a benchmark for evaluating planning, reasoning, and world knowledge in large language models (LLMs).
By Juliusz Ziomek, William Bankes, Lorenz Wolf, Shyam Sundhar Ramesh, Xiaohang Tang, Ilija Bogunovic
SpeedrunBench is a new benchmark that tests large language model agents on their ability to develop and refine strategies for completing nine different video games as quickly as possible. The benchmark requires agents to repeatedly improve, reflect, and reason over long action horizons to beat both themselves and other players. Experiments show that while agents can approach human records on simple platformers, they lag behind humans on longer, more complex games within realistic resource limits.
By Yoshinari Fujinuma, Keisuke Kamahori, Ryuto Koike, Abdelrahman Madkour, Varun Prashant Gangal, Monty Bichouna, Martyna Markiewicz, Shivani Jain, Duncan Curtis, Rebecca Qian, Anand Kannappan
arXiv:2607. 02255v1 Announce Type: new Abstract: Memory for a long-horizon LLM agent is a contract about what each future decision is allowed to see.
By Xiangchen Cheng, Yunwei Jiang, Jianwen Sun, Zizhen Li, Chuanhao Li, Xiangcheng Cao, Yihao Liu, Fanrui Zhang, Li Jin, Kaipeng Zhang
arXiv:2608. 12626v1 Announce Type: cross Abstract: Strategic reasoning in Large Language Models (LLMs) within long-horizon environments is often limited by inconsistent subgoals.
By Yi Wu, Zhimin Hu
The paper introduces the Pitfall Anticipation World Model (PAWM), a selective guidance system for black-box large language model agents. PAWM predicts imminent failures in a bounded multi-step future and intervenes only when a potential pitfall is detected, reducing costly and often uninformative step-by-step predictions. Trained first with supervised fine-tuning on synthetic pitfall-labeled data and then co-trained with a black-box LLM, PAWM boosts Claude Sonnet 4.5’s performance by 24.3% on ScienceWorld and 7.7% on WebArena-Lite, outperforming several baselines.
By Jiongxiao Wang, Han Ding, Youzhi Luo, Yawei Wang, Yun Zhou, Linbo Liu, Panpan Xu
The paper introduces Gauntlet, a framework that lets large language models autonomously build game-playing agents from a bare contract—just a game description, raw observation/action interface, and an empty policy file. In a single session, the model experiments with the game, compiles a standalone controller, and the resulting program is evaluated on held‑out instances without further model calls. The authors demonstrate that these compiled agents can win full‑scale games such as StarCraft II and Civilization, marking the first time a language‑agent system has achieved standalone victory in such complex titles.
By Joey Xiao, Haonan Huang