arXiv:2605. 05138v2 Announce Type: replace Abstract: We evaluate an initial coding-agent system for ARC-AGI-3 in which the agent maintains an executable Python world model, verifies it against previous observations, refactors it toward simpler abstractions as a practical proxy for an MDL-like simplicity bias, and plans through the model before acting.
By Sergey Rodionov
CivBench is an open‑source benchmark that evaluates language‑model agents in the long‑horizon, tool‑mediated game Civilization VI using the Model Context Protocol (MCP). Each episode lasts over 300 turns, generating thousands of tool calls across a 76‑tool action space, and includes a narration layer that translates visual game state into structured text. The study characterises agent behaviour across four model families, introducing Proactive Monitoring Rate (PMR) and RAG@10 as interface‑level metrics, and finds that agents often under‑monitor strategic state and fail to execute near‑term commitments despite tool access and explicit guidance.
By Austin Tudor David Andrews, Liam Wilkinson, Jamie Heagerty, Harry Coppock, Jakob Nicolaus Foerster, Rui Ponte Costa
arXiv:2607. 02255v1 Announce Type: new Abstract: Memory for a long-horizon LLM agent is a contract about what each future decision is allowed to see.
By Xiangchen Cheng, Yunwei Jiang, Jianwen Sun, Zizhen Li, Chuanhao Li, Xiangcheng Cao, Yihao Liu, Fanrui Zhang, Li Jin, Kaipeng Zhang
Kepler is an open‑source harness that represents hypotheses as executable world models and validates them through retrospective transition checks and conditional prediction checks. In the ARC‑AGI‑3 benchmark, a frozen Claude Opus 5 configuration achieved a perfect 100.00 RHAE on all 25 public games without per‑game model selection or score‑conditioned reruns, and matched or outperformed median‑human action counts on 181 of 183 levels. The study also identified three evaluation failures and highlighted that public‑set score alone has limited discriminative value, advocating for first‑attempt, cost‑conditioned, and verification‑aware reporting.
whyItMatters":"The results demonstrate that a purely score‑based evaluation can be misleading, underscoring the need for more rigorous, cost‑aware, and verification‑aware metrics in AI benchmark assessments."
By Wensen Wu
SpireBench evaluates long‑horizon strategic play by having large language models play the full commercial deck‑builder Slay the Spire, a game where early choices determine long‑term success. Across 20 LLMs at the highest difficulty, frontier models beat heuristic baselines and match average human performance but lag far behind experts, especially as runs lengthen. The benchmark reveals that strong models excel in early fights but misjudge risk over a run, and that their performance depends partly on prior game knowledge rather than on memories written during play.
By Scott C. Lowe, Sander W. Keemink
arXiv:2606. 16014v1 Announce Type: cross Abstract: Many games rely on storytelling combined with systems that track levelling, NPC behaviour, and consequence simulation; bridging tightly-authored narrative with deeply-simulated worlds -- most acute in sandbox and open-world settings -- has been prohibitively expensive.
By Yuhang Huang, Chenmiao Li, Chaowei Fang