FM‑Bench is a new benchmark that tests large language model agents on long‑horizon decision making by having them run a football club for 20 in‑game years. The agent must manage a squad, trade players, negotiate contracts, invest in facilities and youth, set lineups, and respond to a board that can fire it, all using 26 tools and roughly 340–400 decision stops, with a deterministic engine producing a final score without human or LLM judges. The benchmark includes a solo track where each of 15 frontier models competes against a frozen scripted world, and an Arena track where the same models plus a scripted anchor share one 20‑year world, allowing the first head‑to‑head evaluation at this scale.
whyItMatters":"FM‑Bench provides a rigorous, large‑scale test of sustained, cumulative decision‑making in language‑model agents, revealing that managerial strategy—not computational scale or vendor—drives performance over long horizons."
By Tianyou Wang, Chongyang Gao, Kezhen Chen, Chen Dong, Yinghao He, Donghan Li, Wangcheng Xu, Hongjiu Zhang, Chi Li
arXiv:2607. 17765v1 Announce Type: cross Abstract: We introduce WC2026-Agents, a benchmark and dataset for evaluating large language models (LLMs) as autonomous forecasting agents on real, future events.
By Jiacheng Ding, Cong Guo, Jason Xu
SpireBench evaluates long‑horizon strategic play by having large language models play the full commercial deck‑builder Slay the Spire, a game where early choices determine long‑term success. Across 20 LLMs at the highest difficulty, frontier models beat heuristic baselines and match average human performance but lag far behind experts, especially as runs lengthen. The benchmark reveals that strong models excel in early fights but misjudge risk over a run, and that their performance depends partly on prior game knowledge rather than on memories written during play.
By Scott C. Lowe, Sander W. Keemink
arXiv:2607. 02255v1 Announce Type: new Abstract: Memory for a long-horizon LLM agent is a contract about what each future decision is allowed to see.
By Xiangchen Cheng, Yunwei Jiang, Jianwen Sun, Zizhen Li, Chuanhao Li, Xiangcheng Cao, Yihao Liu, Fanrui Zhang, Li Jin, Kaipeng Zhang
StoreBench is a live‑commerce environment where an autonomous agent runs a mid‑size online apparel store on a production‑grade backend, facing continuous customer orders, supplier price changes, and market shocks. The agent uses the same 29 merchant tools as a human operator, with a windowed operation budget that ties simulated time to actions, ensuring that model latency does not affect the simulation. Seven large language models were evaluated across 11 scenarios of 30–45 days and a full simulated year, but none matched the scripted smart‑triage policy, and human experts outperformed all models.
By Daksh Raghuvanshi, Ved Vedere, Yifan Wang
arXiv:2606. 29771v1 Announce Type: new Abstract: LLM agents are increasingly cast as autonomous portfolio managers, and benchmarks have moved from financial question-answering to sequential trading.
By Bo Qu, Mingguang Chen