FM‑Bench is a new benchmark that tests large language model agents on long‑horizon decision‑making by having them run a football club for 20 in‑game years. The agent must manage a squad, trade players, negotiate contracts, invest in facilities, set lineups, and respond to a board that can fire it, all while a deterministic engine aggregates the outcomes into a final score without human judgment. The benchmark evaluates six behavioral capabilities and compares 15 frontier models in solo and arena tracks, revealing that managerial behavior—not computational scale—drives performance.
arXiv:2607. 17765v1 Announce Type: cross Abstract: We introduce WC2026-Agents, a benchmark and dataset for evaluating large language models (LLMs) as autonomous forecasting agents on real, future events.
By Jiacheng Ding, Cong Guo, Jason Xu
SpireBench evaluates long‑horizon strategic play by having large language models play the full commercial deck‑builder Slay the Spire, a game where early choices determine long‑term success. Across 20 LLMs at the highest difficulty, frontier models beat heuristic baselines and match average human performance but lag far behind experts, especially as runs lengthen. The benchmark reveals that strong models excel in early fights but misjudge risk over a run, and that their performance depends partly on prior game knowledge rather than on memories written during play.
By Scott C. Lowe, Sander W. Keemink
arXiv:2607. 02255v1 Announce Type: new Abstract: Memory for a long-horizon LLM agent is a contract about what each future decision is allowed to see.
By Xiangchen Cheng, Yunwei Jiang, Jianwen Sun, Zizhen Li, Chuanhao Li, Xiangcheng Cao, Yihao Liu, Fanrui Zhang, Li Jin, Kaipeng Zhang
StoreBench is a live‑commerce environment where an autonomous agent runs a mid‑size online apparel store on a production‑grade backend, facing continuous customer orders, supplier price changes, and market shocks. The agent uses the same 29 merchant tools as a human operator, with a windowed operation budget that ties simulated time to actions, ensuring that model latency does not affect the simulation. Seven large language models were evaluated across 11 scenarios of 30–45 days and a full simulated year, but none matched the scripted smart‑triage policy, and human experts outperformed all models.
By Daksh Raghuvanshi, Ved Vedere, Yifan Wang
The paper presents a continuous Dyna loop that trains a multi‑agent world model for a ten‑hero MOBA game and evaluates the resulting policy in the real game. By using imagined episodes and real‑game data to continuously refine the model, the policy achieves a 70.2% win rate as radiant on unseen seeds, a dramatic improvement over dream‑only training. The study highlights that internal dream metrics can miss model exploitation, that the world model must be repaired on the policy’s own games, and that the policy’s success stems from macro‑level pressure rather than precise micro‑tactics.
By Jordy Kieto
arXiv:2606. 29771v1 Announce Type: new Abstract: LLM agents are increasingly cast as autonomous portfolio managers, and benchmarks have moved from financial question-answering to sequential trading.
By Bo Qu, Mingguang Chen
arXiv:2609.32805v2 Announce Type: replace
Abstract: Long tasks produce more history than an LLM agent can hold in its context, and more than it uses reliably even when the history fits. A growing lin...
By Bingyu Shen, Boyang Li
ERPBench is a benchmark that evaluates large language model agents in enterprise decision-making through a six‑round ERP simulation covering pricing, production, procurement, inventory, finance, and market competition. It tests the same 100 problems in two market ecologies—Solo, where agents compete against rule‑based opponents, and Arena, where six agents compete together—producing 1,200 model trajectories across 7,200 decision rounds. Results show that model performance varies by ecology, with DeepSeek best in Solo and Gemini best in Arena, and only 21 of 100 problems yield the same top performer across both settings.
By Xinran Zhang, Pengrui Lu, Lyumanshan Ye, Pengfei Liu
arXiv:2610.18354v1 Announce Type: new
Abstract: Game-theoretic benchmarks claim that large language model (LLM) strategic decision-making varies across game types, interaction structures, and capabil...
By Arastun Mammadli, Herbert Woisetschl\"ager, Jawad Fayaz, Shiqiang Wang
arXiv:2608. 00355v1 Announce Type: cross Abstract: Progress in large language models is often summarized using a single scalar measure, such as a time horizon, a latent ability estimate, or an aggregate benchmark score.
By Hanwen Xing, Pengyun Wang, BingXu Meng, Kumail Alhamoud, Xiang Li, Jicheng Wang, Xin Yu, Xinyang Han, Xiaomin Li, Philip Torr, Yuexing Hao
arXiv:2608. 09586v1 Announce Type: new Abstract: The Independent Chip Model (ICM) converts tournament chips into reference prize equity, and policies are routinely constructed against those values.
By Boning Li, Longbo Huang