arXiv AI
Build, Don't Play: Self-Improving LLM Research Agents Develop a Hearthstone Battlegrounds AI That Rivals Top Players
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Aug 11
Social Gym and SPaRTan: Benchmarking and Improving LLM Social Reasoning via Multi-Agent Game Tournaments
arXiv:2608. 09128v1 Announce Type: cross Abstract: LLM agents are increasingly deployed in multi-agent social settings where they must cooperate, negotiate, and adapt to other agents.
arXiv AI
Sep 7
$\tau^\tau$-Bench: An Environment for End-To-End, Realistic Agent Construction
The paper introduces $ au^ au$-Bench, a benchmark that turns the construction of AI agents into a measurable task. In this environment a developer agent receives real business records, client requirements, a production API, an existing codebase, and constraints on cost and models, and must deliver a complete customer‑service agent. The benchmark evaluates performance by deploying the agent against simulated users, revealing that current state‑of‑the‑art models achieve only 23.9% success while an expert‑written reference scores 82.2%.
arXiv AI
Sep 17
Compiled Agency: Frontier General-Purpose Coding Agents Build Winning Game Players from Bare Interaction - from Flappy Bird to StarCraft II and Civilization
The paper introduces Gauntlet, a framework that lets large language models autonomously build game-playing agents from a bare contract—just a game description, raw observation/action interface, and an empty policy file. In a single session, the model experiments with the game, compiles a standalone controller, and the resulting program is evaluated on held‑out instances without further model calls. The authors demonstrate that these compiled agents can win full‑scale games such as StarCraft II and Civilization, marking the first time a language‑agent system has achieved standalone victory in such complex titles.
arXiv AI
Oct 12
SpireBench: Evaluating Long-Horizon Strategic Play of Agentic Models with Slay the Spire
SpireBench evaluates long‑horizon strategic play by having large language models play the full commercial deck‑builder Slay the Spire, a game where early choices determine long‑term success. Across 20 LLMs at the highest difficulty, frontier models beat heuristic baselines and match average human performance but lag far behind experts, especially as runs lengthen. The benchmark reveals that strong models excel in early fights but misjudge risk over a run, and that their performance depends partly on prior game knowledge rather than on memories written during play.
arXiv AI
Oct 7
SpeedrunBench: Challenging LLM Agents with Video Game Speedrunning
SpeedrunBench is a new benchmark that tests large language model agents on their ability to develop and refine strategies for completing nine different video games as quickly as possible. The benchmark requires agents to repeatedly improve, reflect, and reason over long action horizons to beat both themselves and other players. Experiments show that while agents can approach human records on simple platformers, they lag behind humans on longer, more complex games within realistic resource limits.
arXiv AI
Jul 3
AgenticSTS: A Bounded-Memory Testbed for Long-Horizon LLM Agents
arXiv:2607. 02255v1 Announce Type: new Abstract: Memory for a long-horizon LLM agent is a contract about what each future decision is allowed to see.