arXiv AI
GameLogicBench: Evaluating Coding Agents on Runtime Game Logic with Tick-Level State Assertions
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Aug 25
GameXpert-Bench: How Far Are Coding Agents from Expert Game Development?
arXiv:2608.21833v1 Announce Type: new Abstract: Recent large language models (LLMs) can operate as coding agents that build complete games from natural language requests. Game development is especial...
arXiv AI
Sep 21
GameASG-Bench: Benchmarking Autonomous Software Generation for Game Development
arXiv:2609.21293v1 Announce Type: new Abstract: Autonomous software generation (ASG) aims to turn human requirements into executable applications, but delivering these applications does not necessari...
arXiv AI
Oct 9
TestJack: Should you trust the results in coding benchmarks? Agentic Coding Benchmarks Auditing via Evaluator Evolution
TestJack is a scalable framework that audits large language model (LLM) coding agents by generating new tests for each trial, focusing on prompt requirements that may be violated. It keeps only the tests passed by the ground‑truth patch and re‑examines failures, providing replayable evidence for each confirmed error. In experiments across six model backends and five benchmarks, TestJack found that 34.4% of trials deemed correct actually violated task requirements, reducing the overall resolution rate from 50.6% to 33.2%.
arXiv AI
Oct 1
A2Z GameSpec-Bench: How Faithfully Can Coding Agents Generate Games from Game Design Specifications?
A2Z GameSpec-Bench introduces a benchmark of 100 long‑form Game Design Documents (GDDs) to evaluate how faithfully coding agents can generate complete games from detailed specifications. The benchmark measures faithfulness by checking that the game satisfies the GDD requirements and preserves the relationships among them, using a dependency‑aware contract and a combination of source‑code inspection and agent‑generated test policies. Evaluations show that current agents struggle to meet interdependent requirements, but requirement‑specific feedback improves GDD fidelity by 10.9% after two revision rounds.
arXiv AI
Oct 5
OpenGameEval: Benchmarking Agentic Programming and Exploration in a Stateful Game Engine
arXiv:2610.02563v1 Announce Type: cross Abstract: We present OpenGameEval, a benchmark and evaluation framework for agentic game development inside Roblox Studio. It runs language models as agents in...
arXiv AI
Sep 23
CraftBench-UE: Deterministic Evaluation for Coding Agents in Unreal Engine
arXiv:2609.23142v1 Announce Type: new Abstract: Building gameplay features in a game engine requires more than code, as code that compiles and runs does not necessarily implement the requested gamepl...