What the Specification Did Not Say: Specification-Oracle Gaps in Test-Graded Coding-Agent Benchmarks

The paper identifies a "specification-oracle gap" in coding‑agent benchmarks, where tests rely on details (constants, module paths, exact messages) that are not stated in the natural‑language task description. The authors formalise this gap, develop a static audit tool that classifies such dependencies across five languages, and demonstrate that many benchmarks (e.g., SWE‑Milestone) contain tasks whose tests depend on unstated information, leading to higher failure rates for agents that do not reproduce upstream implementation details. The study also shows that the prevalence of these gaps varies with benchmark construction and that removing affected tests does not alter leaderboard rankings but shifts scores within repositories, favoring agents that replicate upstream names. whyItMatters":"The findings reveal that benchmark tests can inadvertently measure an agent’s ability to mimic upstream implementation details rather than its true problem‑solving skills, potentially skewing evaluations and comparisons across agents."

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

• llms
• agents
• benchmarks