Choosing Coding-Agent Benchmarks for Practice: Lessons from Task Demands and Agent Behaviour
Read the original on arXiv Computation and Language →The paper analyzes 2,487 tasks from five public coding‑agent benchmarks and 14,922 runs of six agent configurations, introducing the Spread‑Novelty‑Centrality (SNC) profile to describe how reference solutions are distributed, added, and the code’s dependencies, history, and complexity. It finds that every pair of benchmarks differs on at least two SNC axes, even when sharing the same task label, and that resolved runs tend to involve less distributed changes and lower centrality values, with patch sizes varying between Claude and Qwen configurations. The authors recommend that practitioners choose evaluation tasks based on change characteristics, inspect task descriptions, and assess patch scope alongside test outcomes to better align benchmarks with real‑world workloads.
Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.