AutoTuneBench introduces a trustworthy measurement protocol for evaluating how large language model agents auto‑tune GPU kernels and serving engines. The benchmark addresses four failure modes—strawman baselines, machine‑dependent timing, saturated tasks, and infrastructure defects—by enforcing code‑frozen protocols, database validation, anti‑cheat checks, pre‑registered comparisons, and external result anchoring. Using this protocol, the authors demonstrate that previously reported speedups are inflated, revealing more modest improvements across different engines and machines.
By Li Chen
The paper reports a preregistered audit of language‑model judges used as measurement instruments, revealing that the assumption that a model’s responses remain stable over time is invalid. Across nearly 53,000 audited requests, repeat rankings and byte‑identical replays fell far below required reliability thresholds, with three identified mechanisms—label‑to‑meaning bias, candidate gaps below the noise floor, and input permutation noise—explaining the discrepancy. The study proposes a three‑level snapshot‑identity framework, eight design rules, and a reporting checklist to prevent such reliability failures in future evaluations.
By Haoyaun Zhu, Jie Zhang
arXiv:2610.16067v1 Announce Type: cross
Abstract: Agent evaluation harnesses are software systems that run test suites, generate inputs, and decide outcomes with test oracles. This paper audits such...
By Junjie Chen
arXiv:2608. 02464v1 Announce Type: cross Abstract: LLM agents fail mid-episode -- they loop, cascade tool errors, drift off goal, fabricate results, or silently absorb corrupted content -- and the standard remedy, judging every step with a second LLM, costs more than the agent itself.
By Sunny Dubey
The paper investigates whether reviewers with limited expertise can reliably determine if a coding agent’s patch resolves a bug, using 411 execution-labeled traces from three agents and 101 controlled cases. It finds that providing structured but unchecked evidence can improve defect detection and reduce over-rejection, and that a frozen cascade of static errors and generated tests achieves high coverage and catch rates on held‑out GPT‑5.4 and Gemini traces. Reviewer size alone does not predict quality, and the study highlights the challenge of creating reliable checks without official tests.
By Junyu Guo, Shangding Gu, Ming Jin, Javad Lavaei
arXiv:2606. 09863v1 Announce Type: new Abstract: LLM agents can fail silently by asserting task completion when the environment state shows otherwise.
By Laksh Advani