arXiv AI By Li Chen

AutoTuneBench: Trustworthy Measurement for Agent Auto-Tuning of LLM Serving Engines

Read the original on arXiv AI →

AutoTuneBench introduces a trustworthy measurement protocol for evaluating how large language model agents auto‑tune GPU kernels and serving engines. The benchmark addresses four failure modes—strawman baselines, machine‑dependent timing, saturated tasks, and infrastructure defects—by enforcing code‑frozen protocols, database validation, anti‑cheat checks, pre‑registered comparisons, and external result anchoring. Using this protocol, the authors demonstrate that previously reported speedups are inflated, revealing more modest improvements across different engines and machines.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI
Oct 23

VerifierBench: Benchmarking the Generation of Executable Verifiers for LLM Agents

The paper introduces VerifierBench, a benchmark comprising 222 tasks from six agent benchmarks and 4,993 hidden test cases designed to evaluate the generation of executable verifiers by large language model agents. It presents Verifier4Verifier (V4V), a six-check audit that screens source verifiers and test suites, and reports that across 19 models, normalized balanced accuracy ranges from 10.6 to 62.1, with generated verifiers more reliably rejecting invalid solutions than accepting valid ones. Case studies highlight that many false rejections stem from verifiers that accept no tested solution, revealing unsupported assumptions about solution formats, tool use, and code changes that exclude valid solutions, and underscore the need to test both acceptance of correct solutions and other permissible solutions.

By Jianing Yin, Ziyi Chen, Hao Wang, Yiyou Sun, Zijian Jin, Dawn Song, Daoan Zhang
arXiv AI
Sep 4

Clean Engineering, Unstable Measurement: A Preregistered Reliability Failure of Black-Box LLM Observers on Shared Endpoints

The paper reports a preregistered audit of language‑model judges used as measurement instruments, revealing that the assumption that a model’s responses remain stable over time is invalid. Across nearly 53,000 audited requests, repeat rankings and byte‑identical replays fell far below required reliability thresholds, with three identified mechanisms—label‑to‑meaning bias, candidate gaps below the noise floor, and input permutation noise—explaining the discrepancy. The study proposes a three‑level snapshot‑identity framework, eight design rules, and a reporting checklist to prevent such reliability failures in future evaluations.

By Haoyaun Zhu, Jie Zhang
Hugging Face Trending Papers
Sep 3

Clean Engineering, Unstable Measurement: A Preregistered Reliability Failure of Black-Box LLM Observers on Shared Endpoints

The paper investigates the reliability of language‑model judges used as measurement instruments on shared endpoints. Through two preregistered audits of 52,988 requests, the authors found that repeat rankings and byte‑identical replays fell far short of required thresholds, revealing significant instability. They identify three mechanisms—label‑to‑meaning bias, candidate gaps below the noise floor, and input permutation noise—that explain the gap, and propose a snapshot‑identity ladder, design rules, and a reporting checklist to mitigate such failures.