← Back to all news
arXiv Computation and Language

Noise Floor Audit for Agent Benchmarks

The study audits measurement variability for three native tool‑calling endpoints across two providers on the official BFCL benchmark, using matched AST grading. At temperature 0, reruns are nearly deterministic, with very low ever‑flip fractions and high run correlations. However, semantics‑preserving prompt perturbations introduce a larger noise floor, and the failure modes shift, with malformed‑output failures becoming a significant portion of task failures.

Related stories

arXiv AI
Sep 17

AutoTuneBench: Trustworthy Measurement for Agent Auto-Tuning of LLM Serving Engines

AutoTuneBench introduces a trustworthy measurement protocol for evaluating how large language model agents auto‑tune GPU kernels and serving engines. The benchmark addresses four failure modes—strawman baselines, machine‑dependent timing, saturated tasks, and infrastructure defects—by enforcing code‑frozen protocols, database validation, anti‑cheat checks, pre‑registered comparisons, and external result anchoring. Using this protocol, the authors demonstrate that previously reported speedups are inflated, revealing more modest improvements across different engines and machines.

By Li Chen
llmsagentsbenchmarks
More like this →
arXiv AI
Sep 4

Clean Engineering, Unstable Measurement: A Preregistered Reliability Failure of Black-Box LLM Observers on Shared Endpoints

The paper reports a preregistered audit of language‑model judges used as measurement instruments, revealing that the assumption that a model’s responses remain stable over time is invalid. Across nearly 53,000 audited requests, repeat rankings and byte‑identical replays fell far below required reliability thresholds, with three identified mechanisms—label‑to‑meaning bias, candidate gaps below the noise floor, and input permutation noise—explaining the discrepancy. The study proposes a three‑level snapshot‑identity framework, eight design rules, and a reporting checklist to prevent such reliability failures in future evaluations.

By Haoyaun Zhu, Jie Zhang
llms
More like this →
arXiv Machine Learning
Aug 4

Real-Time Detection and Repair of LLM Agent Failures

arXiv:2608. 02464v1 Announce Type: cross Abstract: LLM agents fail mid-episode -- they loop, cascade tool errors, drift off goal, fabricate results, or silently absorb corrupted content -- and the standard remedy, judging every step with a second LLM, costs more than the agent itself.

By Sunny Dubey
llmsagents
More like this →
arXiv AI
Oct 2

Groundability, Not Scale Alone: When Weak Reviewers Can Audit Strong Coding Agents

The paper investigates whether reviewers with limited expertise can reliably determine if a coding agent’s patch resolves a bug, using 411 execution-labeled traces from three agents and 101 controlled cases. It finds that providing structured but unchecked evidence can improve defect detection and reduce over-rejection, and that a frozen cascade of static errors and generated tests achieves high coverage and catch rates on held‑out GPT‑5.4 and Gemini traces. Reviewer size alone does not predict quality, and the study highlights the challenge of creating reliable checks without official tests.

By Junyu Guo, Shangding Gu, Ming Jin, Javad Lavaei
llmsagents
More like this →
Hugging Face Trending Papers
Sep 3

Clean Engineering, Unstable Measurement: A Preregistered Reliability Failure of Black-Box LLM Observers on Shared Endpoints

The paper investigates the reliability of language‑model judges used as measurement instruments on shared endpoints. Through two preregistered audits of 52,988 requests, the authors found that repeat rankings and byte‑identical replays fell far short of required thresholds, revealing significant instability. They identify three mechanisms—label‑to‑meaning bias, candidate gaps below the noise floor, and input permutation noise—that explain the gap, and propose a snapshot‑identity ladder, design rules, and a reporting checklist to mitigate such failures.

llms
More like this →