arXiv AI
Discriminating Fixture Coverage in Agent-Infrastructure Verification Suites
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Oct 1
Hard-Gate Candidacy in a Deployed Validator Suite
The paper evaluates hard‑gate candidacy for validators in a deployed generative‑agent system by measuring how well each validator’s firing separates successful from failed builds. Across 13 validators and thousands of builds, only a few checks show statistically significant separation, while many fail to distinguish or never fire. The study highlights that skipped checks are recorded as passes, limiting detectable failure rates and underscoring the need for clearer evaluation records.
arXiv AI
Jun 10
Regimes: An Auditable, Held-Out-Gated Improvement Loop Demonstrated on LongMemEval with ActiveGraph
arXiv:2606. 10241v1 Announce Type: new Abstract: Autonomous improvement loops are hard to trust because the improvement process is usually external scaffolding bolted onto the agent: failures go unlogged, diagnoses cannot be replayed, and promote-or-discard decisions land in a side database rather than the agent's own history.
arXiv AI
Aug 26
Rebuild Dossier: Mechanically-Enforced Specs for Agentic App Rebuilds, and What Model-Tier Failures Reveal
The paper introduces rebuild‑dossier, an open‑source tool that locks an application’s real interface before code is written and enforces one‑test‑at‑a‑time building through automated checks. In experiments, a compliant agent failed a held‑back test while a rule‑breaking agent passed, showing that a passing test suite can be gamed. The study also demonstrates that the automated check mechanism, rather than interface‑locking alone, is crucial for reliable rebuilds, and that multi‑level verification catches errors that single‑level checks miss.
arXiv AI
Sep 3
ClaimReceipt: Verifying Evidence Sufficiency and Coverage in Agent Evaluations
The paper introduces ClaimReceipt, a specification and verifier that checks whether a claim in an agent evaluation can be recomputed from retained evidence (sufficiency) and whether the evidence covers the entire experiment set (coverage). Using the CR‑2 verifier on 1,392 historical records, the authors demonstrate accurate reproduction of audit verdicts, non‑redundant field groups, and zero false positives on semantic faults. In a prospective CR‑3 run, the system correctly flags missing receipts and preserves coverage when private evidence is withheld, while adding minimal overhead to inference time and transaction size.
arXiv AI
Oct 15
When Agent Evaluation Becomes Unreliable: An Empirical Study of Oracle Failures and Configuration-Dependent Measurement in an LLM Agent Benchmark
arXiv:2610.16067v1 Announce Type: cross Abstract: Agent evaluation harnesses are software systems that run test suites, generate inputs, and decide outcomes with test oracles. This paper audits such...
arXiv AI
Oct 2
A Verifier Can Leak the Answer: Diagnosability Before Optimization in Closed-Loop Agent Debugging
The paper demonstrates that a verifier used in closed‑loop agent debugging can inadvertently reveal the answer it is meant to test, rendering solver comparisons meaningless. In a study of 12 development cases, both exact minimum hitting set and a greedy method returned identical supports, and an audit showed that exact‑anchor predicates always produced the planted fault pair. The authors propose a support‑gated verification contract that requires a clean reference map and runtime evidence before an independently calibrated signal can confirm a detection, and validate this approach on 1,440 held‑out cases with a low false‑admission rate.