The paper introduces Agent Step Value (ASV), a metric that measures how step rewards change across different evaluation channels in black‑box agent traces. Using a matched 2x2 design on frozen PubMed question‑answering transitions, the authors find that cross‑channel gain signs disagree in 48.3% of cases, whereas same‑channel retries disagree only 0.3% of the time. The study shows that negative magnitudes in the generated‑view coordinate drive a positive direct mean and a negative generated‑view mean, and that source‑only generation can restore a positive mean direction on artifact‑bearing retrievals. The results demonstrate how ASV can localize channel sensitivity and test interventions before reusing step rewards.
arXiv:2607. 04419v3 Announce Type: replace Abstract: When evaluator-derived step rewards are pooled or compared across scoring channels, their sign is treated as transportable.
arXiv:2607. 04419v2 Announce Type: replace Abstract: Final-answer scores hide which agent transitions helped or harmed a trace.
arXiv:2607. 04419v1 Announce Type: new Abstract: Most agent evaluations collapse a multi-step trace into a final answer, a success flag, or a trajectory-level score.
arXiv:2606. 10794v3 Announce Type: replace Abstract: Existing black-box LLM provenance methods achieve comparability by querying every candidate model with the same diagnostic prompts.
The paper introduces TRACE, a protocol designed to diagnose whether changes in verifier scores for large language model agents reflect actual changes in agent behavior or merely alterations in the evaluation process. TRACE works by applying targeted changes to evaluation components, running paired experiments, and re‑scoring unchanged trajectories to isolate the source of score variation. Experiments on synthetic tasks and public benchmark tasks demonstrate that seemingly significant score shifts can often be attributed to evaluation artifacts, while TRACE can also detect genuine behavioral changes.
LLM agents are often evaluated by reducing execution traces to labels such as final-answer correctness, tool verification, and fault propagation. These labels are measurements produced by parsers, beh...
arXiv:2606. 29713v1 Announce Type: cross Abstract: Hallucination is the reliability bottleneck for LLM-based agents, and fact attribution verifiers are the last line of defense -- yet today's verifiers emit only opaque binary labels, leaving agents unable to self-correct and operators unable to audit.
The paper argues that classification accuracy alone is insufficient for evaluating AI agent restart monitors. It introduces a decision‑value framework with a decision‑sufficiency criterion and a finite‑sample decision audit to assess whether restarting an agent is beneficial. Empirical results on SymPy issues show that high accuracy can lead to both gains and losses, and that many traces fail to meet the audit’s certification threshold.
arXiv:2610.03448v1 Announce Type: cross Abstract: LLM agents increasingly screen tool outputs with small prompt-injection detectors, and teams choose among detectors by their scores on public benchma...
The paper investigates how standard evaluations of detectors for hijacked large language model (LLM) agents may conceal important weaknesses. By auditing eleven evaluations on the AgentDrift and AgentDojo benchmarks, the authors show that detectors often perform poorly when faced with rare attacks, unseen tools, or data from a different benchmark, with precision dropping dramatically at low attack rates and rankings reversing as tool coverage falls. They also find that a released LLM action monitor, despite high AUROC on its own data, would still generate many false alerts when applied to a new benchmark. "whyItMatters":"The study highlights that current evaluation practices can give a misleading sense of detector reliability, underscoring the need for more robust, cross‑benchmark testing to ensure real‑world safety of LLM agents."
arXiv:2608. 16391v1 Announce Type: cross Abstract: As large language models become increasingly widespread, third-party providers that deploy open-weight models have become an important part of the ecosystem.
arXiv:2610.02267v1 Announce Type: new Abstract: Agent harnesses make many small, typed decisions per task: which model to call, which tool to use, whether retrieved text is relevant, whether an input...