arXiv AI October 12, 2026 By Andrew Zhang, Chengzhan Li

Agent Step Value: Auditing Evaluation-Channel Sensitivity in Black-Box Agent Traces

The paper introduces Agent Step Value (ASV), a metric that measures how step rewards change across different evaluation channels in black‑box agent traces. Using a matched 2x2 design on frozen PubMed question‑answering transitions, the authors find that cross‑channel gain signs disagree in 48.3% of cases, whereas same‑channel retries disagree only 0.3% of the time. The study shows that negative magnitudes in the generated‑view coordinate drive a positive direct mean and a negative generated‑view mean, and that source‑only generation can restore a positive mean direction on artifact‑bearing retrievals. The results demonstrate how ASV can localize channel sensitivity and test interventions before reusing step rewards.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

agents

arXiv AI

Jul 14

Agent Step Value: Auditing Evaluator-Channel Reversals in Black-Box Agent Traces

arXiv:2607. 04419v3 Announce Type: replace Abstract: When evaluator-derived step rewards are pooled or compared across scoring channels, their sign is treated as transportable.

By Andrew Zhang, Chengzhan Li

agents

arXiv AI

Jul 8

Agent Step Value: Probing the Observer Effect in Black-Box Traces

arXiv:2607. 04419v2 Announce Type: replace Abstract: Final-answer scores hide which agent transitions helped or harmed a trace.

By Andrew Zhang, Chengzhan Li

llms agents

arXiv AI

Jul 7

Agent Step Value: State-Transition Measurement with State-Grounded LLM Evaluators

arXiv:2607. 04419v1 Announce Type: new Abstract: Most agent evaluations collapse a multi-step trace into a final answer, a success flag, or a trajectory-level score.

By Andrew Zhang, Chengzhan Li

llms agents

arXiv AI

Aug 11

READER: Dynamic LLM Provenance from Query-Varying Interactions

arXiv:2606. 10794v3 Announce Type: replace Abstract: Existing black-box LLM provenance methods achieve comparability by querying every candidate model with the same diagnostic prompts.

By Jiaxu Liu, Sunnan Mu, Dong Huang, Liuyin Wang, Jing Shao, Jie Zhang

llms agents

arXiv Computation and Language

Oct 9

TRACE: Diagnosing Verifier Brittleness in Agentic Evaluation

The paper introduces TRACE, a protocol designed to diagnose whether changes in verifier scores for large language model agents reflect actual changes in agent behavior or merely alterations in the evaluation process. TRACE works by applying targeted changes to evaluation components, running paired experiments, and re‑scoring unchanged trajectories to isolate the source of score variation. Experiments on synthetic tasks and public benchmark tasks demonstrate that seemingly significant score shifts can often be attributed to evaluation artifacts, while TRACE can also detect genuine behavioral changes.

By Radhika Gaonkar

llms agents benchmarks

Hugging Face Trending Papers

Oct 21

What the Trace Shows and What the Label Says: Auditing Behavioral Detectors in Agent Evaluation

LLM agents are often evaluated by reducing execution traces to labels such as final-answer correctness, tool verification, and fault propagation. These labels are measurements produced by parsers, beh...

llms agents benchmarks

One email a morning, machine-written

Related stories

Agent Step Value: Auditing Evaluator-Channel Reversals in Black-Box Agent Traces

Agent Step Value: Probing the Observer Effect in Black-Box Traces

Agent Step Value: State-Transition Measurement with State-Grounded LLM Evaluators

READER: Dynamic LLM Provenance from Query-Varying Interactions

TRACE: Diagnosing Verifier Brittleness in Agentic Evaluation

What the Trace Shows and What the Label Says: Auditing Behavioral Detectors in Agent Evaluation