arXiv AI
Jul 8
Agent Step Value: Probing the Observer Effect in Black-Box Traces
arXiv:2607. 04419v2 Announce Type: replace Abstract: Final-answer scores hide which agent transitions helped or harmed a trace.
Agent Step Value: State-Transition Measurement with State-Grounded LLM Evaluators
arXiv:2607. 04419v1 Announce Type: new Abstract: Most agent evaluations collapse a multi-step trace into a final answer, a success flag, or a trajectory-level score.
Related stories
arXiv AI
Jul 14
Agent Step Value: Auditing Evaluator-Channel Reversals in Black-Box Agent Traces
arXiv:2607. 04419v3 Announce Type: replace Abstract: When evaluator-derived step rewards are pooled or compared across scoring channels, their sign is treated as transportable.
arXiv Machine Learning
Jul 31
ClawTrack: Towards Trace-Level Evaluation and Improvement of Real-World Autonomous Agents
arXiv:2607. 28037v1 Announce Type: new Abstract: As LLM-based agents are deployed in complex, multi-step workflows, a critical evaluation gap has emerged: most existing benchmarks judge only final outcomes, unable to distinguish reliable reasoning from lucky success or attribute failures to specific process deficiencies, hindering attribution in long-horizon tasks.
arXiv AI
Jul 16
STOCKTAKE: Measuring the Gap Between Perception and Action in LLM Agents with a Fair Oracle
arXiv:2607. 13618v1 Announce Type: new Abstract: LLM agents are increasingly evaluated on multi-week decision tasks in which the state that drives cost is never directly observed.
arXiv AI
Jul 3
GroundEval: A Deterministic Replacement for LLM-as-Judge in Stateful Agent Evaluation
arXiv:2606. 22737v2 Announce Type: replace Abstract: Before letting an agent operate over real context, can you prove it used the right evidence?
arXiv AI
Aug 12
Auditing Automated Evaluation, Error Propagation, and Runtime Mitigation in Tool-Using Language Agents
arXiv:2604. 16706v2 Announce Type: replace Abstract: Automated evaluation of tool-using large language model (LLM) agents is widely assumed to be reliable, yet this assumption is rarely validated against human annotation.
arXiv AI
Jun 30
SEVA: Self-Evolving Verification Agent with Process Reward for Fact Attribution
arXiv:2606. 29713v1 Announce Type: cross Abstract: Hallucination is the reliability bottleneck for LLM-based agents, and fact attribution verifiers are the last line of defense -- yet today's verifiers emit only opaque binary labels, leaving agents unable to self-correct and operators unable to audit.
Hugging Face Trending Papers
Jul 15
STOCKTAKE: Measuring the Gap Between Perception and Action in LLM Agents with a Fair Oracle
LLM agents are increasingly evaluated on multi-week decision tasks in which the state that drives cost is never directly observed. On such tasks the final cost cannot say why an agent failed: it may have misread the world, or read it correctly and still failed to act (the knowing-doing gap).
arXiv AI
Jun 2
AgentProcessBench: Diagnosing Step-Level Process Quality in Tool-Using Agents
arXiv:2603. 14465v2 Announce Type: replace Abstract: While Large Language Models (LLMs) have evolved into tool-using agents, they remain brittle in long-horizon interactions.
arXiv AI
Aug 11
Autorubric: A Unifying Framework for Rubric-Based LLM Evaluation on Non-Verifiable Tasks
arXiv:2603. 00077v3 Announce Type: replace-cross Abstract: Rubric-based LLM judges have become indispensable for evaluating and optimizing systems on non-verifiable tasks, where success cannot be reduced to exact programmatic checks.
arXiv AI
Aug 7
SearchAuditor: Auditing and Attributing Failures in Long-Horizon Search Agents
arXiv:2608. 05212v1 Announce Type: new Abstract: Deep search agents tackle challenging questions through long-horizon web interactions, a process that is both complex and fragile: small reasoning errors may propagate through long, noisy trajectories into fluent but incorrect answers.
arXiv AI
2d ago
Split the Labor: Separating Evidence Interpretation from Decision Aggregation
arXiv:2608. 14509v1 Announce Type: new Abstract: Systems that ask a language model to reach a conclusion from many sources usually concatenate them into one prompt.