arXiv Machine Learning
The Trace Is the State: Exact Credit Assignment for LLM Agent Teams
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv AI
Aug 21
Credit Without Ground Truth: Auditing Step-Level Credit Assignment in LLM Agents Against Executed Replay
arXiv:2608. 19760v1 Announce Type: cross Abstract: Audited against causal ground truth from executed replay in a single-agent tool environment (ALFWorld), none of the step-level credit signals used to train LLM agents -- LLM-judge scores, outcome-conditioned logprob ratios, or the policy's own confidence -- identifies which steps causally matter better than chance.
arXiv Computation and Language
3d ago
TRACE: Diagnosing Verifier Brittleness in Agentic Evaluation
The paper introduces TRACE, a protocol designed to diagnose whether changes in verifier scores for large language model agents reflect actual changes in agent behavior or merely alterations in the evaluation process. TRACE works by applying targeted changes to evaluation components, running paired experiments, and re‑scoring unchanged trajectories to isolate the source of score variation. Experiments on synthetic tasks and public benchmark tasks demonstrate that seemingly significant score shifts can often be attributed to evaluation artifacts, while TRACE can also detect genuine behavioral changes.
arXiv AI
Aug 17
Inducing Reward-Free Judging Rubrics that Reduce Over-Crediting in Agent Evaluation
arXiv:2608. 13564v1 Announce Type: new Abstract: Evaluating language-model agents at scale increasingly relies on a second language model as an automatic judge, because the gold signal, an executable environment reward, is expensive, slow, or unavailable at deployment time.
arXiv Machine Learning
Jun 16
Re-feeding Is Not Replaying: Measuring Replay Noise in Counterfactual Token-Credit Estimation
arXiv:2606. 15621v1 Announce Type: new Abstract: Per-token counterfactual credit estimation asks which token in a language-model rollout caused the final answer to be right or wrong: cut the transcript at a pivot, substitute an alternative token, replay continuations, and compare outcomes.
arXiv Machine Learning
Sep 23
How Strongly Should Task State Influence an LLM Agent?
arXiv:2609.25686v1 Announce Type: cross Abstract: Long-horizon assigned work requires an LLM agent to track the state of a task: which steps are done, blocked, cancelled, or open to repetition. Agent...
arXiv AI
Jul 14
Agent Step Value: Auditing Evaluator-Channel Reversals in Black-Box Agent Traces
arXiv:2607. 04419v3 Announce Type: replace Abstract: When evaluator-derived step rewards are pooled or compared across scoring channels, their sign is treated as transportable.