arXiv AI
Trace Evolution Bench: Measuring Whether Agents Improve from Their Own Environment Traces
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Oct 9
Harness Evolution Hits a Ceiling: When Weight Training Should Begin
The paper investigates when to evolve a harness versus training model weights for long‑horizon LLM agents. By labeling failed trajectories with the first signal that fires, the authors separate process failures from content failures, showing that harness evolution fixes the former while weight training addresses the latter. Experiments on DeepPlanning and WebArena‑Lite demonstrate significant performance gains, with LoRA adapters trained on evolved harness trajectories further improving held‑out scores and reducing content failures.
arXiv Computation and Language
Oct 9
TRACE: Diagnosing Verifier Brittleness in Agentic Evaluation
The paper introduces TRACE, a protocol designed to diagnose whether changes in verifier scores for large language model agents reflect actual changes in agent behavior or merely alterations in the evaluation process. TRACE works by applying targeted changes to evaluation components, running paired experiments, and re‑scoring unchanged trajectories to isolate the source of score variation. Experiments on synthetic tasks and public benchmark tasks demonstrate that seemingly significant score shifts can often be attributed to evaluation artifacts, while TRACE can also detect genuine behavioral changes.
arXiv AI
5d ago
Verifier-Harness Co-Evolution for Training-Free Self-Improvement on Open-Ended Tasks
arXiv:2610.17609v1 Announce Type: new Abstract: Training-free self-improvement lets an agent use feedback to revise the response it is about to return and to update the harness state that shapes how...
arXiv AI
Aug 5
Rethinking Self-Evolving Agent Skills: Feedback Dynamics over Multiple Rounds
arXiv:2608. 02636v1 Announce Type: cross Abstract: Self-evolving skill systems promise to improve agents by turning execution feedback into persistent skill updates without changing the underlying model.
arXiv AI
Jul 15
Who Grades the Grader? Co-Evolving Evaluation Metrics and Skills for Self-Improving LLM Agents
arXiv:2607. 12790v1 Announce Type: new Abstract: Self-evolving agent systems improve by creating, revising, and retiring their own skills, but every such loop rests on a hidden assumption: a reliable evaluation metric already exists.
arXiv Machine Learning
Sep 30
Frozen Judges, Moving Agents: Version-Dependent LLM-Judge Error and the Limits of Judge-Assisted Agent Evaluation
The paper investigates how language‑model judges can make version‑dependent errors when evaluating upgraded agents. Using 35 public coding‑agent submissions, two customer‑service agents, and over a thousand expert‑labeled trajectories, the authors show that fixed judges often reject task‑conditioned error invariance and can incorrectly approve failed patches, especially as agent capability increases. Paired audits of current outputs reduce interval width only marginally, and the study concludes that independent human patch review is still necessary.