Large language model (LLM) agents now fix software, operate computers and serve customers, but their harnesses, the programs that run them, call a large LLM at every step, which makes long tasks slow...
JevAgentHarness is a new framework that benchmarks, deploys, and trains decision models to share control with large language models (LLMs) in agentic tasks. It introduces AgentDecisionBench to evaluate decision models from isolated questions to full agent control, revealing that accurate isolated decisions still miss many tasks and that naïve delegation can reduce success. By fitting confidence thresholds per choice and training a 4B decision model on states after agent mistakes, JevAgentHarness achieves high task success while cutting cost and speeding up execution, with results and resources made publicly available.
By Kunlun Zhu, Tao Feng, Xu Han, Heng Wang, Ling Yang, Xiangru Tang, Jiaxuan You, Heng Ji
The paper introduces TRACE, a protocol designed to diagnose whether changes in verifier scores for large language model agents reflect actual changes in agent behavior or merely alterations in the evaluation process. TRACE works by applying targeted changes to evaluation components, running paired experiments, and re‑scoring unchanged trajectories to isolate the source of score variation. Experiments on synthetic tasks and public benchmark tasks demonstrate that seemingly significant score shifts can often be attributed to evaluation artifacts, while TRACE can also detect genuine behavioral changes.
By Radhika Gaonkar
arXiv:2610.02588v1 Announce Type: new
Abstract: Agents are increasingly given open-ended research tasks: discovering an empirical law from self-designed experiments, improving a heuristic whose optim...
By Chengyang Shi, Xianglin Ji, Jintao Huang, Jicheng Wang, Yifeng He, Jiachen Liu
The paper investigates how language‑model judges can make version‑dependent errors when evaluating upgraded agents. Using 35 public coding‑agent submissions, two customer‑service agents, and over a thousand expert‑labeled trajectories, the authors show that fixed judges often reject task‑conditioned error invariance and can incorrectly approve failed patches, especially as agent capability increases. Paired audits of current outputs reduce interval width only marginally, and the study concludes that independent human patch review is still necessary.
By Jiapeng Li
arXiv:2610.18298v1 Announce Type: new
Abstract: What binds large language models on auditable, hypothesis-driven decision tasks, where every candidate can be checked against the evidence? On the publ...
By Jing Liu, Yongxing Qi, Ping Yao, Ting Wu