The paper investigates how reinforcement learning can cause large language model agents to adopt shortcut policies for tool use, relying on superficial prompt cues rather than actual task needs. By creating synthetic environments that mix factual QA and math reasoning, the authors show that agents often invoke tools when cues are present, even when those tools are unnecessary, with spurious invocation rates rising up to 39%. They find that shortcut learning occurs mainly when agents have already mastered the target tool and that semantic alignment between cues and tools amplifies the effect. To counter this, they propose a dense, decision-level reward where an LLM judge assesses tool necessity, which reduces cue-driven tool use while maintaining performance.
By Yiwei Yang, Haoxiang Zhang, Bingbing Wen, Yao Lu, Yuchen Wu, Lei Zhang, Julian McAuley, Pan Lu, Bill Howe
The paper introduces RUPA, a trajectory‑level uncertainty quantification framework for large language model agents. RUPA models an agent’s execution as a directed graph of reasoning states, tool interactions, and environment feedback, then propagates uncertainty across this graph to capture long‑range dependencies. Experiments on benchmarks such as τ‑2, Terminal‑Bench‑2, and GAIA show that RUPA outperforms existing methods, enabling earlier failure detection and more reliable agent execution.
By Zhengzhao Ma. Boxi Cao, Yaojie Lu, Hongyu Lin, Xianpei Han, Le Sun
Natural Language Actor-Critic (NLAC) is a new actor‑critic algorithm that trains large language model (LLM) agents by using a generative LLM critic to produce natural‑language feedback instead of scalar rewards. By providing richer, actionable explanations for suboptimal actions, NLAC helps LLM policies improve without relying on random exploration, and it can be trained off‑policy without policy gradients. Experiments on reasoning, web browsing, tool‑use, and dialogue tasks show that NLAC outperforms existing training methods and offers a more scalable and stable training paradigm for LLM agents.
By Joey Hong, Kang Liu, Zhan Ling, Jiecao Chen, Sergey Levine
SWE-Critix is a post‑training pipeline that creates generalist verifiers for coding‑agent trajectories. It uses supervised fine‑tuning followed by reinforcement learning to train 30B models on CodeForge‑Preview data, enabling the verifier to predict completion labels and generate natural‑language explanations. The approach raises classification accuracy from 57% to 78% on a held‑out test set.
By Jiajun Hu, Chun Yong Chong
arXiv:2609.08572v1 Announce Type: new
Abstract: Large language model (LLM)-based multi-agent systems (MAS) achieve strong performance by employing specialized multiple agents, yet their performance d...
By Jaewon Chu, Jinwoo Seo, Jaewon Cho, Jeehye Na, Yunyang Xiong, Youngdae Kim, Hyunwoo J. Kim
arXiv:2603. 14465v2 Announce Type: replace Abstract: While Large Language Models (LLMs) have evolved into tool-using agents, they remain brittle in long-horizon interactions.
By Shengda Fan, Xuyan Ye, Yupeng Huo, Zhi-Yuan Chen, Yiju Guo, Shenzhi Yang, Wenkai Yang, Shuqi Ye, Jingwen Chen, Haotian Chen, Xin Cong, Yankai Lin