arXiv AI

Tool-Aware Optimization with Entropy Guidance for Efficient Agentic Reinforcement Learning

arXiv:2606. 03762v1 Announce Type: cross Abstract: Agentic reinforcement learning (RL) equips large language models (LLMs) with tool-use capabilities that substantially improve reasoning on complex tasks.

arXiv Computation and Language
Sep 16

Spurious Tool Use: When RL Agents Learn the Wrong Reason to Act

The paper investigates how reinforcement learning can cause large language model agents to adopt shortcut policies for tool use, relying on superficial prompt cues rather than actual task needs. By creating synthetic environments that mix factual QA and math reasoning, the authors show that agents often invoke tools when cues are present, even when those tools are unnecessary, with spurious invocation rates rising up to 39%. They find that shortcut learning occurs mainly when agents have already mastered the target tool and that semantic alignment between cues and tools amplifies the effect. To counter this, they propose a dense, decision-level reward where an LLM judge assesses tool necessity, which reduces cue-driven tool use while maintaining performance.

By Yiwei Yang, Haoxiang Zhang, Bingbing Wen, Yao Lu, Yuchen Wu, Lei Zhang, Julian McAuley, Pan Lu, Bill Howe
arXiv AI
Aug 25

HiDiffTIR: Hierarchical Difficulty-Aware Policy Optimization for Multi-Turn Tool-Integrated Reasoning

arXiv:2608.21863v1 Announce Type: cross Abstract: Tool-Integrated Reasoning (TIR) is a fundamental capability for LLM agents to solve complex tasks by interacting with external tools iteratively. Rei...

By Yucan Guo, Xiaohan Wang, Miao Su, Saiping Guan, Zhongni Hou, Jiajun Chai, Wei Lin, Guojun Yin, Xiaolong Jin, Jiafeng Guo, Xueqi Cheng
arXiv AI
Sep 3

UniToolCall: Unifying Tool-Use Representation, Data, and Evaluation for LLM Agents

UniToolCall introduces a unified framework for tool-use in large language model agents, standardizing toolset construction, dataset generation, and evaluation. The framework aggregates over 22,000 tools and creates a hybrid training corpus of more than 390,000 instances by combining ten public datasets with synthetically generated, structurally controlled trajectories. It models diverse interaction patterns—single‑hop vs. multi‑hop, single‑turn vs. multi‑turn, serial vs. parallel execution—and adds an Anchor Linkage mechanism to enforce cross‑turn dependencies, while converting seven public benchmarks into a common Query–Action–Observation–Answer format for fine‑grained evaluation.

By Yijuan Liang, Xinghao Chen, Yifan Ge, Ziyi Wu, Hao Wu, Changyu Zeng, Wei Xing, Xiaoyu Shen
arXiv AI
Sep 2

ARISE-RL: Agentic Rubric-Grounded Iterative Self-Evolution with Reinforcement Learning

ARISE‑RL is a full‑cycle self‑evolution framework that couples a task/rubric generator with a reasoning solver, enabling open‑ended agents to learn via reinforcement learning without relying on gold answers. The generator creates tool‑grounded rubric criteria and rewards itself for producing valid, intermediate‑difficulty tasks, while the solver improves through fine‑grained rubric satisfaction signals and multi‑step reasoning. The framework also introduces Reward‑Gated Self‑Evolution Distillation to selectively distill memory‑augmented policies, reducing distribution mismatch, and it is evaluated on the new ECR‑Bench rubric suite, achieving state‑of‑the‑art performance across all benchmarks.

By Fanrui Zhang, Ruixue Ding, Qiang Zhang, Xi Chen, Boli Chen, Shihang Wang, Qiuchen Wang, Hongmin Zhan, Jinxin Bian, Li xingchao, Peijin Zheng, Hao cheng, Pengjun Xie, Kaipeng Zhang, Jiawei Liu, Zheng-Jun Zha
arXiv Computation and Language
6d ago

ToolSearcher: Optimizing Tool Selection at Scale via Reinforcement Learning

ToolSearcher is a reinforcement learning framework designed to improve large‑scale tool selection for large language models. It introduces category‑constrained discrimination, event‑level search modeling, and trajectory‑aligned credit allocation to better distinguish similar tools, optimize multi‑turn search, and provide fine‑grained rewards. Experiments on large‑scale benchmarks show that ToolSearcher outperforms strong baselines in iterative search and complex tool composition scenarios.

By Zhenlong Dai, Xujie Song, Zitong Wang, Tong Niu, Jian liu, Weiqiang Wang, Xiu Tang, Sai Wu, Chang Yao, Jingyuan Chen