arXiv AI
Jul 23

In-the-Flow Agentic System Optimization for Effective Planning and Tool Use

arXiv:2510. 05592v2 Announce Type: replace Abstract: Outcome-driven reinforcement learning has advanced reasoning in large language models (LLMs), but prevailing tool-augmented approaches train a single, monolithic policy that interleaves thoughts and tool calls under full context; this scales poorly with long horizons and diverse tools and generalizes weakly to new scenarios.

By Zhuofeng Li, Haoxiang Zhang, Seungju Han, Sheng Liu, Jianwen Xie, Yu Zhang, Yejin Choi, James Zou, Pan Lu
arXiv AI
6d ago

AEM: Adaptive Entropy Modulation for Multi-Turn Agentic Reinforcement Learning

AEM: Adaptive Entropy Modulation for Multi‑Turn Agentic Reinforcement Learning proposes a supervision‑free credit assignment technique that adjusts entropy dynamics at the response level during RL training. By aligning uncertainty estimation with the granularity of LLM agent actions and rescaling advantages based on a response‑level uncertainty proxy, AEM improves the exploration‑exploitation balance without relying on dense intermediate supervision. Experiments on ALFWorld, WebShop, and SWE‑bench‑Verified show consistent performance gains across models from 1.5B to 32B, including a 1.4% improvement in a state‑of‑the‑art software‑engineering RL framework.

By Haotian Zhao, Songlin Zhou, Yuxin Zhang, Stephen S. -T. Yau, Wenyu Zhang, Lun Tian, Tianshu Zhu, Yifeng Huang, Yucheng Zeng, Jingnan Gu, Daxiang Dong, Jianmin Wu