The paper introduces Actor‑Critic with Action Chunking (AC2), a method that assigns credit to short action chunks instead of entire trajectories, enabling policy updates without waiting for terminal rewards. AC2 employs local readiness, reference solutions, and 10k‑token chunks to make critic‑based credit assignment reliable. Experiments on Qwen3‑4B with FineProofs‑RL show AC2 surpasses GRPO’s peak validation score while using 2.5× fewer decoding FLOPs and fewer training steps.
By Kaiyue Wen, Luke Bailey, Arvind Mahankali, Tengyu Ma
arXiv:2609.36178v1 Announce Type: cross
Abstract: Group Relative Policy Optimization (GRPO) has become a promising approach for training large language model agents. However, its uniform assignment o...
By Dongwon Jung, Hemanth Neelgund Ramesh, Yifan Wang, Xiaomin Li, Yuexing Hao, Yu Hu, Muhao Chen, Varun Chandrasekaran, Andrzej Banburski-Fahey, Jaron Lanier
arXiv:2608. 03223v1 Announce Type: cross Abstract: Agentic reinforcement learning enables LLM agents to learn through interaction, but sparse trajectory-level rewards reveal success without identifying which intermediate decisions deserve credit.
By Ranxu Zhang, Guinan Chen, Chenshaodong, Jinghao Lin, Xiaozhou Xu, Sunzhe, Yanyong Zhang, Chao Wang
arXiv:2606. 09961v1 Announce Type: cross Abstract: Training large language models (LLMs) as autonomous agents via reinforcement learning (RL) has enabled frontier models to achieve superhuman performance in long-horizon tasks.
By Yu Han, Kailing Li, Yang Jiao, Yulin Dai, Yuqian Fu, Linhai Zhuo, Tianwen Qian
arXiv:2609.26355v1 Announce Type: new
Abstract: Reinforcement learning has become a central component of large language model (LLM) post-training, yet token-level credit lacks a generally accepted ma...
By Jiayan Fu, Hang Xu, Yong Zhang, Zhaokai Luo, Yao Hu, Dongyan Zhao, Mu Chuan
arXiv:2608. 13179v1 Announce Type: new Abstract: Reinforcement learning with verifiable rewards (RLVR) offers a verifier-bounded performance ceiling for training multi-turn tool-use agents, yet its trajectory-level credit assignment conflates heterogeneous per-turn outcomes into a single reward signal.
By Zechuan Wang, Siyuan Lu, Hongxuan Zhang, Linjian Mo, Chenyi Zhuang, Leilei Gan
arXiv:2608. 05987v1 Announce Type: new Abstract: Reinforcement learning (RL) with verifiable rewards constructs trajectory-level advantage estimates, yet it often fails to credit the few pivotal decisions that determine outcomes in long-horizon, multi-turn agentic tasks.
By Zi-Han Wang, Zhengxi Lu, Zhiyuan Yao, Jinyang Wu, Jie Wu, Zhengzhou Cai, Yueqing Sun, Ziang Ye, Linji Hao, Qi Gu, Xunliang Cai, Yongliang Shen, Yujiu Yang
The paper introduces TASPO, a method that transforms privileged information (PI) into outcome‑grounded action credit for language‑model agents. TASPO constructs decision‑applicable PI from verified successful experience, aggregates PI‑induced likelihood shifts at the executable‑action level, and converts relative action support into positive, bounded, mean‑preserving weights on the original trajectory advantage. Experiments on three agentic benchmarks show TASPO improves over GRPO by 10.6% and generalizes better to unseen tasks, while reducing supervision mismatch and stabilizing policy optimization.
By Jingxiao Yang, Wangjie Gan, Yingxuan Zhuang, Wenqi Zhang, Jintao Chen, Xuhong Zhang
arXiv:2608.21501v1 Announce Type: new
Abstract: Credit assignment in large-language-model reinforcement learning (LLM RL) can be separated into three objects: evidence about success, a transport oper...
By Qifan Shi, Zhaolu Kang, Chenghua Zhu
arXiv:2605. 17877v2 Announce Type: replace Abstract: A significant hurdle for current LLMs is the execution of complex, multi-stage tasks.
By Wonjoong Kim, Yeonjun In, Sangwu Park, Dongha Lee, Chanyoung Park
The paper introduces Influence-Aware Policy Optimization (IAPO), a method that models multi‑turn agent rollouts as typed influence‑dependency graphs to better assign credit to actions based on how information and errors flow through user and tool interactions. IAPO transforms the structure of support and failure usage into routing weights that redistribute trajectory‑level advantage, enabling more effective learning from sparse final rewards. Experiments with Qwen3‑4B and Qwen3‑8B on three service‑agent benchmarks show that IAPO outperforms existing multi‑turn reinforcement learning baselines without harming function‑calling performance.
By Bo Ren, Yirong Mao, Yi Yang, Wenhui Que
arXiv:2603. 06194v3 Announce Type: replace-cross Abstract: Reinforcement learning (RL) for large language models (LLMs) has shown strong performance in single-turn tasks, but extending it to multi-turn interaction remains challenging due to sparse rewards and poor per-turn credit assignment.
By Naifan Zhang, Ruihan Sun, Jinwei Su, Hengjie Yang, Zhengyuan Pan, Zhaohan Chen, Xiaofan Zhang