Reinforcement learning for multi-turn search reasoning typically relies on terminal outcome rewards, which cannot distinguish useful, redundant, and harmful intermediate interactions. We propose LAPO, a self-generated process-supervision method based on backward leave-one-turn attribution.
arXiv:2607. 13501v2 Announce Type: replace Abstract: Reinforcement learning for multi-turn search reasoning typically relies on terminal outcome rewards, which cannot distinguish useful, redundant, and harmful intermediate interactions.
By Qiang Zhu, Jiajun Wu, Longyi Wang
Reinforcement learning for multi-turn search reasoning typically relies on terminal outcome rewards, which cannot distinguish useful, redundant, and harmful intermediate interactions. We propose LOTAPO , a self-generated process-supervision method based on backward leave-one-turn attribution.
arXiv:2608. 07531v1 Announce Type: cross Abstract: Search-augmented language agents should retrieve external information only when necessary and ground their answers in retrieved evidence.
By Cheng Ruoxi, Ma Haoxuan, Zhang Hongyi, Zhang Junming, Duan Ranjie, Xia Qiaolin, Wang Hao, Lu Yu, Shi Haibo, Ma Xingjun
The paper explores how dense, turn-level reward structures can improve reinforcement learning for large language model agents in multi-turn tasks. It introduces three reward granularity types—terminal, delayed, and per-turn—and adapts Group Relative Policy Optimization and Proximal Policy Optimization to each. Experiments on search and game agents show that per-turn rewards consistently yield better training dynamics, faster convergence, and higher answer correctness compared to sparse terminal or delayed rewards.
By Quan Wei, Siliang Zeng, Chenliang Li, Zhongruo Wang, William Brown, Oana Frunza, Wei Deng, Anderson Schneider, Yuriy Nevmyvaka, Yang Katie Zhao, Alfredo Garcia, Mingyi Hong
arXiv:2608. 12764v1 Announce Type: cross Abstract: Deep search agents operate over trajectories spanning dozens of steps, yet standard reinforcement learning provides only a single outcome reward per trajectory, which is far too sparse for effective credit assignment.
By Haoze Wu, Chuqiao Kuang, Tianyi Zhuang, Xiaoguang Li