arXiv:2607. 14171v1 Announce Type: new Abstract: Reinforcement learning has emerged as the dominant paradigm for training large language model (LLM) agents that interact with executable sandboxes.
By Bowei He, Yankai Chen, Xiaokun Zhang, Xue Liu
arXiv:2609.40035v1 Announce Type: new
Abstract: The policy-gradient theorem gives the exact gradient under the current policy, but finite on-policy samples may miss rare high-return trajectories. We...
By Junyu Lu, Shichao Weng, Zhiqiang Wang, Haojie Luo, Jingfan Zhang, Yuhua Zhou, Cheng Du, Yuzhuo Zhang, Xi Li, Jinwei Du, Tiancheng Feng, Chuan Xiao, Shuyuan Zheng
Contrastive Branch Policy Optimization (CBPO) is a reinforcement learning method that separates the allocation of a fixed rollout budget from the translation of branch outcomes into token-level credit. It uses generation entropy to screen branch positions, path- and node-level decay to distribute the budget, and Contrastive Branch Value (CBV) to estimate local decision sensitivity without changing reward signs. CBPO partitions trajectories into non-overlapping credit segments, preventing duplicated gradients and enabling fine-grained credit assignment using only outcome rewards.
By Ying Wang, Changlin Qiu, Bang Lin, Linbo Jin, Wen Jiang, Zhe Sun, Jingli Yang
arXiv:2606. 25451v1 Announce Type: new Abstract: Estimating token-level advantages in reinforcement learning (RL) for language models remains challenging because scaling up episodic experience collection is expensive.
By Fengdi Che, Yang Liu, Lei Yu, Meng Cao, Tong Che, Rupam Mahmood, Dale Schuurmans
Tree-Aware Posterior Distillation for Long-Horizon Agentic Reinforcement Learning proposes TreeOPSD, a critic‑free framework that treats each interaction turn as a tree node, branches alternative continuations, and recursively backs up terminal rewards to generate implicit turn‑level credit. The method maps this credit to a bounded teacher distribution based on the likelihood ratio to a frozen behavior policy, and minimizes the divergence between teacher and student to recover policy‑gradient directions while limiting updates across mini‑batches. Empirical results show TreeOPSD enhances agentic performance on long‑horizon benchmarks, notably improving Qwen3‑30B‑A3B‑Thinking‑2507’s accuracy by 4.97 points over GRPO in the Deep Research setting.
By Heming Zou, Qi Wang, Yixiu Mao, Yun Qu, Yuyi Guo, Zeyuan Liu, Chenglin Li, Weijie Liu, Saiyong Yang, Xiangyang Ji
arXiv:2608.21501v1 Announce Type: new
Abstract: Credit assignment in large-language-model reinforcement learning (LLM RL) can be separated into three objects: evidence about success, a transport oper...
By Qifan Shi, Zhaolu Kang, Chenghua Zhu