arXiv:2607. 14171v1 Announce Type: new Abstract: Reinforcement learning has emerged as the dominant paradigm for training large language model (LLM) agents that interact with executable sandboxes.
By Bowei He, Yankai Chen, Xiaokun Zhang, Xue Liu
EPIG-Tree proposes a compute‑optimal branching strategy for gradient‑efficient reinforcement learning, arguing that branches should be placed where they most reduce policy‑gradient uncertainty per unit of compute. By deriving allocation laws from a law‑of‑total‑variance decomposition, the method introduces an EPIG‑Tree score that guides branch placement using already computed rollouts, estimating occupancy‑ and score‑weighted value uncertainty. Empirical results show EPIG‑Tree reduces gradient MSE in cloned‑state control, improves frozen‑LLM gradient calibration, and outperforms flat GRPO and entropy branching in both single‑turn math and multi‑turn Wordle tasks.
By Nikita Khomich, Leopold Hermansson, Ido Hakimi
arXiv:2609.39634v1 Announce Type: cross
Abstract: Common policy improvement methods, including TRPO, PPO, and GRPO, estimate policy improvement under the behavioral policy's state-visitation distribu...
By Nima H. Siboni
arXiv:2606. 11119v1 Announce Type: cross Abstract: Reinforcement learning with verifiable rewards (RLVR) is a promising approach for enhancing reasoning and agentic behavior in large language models.
By Heming Zou, Qi Wang, Yun Qu, Yuhang Jiang, Lizhou Cai, Yixiu Mao, Ru Peng, Xin Xu, Weijie Liu, Kai Yang, Saiyong Yang, Xiangyang Ji
Contrastive Branch Policy Optimization (CBPO) is a reinforcement learning method that separates the allocation of a fixed rollout budget from the translation of branch outcomes into token-level credit. It uses generation entropy to screen branch positions, path- and node-level decay to distribute the budget, and Contrastive Branch Value (CBV) to estimate local decision sensitivity without changing reward signs. CBPO partitions trajectories into non-overlapping credit segments, preventing duplicated gradients and enabling fine-grained credit assignment using only outcome rewards.
By Ying Wang, Changlin Qiu, Bang Lin, Linbo Jin, Wen Jiang, Zhe Sun, Jingli Yang
Tree-Aware Posterior Distillation for Long-Horizon Agentic Reinforcement Learning proposes TreeOPSD, a critic‑free framework that treats each interaction turn as a tree node, branches alternative continuations, and recursively backs up terminal rewards to generate implicit turn‑level credit. The method maps this credit to a bounded teacher distribution based on the likelihood ratio to a frozen behavior policy, and minimizes the divergence between teacher and student to recover policy‑gradient directions while limiting updates across mini‑batches. Empirical results show TreeOPSD enhances agentic performance on long‑horizon benchmarks, notably improving Qwen3‑30B‑A3B‑Thinking‑2507’s accuracy by 4.97 points over GRPO in the Deep Research setting.
By Heming Zou, Qi Wang, Yixiu Mao, Yun Qu, Yuyi Guo, Zeyuan Liu, Chenglin Li, Weijie Liu, Saiyong Yang, Xiangyang Ji
arXiv:2608. 12831v1 Announce Type: cross Abstract: Online platforms increasingly compare many adaptive decision policies---ranking systems, recommendation algorithms, pricing rules, and language-model agents---while each reward-bearing interaction can be costly or risky.
By Yuxiao Wen
arXiv:2607. 15610v1 Announce Type: cross Abstract: Reinforcement learning (RL) has become a key approach for training LLM agents, yet popular methods such as GRPO/RLOO rely on multiple independently sampled complete trajectories for advantage estimation.
By Xintong Li, Sha Li, Yuwei Zhang, Changlong Yu, Rongmei Lin, Hongye Jin, Shuyi Guan, Xin Liu, Linwei Li, Qingyu Yin, Jingbo Shang
The paper introduces TSBPO, a tree‑structured sampling‑to‑update framework for training interactive language agents. It tackles three intertwined challenges—sampling task and state continuations, assigning delayed feedback, and aggregating token losses—by using a likelihood‑fitted Root/Branch allocator, trajectory and observation‑group advantages, and a clipped token surrogate aggregation. Experiments with a Qwen2.5‑1.5B actor show high success rates on ALFWorld and WebShop benchmarks, demonstrating the effectiveness of the proposed training configuration.
By Rui Yu, Chong Mu, Jinchuan Zhang
CANOPY is a multi‑fidelity tree bandit algorithm that learns where a piecewise‑smooth prior holds instead of assuming global smoothness. It uses cheap random‑path probes to certify local aggregation bias and then focuses expensive leaf evaluations on cells where smoothness is violated. The method achieves provable fixed‑budget and regret guarantees that scale with the number of discontinuities, matching smooth‑tree rates when no violations exist and approaching structure‑blind search when violations are dense.
By Michael Jerge, Suman Jana
arXiv:2608.21501v1 Announce Type: new
Abstract: Credit assignment in large-language-model reinforcement learning (LLM RL) can be separated into three objects: evidence about success, a transport oper...
By Qifan Shi, Zhaolu Kang, Chenghua Zhu
arXiv:2609.36864v1 Announce Type: new
Abstract: Group-relative methods for reinforcement learning with verifiable rewards (RLVR) learn from differences in rollout outcomes. Independently sampling com...
By Fanchao Chen, Hengyu Fu, Shivaram Venkataraman, Jiantao Jiao