arXiv AI September 18, 2026 By Nikita Khomich, Leopold Hermansson, Ido Hakimi

EPIG-Tree: Compute-Optimal Branching for Gradient-Efficient Reinforcement Learning

EPIG-Tree proposes a compute‑optimal branching strategy for gradient‑efficient reinforcement learning, arguing that branches should be placed where they most reduce policy‑gradient uncertainty per unit of compute. By deriving allocation laws from a law‑of‑total‑variance decomposition, the method introduces an EPIG‑Tree score that guides branch placement using already computed rollouts, estimating occupancy‑ and score‑weighted value uncertainty. Empirical results show EPIG‑Tree reduces gradient MSE in cloned‑state control, improves frozen‑LLM gradient calibration, and outperforms flat GRPO and entropy branching in both single‑turn math and multi‑turn Wordle tasks.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv Machine Learning

Jul 17

Branching Policy Optimization: Sandbox-Native Language Agent Reinforcement Learning

arXiv:2607. 14171v1 Announce Type: new Abstract: Reinforcement learning has emerged as the dominant paradigm for training large language model (LLM) agents that interact with executable sandboxes.

By Bowei He, Yankai Chen, Xiaokun Zhang, Xue Liu

llms agents reinforcement-learning benchmarks

arXiv Computation and Language

1d ago

OPTS-TTPO: Enhancing Finite-Sample Policy-Gradient Learning with Tree Search

arXiv:2609.40035v1 Announce Type: new Abstract: The policy-gradient theorem gives the exact gradient under the current policy, but finite on-policy samples may miss rare high-return trajectories. We...

By Junyu Lu, Shichao Weng, Zhiqiang Wang, Haojie Luo, Jingfan Zhang, Yuhua Zhou, Cheng Du, Yuzhuo Zhang, Xi Li, Jinwei Du, Tiancheng Feng, Chuan Xiao, Shuyuan Zheng

reinforcement-learning safety

arXiv AI

Aug 26

Contrastive Branch Policy Optimization

Contrastive Branch Policy Optimization (CBPO) is a reinforcement learning method that separates the allocation of a fixed rollout budget from the translation of branch outcomes into token-level credit. It uses generation entropy to screen branch positions, path- and node-level decay to distribute the budget, and Contrastive Branch Value (CBV) to estimate local decision sensitivity without changing reward signs. CBPO partitions trajectories into non-overlapping credit segments, preventing duplicated gradients and enabling fine-grained credit assignment using only outcome rewards.

By Ying Wang, Changlin Qiu, Bang Lin, Linbo Jin, Wen Jiang, Zhe Sun, Jingli Yang

llms agents reinforcement-learning benchmarks

arXiv Machine Learning

Jun 25

Learning with a Single Rollout via Monte Carlo Pass@k Critic

arXiv:2606. 25451v1 Announce Type: new Abstract: Estimating token-level advantages in reinforcement learning (RL) for language models remains challenging because scaling up episodic experience collection is expensive.

By Fengdi Che, Yang Liu, Lei Yu, Meng Cao, Tong Che, Rupam Mahmood, Dale Schuurmans

llms reinforcement-learning benchmarks

arXiv AI

Aug 25

Let Credit Follow Computation: Architecture-Aware Credit Transport for Large Language Model Reinforcement Learning

arXiv:2608.21501v1 Announce Type: new Abstract: Credit assignment in large-language-model reinforcement learning (LLM RL) can be separated into three objects: evidence about success, a transport oper...

By Qifan Shi, Zhaolu Kang, Chenghua Zhu

llms reinforcement-learning safety

Hugging Face Trending Papers

Sep 10

Fork Where the Model Changes Its Mind: Belief-Shift Branching for Tree-Structured Reinforcement Learning

The paper introduces belief‑shift branching, a method for placing forks in tree‑structured reinforcement learning rollouts by identifying points where a model’s answer belief changes most. Unlike traditional structural or entropy‑based approaches, belief‑shift uses a probe, logit‑lens depth profile, or learned activation direction to locate pivots in the value curve, incurring minimal computational overhead. Experiments across multiple models and benchmarks show that belief‑shift forking consistently outperforms baseline methods, yielding significant gains in mathematics and code tasks.