The paper introduces belief‑shift branching, a method for placing forks in tree‑structured reinforcement learning rollouts by identifying points where a model’s answer belief changes most. Unlike traditional structural or entropy‑based approaches, belief‑shift uses a probe, logit‑lens depth profile, or learned activation direction to locate pivots in the value curve, incurring minimal computational overhead. Experiments across multiple models and benchmarks show that belief‑shift forking consistently outperforms baseline methods, yielding significant gains in mathematics and code tasks.
arXiv:2609.01274v1 Announce Type: new
Abstract: Reinforcement learning with verifiable rewards (RLVR) improves language-model reasoning, but how these gains relate to inference-time decoding and sear...
By Wenhe Sun, Cunxiang Wang, Zijun Yao, Yixin Cao
arXiv:2607. 14171v1 Announce Type: new Abstract: Reinforcement learning has emerged as the dominant paradigm for training large language model (LLM) agents that interact with executable sandboxes.
By Bowei He, Yankai Chen, Xiaokun Zhang, Xue Liu
EPIG-Tree proposes a compute‑optimal branching strategy for gradient‑efficient reinforcement learning, arguing that branches should be placed where they most reduce policy‑gradient uncertainty per unit of compute. By deriving allocation laws from a law‑of‑total‑variance decomposition, the method introduces an EPIG‑Tree score that guides branch placement using already computed rollouts, estimating occupancy‑ and score‑weighted value uncertainty. Empirical results show EPIG‑Tree reduces gradient MSE in cloned‑state control, improves frozen‑LLM gradient calibration, and outperforms flat GRPO and entropy branching in both single‑turn math and multi‑turn Wordle tasks.
By Nikita Khomich, Leopold Hermansson, Ido Hakimi
arXiv:2606. 29476v1 Announce Type: cross Abstract: Self-distilled agentic reinforcement learning augments trajectory-level reward with a token-level distillation loss, using as its teacher the same policy conditioned on privileged context.
By Zibin Meng, Kani Chen
The paper investigates why reinforcement learning with verifiable rewards (RLVR) reduces the diversity of solutions in reasoning tasks. By analyzing the Countdown task, the authors show that RLVR contracts the solution space mainly at the entrance—before the first arithmetic operation—causing a 67% drop in solution coverage. They demonstrate that providing an unselected entrance prefix or applying entrance‑targeted interventions can restore or even improve coverage without harming accuracy.
By Qiancheng Zhou, Ruizhe Li