The paper introduces a Bellman-consistent trajectory reinforcement learning approach for large language model (LLM) arithmetic reasoning, called Trajectory Bellman Residual Minimization (TBRM). By converting sparse terminal rewards into dense token-level signals and employing a Reinforce Leave-One-Out (RLOO) strategy to reduce gradient variance, the method improves learning stability. On the Countdown task, TBRM-RLOO achieves Pass@1 scores of 0.55–0.64 and a state-of-the-art Pass@16 accuracy of 76.0–84.0%, surpassing several baseline methods.
To address sparse terminal rewards in LLM arithmetic reasoning, we propose a Bellman-consistent trajectory RL method. Our framework introduces Trajectory Bellman Residual Minimization (TBRM) to transf...
arXiv:2601. 11960v3 Announce Type: replace-cross Abstract: Existing reinforcement learning methods for LLM reasoning implicitly assume that the policy generating training trajectories should coincide with the one producing inference responses.
arXiv:2608. 16739v1 Announce Type: new Abstract: Reinforcement learning algorithms for Large Language Models (LLMs) are largely distinguished by their variance reduction strategy.
The paper introduces Trajectory-Aware Policy Optimization (TAPO), a new framework for reinforcement learning that addresses the state distribution shift caused by long-horizon autoregressive rollouts in large language models. TAPO derives a state-distribution correction from the off-policy policy gradient, integrates it into the RL objective, and uses a geometric-mean estimator for variance reduction along with length-aware dynamic clipping to stabilize updates. Experiments on eight competition-level benchmarks show TAPO outperforms existing LLM RL methods, achieving up to 48% improvement on mathematical reasoning and 12% on algorithmic coding tasks while enhancing training stability.
arXiv:2606. 29758v1 Announce Type: cross Abstract: Reinforcement Learning from Human Feedback (RLHF) for Large Language Models increasingly relies on critic-free methods as a practical alternative to actor--critic training.
The paper introduces the Posterior Concentration Phenomenon (PCP), a length‑dependent failure mode where probability‑based rewards collapse to a narrow interval for long reasoning traces, destabilizing verifier‑free reinforcement learning. To address this, the authors propose RLCPR, a framework that uses uncertainty‑aware data sampling and concentration‑aware regularization to mitigate PCP, improving token efficiency and outperforming state‑of‑the‑art baselines on multiple reasoning benchmarks.
arXiv:2606. 24994v1 Announce Type: new Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) for language-model reasoning can fail at both extremes of task difficulty: easy prompts often produce all-correct, low-diversity rollout groups with little gradient signal, while hard prompts can produce all-incorrect groups with no positive reward.
arXiv:2608. 03119v1 Announce Type: new Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) improves LLM reasoning but typically relies on ground-truth (GT) answers, limiting scalability.
arXiv:2603. 25184v2 Announce Type: replace-cross Abstract: Reinforcement learning (RL) has become essential for post-training large language models (LLMs) in reasoning tasks.
arXiv:2606. 01281v1 Announce Type: cross Abstract: Reinforcement learning with verifiable rewards (RLVR) has emerged as a powerful paradigm for enhancing the reasoning capabilities of large language models (LLMs).
arXiv:2608.24696v1 Announce Type: cross Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) and on-policy distillation (OPD) have become two widely adopted paradigms for post-training lar...
The paper introduces FastRL, a reinforcement learning framework designed to enhance the efficiency of Group Relative Policy Optimization (GRPO) and its variants. FastRL employs an advantage-aware pruning strategy that retains high-advantage trajectories while preserving gradient diversity, and an adaptive rollout sampling mechanism that adjusts sampling scale during training based on historical pruning data. Experiments show that FastRL can be integrated into GRPO, DAPO, and GSPO, yielding a 2.07× speedup on Geometry3K and GeoQA8K-R1V and a 1.64% accuracy improvement on visual reasoning benchmarks.