Hugging Face Trending Papers
Bellman-Consistent Trajectory RL for LLM Arithmetic Reasoning
Read the original on Hugging Face Trending Papers →The Flow has not summarised this story yet — read it at Hugging Face Trending Papers.
Related stories
arXiv AI
Oct 16
Bellman-Consistent Trajectory RL for LLM Arithmetic Reasoning
The paper introduces a Bellman-consistent trajectory reinforcement learning approach for large language model (LLM) arithmetic reasoning, called Trajectory Bellman Residual Minimization (TBRM). By converting sparse terminal rewards into dense token-level signals and employing a Reinforce Leave-One-Out (RLOO) strategy to reduce gradient variance, the method improves learning stability. On the Countdown task, TBRM-RLOO achieves Pass@1 scores of 0.55–0.64 and a state-of-the-art Pass@16 accuracy of 76.0–84.0%, surpassing several baseline methods.
arXiv AI
Jul 7
R$^2$PO: Decoupling Rollout and Inference Policies for LLM Reasoning
arXiv:2601. 11960v3 Announce Type: replace-cross Abstract: Existing reinforcement learning methods for LLM reasoning implicitly assume that the policy generating training trajectories should coincide with the one producing inference responses.
arXiv Machine Learning
Aug 18
Le Critique: Privileged Value Functions for LLM Reinforcement Learning
arXiv:2608. 16739v1 Announce Type: new Abstract: Reinforcement learning algorithms for Large Language Models (LLMs) are largely distinguished by their variance reduction strategy.
arXiv AI
Jun 30
PS-PPO: Prefix-Sampling PPO for Critic-Free RLHF
arXiv:2606. 29758v1 Announce Type: cross Abstract: Reinforcement Learning from Human Feedback (RLHF) for Large Language Models increasingly relies on critic-free methods as a practical alternative to actor--critic training.
arXiv Machine Learning
Oct 12
Trajectory-Aware Policy Optimization: Closing the State Distribution Gap in Long-Horizon LLM RL
The paper introduces Trajectory-Aware Policy Optimization (TAPO), a new framework for reinforcement learning that addresses the state distribution shift caused by long-horizon autoregressive rollouts in large language models. TAPO derives a state-distribution correction from the off-policy policy gradient, integrates it into the RL objective, and uses a geometric-mean estimator for variance reduction along with length-aware dynamic clipping to stabilize updates. Experiments on eight competition-level benchmarks show TAPO outperforms existing LLM RL methods, achieving up to 48% improvement on mathematical reasoning and 12% on algorithmic coding tasks while enhancing training stability.
arXiv AI
Aug 26
On-policy Distillation with Verifiable Reward
arXiv:2608.24696v1 Announce Type: cross Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) and on-policy distillation (OPD) have become two widely adopted paradigms for post-training lar...