arXiv Machine Learning
Reward Budgeting Reduces Premature Convergence in Reinforcement Learning for LLM Reasoning
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv AI
Sep 30
Unlocking the Critic: Reward-Free Policy Optimization for LLM Post-Training
arXiv:2609.37119v1 Announce Type: cross Abstract: Recent approaches to reinforcement learning (RL) post-training for large language models increasingly remove the critic to reduce training instabilit...
Hugging Face Trending Papers
Jul 29
Post-Training at the Edge of Detectability: A Game-Theoretic Approach to Fine-Tuning
Reinforcement learning (RL) fine-tuning is widely used in language model training to improve model performance on a target task while limiting drift from a reference policy. A standard way to balance this trade-off is via a KL-regularized RL objective, although this formulation does not by itself provide a principled way to set the regularization coefficient.
arXiv Machine Learning
Oct 16
Gibbs-Gap Policy Optimization for Off-Policy RL with Verifiable Rewards
Gibbs‑Gap Policy Optimization (GGPO) is a new off‑policy reinforcement learning method that avoids high‑variance learner‑to‑behavior importance weights by minimizing a log‑mean‑exp contrast of KL‑adjusted rewards across responses to the same prompt. Derived from KL‑regularized policy optimization, GGPO guarantees that a learner’s single‑response expected reward is at least the behavior policy’s best‑of‑K reward minus bounded terms, and it includes a reference‑refresh mechanism that empirically improves performance. Experiments on benchmark problems and mathematical reasoning tasks show GGPO outperforming GRPO and AGRO, especially under training‑inference mismatch and with fixed offline data. whyItMatters":"The paper demonstrates a practical improvement in off‑policy RL for language models by reducing variance and ensuring better reward guarantees, which is directly supported by the reported empirical gains and theoretical bounds."
arXiv Machine Learning
Jul 30
Post-Training at the Edge of Detectability: A Game-Theoretic Approach to Fine-Tuning
arXiv:2607. 26358v1 Announce Type: new Abstract: Reinforcement learning (RL) fine-tuning is widely used in language model training to improve model performance on a target task while limiting drift from a reference policy.
arXiv Machine Learning
Jul 30
Early Verdicts, Better Budgets: Sequential Adaptive Rollout Allocation for Compute-Efficient RLVR
arXiv:2607. 26253v1 Announce Type: new Abstract: Reinforcement learning with verifiable rewards (RLVR) is bottlenecked by rollout generation, yet many sampled prompts produce saturated groups (all responses correct or all incorrect) whose zero reward variance yields no policy-gradient signal.
arXiv Machine Learning
Oct 15
When Does Momentum Help in Reinforcement Learning with Verifiable Rewards?
The paper investigates the role of first‑moment momentum in reinforcement learning with verifiable rewards (RLVR). Using a bandit formulation with bias‑corrected exponential moving average (EMA) momentum, the authors prove that, when policy gradients are exact, increasing the EMA coefficient β within a safe range accelerates optimization. In practice, they show that maintaining a larger β preserves a non‑vanishing expected gain even as the learning rate shrinks, and experiments on Qwen models confirm that higher β values improve validation accuracy by up to about 2 percentage points.