arXiv:2609.37119v1 Announce Type: cross Abstract: Recent approaches to reinforcement learning (RL) post-training for large language models increasingly remove the critic to reduce training instabilit...
Reinforcement learning (RL) fine-tuning is widely used in language model training to improve model performance on a target task while limiting drift from a reference policy. A standard way to balance this trade-off is via a KL-regularized RL objective, although this formulation does not by itself provide a principled way to set the regularization coefficient.
Gibbs‑Gap Policy Optimization (GGPO) is a new off‑policy reinforcement learning method that avoids high‑variance learner‑to‑behavior importance weights by minimizing a log‑mean‑exp contrast of KL‑adjusted rewards across responses to the same prompt. Derived from KL‑regularized policy optimization, GGPO guarantees that a learner’s single‑response expected reward is at least the behavior policy’s best‑of‑K reward minus bounded terms, and it includes a reference‑refresh mechanism that empirically improves performance. Experiments on benchmark problems and mathematical reasoning tasks show GGPO outperforming GRPO and AGRO, especially under training‑inference mismatch and with fixed offline data. whyItMatters":"The paper demonstrates a practical improvement in off‑policy RL for language models by reducing variance and ensuring better reward guarantees, which is directly supported by the reported empirical gains and theoretical bounds."
arXiv:2607. 26358v1 Announce Type: new Abstract: Reinforcement learning (RL) fine-tuning is widely used in language model training to improve model performance on a target task while limiting drift from a reference policy.
arXiv:2607. 26253v1 Announce Type: new Abstract: Reinforcement learning with verifiable rewards (RLVR) is bottlenecked by rollout generation, yet many sampled prompts produce saturated groups (all responses correct or all incorrect) whose zero reward variance yields no policy-gradient signal.
The paper investigates the role of first‑moment momentum in reinforcement learning with verifiable rewards (RLVR). Using a bandit formulation with bias‑corrected exponential moving average (EMA) momentum, the authors prove that, when policy gradients are exact, increasing the EMA coefficient β within a safe range accelerates optimization. In practice, they show that maintaining a larger β preserves a non‑vanishing expected gain even as the learning rate shrinks, and experiments on Qwen models confirm that higher β values improve validation accuracy by up to about 2 percentage points.
arXiv:2606. 19771v1 Announce Type: new Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) has significantly advanced Large Language Model (LLM) reasoning; however, it faces a fundamental optimization instability: uniform token updates precipitate entropy collapse, leading to premature convergence to suboptimal strategies, whereas excessive Shannon Entropy maximization can cause entropy explosion, driving blind exploration toward incoherent reasoning chains.
arXiv:2608.28771v1 Announce Type: new Abstract: Large reasoning models achieve strong performance on complex tasks by generating extended chain-of-thought (CoT) traces via reinforcement learning with...
arXiv:2610.18704v1 Announce Type: new Abstract: In this paper, we propose an extremely efficient, training-free method to extract token-level reward signals directly from an existing deep reward mode...
arXiv:2609.36945v1 Announce Type: new Abstract: We study the learning dynamics of fine-tuning a policy model on self-generated and reward-weighted data, with particular focus on a generalized version...
ReAPO: Retry-Aware Policy Optimization for Efficient Exploration in LLM Reasoning proposes a new approach to improve language‑model reasoning by addressing the inefficiencies of independent stochastic rollouts. By framing credit assignment as a Stochastic Shortest Path problem, ReAPO modifies GRPO to allocate more generation to failed trajectories, reinforcing successful retries without rewarding failures. Experiments on Qwen3 and OLMo‑2 models across eight mathematical and scientific benchmarks show that ReAPO outperforms GRPO and other baselines, achieving higher first‑attempt accuracy and reducing generated tokens needed to reach peak performance.
arXiv:2608.23830v1 Announce Type: cross Abstract: RL has emerged as a powerful paradigm for enhancing the instruction following capabilities of LLMs. While existing training recipes achieve substanti...