arXiv AI
Variance-Optimal State Resampling for Reinforcement Learning with Verifiable Rewards
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Machine Learning
Jul 30
Early Verdicts, Better Budgets: Sequential Adaptive Rollout Allocation for Compute-Efficient RLVR
arXiv:2607. 26253v1 Announce Type: new Abstract: Reinforcement learning with verifiable rewards (RLVR) is bottlenecked by rollout generation, yet many sampled prompts produce saturated groups (all responses correct or all incorrect) whose zero reward variance yields no policy-gradient signal.
arXiv AI
Jun 4
Rollout-Level Advantage-Prioritized Experience Replay for GRPO
arXiv:2606. 04560v1 Announce Type: cross Abstract: Reinforcement learning from verifiable rewards with GRPO is a standard approach for post-training reasoning LLMs.
arXiv Machine Learning
Jun 5
Cross-Epoch Adaptive Rollout Optimization for RL Post-Training
arXiv:2606. 05606v1 Announce Type: new Abstract: LLM post-training often relies on reinforcement learning methods that sample multiple rollouts per prompt, yet most existing approaches use a fixed rollout budget for every prompt, despite large differences in the training signal different prompts provide.
arXiv Machine Learning
5d ago
Gibbs-Gap Policy Optimization for Off-Policy RL with Verifiable Rewards
Gibbs‑Gap Policy Optimization (GGPO) is a new off‑policy reinforcement learning method that avoids high‑variance learner‑to‑behavior importance weights by minimizing a log‑mean‑exp contrast of KL‑adjusted rewards across responses to the same prompt. Derived from KL‑regularized policy optimization, GGPO guarantees that a learner’s single‑response expected reward is at least the behavior policy’s best‑of‑K reward minus bounded terms, and it includes a reference‑refresh mechanism that empirically improves performance. Experiments on benchmark problems and mathematical reasoning tasks show GGPO outperforming GRPO and AGRO, especially under training‑inference mismatch and with fixed offline data. whyItMatters":"The paper demonstrates a practical improvement in off‑policy RL for language models by reducing variance and ensuring better reward guarantees, which is directly supported by the reported empirical gains and theoretical bounds."
arXiv Machine Learning
Oct 8
BoT-GRPO: Efficient Process-Reward RL for Reasoning via Bag-of-Token Aggregation
arXiv:2610.09804v1 Announce Type: new Abstract: Reinforcement learning is now central to eliciting reasoning in large language models, while in the popular algorithm Group Relative Policy Optimizatio...
arXiv AI
Sep 30
Learn from the Gap: Differential-Aware Advantage Pruning with Adaptive Rollout Sampling for GRPO
The paper introduces FastRL, a reinforcement learning framework designed to enhance the efficiency of Group Relative Policy Optimization (GRPO) and its variants. FastRL employs an advantage-aware pruning strategy that retains high-advantage trajectories while preserving gradient diversity, and an adaptive rollout sampling mechanism that adjusts sampling scale during training based on historical pruning data. Experiments show that FastRL can be integrated into GRPO, DAPO, and GSPO, yielding a 2.07× speedup on Geometry3K and GeoQA8K-R1V and a 1.64% accuracy improvement on visual reasoning benchmarks.