The paper investigates how entropy behaves in reinforcement learning with verifiable rewards (RLVR) when the sampling method used during rollouts differs from the learner’s policy. It shows that a constant advantage shift alters the entropy of a softmax policy in a way that depends on the sampler: positive shifts sharpen the policy for samplers whose likelihood ratio increases with token probability (e.g., top‑k, top‑p, min‑p, and temperatures below one) and flatten it for flatter samplers. In group‑based RLVR with token‑level losses, the entropy drift is governed by the token‑weighted net advantage and a covariance term that can amplify over long horizons, with empirical results on 2B and 9B models demonstrating how different sampling and baseline strategies affect entropy stability and held‑out accuracy.
Reinforcement learning with verifiable rewards (RLVR) makes a language model solve more problems on the first try and fewer when given many tries: it raises pass@1 while the base model overtakes it at...
The paper proves that in reinforcement learning with verifiable rewards (RLVR), any improvement in pass@1 necessarily incurs a cost in pass@k, establishing a precise price for such gains. By modeling RLVR as an exponential tilt of a base policy, the authors show that every update that raises pass@1 strictly lowers success on at least one prompt, and that the number of pass@k crossovers is bounded by the sign variation of per‑prompt success shifts. The work also derives an entropy‑performance law and provides a diagnostic to measure this price from evaluation logs.
arXiv:2611.05942v1 Announce Type: new Abstract: RL for LLM reasoning often reaches an early performance plateau as the policy distribution becomes increasingly sharp. Recent work has studied policy e...
arXiv:2609.09075v1 Announce Type: cross Abstract: In reinforcement learning with verifiable rewards (RLVR) trained with group relative policy optimization (GRPO), the KL-free reward-advantage term st...
arXiv:2606. 19236v1 Announce Type: cross Abstract: Reinforcement Learning with Verifiable Rewards algorithms like GRPO have emerged as the dominant post-training paradigm for complex reasoning in LLMs, yet commonly suffer from policy entropy collapse during training.
arXiv:2606. 19771v1 Announce Type: new Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) has significantly advanced Large Language Model (LLM) reasoning; however, it faces a fundamental optimization instability: uniform token updates precipitate entropy collapse, leading to premature convergence to suboptimal strategies, whereas excessive Shannon Entropy maximization can cause entropy explosion, driving blind exploration toward incoherent reasoning chains.
arXiv:2610.20497v1 Announce Type: new Abstract: Reinforcement learning with verifiable rewards (RLVR) trains language models from checked task outcomes, but assigning a terminal reward to individual...
The paper introduces a new loss function called the horizon loss for training classifiers. It argues that the exact policy gradient used in reinforcement learning is myopic, whereas cross‑entropy is patient, and the horizon loss interpolates between the two by truncating the total error at the remaining learning. Experiments on MNIST and ImageNet with ResNet and ViT models show that horizon loss consistently improves top‑1 accuracy over cross‑entropy, especially when label noise is present.
arXiv:2507.14843v5 Announce Type: replace Abstract: Recent advances highlight Reinforcement Learning with Verifiable Rewards (RLVR) as a promising approach to enhancing LLM capabilities. However, it...
arXiv:2607. 22186v2 Announce Type: replace Abstract: Asynchronous reinforcement learning (RL) accelerates large language model (LLM) post-training by overlapping rollout generation with policy optimization, but the resulting stale, off-policy data can destabilize optimization and ultimately cause policy collapse.
arXiv:2606. 18963v1 Announce Type: new Abstract: We study online reward-punishment learning when the environment provides no scalar reward or evaluative label.
SCARL (Sequence Confidence-Aligned Reinforcement Learning) addresses a misalignment between a reinforcement learning policy’s confidence and its reward signal in language-model reasoning. By reshaping each advantage with an alignment gap that reflects the difference between a response’s group-relative confidence and the outcome it certifies, SCARL drives responses with equal advantage toward equal confidence and mitigates entropy collapse. Experiments on Qwen3‑4B‑Base show a 2.4‑point lift in Pass@32 over DAPO and consistent improvements across multiple estimators and backbones while preserving policy entropy.