The paper proves that in reinforcement learning with verifiable rewards (RLVR), any improvement in pass@1 necessarily incurs a cost in pass@k, establishing a precise price for such gains. By modeling RLVR as an exponential tilt of a base policy, the authors show that every update that raises pass@1 strictly lowers success on at least one prompt, and that the number of pass@k crossovers is bounded by the sign variation of per‑prompt success shifts. The work also derives an entropy‑performance law and provides a diagnostic to measure this price from evaluation logs.
Reinforcement learning with verifiable rewards (RLVR) makes a language model solve more problems on the first try and fewer when given many tries: it raises pass@1 while the base model overtakes it at...
arXiv:2606. 15455v1 Announce Type: cross Abstract: Reinforcement learning with verifiable rewards (RLVR) has become a key approach for enhancing the reasoning abilities of large language models.
The paper investigates how entropy behaves in reinforcement learning with verifiable rewards (RLVR) when the sampling method used during rollouts differs from the learner’s policy. It shows that a constant advantage shift alters the entropy of a softmax policy in a way that depends on the sampler: positive shifts sharpen the policy for samplers whose likelihood ratio increases with token probability (e.g., top‑k, top‑p, min‑p, and temperatures below one) and flatten it for flatter samplers. In group‑based RLVR with token‑level losses, the entropy drift is governed by the token‑weighted net advantage and a covariance term that can amplify over long horizons, with empirical results on 2B and 9B models demonstrating how different sampling and baseline strategies affect entropy stability and held‑out accuracy.
arXiv:2609.01274v1 Announce Type: new Abstract: Reinforcement learning with verifiable rewards (RLVR) improves language-model reasoning, but how these gains relate to inference-time decoding and sear...
The paper argues that the observed crossover in pass@$k$ performance between reinforcement learning with verifiable rewards (RLVR) and its base model is not always statistically confirmed. By constructing confidence bands across sampling budgets and performing power analyses, the authors show that many reported crossovers lack statistical support and that more prompts can improve detection more than more answers per prompt. They further demonstrate that RLVR’s effect on a prompt is better described as a conditional distribution (a Markov kernel) rather than a single curve, allowing predictions of crossovers in new data and clarifying how losses on difficult prompts can overturn early gains.
arXiv:2607. 20543v1 Announce Type: cross Abstract: Reinforcement learning with verifiable rewards (RLVR) can improve one-sample accuracy while making a model worse under repeated sampling.
arXiv:2510. 14807v3 Announce Type: replace Abstract: We revisit exploration collapse in reinforcement learning with verifiable rewards (RLVR), from the perspective of the \emph{candidate distribution} for next-token prediction.
The paper introduces Exploration-Distillation (ExpDis), a method that separates exploration from optimization in reinforcement learning with verifiable rewards (RLVR) for language models. ExpDis trains explorer policies with a novelty bonus, filters their outputs for correctness, and distills the results into a student policy that is optimized without the novelty bonus. This iterative process enables aggressive exploration while preserving model quality, outperforming DAPO on multiple mathematical reasoning benchmarks and improving pass@$k$ scaling.
arXiv:2507.14843v5 Announce Type: replace Abstract: Recent advances highlight Reinforcement Learning with Verifiable Rewards (RLVR) as a promising approach to enhancing LLM capabilities. However, it...
arXiv:2602.21189v3 Announce Type: replace-cross Abstract: Pass@$k$ is a widely used performance metric for verifiable large language model tasks, including mathematical reasoning, code generation, an...
The paper investigates why reinforcement learning with verifiable rewards (RLVR) reduces the diversity of solutions in reasoning tasks. By analyzing the Countdown task, the authors show that RLVR contracts the solution space mainly at the entrance—before the first arithmetic operation—causing a 67% drop in solution coverage. They demonstrate that providing an unselected entrance prefix or applying entrance‑targeted interventions can restore or even improve coverage without harming accuracy.
The paper proposes CANOPY, a minimalist reinforcement learning protocol that addresses two common pitfalls—signal starvation and policy drift—in outcome‑only RL for long‑horizon interactive tasks. By scaling same‑task exploration, keeping updates on‑policy, and anchoring updates with KL divergence, CANOPY enables a Qwen3‑14B agent to achieve top leaderboard results on the AppWorld coding benchmark without auxiliary supervision or elaborate scaffolding. The approach also improves performance on SWE‑bench for a Qwen3.5‑9B model.