The paper proves that in reinforcement learning with verifiable rewards (RLVR), any improvement in pass@1 necessarily incurs a cost in pass@k, establishing a precise price for such gains. By modeling RLVR as an exponential tilt of a base policy, the authors show that every update that raises pass@1 strictly lowers success on at least one prompt, and that the number of pass@k crossovers is bounded by the sign variation of per‑prompt success shifts. The work also derives an entropy‑performance law and provides a diagnostic to measure this price from evaluation logs.
By Yinuo Liu
arXiv:2606. 15455v1 Announce Type: cross Abstract: Reinforcement learning with verifiable rewards (RLVR) has become a key approach for enhancing the reasoning abilities of large language models.
By Suqin Yuan, Jinkun Chen, Jiyang Zheng, Muyang Li, Lei Feng, Dadong Wang, Tao Xiang, Tongliang Liu, Bo An
arXiv:2607. 20543v1 Announce Type: cross Abstract: Reinforcement learning with verifiable rewards (RLVR) can improve one-sample accuracy while making a model worse under repeated sampling.
By Todd Zhou
The paper introduces Exploration-Distillation (ExpDis), a method that separates exploration from optimization in reinforcement learning with verifiable rewards (RLVR) for language models. ExpDis trains explorer policies with a novelty bonus, filters their outputs for correctness, and distills the results into a student policy that is optimized without the novelty bonus. This iterative process enables aggressive exploration while preserving model quality, outperforming DAPO on multiple mathematical reasoning benchmarks and improving pass@$k$ scaling.
By Saif Punjwani, Micah Goldblum
arXiv:2609.01274v1 Announce Type: new
Abstract: Reinforcement learning with verifiable rewards (RLVR) improves language-model reasoning, but how these gains relate to inference-time decoding and sear...
By Wenhe Sun, Cunxiang Wang, Zijun Yao, Yixin Cao
The paper investigates how entropy behaves in reinforcement learning with verifiable rewards (RLVR) when the sampling method used during rollouts differs from the learner’s policy. It shows that a constant advantage shift alters the entropy of a softmax policy in a way that depends on the sampler: positive shifts sharpen the policy for samplers whose likelihood ratio increases with token probability (e.g., top‑k, top‑p, min‑p, and temperatures below one) and flatten it for flatter samplers. In group‑based RLVR with token‑level losses, the entropy drift is governed by the token‑weighted net advantage and a covariance term that can amplify over long horizons, with empirical results on 2B and 9B models demonstrating how different sampling and baseline strategies affect entropy stability and held‑out accuracy.
By Zhiyuan Wang, Jacob Steeves, Chang Liu
arXiv:2510. 14807v3 Announce Type: replace Abstract: We revisit exploration collapse in reinforcement learning with verifiable rewards (RLVR), from the perspective of the \emph{candidate distribution} for next-token prediction.
By Ruotian Peng, Yi Ren, Zhouliang Yu, Weiyang Liu, Yandong Wen
arXiv:2507.14843v5 Announce Type: replace
Abstract: Recent advances highlight Reinforcement Learning with Verifiable Rewards (RLVR) as a promising approach to enhancing LLM capabilities. However, it...
By Fang Wu, Weihao Xuan, Danny Xing, Ximing Lu, Mingjie Liu, Yi Dong, Zaid Harchaoui, Yejin Choi
The paper argues that the observed crossover in pass@$k$ performance between reinforcement learning with verifiable rewards (RLVR) and its base model is not always statistically confirmed. By constructing confidence bands across sampling budgets and performing power analyses, the authors show that many reported crossovers lack statistical support and that more prompts can improve detection more than more answers per prompt. They further demonstrate that RLVR’s effect on a prompt is better described as a conditional distribution (a Markov kernel) rather than a single curve, allowing predictions of crossovers in new data and clarifying how losses on difficult prompts can overturn early gains.
By Chen Yang, Xianyang Zhang, Jun Chen
arXiv:2602.21189v3 Announce Type: replace-cross
Abstract: Pass@$k$ is a widely used performance metric for verifiable large language model tasks, including mathematical reasoning, code generation, an...
By Anas Barakat, Souradip Chakraborty, Khushbu Pahwa, Amrit Singh Bedi
The paper investigates why reinforcement learning with verifiable rewards (RLVR) reduces the diversity of solutions in reasoning tasks. By analyzing the Countdown task, the authors show that RLVR contracts the solution space mainly at the entrance—before the first arithmetic operation—causing a 67% drop in solution coverage. They demonstrate that providing an unselected entrance prefix or applying entrance‑targeted interventions can restore or even improve coverage without harming accuracy.
By Qiancheng Zhou, Ruizhe Li
The paper introduces belief‑shift branching, a method for placing forks in tree‑structured reinforcement learning rollouts by identifying points where a model’s answer belief changes most. Unlike traditional structural or entropy‑based approaches, belief‑shift uses a probe, logit‑lens depth profile, or learned activation direction to locate pivots in the value curve, incurring minimal computational overhead. Experiments across multiple models and benchmarks show that belief‑shift forking consistently outperforms baseline methods, yielding significant gains in mathematics and code tasks.