Reinforcement learning with verifiable rewards (RLVR) makes a language model solve more problems on the first try and fewer when given many tries: it raises pass@1 while the base model overtakes it at...
arXiv:2606. 15455v1 Announce Type: cross Abstract: Reinforcement learning with verifiable rewards (RLVR) has become a key approach for enhancing the reasoning abilities of large language models.
By Suqin Yuan, Jinkun Chen, Jiyang Zheng, Muyang Li, Lei Feng, Dadong Wang, Tao Xiang, Tongliang Liu, Bo An
The paper investigates how entropy behaves in reinforcement learning with verifiable rewards (RLVR) when the sampling method used during rollouts differs from the learner’s policy. It shows that a constant advantage shift alters the entropy of a softmax policy in a way that depends on the sampler: positive shifts sharpen the policy for samplers whose likelihood ratio increases with token probability (e.g., top‑k, top‑p, min‑p, and temperatures below one) and flatten it for flatter samplers. In group‑based RLVR with token‑level losses, the entropy drift is governed by the token‑weighted net advantage and a covariance term that can amplify over long horizons, with empirical results on 2B and 9B models demonstrating how different sampling and baseline strategies affect entropy stability and held‑out accuracy.
By Zhiyuan Wang, Jacob Steeves, Chang Liu
arXiv:2609.01274v1 Announce Type: new
Abstract: Reinforcement learning with verifiable rewards (RLVR) improves language-model reasoning, but how these gains relate to inference-time decoding and sear...
By Wenhe Sun, Cunxiang Wang, Zijun Yao, Yixin Cao
The paper argues that the observed crossover in pass@$k$ performance between reinforcement learning with verifiable rewards (RLVR) and its base model is not always statistically confirmed. By constructing confidence bands across sampling budgets and performing power analyses, the authors show that many reported crossovers lack statistical support and that more prompts can improve detection more than more answers per prompt. They further demonstrate that RLVR’s effect on a prompt is better described as a conditional distribution (a Markov kernel) rather than a single curve, allowing predictions of crossovers in new data and clarifying how losses on difficult prompts can overturn early gains.
By Chen Yang, Xianyang Zhang, Jun Chen
arXiv:2607. 20543v1 Announce Type: cross Abstract: Reinforcement learning with verifiable rewards (RLVR) can improve one-sample accuracy while making a model worse under repeated sampling.
By Todd Zhou