The paper investigates why reinforcement learning with verifiable rewards (RLVR) reduces the diversity of solutions in reasoning tasks. By analyzing the Countdown task, the authors show that RLVR contracts the solution space mainly at the entrance—before the first arithmetic operation—causing a 67% drop in solution coverage. They demonstrate that providing an unselected entrance prefix or applying entrance‑targeted interventions can restore or even improve coverage without harming accuracy.
By Qiancheng Zhou, Ruizhe Li
The study investigates whether post‑training methods—GRPO, SFT, and DPO—improve language models’ ability to follow prompt evidence that conflicts with memorized knowledge. By comparing nine training variants across different scales and families, the authors find that grounding gains are modest for GRPO, moderate for Conflict‑SFT, and near‑ceiling for DPO, but all largely rely on the same causal attention‑head set present in the starting checkpoint. Removing the starting‑model grounding direction suppresses these gains, while adding it back recovers a significant portion of DPO’s improvement, indicating that existing model machinery drives most of the observed gains.
By Prakhar Gupta, Vaibhav Gupta
arXiv:2607. 10203v1 Announce Type: cross Abstract: Adaptive-compute world models -- early-exit or mixture-of-depths predictors that spend variable depth per step -- assume depth buys better predictions and can be routed adaptively.
By Achyuthan Sivasankar
arXiv:2607. 21372v1 Announce Type: cross Abstract: Score Entropy Discrete Diffusion (SEDD) parameterizes discrete reverse processes with unconstrained positive score ratios.
By Jingyuan Li, Xiaoyi Jiang, Yixuan Jiang, Wei Liu, Yi Zhu, Zuoqiang Shi, Pipi Hu
The paper evaluates the true benefit of verifier‑based reinforcement learning (RLVR) over supervised fine‑tuning (SFT) using a deterministic benchmark of 1,877 synthetic GPU and LLM training failure diagnosis tasks. In controlled experiments, SFT alone achieved high performance (0.785 on the hard split), while RLVR added no detectable improvement in‑distribution and only a modest gain under distribution shift (+0.33 points). The study concludes that most performance gains come from SFT, with verifier‑based RL contributing only small additional benefits.
By Amit Sagtani, Saloni Garg
Reinforcement learning from verifiable rewards (RLVR) for mathematical reasoning suffers from a structural blind spot: on "cliff" prompts-those on which every sampled rollout in a group fails-the group-normalized advantage is identically zero, so GRPO produces no gradient on precisely the prompts at the frontier of the model's capability. We introduce LoRA Scaffolded Policy Optimization (LSPO), a sampling-time mechanism that recovers this lost gradient.
arXiv:2609.09075v1 Announce Type: cross
Abstract: In reinforcement learning with verifiable rewards (RLVR) trained with group relative policy optimization (GRPO), the KL-free reward-advantage term st...
By Tommy Sha, Skylar Zhai, Siqi Zhao
arXiv:2610.17093v1 Announce Type: new
Abstract: A model diff credits post-training with whatever separates a base checkpoint from one post-trained descendant, presuming a rerun would land close by. O...
By Xupeng Chen, Chenqian Le, Jiayi Cheng, Runhao Li, Michael Yue
The paper critiques the use of pass@$k$ as a sole metric for evaluating reinforcement‑learning post‑training on language models, showing that it ignores diversity in model outputs. Experiments on Qwen2.5‑1.5B‑Instruct trained with Group Relative Policy Optimization (GRPO) and rejection‑sampling fine‑tuning (RFT) reveal that these methods shift token‑level, answer‑level, and unique‑answer diversity in opposite directions, yet pass@$k$ fails to capture these differences. Even when restricting to verifier‑correct completions, diversity gaps persist, and pass@$k$ does not consistently identify the better performing arm on hard problem subsets, indicating that the metric can mask true capability changes.
By Subham Rath, Raj Dandekar, Rajat Dandekar, Sreedath Panat
arXiv:2605. 17314v2 Announce Type: replace-cross Abstract: We consider whether off-policy experience from a smaller, weaker model can elicit capability in a stronger learner that on-policy RL fine-tuning (e.
By Wei Deng
arXiv:2610.20497v1 Announce Type: new
Abstract: Reinforcement learning with verifiable rewards (RLVR) trains language models from checked task outcomes, but assigning a terminal reward to individual...
By Wei Chen, Yubing Wu, Zhou Xu, Haoyue Zhang, Delu Zeng, Jiahao Wang, Junmei Yang, Pan tiewen, Jingfeng Liu, Zhiheng Zhou, Bo Li, Min Chen
The study measured the impact of a single training example on a GPT‑2 model by running 24 counterfactual experiments. 32 models were trained from scratch on OpenWebText, and at a specific training step a single batch row was replaced with a 194‑token passage under three conditions (fluent prose, fabricated subject, random characters) or left unchanged. Results showed that the passage was learned from one exposure and decayed, with measurable differences in cross‑entropy up to 50 steps after injection but no lasting effect at the final step.
By Zachary Speck, Asa Shepard