arXiv Machine Learning
The Invisible Leash: Why RLVR May or May Not Escape Its Origin
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv AI
Jun 16
Understanding Diversity Collapse in RLVR via the Lens of Overtraining
arXiv:2606. 15455v1 Announce Type: cross Abstract: Reinforcement learning with verifiable rewards (RLVR) has become a key approach for enhancing the reasoning abilities of large language models.
arXiv Computation and Language
Aug 28
Boosting LLM Exploration via Weak-Model Guidance in RLVR
The paper introduces a method to enhance large language model (LLM) exploration in Reinforcement Learning with Verifiable Rewards (RLVR) by guiding the target model with partial reasoning trajectories from smaller, weaker language models. This weak-model guidance disrupts over‑confidence, preserves generative diversity, and mitigates entropy collapse without extra fine‑tuning or complex reward designs. Experiments on mathematical benchmarks show consistent improvements over vanilla RLVR, especially as the number of allowed attempts ($k$) increases, indicating broader reasoning coverage.
arXiv AI
Jun 17
Beyond the Sampled Token: Preserving Candidate Support in RLVR
arXiv:2510. 14807v3 Announce Type: replace Abstract: We revisit exploration collapse in reinforcement learning with verifiable rewards (RLVR), from the perspective of the \emph{candidate distribution} for next-token prediction.
arXiv Machine Learning
Aug 27
Demystifying Reinforcement Learning Post-Training of Language Models
The paper "Demystifying Reinforcement Learning Post-Training of Language Models" investigates how reinforcement learning (RL) post‑training enhances large language models (LLMs) for tasks such as reasoning, math, and coding. By isolating RL components in a controlled setting, the authors analyze how the base model’s prior distribution, reward granularity, prompt diversity, and model scale influence outcomes, using policy entropy to compare pre‑training, supervised fine‑tuning (SFT), and RL stages. The study clarifies the role of spurious rewards, the importance of the base model’s probability mass on desired behaviors, and how these factors interact to determine post‑training success, offering a practical primer for NLP researchers. "whyItMatters":"The work provides a clearer understanding of RL post‑training mechanics, helping researchers and practitioners effectively apply RL to improve LLM capabilities."
arXiv AI
Sep 10
Boosting LLM Reasoning via Human-Inspired Reward Shaping
The paper introduces T2T (Thickening-to-Thinning), a dynamic reward framework for large language models that mimics human learning by separating exploration and consolidation phases. During incorrect attempts, T2T encourages exploration to broaden the search space, while after correct solutions it applies length penalties to promote concise reasoning. Experiments on mathematical benchmarks across five mainstream LLMs show that T2T outperforms standard GRPO and recent baselines, improving overall reasoning performance.
arXiv Machine Learning
Oct 9
Measuring and Mitigating Solution Mode Collapse in RLVR
The paper introduces ModeBench, a benchmark for multi-solution tasks that evaluates both correctness and the diversity of solutions found by a language model. It shows that reinforcement learning with verifiable rewards (RLVR) tends to collapse solution modes, concentrating probability on fewer correct answers even while accuracy remains high. To counter this, the authors propose Re:Max, a replay-based method that stores one verified example per discovered mode and trains uniformly on these stored modes, improving both success rates and solution diversity across various model scales and RL objectives.