arXiv AI
Where and What to Reward? Understanding Generalization Between Verifiable and Open-Ended Domains in Reinforcement Learning
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Machine Learning
Aug 27
Demystifying Reinforcement Learning Post-Training of Language Models
The paper "Demystifying Reinforcement Learning Post-Training of Language Models" investigates how reinforcement learning (RL) post‑training enhances large language models (LLMs) for tasks such as reasoning, math, and coding. By isolating RL components in a controlled setting, the authors analyze how the base model’s prior distribution, reward granularity, prompt diversity, and model scale influence outcomes, using policy entropy to compare pre‑training, supervised fine‑tuning (SFT), and RL stages. The study clarifies the role of spurious rewards, the importance of the base model’s probability mass on desired behaviors, and how these factors interact to determine post‑training success, offering a practical primer for NLP researchers. "whyItMatters":"The work provides a clearer understanding of RL post‑training mechanics, helping researchers and practitioners effectively apply RL to improve LLM capabilities."
arXiv AI
Sep 10
To Mix or To Merge: Toward Multi-Domain Reinforcement Learning for Large Language Models
The paper investigates how to apply Reinforcement Learning with Verifiable Rewards (RLVR) to large language models across multiple domains. It compares two training paradigms—mixed multi-task RLVR and separate RLVR followed by model merging—using tasks such as math, coding, science, instruction following, and agent. Experiments show that RLVR across domains causes minimal interference and that reasoning-intensive domains can synergize, with insights drawn from information constraints, prediction behavior, and self-verification.
arXiv AI
Sep 25
Reinforcement Learning with Verifiable Rewards for Small Search Agents
The paper introduces Reinforcement Learning with Verifiable Rewards (RLVR) applied to small search agents, specifically training a Qwen3.5-0.8B model with Group Relative Policy Optimization and an interleaved Wikipedia-search tool on the MuSiQue dataset. Experiments varying reward shapes across three seeds show that RLVR can achieve a 3.8‑fold improvement over an untrained baseline, with the best run reaching a 0.352 average exact match. The study finds that the sparse exact‑match reward, standard in larger models, performs poorly for small models, indicating that reward design must be tailored rather than scaled down from large‑model recipes.
arXiv Machine Learning
Oct 15
The Invisible Leash: Why RLVR May or May Not Escape Its Origin
arXiv:2507.14843v5 Announce Type: replace Abstract: Recent advances highlight Reinforcement Learning with Verifiable Rewards (RLVR) as a promising approach to enhancing LLM capabilities. However, it...
arXiv Computation and Language
Oct 16
Composition-RL: Compose Your Verifiable Prompts for Reinforcement Learning of Large Language Models
Composition‑RL is an offline data‑augmentation method that composes existing prompts before reinforcement learning to address the pass‑rate‑1 bottleneck in large‑language‑model training. By generating and verifying a composed dataset once from a fixed source pool, it avoids modifying the RL optimizer and improves reasoning performance across models ranging from 4B to 30B. A curriculum variant that gradually increases compositional depth further boosts performance, and the approach also facilitates cross‑domain RL by combining prompts from different domains.
arXiv AI
Aug 18
Delay, Plateau, or Collapse: Evaluating the Impact of Systematic Verification Error on RLVR
arXiv:2605. 02909v2 Announce Type: replace-cross Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) has become a powerful approach for improving the reasoning capabilities of large language models (LLMs).