The paper "Demystifying Reinforcement Learning Post-Training of Language Models" investigates how reinforcement learning (RL) post‑training enhances large language models (LLMs) for tasks such as reasoning, math, and coding. By isolating RL components in a controlled setting, the authors analyze how the base model’s prior distribution, reward granularity, prompt diversity, and model scale influence outcomes, using policy entropy to compare pre‑training, supervised fine‑tuning (SFT), and RL stages. The study clarifies the role of spurious rewards, the importance of the base model’s probability mass on desired behaviors, and how these factors interact to determine post‑training success, offering a practical primer for NLP researchers.
"whyItMatters":"The work provides a clearer understanding of RL post‑training mechanics, helping researchers and practitioners effectively apply RL to improve LLM capabilities."
By Donovan Clay, Saket Gollapudi, Sankar Harilal, Min Jang, Jacob Morrison, Sewoong Oh, Natasha Jaques
The paper investigates how to apply Reinforcement Learning with Verifiable Rewards (RLVR) to large language models across multiple domains. It compares two training paradigms—mixed multi-task RLVR and separate RLVR followed by model merging—using tasks such as math, coding, science, instruction following, and agent. Experiments show that RLVR across domains causes minimal interference and that reasoning-intensive domains can synergize, with insights drawn from information constraints, prediction behavior, and self-verification.
By Haoqing Wang, Xiang Long, Ziheng Li, Yilong Xu, Tingguang Li, Yehui Tang
The paper introduces Reinforcement Learning with Verifiable Rewards (RLVR) applied to small search agents, specifically training a Qwen3.5-0.8B model with Group Relative Policy Optimization and an interleaved Wikipedia-search tool on the MuSiQue dataset. Experiments varying reward shapes across three seeds show that RLVR can achieve a 3.8‑fold improvement over an untrained baseline, with the best run reaching a 0.352 average exact match. The study finds that the sparse exact‑match reward, standard in larger models, performs poorly for small models, indicating that reward design must be tailored rather than scaled down from large‑model recipes.
By Gaurisankar Jayadas, Aske Plaat, \'Alvaro Serra-G\'omez, Sandheep P
arXiv:2507.14843v5 Announce Type: replace
Abstract: Recent advances highlight Reinforcement Learning with Verifiable Rewards (RLVR) as a promising approach to enhancing LLM capabilities. However, it...
By Fang Wu, Weihao Xuan, Danny Xing, Ximing Lu, Mingjie Liu, Yi Dong, Zaid Harchaoui, Yejin Choi
Composition‑RL is an offline data‑augmentation method that composes existing prompts before reinforcement learning to address the pass‑rate‑1 bottleneck in large‑language‑model training. By generating and verifying a composed dataset once from a fixed source pool, it avoids modifying the RL optimizer and improves reasoning performance across models ranging from 4B to 30B. A curriculum variant that gradually increases compositional depth further boosts performance, and the approach also facilitates cross‑domain RL by combining prompts from different domains.
By Xin Xu, Clive Bai, Kai Yang, Tianhao Chen, Cai Yuchen, Yangkun Chen, Weijie Liu, Hao Chen, Yang Wang, Saiyong Yang, Can Yang
arXiv:2605. 02909v2 Announce Type: replace-cross Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) has become a powerful approach for improving the reasoning capabilities of large language models (LLMs).
By Kazuki Egashira, Mark Vero, Jasper Dekoninck, Florian E. Dorner, Robin Staab, Martin Vechev
arXiv:2607. 02869v1 Announce Type: new Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a promising paradigm for improving mathematical reasoning in language models.
By Anagha Radhakrishna Palandye, Rebecca Glick, Osheen Kaul
The paper introduces Circuit Reasoning Score (CRS), a data‑selection signal for reinforcement learning with verifiable rewards that uses attention‑head activity from a frozen base model to gauge reasoning engagement. CRS is computed in a single forward pass without reward labels or rollouts, and it shows that selecting problems with the lowest reasoning‑circuit engagement can outperform random selection on several medium‑difficulty benchmarks. However, the benefit depends on domain, model scale, and reward conditions, indicating that data selection in this setting is regime‑dependent rather than a fixed ranking of problem quality.
By Zhuofan Chen, Ziqian Jiao, Yikai Cui, Zhixin Cai, Jun Bai, Wenge Rong
arXiv:2607. 23802v1 Announce Type: new Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) has driven recent progress in reasoning-oriented large language models (LLMs) by enabling large-scale optimization.
By Qinsi Wang, Jing Shi, Huazheng Wang, Kun Wan, Yiran Wu, Bo Liu, Qingyun Wu, Hai Helen Li, Yiran Chen, Handong Zhao, Wentian Zhao
arXiv:2607. 16097v1 Announce Type: cross Abstract: Reinforcement learning (RL) has become central to improving large language models (LLMs) on complex reasoning tasks, yet RL post-training is largely studied in isolation from the pretraining that precedes it.
By Jingyan Shen, Ang Li, Salman Rahman, Yifan Sun, Micah Goldblum, Matus Telgarsky, Pavel Izmailov
The paper introduces T2T (Thickening-to-Thinning), a dynamic reward framework for large language models that mimics human learning by separating exploration and consolidation phases. During incorrect attempts, T2T encourages exploration to broaden the search space, while after correct solutions it applies length penalties to promote concise reasoning. Experiments on mathematical benchmarks across five mainstream LLMs show that T2T outperforms standard GRPO and recent baselines, improving overall reasoning performance.
By Wenze Lin, Zhen Yang, Xitai Jiang, Xiaoteng Ma, Gao Huang
The paper investigates the reliability of rule- and model-based verifiers used in reinforcement learning with verifiable reward (RLVR) for mathematical reasoning. It finds that rule-based verifiers often miss equivalent answers in different formats, causing false negatives that degrade RL performance as models improve. Model-based verifiers achieve higher static accuracy but become vulnerable to reward hacking during RL, misclassifying certain response patterns as correct after fine-tuning.
By Yuzhen Huang, Weihao Zeng, Xingshan Zeng, Qi Zhu, Junxian He