The paper examines the computational challenges of training Reasoning Language Models (RLMs) using reinforcement learning with verifiable rewards (RLVR) and similar post‑training methods. It provides a compute‑centric analysis of popular RL algorithms such as PPO and GRPO, and introduces a taxonomy of intra‑ and inter‑model parallelism strategies—including traditional and novel techniques—to improve scalability and cost‑efficiency. The authors also evaluate existing RLM frameworks and offer practical guidelines and research directions for building high‑performance, scalable RLMs.
arXiv:2507. 04136v2 Announce Type: replace Abstract: This survey offers a comprehensive foundation on the integration of RL with language models, highlighting prominent algorithms such as Proximal Policy Optimization (PPO), Q-Learning, and Actor-Critic methods.
By Saksham Sahai Srivastava, Vaneet Aggarwal
arXiv:2607. 07508v1 Announce Type: cross Abstract: Reinforcement learning (RL) is becoming increasingly important for post-training large language models (LLMs).
By Zhenyu Hou, Yujiang Li, Jie Tang, Yuxiao Dong
Parason is a new framework that discovers and exploits both subtask and trial parallelism in large language model (LLM) reasoning. By converting sequential reasoning traces into structured parallel trajectories and training with Parallelism-Aware Group Relative Policy Optimization, it balances accuracy, latency, and parallelism. Experiments on mathematical reasoning benchmarks such as AIME24 and AIME25 show that Parason achieves an average acceleration of about 1.7× while maintaining competitive accuracy.
By Zhengyang Zhang, Zijian Zhang, Jiaxuan Gao, Shusheng Xu, Yi Wu, Song Han, Ligeng Zhu
Co‑RL is a multi‑agent reinforcement learning framework that trains several decoupled models without shared parameters, using rewards generated by their peers. By increasing cohort diversity—through heterogeneous model families, varying sizes, and rephrased training samples—Co‑RL reduces self‑reinforcing feedback loops, preserves behavioral diversity, and prevents training collapse. Across both text‑only and multimodal benchmarks, Co‑RL outperforms base models and prior label‑free methods, achieving gains of 3.0‑8.6% on seven text benchmarks and 2.3‑7.2% on four multimodal benchmarks, while matching or surpassing supervised approaches without any ground‑truth labels.
By Yunhao Yang, Yuexin Bian, Yunjie Tian, Di Fu, Tianjin Huang, Yuanyuan Shi, Ziang Xiao, Nuno Vasconcelos, Yijiang Li
The paper investigates how to apply Reinforcement Learning with Verifiable Rewards (RLVR) to large language models across multiple domains. It compares two training paradigms—mixed multi-task RLVR and separate RLVR followed by model merging—using tasks such as math, coding, science, instruction following, and agent. Experiments show that RLVR across domains causes minimal interference and that reasoning-intensive domains can synergize, with insights drawn from information constraints, prediction behavior, and self-verification.
By Haoqing Wang, Xiang Long, Ziheng Li, Yilong Xu, Tingguang Li, Yehui Tang