The paper introduces a method to enhance large language model (LLM) exploration in Reinforcement Learning with Verifiable Rewards (RLVR) by guiding the target model with partial reasoning trajectories from smaller, weaker language models. This weak-model guidance disrupts over‑confidence, preserves generative diversity, and mitigates entropy collapse without extra fine‑tuning or complex reward designs. Experiments on mathematical benchmarks show consistent improvements over vanilla RLVR, especially as the number of allowed attempts ($k$) increases, indicating broader reasoning coverage.
By Xingyu Shen, Huishuai Zhang, Peng Li, Yinchun Wang, Dongyan Zhao
arXiv:2606. 03608v1 Announce Type: cross Abstract: Test-time reinforcement learning has emerged as a promising paradigm for enhancing the complex reasoning abilities of large language models in a completely label-free manner.
By Jiahui Li, Jianfeng Shan, Wenpei Chen, Shunyu Wu, Jian Lou, Wenjie Feng, Dan Li, See-Kiong Ng
arXiv:2607. 16205v1 Announce Type: new Abstract: Reinforcement learning with verifiable rewards has emerged as a standard approach for enhancing reasoning in large language models, which typically optimizes the policy by contrasting multiple self generated rollouts.
By Dayu Wang, Jiaye Yang, Weikang Li, Jiahui Liang, Liwei Qian, Xin Pei, Jizhou Huang
arXiv:2605.28295v2 Announce Type: replace
Abstract: Reinforcement learning with verifiable rewards (RLVR) trains reasoning models without labeled trajectories, using groups of verifier-scored rollout...
By Soeun Kim, Albert No
arXiv:2603. 25184v2 Announce Type: replace-cross Abstract: Reinforcement learning (RL) has become essential for post-training large language models (LLMs) in reasoning tasks.
By Jiahao Wu, Ning Lu, Shengcai Liu, Kun Wang, Yanting Yang, Bailong Lin, Chen Jason Zhang, Li Qing, Ke Tang
arXiv:2606. 01281v1 Announce Type: cross Abstract: Reinforcement learning with verifiable rewards (RLVR) has emerged as a powerful paradigm for enhancing the reasoning capabilities of large language models (LLMs).
By Yixiu Mao, Yun Qu, Qi Wang, Heming Zou, Xiangyang Ji
The paper introduces APIVIS, a training-time framework that integrates finite-budget Gumbel search into reinforcement learning with verifiable rewards (RLVR) for mathematical reasoning. APIVIS combines direct and searched responses within each rollout group, uses selective supervision on search-improved tokens, and applies value-guided selection to improve verifier rewards at each searched state. Experiments on standard mathematical reasoning benchmarks and various model scales show significant performance gains over existing search-based methods.
By Shaohuai Liu, Yuning Wu, Haoran Liu, Enzo Jia, Devin Chen, Kai Wei
The paper introduces EquivAug, a data augmentation framework that applies symbolic equivalence transformations to chain-of-thought (CoT) solutions, generating diverse yet provably correct reasoning paths for supervised fine-tuning (SFT). By converting CoT solutions into formal symbolic representations, applying equivalence transformations, and reconstructing natural-language explanations, EquivAug produces training data that varies in reasoning steps rather than surface-level lexical differences. Experiments on GSM8K and FOLIO show that EquivAug consistently achieves the highest reasoning path diversity after SFT and the strongest accuracy after GRPO, establishing a positive correlation between CoT diversity during SFT and downstream reinforcement learning with verifiable rewards (RLVR) performance.
By Xiaoou Liu, Tiejin Chen, Hanyang Chen, Yuan Ling, Hua Wei
Guided-RLVR introduces a lightweight controller to reinforcement learning with verifiable rewards (RLVR), injecting semantically meaningful steering tokens into the sampling distribution. This strategy encourages large language models to explore multiple reasoning perspectives, expanding the set of reasoning trajectories. Experiments on mathematical reasoning and code generation benchmarks demonstrate that Guided-RLVR consistently outperforms existing RLVR methods while producing more diverse reasoning paths.
By Bo Yang, Zishuai Zhang, Weihua Li, Qinnan Zhang, Sijia Wen, Hainan Zhang
The paper introduces Policy-Compatible Reinforcement Learning (PICO), a method that addresses the learning cliff in reinforcement learning with verifiable rewards (RLVR) by performing distribution-constrained joint decoding. PICO guides token-level decoding with expert data while preserving the model’s token probabilities, and uses a dynamic top‑k mechanism to keep sampling within a finite divergence budget. The resulting trajectories are verified and policy‑compatible, and are jointly optimized with on‑policy rollouts using a mixed‑behavior objective that mitigates distribution mismatch via importance sampling, leading to consistent performance gains over standard RLVR and prior off‑policy baselines on diverse reasoning datasets and agentic benchmarks.
By Zehao Wang, Hanzhu Chen, Runquan Gui, Zhe Wang, Lin Yang
The paper introduces Policy-Compatible Reinforcement Learning (PICO), a method that combines distribution-constrained joint decoding with on-policy rollouts to address the learning cliff in reinforcement learning with verifiable rewards (RLVR). PICO guides token-level decoding using expert data while preserving the model’s relative token probabilities, and employs a dynamic top‑k mechanism to keep sampling within a finite divergence budget. This approach yields verified, policy‑compatible trajectories that are jointly optimized with on‑policy rollouts, leading to consistent performance gains over standard RLVR and prior off‑policy baselines across diverse reasoning datasets and agentic benchmarks.
arXiv:2606. 24994v1 Announce Type: new Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) for language-model reasoning can fail at both extremes of task difficulty: easy prompts often produce all-correct, low-diversity rollout groups with little gradient signal, while hard prompts can produce all-incorrect groups with no positive reward.
By Wenyang Hu, Junxiang Jia, Zhen Shu, Daniel Dahlmeier, See-Kiong Ng, Bryan Kian Hsiang Low