The paper introduces Policy-Compatible Reinforcement Learning (PICO), a method that addresses the learning cliff in reinforcement learning with verifiable rewards (RLVR) by performing distribution-constrained joint decoding. PICO guides token-level decoding with expert data while preserving the model’s token probabilities, and uses a dynamic top‑k mechanism to keep sampling within a finite divergence budget. The resulting trajectories are verified and policy‑compatible, and are jointly optimized with on‑policy rollouts using a mixed‑behavior objective that mitigates distribution mismatch via importance sampling, leading to consistent performance gains over standard RLVR and prior off‑policy baselines on diverse reasoning datasets and agentic benchmarks.
By Zehao Wang, Hanzhu Chen, Runquan Gui, Zhe Wang, Lin Yang
arXiv:2606. 01281v1 Announce Type: cross Abstract: Reinforcement learning with verifiable rewards (RLVR) has emerged as a powerful paradigm for enhancing the reasoning capabilities of large language models (LLMs).
By Yixiu Mao, Yun Qu, Qi Wang, Heming Zou, Xiangyang Ji
arXiv:2607. 16205v1 Announce Type: new Abstract: Reinforcement learning with verifiable rewards has emerged as a standard approach for enhancing reasoning in large language models, which typically optimizes the policy by contrasting multiple self generated rollouts.
By Dayu Wang, Jiaye Yang, Weikang Li, Jiahui Liang, Liwei Qian, Xin Pei, Jizhou Huang
arXiv:2603.16157v2 Announce Type: replace-cross
Abstract: While Reinforcement Learning (RL) enhances Large Language Model reasoning, on-policy algorithms like GRPO are sample-inefficient as they disc...
By Long Li, Zhijian Zhou, Tianyi Wang, Weidi Xu, Zuming Huang, Wei Chu, Zhe Wang, Shirui Pan, Chao Qu, Yuan Qi
arXiv:2607. 06987v1 Announce Type: new Abstract: Reinforcement learning (RL) has become the standard paradigm for enhancing the complex reasoning capabilities of large language models (LLMs).
By Chongyu Fan, Pengfei Liu, Jingjia Huang, Sijia Liu, Yi Lin
The paper introduces POISE, a reinforcement learning algorithm that uses a model’s internal states as a value estimator to reduce variance in reinforcement learning with verifiable rewards (RLVR). By employing a lightweight probe that reads internal signals during the forward pass, POISE predicts baselines online and uses a cross‑rollout construction to keep gradients unbiased. Experiments on Qwen3‑4B and OLMo3‑7B‑Instruct‑DPO across six domains show POISE outperforms existing RLVR baselines, offering more stable training and a value model that generalizes across tasks and scales with the policy.
By Yunho Choi, Jongwon Lim, Woojin Ahn, Minjae Oh, Jeonghoon Shim, Yohan Jo