arXiv AI

Selective Off-Policy Reference Tuning with Plan Guidance

Selective Off-Policy Reference Tuning with Plan Guidance (SORT) is a reinforcement learning approach that improves reasoning by adding a repair update for failed rollouts. It derives a plan from a reference solution, compares token probabilities with and without that plan, and gives higher weight to tokens that become more predictable under plan conditioning. This method turns all-wrong prompts into selective, structure-aware learning signals, yielding improvements over GRPO and guidance baselines across three backbones and eight reasoning benchmarks, especially on weaker models.

arXiv Machine Learning

Oct 12

Cross-Behavior Policy Optimization: Unified Learning from Offline Demonstrations and On- and Off-Policy Rollouts via a Shared Regression Target

The paper introduces Cross-Behavior Policy Optimization (CBPO), a method that unifies offline demonstrations and policy-generated rollouts into a single regression loss using a fixed-reference target. CBPO eliminates the need for separate supervised fine‑tuning and reinforcement learning objectives, allowing positive and negative offline completions, reference responses, and current and historical rollouts to share a common reward‑labeled format and replay buffer. Experiments on two model scales and four mathematical reasoning benchmarks show that CBPO, especially the CBPO‑Unified variant, outperforms baseline methods in avg@32 and pass@32 metrics, with historical rollouts providing additional gains.

By Du Cheng, Junyu Lu, Jingfan Zhang, Yuzhuo Zhang, Haoxuan Jia, Shichao Weng

reinforcement-learning fine-tuning benchmarks

arXiv AI

Sep 4

Headroom-Drift Replay: A Primitive for Principled Replay Control in GRPO

Headroom-Drift Replay is a replay control primitive designed for GRPO that separates reuse into two decisions: Headroom ranks stored groups by remaining learning value, while Drift gates them by compatibility with the current policy. The method keeps the fresh on‑policy stream unchanged and adds no auxiliary generation or training machinery. Across mathematical reasoning, multimodal reasoning, and Agentic Search benchmarks, it outperforms naive replay and matches or exceeds broader replay methods on Avg Mean@32, delivering comparable quality at materially lower wall‑clock time in Agentic Search.

By Hyun Bin Park, Du-Seong Chang

agents reinforcement-learning multimodal benchmarks

arXiv AI

6d ago

Exploration via Activation-Space Steering for Reinforcement Learning with Verifiable Rewards

arXiv:2610.27012v1 Announce Type: cross Abstract: Reinforcement learning with verifiable rewards (RLVR) trains a language model by sampling responses, scoring them with an automatic verifier, and upd...

By Yunan Zeng, Jinhao Jing, Xi Cheng, Yuwang Wang

llms reinforcement-learning benchmarks

arXiv AI

Jun 2

RLVR without Ineffective Samples: Group Prioritized Off-Policy Optimization for LLM Reasoning

arXiv:2606. 01281v1 Announce Type: cross Abstract: Reinforcement learning with verifiable rewards (RLVR) has emerged as a powerful paradigm for enhancing the reasoning capabilities of large language models (LLMs).

By Yixiu Mao, Yun Qu, Qi Wang, Heming Zou, Xiangyang Ji

llms reinforcement-learning fine-tuning benchmarks safety

arXiv Machine Learning

4d ago

MOER: Mix-Policy Optimization under Expert Prefix Guidance with Selective Rollback

MOER is a mix‑policy optimization framework that blends expert guidance with self‑exploration in reinforcement learning. It injects a decaying expert chain‑of‑thought prefix into prompts, progressively annealing the prefix length to encourage exploration, and uses a selective rollback mechanism to replace failed completions with the full expert trace, ensuring correctness. The method employs a harmonized mix‑policy objective that optimizes on‑policy tokens with a GRPO loss while supervising both expert prefix and completion tokens with a trust‑region SFT loss, leading to stable training and superior performance on mathematical and general reasoning tasks.

By Xuyang Zhong, Jiuding Yang, Yiwen Guo, Chen Liu

reinforcement-learning fine-tuning

arXiv Machine Learning

Jun 30

Experience Augmented Policy Optimization for LLM Reasoning

arXiv:2606. 30420v1 Announce Type: new Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) is a powerful paradigm for improving the reasoning capabilities of large language models (LLMs).

By Jinda Lu, Kexin Huang, Junkang Wu, Shuo Yang, Jinghan Li, Chiyu Ma, Shaohang Wei, Xiang Wang, Guoyin Wang, Jingren Zhou

llms reinforcement-learning benchmarks

arXiv AI

Jul 22

Off-Context GRPO: Learning to Reason on Hard Problems using Privileged Information

arXiv:2607. 19313v1 Announce Type: cross Abstract: Reinforcement learning with verifiable rewards (RLVR) improves reasoning in large language models.

By Priyank Agrawal, Ankur Samanta, Shervin Ghasemlou, Jalaj Bhandari, Kavosh Asadi, Daniel Jiang, Aditya Modi

llms reinforcement-learning benchmarks

arXiv AI

Aug 11

Parameter Exploration for RLVR via Variational Learning

arXiv:2608. 09805v1 Announce Type: cross Abstract: Exploration has been a focus of reinforcement learning research for a long time.

By Vatsal Venkatkrishna, Nico Daheim, Iryna Gurevych

llms reinforcement-learning

arXiv AI

Jul 7

R$^2$PO: Decoupling Rollout and Inference Policies for LLM Reasoning

arXiv:2601. 11960v3 Announce Type: replace-cross Abstract: Existing reinforcement learning methods for LLM reasoning implicitly assume that the policy generating training trajectories should coincide with the one producing inference responses.

By Jingchu Wang, Bingbing Xu, Yige Yuan, Dan Zhang, Bin Xie, Xiaoqian Sun, Huawei Shen

llms reinforcement-learning safety

arXiv Machine Learning

Sep 16

BASIS: Batchwise Advantage Estimation from Single-Rollout Information Sharing for LLM Reasoning

arXiv:2605.27293v2 Announce Type: replace Abstract: Reinforcement learning with verifiable rewards has become a standard recipe for improving the reasoning abilities of large language models. Existin...

By Shijin Gong, Erhan Xu, Kai Ye, Giulia Livieri, Francesco Quinzan, Chengchun Shi

llms reinforcement-learning

arXiv AI

Jul 14

ARMOR: Stabilizing On-Policy LLM RL with Off-Policy Anchor Samples

arXiv:2607. 10481v1 Announce Type: cross Abstract: Reinforcement learning (RL) has significantly enhanced the reasoning capabilities of large language models (LLMs), yet the training process remains notoriously fragile.

By Kexin Huang, Junkang Wu, Jinda Lu, Shuo Yang, Chiyu Ma, Jiancan Wu, Xiang Wang, Xiangnan He, Guoyin Wang, Jingren Zhou

llms reinforcement-learning benchmarks

arXiv Machine Learning

Jun 16

Replay What Matters: Off-Policy Replay for Efficient LLM Reinforcement Unlearning

arXiv:2606. 15333v1 Announce Type: cross Abstract: LLM unlearning has emerged as a cost-effective alternative to full retraining for removing hazardous knowledge from pretrained models while preserving general utility.

By Zirui Pang, Chenlong Zhang, Haosheng Tan, Zhuoran Jin, Jiaheng Wei, Zixin Zhong

llms reinforcement-learning benchmarks

Related stories