← Back to all news
arXiv AI By Anh Duc, Tien-Phat Nguyen, Thien Huu Nguyen, Linh Ngo Van, Trung Le

Selective Off-Policy Reference Tuning with Plan Guidance

Read the original on arXiv AI →

Selective Off-Policy Reference Tuning with Plan Guidance (SORT) is a reinforcement learning approach that improves reasoning by adding a repair update for failed rollouts. It derives a plan from a reference solution, compares token probabilities with and without that plan, and gives higher weight to tokens that become more predictable under plan conditioning. This method turns all-wrong prompts into selective, structure-aware learning signals, yielding improvements over GRPO and guidance baselines across three backbones and eight reasoning benchmarks, especially on weaker models.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv Machine Learning
Oct 12

Cross-Behavior Policy Optimization: Unified Learning from Offline Demonstrations and On- and Off-Policy Rollouts via a Shared Regression Target

The paper introduces Cross-Behavior Policy Optimization (CBPO), a method that unifies offline demonstrations and policy-generated rollouts into a single regression loss using a fixed-reference target. CBPO eliminates the need for separate supervised fine‑tuning and reinforcement learning objectives, allowing positive and negative offline completions, reference responses, and current and historical rollouts to share a common reward‑labeled format and replay buffer. Experiments on two model scales and four mathematical reasoning benchmarks show that CBPO, especially the CBPO‑Unified variant, outperforms baseline methods in avg@32 and pass@32 metrics, with historical rollouts providing additional gains.

By Du Cheng, Junyu Lu, Jingfan Zhang, Yuzhuo Zhang, Haoxuan Jia, Shichao Weng
reinforcement-learningfine-tuningbenchmarks
More like this →
arXiv AI
Sep 4

Headroom-Drift Replay: A Primitive for Principled Replay Control in GRPO

Headroom-Drift Replay is a replay control primitive designed for GRPO that separates reuse into two decisions: Headroom ranks stored groups by remaining learning value, while Drift gates them by compatibility with the current policy. The method keeps the fresh on‑policy stream unchanged and adds no auxiliary generation or training machinery. Across mathematical reasoning, multimodal reasoning, and Agentic Search benchmarks, it outperforms naive replay and matches or exceeds broader replay methods on Avg Mean@32, delivering comparable quality at materially lower wall‑clock time in Agentic Search.

By Hyun Bin Park, Du-Seong Chang
agentsreinforcement-learningmultimodalbenchmarks
More like this →
arXiv Machine Learning
4d ago

MOER: Mix-Policy Optimization under Expert Prefix Guidance with Selective Rollback

MOER is a mix‑policy optimization framework that blends expert guidance with self‑exploration in reinforcement learning. It injects a decaying expert chain‑of‑thought prefix into prompts, progressively annealing the prefix length to encourage exploration, and uses a selective rollback mechanism to replace failed completions with the full expert trace, ensuring correctness. The method employs a harmonized mix‑policy objective that optimizes on‑policy tokens with a GRPO loss while supervising both expert prefix and completion tokens with a trust‑region SFT loss, leading to stable training and superior performance on mathematical and general reasoning tasks.

By Xuyang Zhong, Jiuding Yang, Yiwen Guo, Chen Liu
reinforcement-learningfine-tuning
More like this →