arXiv AI
OISD: On-Policy Internal Self-Distillation of Language Models
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
4d ago
SIPO: Unifying Reinforcement Learning with On-Policy Self-Distillation
SIPO (Self‑Instructing Policy Optimization) unifies reinforcement learning with on‑policy self‑distillation by using a contrastive self‑teacher to generate token‑level credit signals. The method samples multiple rollouts per prompt, pairs each with a reference answer and its mistakes, and uses the difference in teacher log‑probabilities to provide dense feedback while still respecting the overall task reward. Experiments on reasoning and code‑generation benchmarks show that SIPO outperforms both RLVR and OPSD baselines without requiring an external teacher or extra generation steps.
arXiv AI
2d ago
Your Language Model is Its Own Critic: Reinforcement Learning with Value Estimation from Actor's Internal States
The paper introduces POISE, a reinforcement learning algorithm that uses a model’s internal states as a value estimator to reduce variance in reinforcement learning with verifiable rewards (RLVR). By employing a lightweight probe that reads internal signals during the forward pass, POISE predicts baselines online and uses a cross‑rollout construction to keep gradients unbiased. Experiments on Qwen3‑4B and OLMo3‑7B‑Instruct‑DPO across six domains show POISE outperforms existing RLVR baselines, offering more stable training and a value model that generalizes across tasks and scales with the policy.
arXiv AI
Jul 21
It Takes 8 Tokens: Weak-to-Strong Off-Policy RL via Auxiliary Branches
arXiv:2607. 16205v1 Announce Type: new Abstract: Reinforcement learning with verifiable rewards has emerged as a standard approach for enhancing reasoning in large language models, which typically optimizes the policy by contrasting multiple self generated rollouts.
arXiv Machine Learning
Jun 16
ExpRL: Exploratory RL for LLM Mid-Training
arXiv:2606. 17024v1 Announce Type: new Abstract: Sparse reward reinforcement learning (RL) has become a standard tool for improving LLM reasoning, but its success depends critically on the coverage present in the base model.
arXiv Machine Learning
Jul 31
Contrastive Reinforced Policy Optimization via Privileged Self-Distillation
arXiv:2607. 28026v1 Announce Type: new Abstract: Recent advances in post-training Large Language Models (LLMs) increasingly rely on Reinforcement Learning with Verifiable Rewards (RLVR) or On-Policy Self-Distillation (OPSD).
arXiv AI
Aug 26
On-policy Distillation with Verifiable Reward
arXiv:2608.24696v1 Announce Type: cross Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) and on-policy distillation (OPD) have become two widely adopted paradigms for post-training lar...