arXiv AI
Capable Teacher, Myopic Scorer: Reward-Free Teacher Repair for Agentic On-Policy Distillation
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Oct 1
PivotOPD: Learning to Recover from Pivotal Mistakes in Multi-Turn Agents
arXiv:2609. 40285v1 Announce Type: new Abstract: On-policy distillation (OPD) is a promising approach for training language agents, providing dense teacher supervision on student-generated trajectories.
arXiv AI
Sep 30
Guide, Then Let Go: Gap-Adaptive Teacher Scheduling for Sparse-Reward Agentic RL
arXiv:2609.37898v1 Announce Type: new Abstract: Reinforcement learning for long-horizon agents typically relies on sparse outcome-based rewards. This leads to a severe cold-start problem, as early-st...
arXiv Computation and Language
Oct 9
Residual Advantage: Student-Relative Teacher Guidance for RL with Verifiable Rewards
The paper introduces Residual Advantage (RA), a new guidance method for reinforcement learning with verifiable rewards (RLVR) that treats the teacher–student probability residual as a bounded one‑step reward, subtracts the student’s state value to form an advantage, and centers it within each response. RA preserves the verifier’s mean label while redistributing credit among response steps, and its extension CoRA updates the teacher model using verifier advantages to adapt guidance to the student’s attempts. Experiments with Qwen3 models show that RA combined with GRPO or REINFORCE++ consistently outperforms teacher‑only on‑policy distillation across three mathematical benchmarks, improving macro Avg@8 and Pass@8 scores, with CoRA providing additional gains.
arXiv Machine Learning
Sep 15
Know When to Stop, Where to Restart: Accelerating Multi-Turn Agentic On-Policy Distillation
arXiv:2609.14636v1 Announce Type: new Abstract: On-policy distillation (OPD) has become a standard approach for transferring capabilities from large teachers to compact students. Its cost, however, i...
arXiv AI
Oct 12
A Privileged Teacher That Only Scores Cannot Search: Critique-Conditioned On-Policy Self-Distillation for Reasoning
The paper examines on‑policy self‑distillation, where a teacher scores a student’s rollout to provide dense token‑level credit. Experiments show that the raw scoring signals neither localize errors nor effectively penalize incorrect tokens, because locating errors is a search problem that the teacher is not explicitly asked to perform. Introducing a critique‑conditioned approach—first generating a critique of the rollout and then conditioning the teacher on it—produces signals that accurately localize errors, correct erroneous spans, and guide the policy toward backtracking tokens, improving performance on math problems and in cold‑start GRPO settings.
arXiv Machine Learning
Sep 30
Graph-Conditioned On-Policy Agent Distillation from Off-the-Shelf Teachers
arXiv:2609.37522v1 Announce Type: new Abstract: On-policy distillation (OPD) trains compact language agents with teacher feedback on student-generated trajectories. In multi-turn tasks, compounding e...