A Privileged Teacher That Only Scores Cannot Search: Critique-Conditioned On-Policy Self-Distillation for Reasoning

The paper examines on‑policy self‑distillation, where a teacher scores a student’s rollout to provide dense token‑level credit. Experiments show that the raw scoring signals neither localize errors nor effectively penalize incorrect tokens, because locating errors is a search problem that the teacher is not explicitly asked to perform. Introducing a critique‑conditioned approach—first generating a critique of the rollout and then conditioning the teacher on it—produces signals that accurately localize errors, correct erroneous spans, and guide the policy toward backtracking tokens, improving performance on math problems and in cold‑start GRPO settings.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

• reinforcement-learning
• efficiency