Branching On-Policy Distillation

Branching On-Policy Distillation (BranchOPD) enhances on‑policy distillation by selecting informative decision points based on student uncertainty and teacher‑student disagreement, resampling continuations from shared prefixes, and applying token‑level teacher supervision to both main trajectories and branches. This approach reduces student‑generated token costs by about 19% on ALFWorld and 31% on WebShop compared to seven independent trajectories, and introduces protocol‑boundary repair (PBR) to mitigate repetition under protocol constraints. BranchOPD outperforms Vanilla OPD and Multi‑Rollout OPD across benchmarks, achieving significant gains in mathematics accuracy and ALFWorld success rates, with mechanistic analyses showing that branching at uncertain decisions expands behavior coverage.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

• agents
• efficiency
• benchmarks