Prompt Breadth and Rollout Refresh Interact in On-Policy Distillation

The study investigates how the number of prompts and the strategy of refreshing rollout responses affect on‑policy distillation (OPD). Using a 3×3 experiment with 14,080 trajectories and 110 optimizer updates, the authors find that with ten policy snapshots, eight prompts achieve 24.09% accuracy—nearly matching the 24.51% obtained with 14,080 distinct prompts. However, when responses are frozen at the initial policy, increasing prompt breadth actually reduces accuracy, whereas per‑update refresh raises it, producing a 4.07‑point interaction effect. Comparisons with two teacher models show that periodic models excel in short‑budget accuracy and answer completion, but frozen‑response models surpass them in overall accuracy at a 32K output limit, using 1.7–1.8× more response tokens. whyItMatters":"The findings demonstrate that prompt efficiency in OPD is contingent on both the refresh strategy and the inference budget, informing how to design more effective distillation pipelines."