The Distillation Game: Adaptive Evaluations & Efficient Defenses
Read the original on arXiv AI →The paper introduces a minimax framework to analyze the trade-off between model utility and vulnerability to distillation attacks, featuring an adaptive evaluation rule that reweights high-value examples and a teacher-side defense that suppresses outputs most useful for imitation. It proposes a cheap, forward-pass-only defense called Product-of-Experts (PoE) that combines the teacher with a proxy student during generation. Empirical results show a significant passive–adaptive gap, with adaptive students recovering more capability than passive evaluation suggests, narrowing the robustness gap between expensive defenses and PoE while keeping PoE cheaper and preserving higher-quality reasoning traces.
Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.