arXiv AI September 11, 2026 By Xiaofei Feng

Decision Shifts, Lost Label Functionality, and an Inconclusive Grounding Audit in Correctness-Gated Multi-Teacher Distillation

The paper investigates correctness‑gated multi‑teacher distillation, comparing a weighted arm to unfiltered distillation across eight experimental arms. While the weighted arm shows modest gains in accuracy (+0.1660) and macro‑F1 (+0.1323) and a reduction in unsafe action rate (−0.4979), it also exhibits lost label functionality, such as zero Refuted recall and over‑assignment of NotEnoughInfo. A subsequent grounding audit was inconclusive, failing to demonstrate a clear improvement in evidence grounding or overall system performance.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv Machine Learning

Jul 30

When Top-K Misses the Decision: Tool-Call Drift in Multi-Teacher On-Policy Distillation

arXiv:2607. 07050v3 Announce Type: replace-cross Abstract: Top-K teacher logits make on-policy distillation tractable, but probability mass is not the same as decision support.

By Jiabin Shen, Guang Chen, Chengjun Mao

efficiency safety

arXiv AI

Aug 3

The Asymmetric Effects of Knowledge Distillation on Bias in Small Language Models

arXiv:2607. 28639v1 Announce Type: cross Abstract: We show that knowledge distillation in small instruction-tuned language models has asymmetric effects on bias.

By Plawan Kumar Rath

llms fine-tuning efficiency safety

arXiv Machine Learning

Aug 18

DUET: Dual-Teacher On-Policy Distillation via Same-Weight Disagreement for Prohibition Compliance

arXiv:2608. 14644v1 Announce Type: new Abstract: Real-world LLM deployments increasingly rely on runtime-injected prohibitions--enterprise policies, PII redlines, tool boundaries--that vary per request and per tenant.

By Zihan Li, Feifei Li, Wenhui Que

llms rag reinforcement-learning fine-tuning efficiency benchmarks safety

arXiv Machine Learning

Sep 22

A Shared Learning Rate Is Not a Neutral Control in Selective On-Policy Distillation

The paper investigates selective on‑policy distillation, where a student model is trained only on token positions chosen by a selector. It demonstrates that the commonly used shared learning rate is not neutral: performance varies significantly with the learning rate for different selectors, leading to inconsistent comparisons. The authors attribute this selector‑rate entanglement to the selection process itself and recommend reporting the full arm‑by‑rate matrix for fair evaluation.

By Chencheng Zhu

fine-tuning efficiency benchmarks

arXiv AI

Aug 6

Not Every Divergence Should Be Suppressed: Counterfactual Recoverability in On-Policy Distillation

arXiv:2608. 04408v1 Announce Type: cross Abstract: On-policy distillation (OPD) supervises student-visited trajectories, yet divergence-based rules cannot determine whether an erroneous prefix remains correctable.

By De Jiang, Zhengyang Zhang, Kehong Yuan, Shaohua Ma

efficiency

arXiv Machine Learning

Jul 14

Student Capacity Moderates Knowledge Distillation Effectiveness: A Systematic Study Across ResNet Teacher-Student Pairs on CIFAR-10

arXiv:2605. 31191v2 Announce Type: replace Abstract: We investigate how teacher-student capacity relationships modulate knowledge distillation (KD) effectiveness in ResNet-based image classification on CIFAR-10.

By Umut Onur Yasar

computer-vision efficiency

One email a morning, machine-written

Related stories

When Top-K Misses the Decision: Tool-Call Drift in Multi-Teacher On-Policy Distillation

The Asymmetric Effects of Knowledge Distillation on Bias in Small Language Models

DUET: Dual-Teacher On-Policy Distillation via Same-Weight Disagreement for Prohibition Compliance

A Shared Learning Rate Is Not a Neutral Control in Selective On-Policy Distillation

Not Every Divergence Should Be Suppressed: Counterfactual Recoverability in On-Policy Distillation

Student Capacity Moderates Knowledge Distillation Effectiveness: A Systematic Study Across ResNet Teacher-Student Pairs on CIFAR-10