← Back to all news
arXiv Machine Learning By Mingyu Yang, Sharath Turuvekere Sreenivas, Adithyakrishna Venkatesh Hanasoge, Nima Tajbakhsh, Ashwath Aithal, Pavlo Molchanov

CSD: Common-Support Distillation for Cross-Tokenizer Knowledge Transfer

Read the original on arXiv Machine Learning →

The paper introduces Common‑Support Distillation (CSD), a logits‑based framework for transferring knowledge across different tokenizers by constructing a shared support between teacher and student distributions. CSD handles both one‑to‑one alignments using explicit common‑token pairs and other alignments by creating a virtual support via the autoregressive chain rule, while a residual event ensures normalization. Experiments on Llama‑3.2‑1B, Gemma3‑1B‑PT, Qwen, and Phi models show that CSD consistently outperforms existing cross‑tokenizer methods, achieving up to 1.8‑point gains over the best alternative and surpassing both continued pretraining and same‑tokenizer distillation.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Machine Learning
Oct 12

CAST: Credit Alignment for Cross-Tokenizer On-Policy Distillation

CAST (Chunk-aware credit Assignment with Selective Token Anchoring) is a new framework for cross-tokenizer on-policy distillation that tackles two key problems: credit alignment and unaligned token positions. It redistributes chunk-level advantage signals across student tokens using predictive uncertainty and teacher-signal smoothness, while anchoring unaligned tokens to the initial policy via a KL penalty. Experiments on six mathematics, science, and code benchmarks show that CAST improves average performance over the strongest baseline by up to 2.45 points in cross-tokenizer transfer scenarios.

By Jingru Yang, Jinhang Wu, Minjing Shi, Shengfeng He, Xuyan Li, Man Tan, Biye Li
llmsnlpefficiencybenchmarkssafety
More like this →
arXiv AI
Jul 23

When Does Knowledge Distillation Hurt? Reliability-Aware Distillation for Low-Resource Language Summarization

arXiv:2607. 19956v1 Announce Type: cross Abstract: Knowledge distillation (KD) is a standard approach for compressing sequence-to-sequence models, but its per-sample effects are rarely examined.

By Dipto Sumit, Ankan Kumar Roy Srizon, Sadia Khair Rodela, Atia Haque Asha, Mourchona Afrin, Niloy Farhan, Farig Sadeque
nlpfine-tuningefficiencybenchmarkssafety
More like this →