arXiv AI
MuSID-KD: Multigranularity Knowledge Distillation for Generative Recommendation with Semantic IDs
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Machine Learning
Oct 27
Knowledge Distillation across Item and Code Spaces for Parallel Generative Recommendation
arXiv:2610.32943v1 Announce Type: cross Abstract: Generative recommendation with semantic IDs represents each item as a sequence of discrete codes derived from its semantic representation and predict...
arXiv AI
Oct 28
REGER: Modeling inter-item Relations with Semantic IDs for Generative Recommendation
REGER is a unified framework that enhances generative recommendation by explicitly modeling inter-item relations through semantic pretraining, history encoding, and SID generation. It pretrains the model to generate semantic neighbors’ IDs, uses combined global and local attention to capture historical context, and introduces a hierarchical preference decomposition over the SID prefix tree. The approach also distills a discriminative teacher’s full conditional distributions to improve next-token predictions, yielding consistent performance gains on real-world datasets.
arXiv Computation and Language
3d ago
DecisionRank: On-Policy Distillation for Decision-Model Rerankers
DecisionRank introduces an on-policy distillation (OPD) method that trains decision models to replicate a generative LLM’s ranking knowledge by converting the student’s document scores into a ranked sequence and using a teacher to provide conditional feedback. The approach combines item‑space OPD with offline ranking supervision, achieving state‑of‑the‑art results on retrieval benchmarks such as TREC DL, BEIR, and BRIGHT. Even with a 1.7B parameter model, DecisionRank outperforms its 27B generative teacher on TREC DL19–23, and at 8B it surpasses the generative listwise counterpart while running 24.5× faster on TREC DL and 16.7× faster on BRIGHT.
Hugging Face Trending Papers
Oct 27
REGER: Modeling inter-item Relations with Semantic IDs for Generative Recommendation
REGER is a framework that enhances generative recommendation by explicitly modeling inter-item relations through semantic pretraining, refined history encoding, and SID generation. It pretrains the model to generate semantic neighbors’ IDs, uses a combination of global and local attention to capture context, and introduces a hierarchical preference decomposition that allows exact factorization of collaborative preferences. The method also distills a discriminative teacher’s full conditional distributions to improve next-token predictions, yielding consistent performance gains on real-world datasets.
arXiv AI
Jun 10
Reasoning over Semantic IDs Enhances Generative Recommendation
arXiv:2603. 23183v2 Announce Type: replace-cross Abstract: Recent advances in generative recommendation have leveraged pretrained LLMs by formulating sequential recommendation as autoregressive generation over a unified token space comprising language tokens and itemic identifiers, where each item is represented by a compact sequence of discrete tokens, namely Semantic IDs (SIDs).
arXiv Machine Learning
Oct 22
CSD: Common-Support Distillation for Cross-Tokenizer Knowledge Transfer
The paper introduces Common‑Support Distillation (CSD), a logits‑based framework for transferring knowledge across different tokenizers by constructing a shared support between teacher and student distributions. CSD handles both one‑to‑one alignments using explicit common‑token pairs and other alignments by creating a virtual support via the autoregressive chain rule, while a residual event ensures normalization. Experiments on Llama‑3.2‑1B, Gemma3‑1B‑PT, Qwen, and Phi models show that CSD consistently outperforms existing cross‑tokenizer methods, achieving up to 1.8‑point gains over the best alternative and surpassing both continued pretraining and same‑tokenizer distillation.