arXiv:2610.07032v1 Announce Type: cross Abstract: Large-scale pretrained transformer models have achieved state-of-the-art performance across diverse machine translation tasks, including multilingual...
arXiv:2606. 31796v1 Announce Type: cross Abstract: We study three complementary techniques for training compute-efficient language models.
arXiv:2603. 01875v3 Announce Type: replace-cross Abstract: Knowledge distillation (KD) is an essential technique to compress large language models (LLMs) into smaller ones.
arXiv:2412. 01282v2 Announce Type: replace-cross Abstract: Vision-Language Models (VLMs) bring powerful understanding and reasoning capabilities to multimodal tasks.
The paper introduces ResMem, a method that transforms retrieved knowledge into a parametric memory for large language models. By using the LLM as an offline teacher that reads retrieved text, ResMem learns an additive logit residual, avoiding large token storage and improving question‑answering performance. Experiments on Wikipedia and five QA datasets show significant storage savings and higher token F1 scores compared to k‑NN baselines.
arXiv:2510. 20535v2 Announce Type: replace-cross Abstract: Recent techniques such as retrieval-augmented generation or chain-of-thought reasoning have led to longer contexts and increased inference costs.
arXiv:2505.12781v5 Announce Type: replace-cross Abstract: Training high-performing Small Language Models (SLMs) remains costly, even with knowledge distillation and pruning from larger teacher models...
arXiv:2609.27233v1 Announce Type: new Abstract: Transformer language models process sequences token by token in an autoregressive manner, making growing contexts increasingly expensive. Yet many adja...
Transformer language models process sequences token by token in an autoregressive manner, making growing contexts increasingly expensive. Yet many adjacent token spans are highly predictable or freque...
arXiv:2610.33006v1 Announce Type: new Abstract: Transformer-based large language models (LLMs) impose substantial computation and memory costs as context sizes increase, motivating methods that compr...
arXiv:2603. 26556v2 Announce Type: replace-cross Abstract: Converting a pretrained Transformer into a more efficient hybrid model through distillation offers a promising approach to reducing inference costs.
The paper introduces Common‑Support Distillation (CSD), a logits‑based framework for transferring knowledge across different tokenizers by constructing a shared support between teacher and student distributions. CSD handles both one‑to‑one alignments using explicit common‑token pairs and other alignments by creating a virtual support via the autoregressive chain rule, while a residual event ensures normalization. Experiments on Llama‑3.2‑1B, Gemma3‑1B‑PT, Qwen, and Phi models show that CSD consistently outperforms existing cross‑tokenizer methods, achieving up to 1.8‑point gains over the best alternative and surpassing both continued pretraining and same‑tokenizer distillation.