The paper investigates how quantization and pruning affect language models of different sizes. It shows that compression damages specific components—particularly attention heads used for in‑context retrieval—more severely in larger models, and that the network’s ability to compensate for this damage diminishes as model size grows. Consequently, compression techniques that appear benign on small models can become costly on larger ones.
arXiv:2601.06787v2 Announce Type: replace
Abstract: Large Language Models (LLMs) are known to contain significant redundancy, yet a systematic explanation for why certain components, particularly in...
By Jaewon Sok, Jewon Yeom, Seonghyeon Park, Jeongjae Park, Taesup Kim
Why do language models fail on long inputs? The common answer is length. We show that, for retrieval, length is cheap and competition is not. On five open models (0.5B to 3B parameters, four families)...
arXiv:2607. 23054v1 Announce Type: cross Abstract: Multi-head Latent Attention (MLA), introduced in DeepSeek-V2, compresses key-value pairs through a shared low-rank bottleneck (cKV), achieving 81% KV-cache reduction during inference.
By Dhruvil S, Fenil Sojitra, Ravirajsinh Chauhan
The paper investigates how the dimension of attention heads influences long‑context retrieval in language models. Experiments show that while pretraining loss and short‑context tasks are largely unaffected, wider heads significantly improve retrieval performance at fixed model sizes. Ablation studies indicate that this benefit persists even when attention weights are frozen, suggesting that wider heads enhance adaptation of other model components such as embeddings and MLPs.
By Seiya Tokui, Shoichiro Yamaguchi, Sosuke Kobayashi
arXiv:2606. 19150v1 Announce Type: new Abstract: The remarkable success of Transformer-based models in natural language processing stems from architectural scaling, which leads to a large number of parameters and hinders deployment in resource-constrained environments.
By Yaniv Livertovsky, Shahar Somin, Gonen Singer
arXiv:2606. 24970v1 Announce Type: new Abstract: Pruning Large Language Models (LLMs) reduces memory and inference costs by removing parts of the network, producing smaller models that retain most of their accuracy.
By Pietro Tropeano, Maria Maistro, Tuukka Ruotsalo, Christina Lioma
ACE-Compression introduces an ability‑centric evaluation framework for compressed large language models, moving beyond traditional task‑performance metrics. It probes four deployment‑relevant dimensions—procedural reasoning, knowledge grounding, social cognition, and persona fidelity—to determine which core abilities survive compression. The study finds that compression redistributes ability profiles, with coarse signals persisting while fine‑grained abilities become fragile, and that quantization generally retains more abilities than pruning.
By Jiahui Ning, Liang Chen, Yuchen Liang, Yi Zhang, Fenglei Fan
Modern large language models are pretrained on massive datasets, making it difficult to prevent benchmark data from entering their training sets and undermining the reliability of evaluation results....
arXiv:2609.27510v1 Announce Type: cross
Abstract: Modern large language models are pretrained on massive datasets, making it difficult to prevent benchmark data from entering their training sets and...
By Kaifeng Tan, Yudong Li, Linlin Shen
arXiv:2609.37879v1 Announce Type: cross
Abstract: How many tokens from its context does a language model actually use, and what determines that number? We study this question through self-attention....
By Timur Mudarisov, Mikhail Burtsev, Radu State
The paper introduces Declarative Attention (DA), a protocol that lets language models explicitly declare which parts of their context to focus on during generation. By partitioning decoding into full-context, region-specific, and recent-output-only modes, the inference engine can skip large portions of the KV cache, dramatically reducing attended tokens. Experiments on 15 long-context tasks with off-the-shelf models show significant savings (52.0% and 31.1% reductions) with only modest accuracy drops that diminish as model size increases.
By Namgyu Ho, Huzama Ahmad, Woosung Koh, Se-Young Yun, Tal Schuster, Cicero Nogueira dos Santos