arXiv AI

When Bigger Becomes Brittle: Why Compression Hurts More as Language Models Scale

The paper investigates how quantization and pruning—common compression techniques—affect language models of different sizes. By isolating a small set of attention heads responsible for in-context retrieval, the authors decompose the loss from ablating these heads into components that are compensated by the rest of the network and components that directly impact the output. Their findings show that while smaller models largely recover from such damage, larger models retain most of the loss because their compensatory mechanisms are less effective, indicating that compression can be far more detrimental as models scale.

Hugging Face Trending Papers
Nov 5

When Bigger Becomes Brittle: Why Compression Hurts More as Language Models Scale

The paper investigates how quantization and pruning affect language models of different sizes. It shows that compression damages specific components—particularly attention heads used for in‑context retrieval—more severely in larger models, and that the network’s ability to compensate for this damage diminishes as model size grows. Consequently, compression techniques that appear benign on small models can become costly on larger ones.

arXiv AI
Oct 26

Head Dimension Matters in Language Models for Long-Context Retrieval

The paper investigates how the dimension of attention heads influences long‑context retrieval in language models. Experiments show that while pretraining loss and short‑context tasks are largely unaffected, wider heads significantly improve retrieval performance at fixed model sizes. Ablation studies indicate that this benefit persists even when attention weights are frozen, suggesting that wider heads enhance adaptation of other model components such as embeddings and MLPs.

By Seiya Tokui, Shoichiro Yamaguchi, Sosuke Kobayashi
arXiv AI
Oct 21

ACE-Compression: An Ability-Centric Evaluation of Compressed Large Language Models

ACE-Compression introduces an ability‑centric evaluation framework for compressed large language models, moving beyond traditional task‑performance metrics. It probes four deployment‑relevant dimensions—procedural reasoning, knowledge grounding, social cognition, and persona fidelity—to determine which core abilities survive compression. The study finds that compression redistributes ability profiles, with coarse signals persisting while fine‑grained abilities become fragile, and that quantization generally retains more abilities than pruning.

By Jiahui Ning, Liang Chen, Yuchen Liang, Yi Zhang, Fenglei Fan
arXiv AI
Sep 3

Language Models Can Control Their Own Attention

The paper introduces Declarative Attention (DA), a protocol that lets language models explicitly declare which parts of their context to focus on during generation. By partitioning decoding into full-context, region-specific, and recent-output-only modes, the inference engine can skip large portions of the KV cache, dramatically reducing attended tokens. Experiments on 15 long-context tasks with off-the-shelf models show significant savings (52.0% and 31.1% reductions) with only modest accuracy drops that diminish as model size increases.

By Namgyu Ho, Huzama Ahmad, Woosung Koh, Se-Young Yun, Tal Schuster, Cicero Nogueira dos Santos