arXiv AI By Gautam Gupta, Dikshant Kukreja, Unique Kumar, Avni Mittal, Deepanshu Dabas, Kshitij Sah, Erik Cambria, Rajiv Ratn Shah, Zhengkui Wang, Avinash Anand

When Bigger Becomes Brittle: Why Compression Hurts More as Language Models Scale

Read the original on arXiv AI →

The paper investigates how quantization and pruning—common compression techniques—affect language models of different sizes. By isolating a small set of attention heads responsible for in-context retrieval, the authors decompose the loss from ablating these heads into components that are compensated by the rest of the network and components that directly impact the output. Their findings show that while smaller models largely recover from such damage, larger models retain most of the loss because their compensatory mechanisms are less effective, indicating that compression can be far more detrimental as models scale.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Hugging Face Trending Papers
Nov 5

When Bigger Becomes Brittle: Why Compression Hurts More as Language Models Scale

The paper investigates how quantization and pruning affect language models of different sizes. It shows that compression damages specific components—particularly attention heads used for in‑context retrieval—more severely in larger models, and that the network’s ability to compensate for this damage diminishes as model size grows. Consequently, compression techniques that appear benign on small models can become costly on larger ones.

arXiv AI
Oct 26

Head Dimension Matters in Language Models for Long-Context Retrieval

The paper investigates how the dimension of attention heads influences long‑context retrieval in language models. Experiments show that while pretraining loss and short‑context tasks are largely unaffected, wider heads significantly improve retrieval performance at fixed model sizes. Ablation studies indicate that this benefit persists even when attention weights are frozen, suggesting that wider heads enhance adaptation of other model components such as embeddings and MLPs.

By Seiya Tokui, Shoichiro Yamaguchi, Sosuke Kobayashi