← Back to all news
arXiv Machine Learning By Anh Tuan Nguyen, Anna Bair, Zico Kolter

Can Capability Enhancement be Localized?

Read the original on arXiv Machine Learning →

The paper investigates whether removing a small set of attention heads in transformer-based language models can improve task performance, a phenomenon termed Capability Enhancement Localization (CEL). Empirical experiments across various models identify sparse head combinations whose ablation boosts performance without fine-tuning. The authors also provide a theoretical explanation using a simplified attention model, showing that training dynamics can concentrate conflicting task features into specific heads, whose removal selectively benefits one task while harming another.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Computation and Language
3d ago

Putting Transformer Theories to the Test

arXiv:2610.32104v1 Announce Type: cross Abstract: Several formal theories have been developed to characterize the capabilities and limitations of the transformer architecture, each employing differen...

By Awni Altabaa, Tobias Schnabel, Kiran Tomlinson
llms
More like this →
arXiv AI
Oct 16

Language Models Share a Readout Bottleneck in Working Memory

Large language models (LLMs) exhibit a human‑like working‑memory pattern on the N‑back task, with performance dropping as memory load increases and errors mainly on lure items. Two hypotheses—precision limits of softmax attention and interference from competing items—were tested across 15 pretrained LLMs and minimal transformers. Results show that interference, not precision, constrains LLMs’ working‑memory performance, as suppressing non‑target item identities improves sensitivity and larger models benefit more from reducing interference.

By Dongyu Gong
llms
More like this →