The paper investigates whether removing a small set of attention heads in transformer-based language models can improve task performance, a phenomenon termed Capability Enhancement Localization (CEL). Empirical experiments across various models identify sparse head combinations whose ablation boosts performance without fine-tuning. The authors also provide a theoretical explanation using a simplified attention model, showing that training dynamics can concentrate conflicting task features into specific heads, whose removal selectively benefits one task while harming another.
arXiv:2610.32104v1 Announce Type: cross Abstract: Several formal theories have been developed to characterize the capabilities and limitations of the transformer architecture, each employing differen...
arXiv:2607. 14111v1 Announce Type: cross Abstract: Can small language models detect and report on perturbations their own internal activations?
Large language models (LLMs) exhibit a human‑like working‑memory pattern on the N‑back task, with performance dropping as memory load increases and errors mainly on lure items. Two hypotheses—precision limits of softmax attention and interference from competing items—were tested across 15 pretrained LLMs and minimal transformers. Results show that interference, not precision, constrains LLMs’ working‑memory performance, as suppressing non‑target item identities improves sensitivity and larger models benefit more from reducing interference.
arXiv:2607. 25907v1 Announce Type: cross Abstract: Activation steering controls model behavior by editing internal activations at inference time.
arXiv:2610.10526v1 Announce Type: cross Abstract: Vision-language-action models (VLAs) are strikingly sensitive to instruction phrasing and do not inherit the language robustness of the vision-langua...
arXiv:2409. 02228v2 Announce Type: replace Abstract: When language models (LMs) are trained to forget (or "unlearn'') a skill, how precisely does their behavior change?
The paper introduces Declarative Attention (DA), a protocol that lets language models explicitly declare which parts of their context to focus on during generation. By partitioning decoding into full-context, region-specific, and recent-output-only modes, the inference engine can skip large portions of the KV cache, dramatically reducing attended tokens. Experiments on 15 long-context tasks with off-the-shelf models show significant savings (52.0% and 31.1% reductions) with only modest accuracy drops that diminish as model size increases.
arXiv:2608. 02830v1 Announce Type: cross Abstract: Many-shot in-context learning (ICL) lets vision-language models (VLMs) adapt from image--label demonstrations without weight updates, and is widely assumed to improve as more demonstrations are supplied.
arXiv:2604. 09670v2 Announce Type: replace-cross Abstract: Intelligent systems must maintain and manipulate task-relevant information online to adapt to dynamic environments and changing goals.
arXiv:2610.27724v1 Announce Type: new Abstract: Dense linear probes assess task readability from language model (LM) representations, but do not directly reveal how many features suffice for predicti...
arXiv:2608. 20274v1 Announce Type: new Abstract: Large language model (LLM) agents can induce skills from completed tasks and reuse them later to grow more capable with experience.
arXiv:2609.00756v1 Announce Type: new Abstract: The Mutual Reinforcement Effect (MRE) asks whether a fine, span-level and a coarse, document-level task help each other when one model handles both. We...