arXiv AI
Jul 16
Data-Efficient Adaptation of LLMs via Attention Head Reweighting
arXiv:2607. 13425v1 Announce Type: cross Abstract: Learning effectively from limited data is critical in domains like security where labeled examples are scarce.
Adaptive Head Budgeting for Efficient Multi-Head Attention
arXiv:2604. 22583v2 Announce Type: replace Abstract: Multi-head attention enables Transformers to capture diverse representations, but all attention heads are typically activated for every input, regardless of task complexity.
Related stories
arXiv AI
Jun 17
Top-Theta Attention: Sparsifying Transformers by Compensated Thresholding
arXiv:2502. 08363v3 Announce Type: replace-cross Abstract: We present Top-Theta (Top-$\theta$) Attention, a training-free method for sparsifying transformer attention during inference.
arXiv Computation and Language
Sep 22
Garbage Attention in Large Language Models: BOS Sink Heads and Sink-aware Pruning
arXiv:2601.06787v2 Announce Type: replace Abstract: Large Language Models (LLMs) are known to contain significant redundancy, yet a systematic explanation for why certain components, particularly in...
arXiv Machine Learning
Aug 28
Importance Scoring of Transformer Attention Heads in Learning Tabular Data
The paper introduces an importance‑scoring metric for multi‑head transformer attention heads applied to tabular data, a domain where transformers have been less studied. Experiments on 40 diverse tabular datasets show that removing heads with the lowest importance scores has minimal impact on performance, while removing the most important head first causes the largest drop. The study finds that important heads are distributed across layers and vary significantly across different tabular schemas, suggesting that the proposed score can help reduce redundancy and improve transformer efficiency.
arXiv Machine Learning
Sep 1
LoGo: Token-Level Dynamic Local-Global Attention
arXiv:2608.29539v1 Announce Type: cross Abstract: As context lengths scale, attention increasingly becomes a primary computational bottleneck in large language models. Standard Transformers remain po...
arXiv Machine Learning
Jun 18
Complementary Attention Head Pruning for Efficient Transformers
arXiv:2606. 19150v1 Announce Type: new Abstract: The remarkable success of Transformer-based models in natural language processing stems from architectural scaling, which leads to a large number of parameters and hinders deployment in resource-constrained environments.
arXiv Machine Learning
Jun 29
Prism Transformer: Progressive Head Schedules for Hierarchical Attention Processing
arXiv:2606. 27449v1 Announce Type: new Abstract: Multi-head attention conventionally partitions the hidden dimension equally across all heads at every layer, enforcing an identical representational subspace dimension (dh = dmodel/h) throughout the models depth.
arXiv Computation and Language
Sep 1
Switch Attention: Towards Dynamic and Fine-grained Hybrid Transformers
arXiv:2603.26380v3 Announce Type: replace Abstract: The attention mechanism has been the core component in modern transformer architectures. However, the computation of standard full attention scales...
arXiv Machine Learning
Jun 8
Attention Sink in Transformers: A Survey on Utilization, Interpretation, and Mitigation
arXiv:2604. 10098v2 Announce Type: replace Abstract: As the foundational architecture of modern machine learning, Transformers have driven remarkable progress across diverse AI domains.
arXiv AI
Jun 29
Flexformer: Flexible Linear Transformer with Learnable Attention Kernel
arXiv:2606. 27748v1 Announce Type: cross Abstract: Transformer models rely on attention mechanism to capture long-range dependencies but suffer from quadratic complexity, limiting their scalability to long sequences.
arXiv Computer Vision
Aug 26
HAP: Head-Adaptive Visual Token Pruning via Cross-Modal Alignment
arXiv:2608.23921v1 Announce Type: new Abstract: Recent Vision-Language Models encode high-resolution images into long visual token sequences, incurring prohibitive prefill costs. To compress them, ex...
arXiv AI
Aug 6
Calibrating Transformer Attention via Task-Space Sensitivity Feedback
arXiv:2512. 20661v2 Announce Type: replace Abstract: Transformer-based pre-trained language models (PLMs) excel in text classification but suffer from attention dilution and attention sink effects, forcing models to over-focus on task-irrelevant tokens.