arXiv Machine Learning August 28, 2026 By Ahmad Jad Allah, Kazi F. Akhter, Md. Kamrozzaman Bhuiyan, Manar D. Samad

Importance Scoring of Transformer Attention Heads in Learning Tabular Data

Read the original on arXiv Machine Learning →

The paper introduces an importance‑scoring metric for multi‑head transformer attention heads applied to tabular data, a domain where transformers have been less studied. Experiments on 40 diverse tabular datasets show that removing heads with the lowest importance scores has minimal impact on performance, while removing the most important head first causes the largest drop. The study finds that important heads are distributed across layers and vary significantly across different tabular schemas, suggesting that the proposed score can help reduce redundancy and improve transformer efficiency.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv AI

Aug 10

Learning to Predict Middle-Layer Attention in MLLMs for Visual Token Prunin

arXiv:2608. 06411v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) achieve strong performance across diverse vision-language tasks, but their efficiency is limited by the cost of processing numerous visual tokens.

By Yuyao Sun, Tao Deng, Shuang Li, Deqing Wang, Hao Geng, Minjun Yu

llms efficiency multimodal benchmarks

arXiv Computer Vision

Sep 10

Learning to Predict Middle-Layer Attention in MLLMs for Visual Token Pruning

arXiv:2608.06411v2 Announce Type: replace-cross Abstract: Multimodal large language models (MLLMs) achieve strong performance across diverse vision-language tasks, but their efficiency is limited by...

By Yuyao Sun, Tao Deng, Shuang Li, Deqing Wang, Hao Geng, Minjun Yu

llms efficiency multimodal benchmarks

arXiv Computation and Language

Aug 31

Semantic Head Specialization Guides Hybrid ViT Attention for Multimodal LLMs

The paper introduces Semantic Head Specialization (SHS), a phenomenon where Vision Transformer (ViT) attention heads specialize as either object- or background-focused, most evident under full attention. It proposes the SHS-Index to quantify this specialization, demonstrating its ability to distinguish full-attention from chunk-window ViTs and its strong correlation with downstream benchmark performance. Leveraging insights into window interaction, token serialization, and local softmax allocation, the authors design Ariadne Attention, a hybrid attention mechanism that matches full-attention performance on 22 image and video tasks while reducing attention compute by 6.5×.

By Chenhong He, Lei Li, Shicheng Li, Hanglong Lv, Lingpeng Kong, Qi Liu, Tong Yang, Shuhuai Ren

llms multimodal benchmarks

arXiv Machine Learning

Jun 18

Complementary Attention Head Pruning for Efficient Transformers

arXiv:2606. 19150v1 Announce Type: new Abstract: The remarkable success of Transformer-based models in natural language processing stems from architectural scaling, which leads to a large number of parameters and hinders deployment in resource-constrained environments.

By Yaniv Livertovsky, Shahar Somin, Gonen Singer

llms nlp efficiency benchmarks safety

arXiv AI

Jun 17

Top-Theta Attention: Sparsifying Transformers by Compensated Thresholding

arXiv:2502. 08363v3 Announce Type: replace-cross Abstract: We present Top-Theta (Top-$\theta$) Attention, a training-free method for sparsifying transformer attention during inference.

By Konstantin Berestizshevsky, Renzo Andri, Lukas Cavigelli

llms nlp efficiency

arXiv Machine Learning

Jun 29

Prism Transformer: Progressive Head Schedules for Hierarchical Attention Processing

arXiv:2606. 27449v1 Announce Type: new Abstract: Multi-head attention conventionally partitions the hidden dimension equally across all heads at every layer, enforcing an identical representational subspace dimension (dh = dmodel/h) throughout the models depth.

By Shubham Aggarwal

llms benchmarks