arXiv Machine Learning September 23, 2026 By Kasun Dewage, Marianna Pensky, Heranga K. Rathnasekara, Suranadi De Silva

Magnitude Profile Pruning: Calibration-Free Structured Attention Head Removal for Transformer Compression

Read the original on arXiv Machine Learning →

Magnitude Profile Pruning introduces a training‑free, calibration‑free method for removing attention heads in Transformer models by statistically detecting outliers in weight row norms. Heads whose projection weights fall within the bulk of the distribution are pruned, while outlier heads are retained. Across several models, the MP‑G variant achieves superior perplexity at various sparsity levels and yields significant parameter and FLOP reductions without requiring forward passes, calibration data, or gradient computations.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv Machine Learning

Jun 18

Complementary Attention Head Pruning for Efficient Transformers

arXiv:2606. 19150v1 Announce Type: new Abstract: The remarkable success of Transformer-based models in natural language processing stems from architectural scaling, which leads to a large number of parameters and hinders deployment in resource-constrained environments.

By Yaniv Livertovsky, Shahar Somin, Gonen Singer

llms nlp efficiency benchmarks safety

arXiv Computer Vision

Aug 26

HAP: Head-Adaptive Visual Token Pruning via Cross-Modal Alignment

arXiv:2608.23921v1 Announce Type: new Abstract: Recent Vision-Language Models encode high-resolution images into long visual token sequences, incurring prohibitive prefill costs. To compress them, ex...

By Yuanhao Sun, Huawei Ji, Yuan Jin, Cheng Deng, Luoyi Fu, Xinbing Wang

llms efficiency multimodal benchmarks safety

arXiv Machine Learning

Aug 28

Importance Scoring of Transformer Attention Heads in Learning Tabular Data

The paper introduces an importance‑scoring metric for multi‑head transformer attention heads applied to tabular data, a domain where transformers have been less studied. Experiments on 40 diverse tabular datasets show that removing heads with the lowest importance scores has minimal impact on performance, while removing the most important head first causes the largest drop. The study finds that important heads are distributed across layers and vary significantly across different tabular schemas, suggesting that the proposed score can help reduce redundancy and improve transformer efficiency.

By Ahmad Jad Allah, Kazi F. Akhter, Md. Kamrozzaman Bhuiyan, Manar D. Samad

llms computer-vision nlp

arXiv Computation and Language

Sep 22

Garbage Attention in Large Language Models: BOS Sink Heads and Sink-aware Pruning

arXiv:2601.06787v2 Announce Type: replace Abstract: Large Language Models (LLMs) are known to contain significant redundancy, yet a systematic explanation for why certain components, particularly in...

By Jaewon Sok, Jewon Yeom, Seonghyeon Park, Jeongjae Park, Taesup Kim

llms efficiency

arXiv AI

Jul 21

CoCurve: Cross-Module Co-Pruning Curvature for Training-Free Structured LLM Pruning

arXiv:2607. 17568v1 Announce Type: cross Abstract: Structured pruning compresses large language models (LLMs) by removing whole computational units, such as attention heads and feed-forward (FFN) channel groups.

By Zhiren Gong, Zihao Zeng, Zijie Wang, Tiantong Wang, Chau Yuen, Wei Yang Bryan Lim

llms fine-tuning efficiency benchmarks

arXiv AI

Jul 28

TriSP: Tri-Signal Structured Pruning for Large Language Models

arXiv:2607. 22587v1 Announce Type: new Abstract: Large language models (LLMs) achieve strong performance across diverse tasks but their deployment is constrained by the memory and compute cost of their parameters.

By Manel Kara laoua, Soumia Bouyahiaoui, Aicha Boutorh

llms fine-tuning efficiency