SVD-Attention: Efficiently Analyzing Attention Blocks in Transformers

SVD-Attention is a post‑hoc diagnostic tool that decomposes each self‑attention head’s interaction matrix into singular modes, scoring them by spectral magnitude and token‑wise activation variance on a small calibration set. The method retains only the modes that the head can amplify and that are actually activated by real data, producing an exact factorization without finetuning. Across seven pretrained models and five tasks, the analysis reveals task‑dependent redundancy patterns and shows that truncating the identified redundant layers incurs minimal performance loss while truncating other layers can cause significant degradation.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

• llms
• computer-vision
• nlp
• fine-tuning
• efficiency