arXiv AI
Linear Algebra Foundations of Efficient Attention: A Phase Reversal in Rank Collapse Under SVD Compression
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
Hugging Face Trending Papers
Sep 30
Learning Functional Subspaces for Neural Network Compression
Modern transformers pair impressive capabilities with substantial memory and compute demands. Low-rank weight factorization reduces both while keeping the matrices dense, and thus efficient on standar...
arXiv Computation and Language
Oct 1
Learning Functional Subspaces for Neural Network Compression
arXiv:2609.40127v1 Announce Type: cross Abstract: Modern transformers pair impressive capabilities with substantial memory and compute demands. Low-rank weight factorization reduces both while keepin...
arXiv AI
6d ago
SVD-Attention: Efficiently Analyzing Attention Blocks in Transformers
SVD-Attention is a post‑hoc diagnostic tool that decomposes each self‑attention head’s interaction matrix into singular modes, scoring them by spectral magnitude and token‑wise activation variance on a small calibration set. The method retains only the modes that the head can amplify and that are actually activated by real data, producing an exact factorization without finetuning. Across seven pretrained models and five tasks, the analysis reveals task‑dependent redundancy patterns and shows that truncating the identified redundant layers incurs minimal performance loss while truncating other layers can cause significant degradation.
arXiv Machine Learning
Aug 11
A Mechanistic Diagnostic of Rank Collapse in Post-Norm Decoder Transformers
arXiv:2608. 09417v1 Announce Type: new Abstract: Deep decoder-only Transformers often replace the original Post-Norm architecture with Pre-Norm variants because Post-Norm training is highly sensitive to warmup and learning rate under conventional initialization schemes.
arXiv AI
Sep 10
Attention-Weighted Value Projection for KV-Cache Compression
arXiv:2604.11501v2 Announce Type: replace-cross Abstract: Rank reduction discards dimensions; quantization keeps them at lower precision. Comparing the two requires a choice of what compression shoul...
arXiv Computer Vision
Oct 22
Training-free Acceleration for Blind Compressed Image Restoration via Activation-Calibrated Singular Value Decomposition
The paper proposes a training‑free acceleration technique for transformer‑based blind compressed image restoration models. By profiling computation hotspots and channel‑wise activation distributions, the authors identify recurrent projection operators that can be replaced with low‑rank approximations. They introduce Activation‑Calibrated Singular Value Decomposition (ASVD), which derives low‑rank factors offline from pretrained weights and a few calibration images, achieving significant parameter and computation reductions while improving PSNR‑B over a TSVD baseline.