Modern transformers pair impressive capabilities with substantial memory and compute demands. Low-rank weight factorization reduces both while keeping the matrices dense, and thus efficient on standar...
arXiv:2609.40127v1 Announce Type: cross Abstract: Modern transformers pair impressive capabilities with substantial memory and compute demands. Low-rank weight factorization reduces both while keepin...
SVD-Attention is a post‑hoc diagnostic tool that decomposes each self‑attention head’s interaction matrix into singular modes, scoring them by spectral magnitude and token‑wise activation variance on a small calibration set. The method retains only the modes that the head can amplify and that are actually activated by real data, producing an exact factorization without finetuning. Across seven pretrained models and five tasks, the analysis reveals task‑dependent redundancy patterns and shows that truncating the identified redundant layers incurs minimal performance loss while truncating other layers can cause significant degradation.
arXiv:2608. 09417v1 Announce Type: new Abstract: Deep decoder-only Transformers often replace the original Post-Norm architecture with Pre-Norm variants because Post-Norm training is highly sensitive to warmup and learning rate under conventional initialization schemes.
arXiv:2604.11501v2 Announce Type: replace-cross Abstract: Rank reduction discards dimensions; quantization keeps them at lower precision. Comparing the two requires a choice of what compression shoul...
The paper proposes a training‑free acceleration technique for transformer‑based blind compressed image restoration models. By profiling computation hotspots and channel‑wise activation distributions, the authors identify recurrent projection operators that can be replaced with low‑rank approximations. They introduce Activation‑Calibrated Singular Value Decomposition (ASVD), which derives low‑rank factors offline from pretrained weights and a few calibration images, achieving significant parameter and computation reductions while improving PSNR‑B over a TSVD baseline.
The paper investigates the relationship between rank compression of weight matrices and the phenomenon of grokking in neural networks. Using a mean‑field MLP and a one‑layer transformer trained on modular arithmetic, the authors find that rank compression occurs after the onset of grokking, not during memorization, and that the effective rank drops sharply around 900 steps following the grokking onset. They also observe a subspace crystallization effect, where the number of singular directions needed to preserve test accuracy shrinks dramatically during the transition, and demonstrate that artificially imposing low rank does not accelerate grokking, indicating that compression is a signature rather than a cause of the phenomenon.
arXiv:2610.06927v1 Announce Type: cross Abstract: The key-value (KV) cache of autoregressive transformers grows linearly with context length and dominates memory at long context. Most training-free r...
arXiv:2608. 09417v2 Announce Type: replace Abstract: Deep decoder-only Transformers often replace the original Post-Norm architecture with Pre-Norm variants because Post-Norm training is highly sensitive to warmup and learning rate under conventional initialization schemes.
arXiv:2607. 18284v1 Announce Type: cross Abstract: To excel at their domain large language models are comprised of billions of parameters.
arXiv:2607. 14018v1 Announce Type: cross Abstract: We investigate how each component of the Transformer feedforward block architecture design determines how much rank survives across depth at initialization.
arXiv:2607. 12550v1 Announce Type: new Abstract: The key-value (KV) cache has become the dominant memory cost of transformer inference.