The paper introduces Layer-wise Error Modeling Search (LEMS) and KFAC-SVD to improve low-rank SVD-based compression of large language models. LEMS addresses global rank allocation by using a layer-wise error surrogate and an ILP formulation, while KFAC-SVD enhances decomposition quality by employing token-wise statistics to avoid rank collapse. Experiments on Mistral, Qwen3, and Llama 3 show LEMS yields up to 4.8 pp zero‑shot accuracy gains and KFAC‑SVD achieves an average perplexity improvement of 15%.
arXiv:2607. 03057v1 Announce Type: cross Abstract: The rapid growth in the parameter scale of large language models (LLMs) has created a strong demand for efficient compression techniques.
The paper introduces FACTS, a structured Fisher Approximation for compressing Vision Transformers (ViTs) using Fisher-weighted SVD, which enforces token‑local aggregation while preserving within‑token activation‑gradient dependence. It also presents Constrained Rank Search (CoRS) to optimize layer‑wise rank allocation under a fixed FLOP budget. Experiments on ViTs and hybrid architectures show that FACTS improves accuracy‑efficiency trade‑offs, outperforming the strongest SVD baseline by up to +5.8 percentage points on Swin‑B without requiring finetuning.
arXiv:2602.02848v2 Announce Type: replace Abstract: Advances in large language models have driven strong performance across many tasks, but their memory and compute costs still hinder deployment. SVD...
arXiv:2608. 08506v1 Announce Type: new Abstract: Training-free low-rank compression frameworks have been gaining prominence for LLM compression given their effectiveness in reducing model parameter count while maintaining task-level accuracy.
arXiv:2606. 07098v1 Announce Type: cross Abstract: We present SigmaScale, a method for learning auxiliary scaling matrices $S$ to aid truncated Singular Value Decomposition (SVD) based Large Language Model (LLM) compression.
LowRankArena is a standardized evaluation platform for SVD‑based low‑rank compression of large language models, unifying task versions, compression budgets, comparison regimes, and inference measurements. It provides a reproducible pipeline with over 3 TiB of released compressed checkpoints, enabling consistent comparisons across methods. An audit of five representative SVD techniques using LowRankArena shows that prior reported gains are highly conditional, with performance leaders and tiers shifting across backbones and keep ratios, and that nominal low‑rank savings often yield limited end‑to‑end speedups.
arXiv:2606.21847v2 Announce Type: replace-cross Abstract: Low-rank decomposition is a promising compression paradigm for large language models (LLMs), yet its effectiveness hinges on rank budget allo...
arXiv:2606. 00573v1 Announce Type: new Abstract: Vision-language models (VLMs) deliver strong multimodal reasoning capabilities, but their large computational cost and high parameter counts make deployment challenging on resource-constrained devices.
arXiv:2607. 19391v1 Announce Type: cross Abstract: Low-Rank Adaptation is widely used for parameter-efficient fine-tuning, yet existing methods typically assign the same adapter rank to every transformer layer despite their heterogeneous adaptation requirements.
The paper introduces LASER, a loss‑aware low‑rank compression framework for vision‑language models. It derives a curvature‑weighted SVD objective from a second‑order loss approximation and uses Kronecker‑factored Fisher information to guide decomposition toward downstream performance. LASER also proposes a cross‑layer rank allocation strategy based on calibration gradients and extends compression to feed‑forward networks with a hybrid SVD‑quantization scheme, achieving over 2.3× decoding speedup while maintaining accuracy under low‑precision inference.
The paper introduces SoCo, a compression framework that optimizes the singular spectrum of large language models using a learnable diagonal matrix to assign importance scores to SVD components. By progressively refining these scores through a three‑stage training process, SoCo adaptively prunes components while amplifying the importance of remaining ones, achieving a balance between aggressive compression and performance preservation. Experiments on multiple LLMs and benchmarks show that SoCo outperforms state‑of‑the‑art compression methods.
arXiv:2606. 19993v1 Announce Type: new Abstract: We present Activation- and Influence-Aware Ranks (AIR), an SVD-based LLM compression framework that guides each weight matrix's low-rank approximation with a backward-signal influence metric.