arXiv Machine Learning
Zero Sum SVD: Balancing Loss Sensitivity for Low Rank LLM Compression
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv AI
Sep 15
UniRank: Unified Rank Allocation for Low-Rank LLM Compression
arXiv:2606.21847v2 Announce Type: replace-cross Abstract: Low-rank decomposition is a promising compression paradigm for large language models (LLMs), yet its effectiveness hinges on rank budget allo...
arXiv AI
Jul 7
LACE-SVD: Loss-Aware SVD with Cumulative Error Correction for LLM Compression
arXiv:2607. 03057v1 Announce Type: cross Abstract: The rapid growth in the parameter scale of large language models (LLMs) has created a strong demand for efficient compression techniques.
arXiv Machine Learning
Aug 28
LowRankArena: A Standardized Evaluation Platform for SVD-Based LLM Compression
LowRankArena is a standardized evaluation platform for SVD‑based low‑rank compression of large language models, unifying task versions, compression budgets, comparison regimes, and inference measurements. It provides a reproducible pipeline with over 3 TiB of released compressed checkpoints, enabling consistent comparisons across methods. An audit of five representative SVD techniques using LowRankArena shows that prior reported gains are highly conditional, with performance leaders and tiers shifting across backbones and keep ratios, and that nominal low‑rank savings often yield limited end‑to‑end speedups.
arXiv Machine Learning
Jun 8
SigmaScale: LLM Compression with SVD-based Low-Rank Decomposition and Learned Scaling Matrices
arXiv:2606. 07098v1 Announce Type: cross Abstract: We present SigmaScale, a method for learning auxiliary scaling matrices $S$ to aid truncated Singular Value Decomposition (SVD) based Large Language Model (LLM) compression.
arXiv Machine Learning
Jun 5
Activation-Informed Pareto-Guided Low-Rank Compression for Efficient LLM/VLM
arXiv:2510. 05544v2 Announce Type: replace-cross Abstract: Large language models (LLM) and vision-language models (VLM) have achieved state-of-the-art performance, but they impose significant memory and computing challenges in deployment.
arXiv AI
Sep 10
Mind the Approximation: Fisher-Weighted SVD Compression for ViTs
The paper introduces FACTS, a structured Fisher Approximation for compressing Vision Transformers (ViTs) using Fisher-weighted SVD, which enforces token‑local aggregation while preserving within‑token activation‑gradient dependence. It also presents Constrained Rank Search (CoRS) to optimize layer‑wise rank allocation under a fixed FLOP budget. Experiments on ViTs and hybrid architectures show that FACTS improves accuracy‑efficiency trade‑offs, outperforming the strongest SVD baseline by up to +5.8 percentage points on Swin‑B without requiring finetuning.