arXiv AI
UniRank: Unified Rank Allocation for Low-Rank LLM Compression
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Machine Learning
Sep 30
Zero Sum SVD: Balancing Loss Sensitivity for Low Rank LLM Compression
arXiv:2602.02848v2 Announce Type: replace Abstract: Advances in large language models have driven strong performance across many tasks, but their memory and compute costs still hinder deployment. SVD...
arXiv Computation and Language
Oct 19
Optimizing Singular Spectrum for Large Language Model Compression
The paper introduces SoCo, a compression framework that optimizes the singular spectrum of large language models using a learnable diagonal matrix to assign importance scores to SVD components. By progressively refining these scores through a three‑stage training process, SoCo adaptively prunes components while amplifying the importance of remaining ones, achieving a balance between aggressive compression and performance preservation. Experiments on multiple LLMs and benchmarks show that SoCo outperforms state‑of‑the‑art compression methods.
arXiv AI
Sep 10
Mind the Approximation: Fisher-Weighted SVD Compression for ViTs
The paper introduces FACTS, a structured Fisher Approximation for compressing Vision Transformers (ViTs) using Fisher-weighted SVD, which enforces token‑local aggregation while preserving within‑token activation‑gradient dependence. It also presents Constrained Rank Search (CoRS) to optimize layer‑wise rank allocation under a fixed FLOP budget. Experiments on ViTs and hybrid architectures show that FACTS improves accuracy‑efficiency trade‑offs, outperforming the strongest SVD baseline by up to +5.8 percentage points on Swin‑B without requiring finetuning.
arXiv Machine Learning
Jun 8
SigmaScale: LLM Compression with SVD-based Low-Rank Decomposition and Learned Scaling Matrices
arXiv:2606. 07098v1 Announce Type: cross Abstract: We present SigmaScale, a method for learning auxiliary scaling matrices $S$ to aid truncated Singular Value Decomposition (SVD) based Large Language Model (LLM) compression.
arXiv AI
Jul 22
Compressing What Matters: Neuron Importance Meets Data-Aware Low Rank Approximation for Language Model Compression
arXiv:2607. 18284v1 Announce Type: cross Abstract: To excel at their domain large language models are comprised of billions of parameters.
arXiv AI
Jul 21
Sparsity-Aware Low-Rank Representation for Efficient Fine-Tuning of Large Language Models
arXiv:2601. 16991v3 Announce Type: replace-cross Abstract: Adapting large pre-trained language models to downstream tasks often entails fine-tuning millions of parameters or deploying costly dense weight updates, which hinders their use in resource-constrained environments.