arXiv AI September 10, 2026 By Moritz Thoma, Maximilian Groezinger, Maximilian Forstenh\"ausler, Emad Aghajanzadeh, Ryan Pegoud, Manoj Rohit Vemparala, Pierpaolo Mori, Alexander Frickenstein, Daniel Mueller-Gritschneder, Ulf Schlichtmann

Mind the Approximation: Fisher-Weighted SVD Compression for ViTs

Read the original on arXiv AI →

The paper introduces FACTS, a structured Fisher Approximation for compressing Vision Transformers (ViTs) using Fisher-weighted SVD, which enforces token‑local aggregation while preserving within‑token activation‑gradient dependence. It also presents Constrained Rank Search (CoRS) to optimize layer‑wise rank allocation under a fixed FLOP budget. Experiments on ViTs and hybrid architectures show that FACTS improves accuracy‑efficiency trade‑offs, outperforming the strongest SVD baseline by up to +5.8 percentage points on Swin‑B without requiring finetuning.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv Machine Learning

Jun 2

LASER: Loss-Aware Singular-value Decomposition and Rank Allocation for Efficient Low-Precision Vision-Language Models

arXiv:2606. 00573v1 Announce Type: new Abstract: Vision-language models (VLMs) deliver strong multimodal reasoning capabilities, but their large computational cost and high parameter counts make deployment challenging on resource-constrained devices.

By Haiyu Wang, Yutong Wang, Leshu Li, Yihui Ren, Sai Qian Zhang

llms efficiency multimodal

arXiv AI

Sep 15

UniRank: Unified Rank Allocation for Low-Rank LLM Compression

arXiv:2606.21847v2 Announce Type: replace-cross Abstract: Low-rank decomposition is a promising compression paradigm for large language models (LLMs), yet its effectiveness hinges on rank budget allo...

By Chao Han, Yongjie Du, Junjie Tan, Zihao Xuan

llms fine-tuning efficiency

arXiv AI

Jul 17

Generalized Fisher-Weighted SVD: Scalable Kronecker-Factored Fisher Approximation for Compressing Large Language Models

arXiv:2505. 17974v2 Announce Type: replace-cross Abstract: The Fisher information is a fundamental concept for characterizing the sensitivity of parameters in neural networks.

By Viktoriia Chekalina, Daniil Moskovskiy, Tatiana Matveeva, Andrey Kuznetsov, Evgeny Frolov

llms benchmarks

arXiv Machine Learning

Jun 5

Activation-Informed Pareto-Guided Low-Rank Compression for Efficient LLM/VLM

arXiv:2510. 05544v2 Announce Type: replace-cross Abstract: Large language models (LLM) and vision-language models (VLM) have achieved state-of-the-art performance, but they impose significant memory and computing challenges in deployment.

By Ryan Solgi, Parsa Madinei, Jiayi Tian, Rupak Swaminathan, Jing Liu, Nathan Susanj, Zheng Zhang

llms efficiency multimodal benchmarks

arXiv Machine Learning

Jul 3

BALF: Budgeted Activation-Aware Low-Rank Factorization for Fine-Tuning-Free Model Compression

arXiv:2509. 25136v3 Announce Type: replace Abstract: Activation-aware low-rank factorization techniques yield strong compression results but are generally confined to linear layers, while existing whitening-based theory typically makes an implicit full-rank assumption on activations.

By David Gonz\'alez-Mart\'inez

llms fine-tuning efficiency

arXiv AI

Jul 10

Beyond Attention Scores: SVD-Based Vision Token Pruning for Efficient Vision-Language Models

arXiv:2604. 11530v3 Announce Type: replace-cross Abstract: Vision-Language Models (VLMs) have revolutionized multi-modal learning by jointly processing visual and textual information.

By Yvon Apedo, Martyna Poreba, Michal Szczepanski, Samia Bouchafa

llms efficiency multimodal safety

One email a morning, machine-written

Related stories

LASER: Loss-Aware Singular-value Decomposition and Rank Allocation for Efficient Low-Precision Vision-Language Models

UniRank: Unified Rank Allocation for Low-Rank LLM Compression

Generalized Fisher-Weighted SVD: Scalable Kronecker-Factored Fisher Approximation for Compressing Large Language Models

Activation-Informed Pareto-Guided Low-Rank Compression for Efficient LLM/VLM

BALF: Budgeted Activation-Aware Low-Rank Factorization for Fine-Tuning-Free Model Compression

Beyond Attention Scores: SVD-Based Vision Token Pruning for Efficient Vision-Language Models