arXiv Machine Learning June 19, 2026 By Nico Harder, Daniel Becking, Karsten Mueller, Wojciech Samek

Activation- and Influence-Aware Ranks (AIR): Function-Preserving SVD Compression for LLMs

Read the original on arXiv Machine Learning →

arXiv:2606. 19993v1 Announce Type: new Abstract: We present Activation- and Influence-Aware Ranks (AIR), an SVD-based LLM compression framework that guides each weight matrix's low-rank approximation with a backward-signal influence metric.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv AI

4d ago

Efficient Pre-Training of LLMs through Truncated SVD Representations

arXiv:2605.28573v2 Announce Type: replace-cross Abstract: LLM pretraining is extremely costly; therefore, parameter-efficient LLM architectures have recently emerged as a compelling research directio...

By Kaivan Kamali, Kajetan Schweighofer, Hormoz Shahrzad, Olivier Francon, Babak Hodjat, Risto Miikkulainen

llms

arXiv AI

Jul 7

LACE-SVD: Loss-Aware SVD with Cumulative Error Correction for LLM Compression

arXiv:2607. 03057v1 Announce Type: cross Abstract: The rapid growth in the parameter scale of large language models (LLMs) has created a strong demand for efficient compression techniques.

By Zhuowen Liu, Longkun Hao, Shiyu Feng, Xiaowen Chang, Ruiqun Li, Changqun Li

llms efficiency

arXiv Machine Learning

Jun 8

SigmaScale: LLM Compression with SVD-based Low-Rank Decomposition and Learned Scaling Matrices

arXiv:2606. 07098v1 Announce Type: cross Abstract: We present SigmaScale, a method for learning auxiliary scaling matrices $S$ to aid truncated Singular Value Decomposition (SVD) based Large Language Model (LLM) compression.

By Ernests Lavrinovics, Marco Letizia, Roy Janco, Shai Segal, Johannes Bjerva, Maurizio Pierini

llms benchmarks

arXiv Machine Learning

4d ago

Zero Sum SVD: Balancing Loss Sensitivity for Low Rank LLM Compression

arXiv:2602.02848v2 Announce Type: replace Abstract: Advances in large language models have driven strong performance across many tasks, but their memory and compute costs still hinder deployment. SVD...

By Ali Abbasi, Chayne Thrash, Haoran Qin, Shansita Sharma, Sepehr Seifi, Soheil Kolouri

llms benchmarks

arXiv AI

Aug 11

Understanding Calibration and Truncation Error Propagation in Training-Free Low-Rank Compression for LLMs

arXiv:2608. 08506v1 Announce Type: new Abstract: Training-free low-rank compression frameworks have been gaining prominence for LLM compression given their effectiveness in reducing model parameter count while maintaining task-level accuracy.

By Mohanad Odema, Gabrielle De Micheli, Dayin Gou, Nilesh Malpeddi, Prathamesh Vaste, Jacob Song

llms benchmarks

arXiv AI

Sep 10

Mind the Approximation: Fisher-Weighted SVD Compression for ViTs

The paper introduces FACTS, a structured Fisher Approximation for compressing Vision Transformers (ViTs) using Fisher-weighted SVD, which enforces token‑local aggregation while preserving within‑token activation‑gradient dependence. It also presents Constrained Rank Search (CoRS) to optimize layer‑wise rank allocation under a fixed FLOP budget. Experiments on ViTs and hybrid architectures show that FACTS improves accuracy‑efficiency trade‑offs, outperforming the strongest SVD baseline by up to +5.8 percentage points on Swin‑B without requiring finetuning.

By Moritz Thoma, Maximilian Groezinger, Maximilian Forstenh\"ausler, Emad Aghajanzadeh, Ryan Pegoud, Manoj Rohit Vemparala, Pierpaolo Mori, Alexander Frickenstein, Daniel Mueller-Gritschneder, Ulf Schlichtmann

llms fine-tuning efficiency

One email a morning, machine-written

Related stories

Efficient Pre-Training of LLMs through Truncated SVD Representations

LACE-SVD: Loss-Aware SVD with Cumulative Error Correction for LLM Compression

SigmaScale: LLM Compression with SVD-based Low-Rank Decomposition and Learned Scaling Matrices

Zero Sum SVD: Balancing Loss Sensitivity for Low Rank LLM Compression

Understanding Calibration and Truncation Error Propagation in Training-Free Low-Rank Compression for LLMs

Mind the Approximation: Fisher-Weighted SVD Compression for ViTs