arXiv Computation and Language October 19, 2026 By Dengjie Li, Tiancheng Shen, Yao Zhou, Bernard Ghanem, Yibo Yang, Ming-Hsuan Yang

Optimizing Singular Spectrum for Large Language Model Compression

Read the original on arXiv Computation and Language →

The paper introduces SoCo, a compression framework that optimizes the singular spectrum of large language models using a learnable diagonal matrix to assign importance scores to SVD components. By progressively refining these scores through a three‑stage training process, SoCo adaptively prunes components while amplifying the importance of remaining ones, achieving a balance between aggressive compression and performance preservation. Experiments on multiple LLMs and benchmarks show that SoCo outperforms state‑of‑the‑art compression methods.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

arXiv Machine Learning

Jun 8

SigmaScale: LLM Compression with SVD-based Low-Rank Decomposition and Learned Scaling Matrices

arXiv:2606. 07098v1 Announce Type: cross Abstract: We present SigmaScale, a method for learning auxiliary scaling matrices $S$ to aid truncated Singular Value Decomposition (SVD) based Large Language Model (LLM) compression.

By Ernests Lavrinovics, Marco Letizia, Roy Janco, Shai Segal, Johannes Bjerva, Maurizio Pierini

llms benchmarks

arXiv AI

Sep 15

UniRank: Unified Rank Allocation for Low-Rank LLM Compression

arXiv:2606.21847v2 Announce Type: replace-cross Abstract: Low-rank decomposition is a promising compression paradigm for large language models (LLMs), yet its effectiveness hinges on rank budget allo...

By Chao Han, Yongjie Du, Junjie Tan, Zihao Xuan

llms fine-tuning efficiency

arXiv AI

Jul 22

Compressing What Matters: Neuron Importance Meets Data-Aware Low Rank Approximation for Language Model Compression

arXiv:2607. 18284v1 Announce Type: cross Abstract: To excel at their domain large language models are comprised of billions of parameters.

By Athanasios Ntovas, Alexandros Doumanoglou, Petros Drakoulis, Dimitris Zarpalas

llms efficiency benchmarks

arXiv Machine Learning

Sep 30

Zero Sum SVD: Balancing Loss Sensitivity for Low Rank LLM Compression

arXiv:2602.02848v2 Announce Type: replace Abstract: Advances in large language models have driven strong performance across many tasks, but their memory and compute costs still hinder deployment. SVD...

By Ali Abbasi, Chayne Thrash, Haoran Qin, Shansita Sharma, Sepehr Seifi, Soheil Kolouri

llms benchmarks

arXiv AI

Jul 17

Generalized Fisher-Weighted SVD: Scalable Kronecker-Factored Fisher Approximation for Compressing Large Language Models

arXiv:2505. 17974v2 Announce Type: replace-cross Abstract: The Fisher information is a fundamental concept for characterizing the sensitivity of parameters in neural networks.

By Viktoriia Chekalina, Daniil Moskovskiy, Tatiana Matveeva, Andrey Kuznetsov, Evgeny Frolov

llms benchmarks

arXiv AI

Sep 2

Residual Sparsification via Output Importance for Compressing Mixture-of-Experts LLMs

Residual sparsification via output importance (PARSER) is a new compression technique for mixture-of-experts large language models that shifts the compression objective from minimizing isolated matrix errors to preserving the expert output error. By introducing output importance, PARSER measures each residual’s contribution to the final expert output and compresses accordingly. Experiments show that PARSER reduces the accuracy gap to the uncompressed model by 1.41× on Qwen and 1.44× on DeepSeek while achieving the same peak memory reduction.

By Seungwoo Jung, Dohyeok Kwon, Seungmin Cha, Junseok Lee, Yeonho Yoo, Chuck Yoo, Gyeongsik Yang

llms efficiency

One email a morning, machine-written

Related stories

SigmaScale: LLM Compression with SVD-based Low-Rank Decomposition and Learned Scaling Matrices

UniRank: Unified Rank Allocation for Low-Rank LLM Compression

Compressing What Matters: Neuron Importance Meets Data-Aware Low Rank Approximation for Language Model Compression

Zero Sum SVD: Balancing Loss Sensitivity for Low Rank LLM Compression

Generalized Fisher-Weighted SVD: Scalable Kronecker-Factored Fisher Approximation for Compressing Large Language Models

Residual Sparsification via Output Importance for Compressing Mixture-of-Experts LLMs