arXiv Computation and Language
Oct 1

Learning Functional Subspaces for Neural Network Compression

arXiv:2609.40127v1 Announce Type: cross Abstract: Modern transformers pair impressive capabilities with substantial memory and compute demands. Low-rank weight factorization reduces both while keepin...

By Massimo Bini, Anders Christensen, Stephan Alaniz, Judah Goldfeder, Ole Winther, Yann LeCun, Ravid Shwartz-Ziv, Zeynep Akata
arXiv Machine Learning
Oct 28

COMPOT: Calibration-Optimized Matrix Procrustes Orthogonalization for Transformers Compression

COMPOT is a training‑free compression framework for transformer models that uses a small calibration dataset to estimate a sparse weight factorization. It employs orthogonal dictionaries, enabling closed‑form Procrustes updates and analytical single‑step sparse coding, which dramatically reduces optimization cost. The method also introduces a one‑shot dynamic allocation strategy to adaptively redistribute compression rates across layers, achieving a superior quality‑compression trade‑off compared to strong low‑rank and sparse baselines while remaining compatible with post‑training quantization.

By Denis Makhov, Dmitriy Shopkhoev, Magauiya Zhussip, Ammar Ali, Baher Mohammad, Stamatios Lefkimmiatis