Activation-Aware Weight Tensorization (AWT) is a training‑free calibration wrapper that preconditions each weight matrix with a diagonal activation‑derived scale before applying standard Tensor Train (TT) or Tree Tensor Network (TTN) decompositions. By adding only an input‑side elementwise rescaling, AWT improves functional fidelity of post‑training tensor‑network compression across several large language models (Llama 3.1 8B, Ministral 8B, Qwen2.5 7B) at 2–6× compression, closing a significant portion of the perplexity gap to dense baselines and enhancing downstream task performance. The method demonstrates that diagonal preconditioning offers a robustness‑modularity trade‑off, outperforming full‑covariance oracles in many cases while remaining modular and solver‑agnostic.
By Alessandro Beatini, Marco Maronese, Emanuele Rodol\`a
COMPOT is a training‑free compression framework for transformer models that uses a small calibration dataset to estimate a sparse weight factorization. It employs orthogonal dictionaries, enabling closed‑form Procrustes updates and analytical single‑step sparse coding, which dramatically reduces optimization cost. The method also introduces a one‑shot dynamic allocation strategy to adaptively redistribute compression rates across layers, achieving a superior quality‑compression trade‑off compared to strong low‑rank and sparse baselines while remaining compatible with post‑training quantization.
By Denis Makhov, Dmitriy Shopkhoev, Magauiya Zhussip, Ammar Ali, Baher Mohammad, Stamatios Lefkimmiatis
arXiv:2607. 20652v1 Announce Type: cross Abstract: Language models are thought to exhibit the phenomenon of superposition, representing many more features than dimensions in their residual streams.
By Andrew Mack, Kraig Yuheng Tou, Mark Henry, Zhengxun Wu, Lauren Greenspan
SparseDecoding is a decoding‑aware pruning framework that improves large language model inference by aligning pruning with the activations seen during actual autoregressive generation. It constructs calibration matrices from dense‑model activations (excluding prefill) and introduces an optimized N:M sparse matrix‑vector kernel with bitmask indexing to accelerate decoding. Experiments on models such as Llama‑3.1‑8B, Llama‑3.3‑70B, and Qwen3‑14B/32B show consistent performance gains over standard fixed‑text calibration and up to 1.48× wall‑clock speedup on A100 GPUs.
By Qitong Wang, Xinwei Niu, Mingluo Su, Shanwei Zhao, Shiai Zhu, Huan Wang
Modern transformers pair impressive capabilities with substantial memory and compute demands. Low-rank weight factorization reduces both while keeping the matrices dense, and thus efficient on standar...
arXiv:2609.40127v1 Announce Type: cross
Abstract: Modern transformers pair impressive capabilities with substantial memory and compute demands. Low-rank weight factorization reduces both while keepin...
By Massimo Bini, Anders Christensen, Stephan Alaniz, Judah Goldfeder, Ole Winther, Yann LeCun, Ravid Shwartz-Ziv, Zeynep Akata