Activation-Aware Weight Tensorization: A Calibration-Time Preconditioner for Tensor-Network LLM Compression
Read the original on arXiv Machine Learning →Activation-Aware Weight Tensorization (AWT) is a training‑free calibration wrapper that preconditions each weight matrix with a diagonal activation‑derived scale before applying standard Tensor Train (TT) or Tree Tensor Network (TTN) decompositions. By adding only an input‑side elementwise rescaling, AWT improves functional fidelity of post‑training tensor‑network compression across several large language models (Llama 3.1 8B, Ministral 8B, Qwen2.5 7B) at 2–6× compression, closing a significant portion of the perplexity gap to dense baselines and enhancing downstream task performance. The method demonstrates that diagonal preconditioning offers a robustness‑modularity trade‑off, outperforming full‑covariance oracles in many cases while remaining modular and solver‑agnostic.
Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.