arXiv:2505. 17974v2 Announce Type: replace-cross Abstract: The Fisher information is a fundamental concept for characterizing the sensitivity of parameters in neural networks.
By Viktoriia Chekalina, Daniil Moskovskiy, Tatiana Matveeva, Andrey Kuznetsov, Evgeny Frolov
The paper introduces a unified Hessian-guided framework for post‑training quantization of large language models. By deriving a Hessian‑based approximation of output changes, it jointly optimizes quantization ranges, linear transformations, and weight rounding under a single output‑sensitivity objective. The method remains practical for modern models and consistently outperforms existing transformation and rounding techniques in both channel‑wise and block‑wise settings.
By Mads Toftrup, Marco Federici, Boris van Breugel, Paul Whatmough, Markus Nagel
arXiv:2608.30226v1 Announce Type: new
Abstract: Modular compression has enabled considerable parameter reduction in LLMs while preserving strong language understanding and downstream task accuracy. H...
By Mohanad Odema, Jacob Song
The paper introduces a unified Hessian-guided framework for post-training quantization of large language models. It derives a Hessian-based approximation of output changes due to joint weight and activation quantization, enabling simultaneous optimization of quantization ranges, linear transformations, and weight rounding under a single output-sensitivity objective. The method remains practical for modern models and consistently outperforms existing transformation and rounding techniques in both channel-wise and block-wise settings.
arXiv:2607. 07964v1 Announce Type: new Abstract: Post-training quantization (PTQ) is a widely adopted technique for compressing large language models (LLMs) without retraining.
By Donghyun Lee, Yuhang Li, Ruokai Yin, Priyadarshini Panda
arXiv:2607. 18284v1 Announce Type: cross Abstract: To excel at their domain large language models are comprised of billions of parameters.
By Athanasios Ntovas, Alexandros Doumanoglou, Petros Drakoulis, Dimitris Zarpalas
The paper introduces MSign, an optimizer designed to prevent training instability in large language models by restoring the stable rank of weight matrices. It identifies two precursors to gradient explosions—rapid stable rank decline and increased Jacobian alignment—and proves that these jointly cause exponential gradient growth. Experiments on models ranging from 5 M to 3 B parameters show that MSign stops training failures while adding less than 7.0% computational overhead.
By Lianhai Ren, Yucheng Ding, Xiao Liu, Peng Cheng, Yeyun Gong
arXiv:2609.25916v1 Announce Type: new
Abstract: Mixed-precision weight quantization is commonly formulated as a Multiple-Choice Knapsack Problem (MCKP), yet existing solvers rely on scalar sensitivit...
By Akihiro Yoshida, Yuma Ichikawa
The paper introduces GaugeLasso, a method that applies symmetric group‑lasso penalties to transformer channels during training, enabling entire tensor slices to be zeroed out while maintaining dense tensors for GPU efficiency. By calibrating channel penalties based on inference utility per compute, the network self‑organizes into depth‑dependent structural profiles that can be dramatically smaller than the original architecture, achieving up to 255‑fold compression on a polynomial division task and outperforming hand‑designed baselines on language modeling and autoencoding benchmarks. The approach also accelerates training and reveals over‑provisioned axes that guide subsequent design iterations.
By Jed A. Duersch, Na\"im Es-Sebbani, Nathana\"el Haas, Zied Bouraoui
Large language models (LLMs) are built from structured high-dimensional objects such as token representations, weights, adaptation updates, caches, and activations, whose multilinear structure is unde...
This survey reviews tensor methods applied to large language models, framing them through a seven‑stage lifecycle (tokenization, embeddings, pre‑training, adaptation, compression, inference, interpretability) and a component view (embeddings, attention, feed‑forward networks). It offers unified notation, theoretical foundations, and comparative analyses of tensorization strategies for Transformer components, while highlighting evaluation protocol differences and model scale effects. The paper also introduces a new metric, ρ_gap, to quantify the gap between theoretical memory savings and actual system‑level speedup, and connects tensor techniques to related efficiency and probabilistic methods.
By Matvei Tarasov, Salman Ahmadi-Asl, Andre L. F. de Almeida, Andrzej Cichocki
The paper introduces Layer-wise Error Modeling Search (LEMS) and KFAC-SVD to improve low-rank SVD-based compression of large language models. LEMS addresses global rank allocation by using a layer-wise error surrogate and an ILP formulation, while KFAC-SVD enhances decomposition quality by employing token-wise statistics to avoid rank collapse. Experiments on Mistral, Qwen3, and Llama 3 show LEMS yields up to 4.8 pp zero‑shot accuracy gains and KFAC‑SVD achieves an average perplexity improvement of 15%.
By Moritz Thoma, Maximilian Groezinger, Maximilian Forstenh\"ausler, Emad Aghajanzadeh, Manoj Rohit Vemparala, Christos Anagnostopoulos, Pierpaolo Mori, Nael Fasfous, Alexander Frickenstein, Daniel Mueller-Gritschneder, Ulf Schlichtmann