arXiv AI
LaMoC: Loss-Aware Modular Compression for LLMs
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Jul 17
Generalized Fisher-Weighted SVD: Scalable Kronecker-Factored Fisher Approximation for Compressing Large Language Models
arXiv:2505. 17974v2 Announce Type: replace-cross Abstract: The Fisher information is a fundamental concept for characterizing the sensitivity of parameters in neural networks.
arXiv AI
Aug 11
Understanding Calibration and Truncation Error Propagation in Training-Free Low-Rank Compression for LLMs
arXiv:2608. 08506v1 Announce Type: new Abstract: Training-free low-rank compression frameworks have been gaining prominence for LLM compression given their effectiveness in reducing model parameter count while maintaining task-level accuracy.
arXiv AI
Jun 6
LLMCodec: Adapting Video Codecs for Efficient Weight Compression of Large Language Models
arXiv:2606. 05861v1 Announce Type: cross Abstract: The rapid development of large language models(LLMs) has led to remarkable advances in natural language processing.
arXiv Computer Vision
Aug 27
SHIFT-LLM: Distribution Shift Correction in Depth-Pruned LLMs
SHIFT-LLM is a training‑free post‑pruning correction framework that inserts a Linear Residual Adapter (LRA) at each depth‑pruned site in large language models. Each LRA preserves the original residual identity while adding a lightweight affine correction calibrated via closed‑form least‑squares regression on a small held‑out set, thereby approximating the hidden state that would have been produced by the removed block. Experiments across multiple model families and benchmarks show that SHIFT‑LLM consistently recovers accuracy lost to depth pruning, achieving gains up to +15.7 points on Llama‑3.1‑8B‑Instruct with only a few hundred calibration samples and no gradient computation.
arXiv AI
Jul 7
LACE-SVD: Loss-Aware SVD with Cumulative Error Correction for LLM Compression
arXiv:2607. 03057v1 Announce Type: cross Abstract: The rapid growth in the parameter scale of large language models (LLMs) has created a strong demand for efficient compression techniques.
arXiv AI
Sep 3
Scalable Kronecker-Fisher Approximation: Efficient Hessian Analysis for Billion-Parameter Language Models Compression
The paper introduces a scalable Kronecker-based approximation that captures cross-layer interactions without storing the full Fisher matrix, making Hessian analysis feasible for billion-parameter language models. It identifies consistent vulnerability patterns, notably that value projection layers are the most sensitive and exhibit strong cross-layer correlations across various model families. Experiments on quantization, sparsification, inter-layer corruption, and fine-tuning show that the approximation correlates strongly with performance degradation and recovery, providing a practical tool for identifying fragile components and guiding compression and optimization strategies.