arXiv:2505. 17974v2 Announce Type: replace-cross Abstract: The Fisher information is a fundamental concept for characterizing the sensitivity of parameters in neural networks.
By Viktoriia Chekalina, Daniil Moskovskiy, Tatiana Matveeva, Andrey Kuznetsov, Evgeny Frolov
arXiv:2608. 08506v1 Announce Type: new Abstract: Training-free low-rank compression frameworks have been gaining prominence for LLM compression given their effectiveness in reducing model parameter count while maintaining task-level accuracy.
By Mohanad Odema, Gabrielle De Micheli, Dayin Gou, Nilesh Malpeddi, Prathamesh Vaste, Jacob Song
arXiv:2606. 05861v1 Announce Type: cross Abstract: The rapid development of large language models(LLMs) has led to remarkable advances in natural language processing.
By Rui Wang, Yan Zhao, Li Song, Zhengxue Cheng
SHIFT-LLM is a training‑free post‑pruning correction framework that inserts a Linear Residual Adapter (LRA) at each depth‑pruned site in large language models. Each LRA preserves the original residual identity while adding a lightweight affine correction calibrated via closed‑form least‑squares regression on a small held‑out set, thereby approximating the hidden state that would have been produced by the removed block. Experiments across multiple model families and benchmarks show that SHIFT‑LLM consistently recovers accuracy lost to depth pruning, achieving gains up to +15.7 points on Llama‑3.1‑8B‑Instruct with only a few hundred calibration samples and no gradient computation.
By Ali Bahri, Hang Li, Hongliang Li, Zhitang Chen
arXiv:2607. 03057v1 Announce Type: cross Abstract: The rapid growth in the parameter scale of large language models (LLMs) has created a strong demand for efficient compression techniques.
By Zhuowen Liu, Longkun Hao, Shiyu Feng, Xiaowen Chang, Ruiqun Li, Changqun Li
arXiv:2605.05794v2 Announce Type: replace-cross
Abstract: Large language models (LLMs) contain functionally distinct modules with heterogeneous gradient statistics. Although Adam(W) adapts updates at...
By Ziqing Wen, Zhouyang Liu, Jiahuan Wang, Ping Luo, Junlin Zeng, Li Shen, Dongsheng Li, Tao Sun
The paper introduces a scalable Kronecker-based approximation that captures cross-layer interactions without storing the full Fisher matrix, making Hessian analysis feasible for billion-parameter language models. It identifies consistent vulnerability patterns, notably that value projection layers are the most sensitive and exhibit strong cross-layer correlations across various model families. Experiments on quantization, sparsification, inter-layer corruption, and fine-tuning show that the approximation correlates strongly with performance degradation and recovery, providing a practical tool for identifying fragile components and guiding compression and optimization strategies.
By Viacheslav Yusupov, Daria Cherniuk, Evgeny Frolov
ITC-MoE introduces an Importance-guided Token-aware Compression framework for Mixture-of-Experts Diffusion Language Models. It combines Adaptive Tucker Compression, which uses activation and gradient importance to jointly factorize expert weights and allocate ranks, with Token-aware Compensation and Routing that applies low‑rank adjustments to hot tokens and limits expert candidates for cold tokens. The method achieves significant reductions in computation and storage while maintaining generation quality, exemplified by a 30% compression budget that preserves 96.33% accuracy on MultiArith and delivers up to a 7.22× speedup.
By Lianjun Liu, Shipeng Li, You Huang, Weiqi Yan, Mingte Qiu, Huazhong Liu, Xiaofeng Zhu, Yunshan Zhong
arXiv:2509. 14230v2 Announce Type: replace Abstract: While structured pruning presents a highly effective pathway for accelerating Large Language Model (LLM) inference, existing methods frequently suffer from significant performance degradation and demand computationally retraining to recover capabilities.
By Mengting Ai, Tianxin Wei, Sirui Chen, Jingrui He
arXiv:2608. 06901v1 Announce Type: cross Abstract: Vision-language models (VLMs) have achieved remarkable generalization across diverse multimodal tasks through large-scale pre-training, yet their rapidly increasing computational and memory requirements pose significant challenges for deployment in constrained environments.
By Minseok Kang, Hyunwoo Kim, Chanyoung Kim, Minwoo Kim, Jaekoo Lee, Dahuin Jung
The paper introduces a geometry-preserving output alignment framework for 1‑bit post‑training quantization of large language models. It formulates output alignment as a constrained optimization that balances reconstruction quality with token similarity preservation, and proposes a geometry‑aware salient‑mask selection strategy to choose quantization settings that maintain token relations. Experiments on various LLM families and downstream tasks show consistent gains over existing 1‑bit PTQ methods.
By Dung Anh Hoang, Cuong Pham, Cuong Nguyen, Trung le, Jianfei Cai, Thanh-Toan Do
arXiv:2504.04342v2 Announce Type: replace
Abstract: Scaling up model parameters and training data consistently improves the performance of large language models (LLMs), but at the cost of rapidly gro...
By Ayan Sengupta, Siddhant Chaudhary, Tanmoy Chakraborty