GAMMA is a post‑training framework that learns module‑wise precision preferences for mixed‑precision quantization of large language models. It optimizes a teacher‑forced hidden‑state reconstruction objective under an augmented Lagrangian constraint and then projects the learned preferences into exact budget‑feasible discrete assignments via integer programming. Because the learned preferences encode a stable sensitivity ranking, a single training run can be reused for any deployment budget, reducing per‑budget adaptation from hours to minutes and outperforming fixed‑precision baselines and search‑based methods on Llama and Qwen models.
By Zhangyang Yao, Haiyan Zhao, Haoyu Wang, Xu Han
arXiv:2606. 04115v1 Announce Type: cross Abstract: Quantizing large language models (LLMs) to low-precision floating-point representations is central to efficient deployment, yet applying a single bit-width uniformly across all layers is sub-optimal in terms of both performance and accuracy.
By Giuseppe Franco, Ian Colbert, Pablo Monteagudo-Lago, Felix Marty, Nicholas Fraser
arXiv:2605. 26660v2 Announce Type: replace Abstract: Quantization is an effective approach to reduce the memory footprint and inference cost of large language models (LLMs), yet maintaining performance in the ultra-low-bit regime remains challenging.
By Phong Nam Huu Nguyen, Khoi M. Le, Cong-Duy T Nguyen, Anh Tuan Luu, Thong Thanh Nguyen, Tho Quan
arXiv:2605. 06675v2 Announce Type: replace Abstract: Large language models cache all previously computed key-value (KV) pairs during generation, and this KV cache grows linearly with sequence length, making it a primary memory bottleneck for serving.
By Fei Zuo, Zikang Zhou, Hao Cong, Xiaoyan Xi, Ho Fai Leung
The paper introduces FAMPWQ, a Fisher information-based Adaptive Mixed Precision Weight Quantization method designed to improve Large Language Model inference on commodity GPUs. It uses a Fisher information metric to assess layer-wise sensitivity and a reinforcement learning-based bit-width allocator to adaptively assign precision per layer. Experiments across seven models and five benchmarks show significant gains in perplexity, accuracy, and LLM-as-a-judge performance compared to seven baseline approaches.
By Gongwei Lee, Ji Liu, Juncheng Jia, Ji Wu
arXiv:2606. 07819v1 Announce Type: new Abstract: Recently, the efficiency of Large Language Models (LLMs) deployment has become a critical concern in practical applications.
By Hoang-Loc La, Truong-Thanh Le, Amir Taherkordi, Phuong Hoai Ha