arXiv:2606. 26650v1 Announce Type: cross Abstract: In this paper, we present CAT-Q, Cost-efficient and Accurate Ternary Quantization, for compressing and accelerating LLMs.
By Shigeng Wang, Chao Li, Yangyuxuan Kang, Jiawei Fan, Anbang Yao
FedSAP is a federated learning framework that addresses heterogeneous edge devices by using structured pruning as a budget-constrained tri-state channel allocation. It partitions model channels into a Global pool, pseudo-domain-specific Private pools, and a Dropped state, allowing broadly useful features to be shared while isolating domain-sensitive updates. Experiments on Digits and Office-Caltech datasets show FedSAP achieving higher mean global accuracy than the strongest baseline while supporting up to 80% client pruning ratios.
By Wentao Yue, Tianyou Lai, Hongji Li, Qingyu Mao, Qilei Li
In this paper, we present CAT-Q, Cost-efficient and Accurate Ternary Quantization, for compressing and accelerating LLMs. Unlike existing state-of-the-art ternary quantization methods that rely on data-intensive and costly quantization-aware training to mitigate severe performance degradation, CAT-Q is a simple yet effective post-training quantization scheme that is readily applicable to LLMs with diverse architectures and model sizes.
arXiv:2602. 10431v4 Announce Type: replace Abstract: Large language models (LLMs) demand substantial computational and memory resources, posing challenges for efficient deployment.
By Kanghyun Noh, Jinheon Choi, Yulhwa Kim
The paper introduces FAMPWQ, a Fisher information-based Adaptive Mixed Precision Weight Quantization method designed to improve Large Language Model inference on commodity GPUs. It uses a Fisher information metric to assess layer-wise sensitivity and a reinforcement learning-based bit-width allocator to adaptively assign precision per layer. Experiments across seven models and five benchmarks show significant gains in perplexity, accuracy, and LLM-as-a-judge performance compared to seven baseline approaches.
By Gongwei Lee, Ji Liu, Juncheng Jia, Ji Wu
arXiv:2605. 02404v2 Announce Type: replace Abstract: Model quantization has become essential for efficient large language model deployment, yet existing approaches present clear trade-offs: methods such as GPTQ and AWQ achieve practical compression but are lossy, while lossless techniques preserve fidelity but lack inference acceleration.
By Michael Helcig, Eldar Kurtic, Dan Alistarh