arXiv AI June 6, 2026 By Zihan Chen, Bike Xie, Jundong Li, Cong Shen

Channel-Wise Mixed-Precision Quantization for Large Language Models

arXiv:2410. 13056v4 Announce Type: replace-cross Abstract: Large Language Models (LLMs) have demonstrated remarkable success across a wide range of language tasks, but their deployment on edge devices remains challenging due to the substantial memory requirements imposed by their large parameter sizes.

Summary generated by The Flow from the publisher's feed. The full article lives at arXiv AI.

arXiv Machine Learning

Jun 4

SFMP: Fine-Grained, Hardware-Friendly and Search-Free Mixed-Precision Quantization for Large Language Models

arXiv:2602. 01027v2 Announce Type: replace Abstract: Mixed-precision quantization is a promising approach for compressing large language models under tight memory budgets.

By Xin Nie, Haicheng Zhang, Liang Dong, Beining Feng, Jinhong Weng, Guiling Sun

llms efficiency benchmarks

arXiv AI

Aug 6

Recurrent Residual Quantization: A Progressive Multi-Precision Representation for LLMs

arXiv:2608. 04048v1 Announce Type: cross Abstract: Serving large language models (LLMs) under diverse deployment constraints requires flexible trade-offs between accuracy, memory footprint, and throughput.

By Yu Luo, Bo Dong, Wenhua Cheng, Haihao Shen

llms efficiency

arXiv Machine Learning

Jun 16

MosaicQuant: Inlier-Outlier Disaggregation for Unified 4-Bit LLM Quantization

arXiv:2606. 15652v1 Announce Type: new Abstract: 4-bit quantization significantly reduces the memory footprint and accelerates the inference of large language models (LLMs).

By Yangjia Hu, Haodong Wang, Zicong Hong, Qianli Liu, Quanxin Shou, Jian Lin, Song Guo, Xiaowei Shen, Xiangjun Huang, Dian Wang, Jian Yang

llms efficiency

arXiv AI

Jun 9

Joint Structural Pruning and Mixed-Precision Quantization for LLM Compression

arXiv:2606. 07819v1 Announce Type: new Abstract: Recently, the efficiency of Large Language Models (LLMs) deployment has become a critical concern in practical applications.

By Hoang-Loc La, Truong-Thanh Le, Amir Taherkordi, Phuong Hoai Ha

llms efficiency benchmarks

arXiv AI

Jun 4

Recover-LoRA for Aggressive Quantization: Reclaiming Accuracy in 2-Bit Language Models via Low-Rank Adaptation with Knowledge Distillation on Synthetic Data

arXiv:2606. 04238v1 Announce Type: cross Abstract: Aggressive weight quantization to 2-bit precision offers substantial throughput and memory gains for large language model (LLM) inference, but typically incurs severe accuracy degradation.

By Devleena Das, Rajeev Patwari, Elliott Delaye, Ashish Sirasao

llms fine-tuning efficiency benchmarks

arXiv Machine Learning

Jul 17

PolyQ: Codesigning End-to-End Quantization Framework for Scalable Edge CPU LLM Inference

arXiv:2607. 14618v1 Announce Type: new Abstract: CPUs are the most universal target for on-device LLM inference, but existing low-bit quantization methods offer either coarse operating points or fine-grained mixed precision that is difficult to execute efficiently on CPUs.

By Hyunwoo Oh, Suyeon Jang, Hanning Chen, KyungIn Nam, Sanggeon Yun, Ryozo Masukawa, Mohsen Imani

llms efficiency

Related stories

SFMP: Fine-Grained, Hardware-Friendly and Search-Free Mixed-Precision Quantization for Large Language Models

Recurrent Residual Quantization: A Progressive Multi-Precision Representation for LLMs

MosaicQuant: Inlier-Outlier Disaggregation for Unified 4-Bit LLM Quantization

Joint Structural Pruning and Mixed-Precision Quantization for LLM Compression

Recover-LoRA for Aggressive Quantization: Reclaiming Accuracy in 2-Bit Language Models via Low-Rank Adaptation with Knowledge Distillation on Synthetic Data

PolyQ: Codesigning End-to-End Quantization Framework for Scalable Edge CPU LLM Inference