arXiv Machine Learning August 31, 2026 By Artem Safronov

A Method for Layer Bit-Width Allocation in LLM Quantization via Performance Maximization Under a Quality-Degradation Constraint

Read the original on arXiv Machine Learning →

The paper introduces a method for allocating bit-widths to individual layers in Gemma-3-1B to maximize performance (latency reduction) while staying within a specified quality-degradation budget. Using a layer sensitivity profile from SA-PTQ and TensorRT-LLM, the authors evaluate 13 W8A8 variants on an RTX 5090, finding that FFN 5+5 with lm_head yields an 11.0% latency reduction with negligible quality loss. They also discuss trade-offs for Attention layers and propose further optimizations such as fused INT8 attention kernels and FP8 usage.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv Machine Learning

Jul 17

PolyQ: Codesigning End-to-End Quantization Framework for Scalable Edge CPU LLM Inference

arXiv:2607. 14618v1 Announce Type: new Abstract: CPUs are the most universal target for on-device LLM inference, but existing low-bit quantization methods offer either coarse operating points or fine-grained mixed precision that is difficult to execute efficiently on CPUs.

By Hyunwoo Oh, Suyeon Jang, Hanning Chen, KyungIn Nam, Sanggeon Yun, Ryozo Masukawa, Mohsen Imani

llms efficiency

arXiv Machine Learning

Jun 29

RateQuant: Optimal Mixed-Precision KV Cache Quantization via Rate-Distortion Theory

arXiv:2605. 06675v2 Announce Type: replace Abstract: Large language models cache all previously computed key-value (KV) pairs during generation, and this KV cache grows linearly with sequence length, making it a primary memory bottleneck for serving.

By Fei Zuo, Zikang Zhou, Hao Cong, Xiaoyan Xi, Ho Fai Leung

llms efficiency

arXiv Machine Learning

Jun 24

RoPE-Aware Bit Allocation for KV-Cache Quantization

arXiv:2606. 24033v1 Announce Type: new Abstract: Existing low-bit KV-cache quantizers often treat each cached key as a flat vector.

By Fengfeng Liang, Yuechen Zhang, Jiaya Jia

llms efficiency

arXiv AI

Jul 28

MixQuant: Adaptive Mixed-Precision Quantization for Large Language Models

arXiv:2607. 23047v1 Announce Type: cross Abstract: Mixed-precision quantization improves the accuracy of post-training quantization by allocating higher bitwidths to sensitive layers, but existing methods solve the allocation for a single fixed memory budget.

By Ashitabh Misra, Madhav Agrawal, Arham Jain, Tarek Abdelzaher

llms efficiency

arXiv AI

Jun 2

BitsMoE: Efficient Spectral Energy-Guided Bit Allocation for MoE LLM Quantization

arXiv:2606. 00079v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) large language models reduce per-token computation through sparse expert activation, but their deployment remains memory-intensive because all expert weights must be kept resident in memory.

By Jiayu Zhao, Zihan Teng, Minhao Fan, Tianrui Ma, Wentao Ren, Song Chen, Weichen Liu

llms efficiency

arXiv Machine Learning

Jul 7

Variable Bit-width Quantization: Learning Per-Group Precision for "Bigger-but-Smaller" Language Models

arXiv:2607. 02893v1 Announce Type: new Abstract: Low-bit quantization shrinks language models but treats precision as a single global hyper-parameter: every weight uses the same bit-width.

By Hamish Ogilvy

llms efficiency

One email a morning, machine-written

Related stories

PolyQ: Codesigning End-to-End Quantization Framework for Scalable Edge CPU LLM Inference

RateQuant: Optimal Mixed-Precision KV Cache Quantization via Rate-Distortion Theory

RoPE-Aware Bit Allocation for KV-Cache Quantization

MixQuant: Adaptive Mixed-Precision Quantization for Large Language Models

BitsMoE: Efficient Spectral Energy-Guided Bit Allocation for MoE LLM Quantization

Variable Bit-width Quantization: Learning Per-Group Precision for "Bigger-but-Smaller" Language Models