← Back to all news
arXiv Machine Learning By Hamish Ogilvy

Variable Bit-width Quantization: Learning Per-Group Precision for "Bigger-but-Smaller" Language Models

Read the original on arXiv Machine Learning →

arXiv:2607. 02893v1 Announce Type: new Abstract: Low-bit quantization shrinks language models but treats precision as a single global hyper-parameter: every weight uses the same bit-width.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Machine Learning
Jul 17

PolyQ: Codesigning End-to-End Quantization Framework for Scalable Edge CPU LLM Inference

arXiv:2607. 14618v1 Announce Type: new Abstract: CPUs are the most universal target for on-device LLM inference, but existing low-bit quantization methods offer either coarse operating points or fine-grained mixed precision that is difficult to execute efficiently on CPUs.

By Hyunwoo Oh, Suyeon Jang, Hanning Chen, KyungIn Nam, Sanggeon Yun, Ryozo Masukawa, Mohsen Imani
llmsefficiency
More like this →
arXiv AI
Sep 4

Why Gated DeltaNet Survives 4-Bit Quantization: NVFP4 W4A4 for the Recurrent Half of a Hybrid 27B LLM

The paper presents a 4‑bit quantization recipe, Minima: NVFP4 W4A4, that fully quantizes all linear layers—including the Gated DeltaNet (GDN) recurrent blocks—of the 27‑billion‑parameter Qwen3.8 LLM. Across a suite of benchmarks (perplexity, MMLU‑Pro, GSM8K, AIME'25, GPQA‑Diamond, LiveCodeBench, and RULER retrieval), the quantized model matches BF16 performance within seed noise while being 17.5 GiB in size and 14–19 % faster at prefill. The authors attribute this success to four mechanisms: block‑scaling of residuals, robust gate projections, the delta‑rule recurrence’s noise‑plateau behavior, and the per‑token quantization cost’s dilution over long contexts.

By Sergii Kozyrev, Davyd Maiboroda
llmsefficiencybenchmarks
More like this →
arXiv Machine Learning
Aug 11

RotaryQuant: Fitting 120B MoE Models on Consumer Hardware via Fused Compressed-Space Attention

arXiv:2608. 08081v1 Announce Type: cross Abstract: Large mixture-of-experts (MoE) language models with 26--120 billion parameters exceed the memory capacity of consumer devices through three simultaneous pressures: resident weight matrices, key-value (KV) cache state that grows linearly with context, and dozens of expert sublayers that must be paged on demand.

By Anthony. Lui, Mohamed. Elsaied, N. P. Savani
llmsefficiency
More like this →