Hugging Face Trending Papers
Sep 28

Tetra: Serving Leech-Lattice Quantized LLMs at 2.7 Bits per Parameter

Tetra introduces a new Leech‑lattice based codebook that reduces the memory footprint of quantized LLMs to about 2.15 bits per weight, enabling efficient 2‑bit quantization without a massive lookup table. The method employs a 64‑state Golay trellis and a shared 16 KiB table, decoding each 24‑weight block with only six table loads and two small lookups. When applied to Qwen3 models (4B, 8B, 14B), Tetra achieves 2.70–2.73 bits per parameter, scoring 63–75 on MMLU and generating 57–114 tokens per second, while maintaining close performance to 4‑bit AWQ and outperforming llama.cpp’s IQ2_XXS on 4B.

arXiv Machine Learning
Oct 12

DuoRate: Two-Rate Per-Expert Precision Allocation for Mixture-of-Experts Serving

The paper introduces DuoRate, a method for allocating precision in Mixture-of-Experts (MoE) models that separates storage and decoding rates. It stores a uniform tier of weights once and then adapts read precision per workload using a Lagrangian rate‑distortion optimization, reducing the store‑read gap. Experiments on Qwen3-30B-A3B and GSM8K show that DuoRate achieves higher accuracy at lower bit‑per‑weight (bpw) rates compared to fixed‑format baselines.

By Mengqi Guo, Zhe Wei, Yuan Yuan, Jiunn Bin Lim, Boyi Pan, Michael Bi Mi
arXiv AI
Oct 31

TableTune: Measured, Self-Correcting Draft-Length Tables for Dynamic Speculative Decoding

TableTune is a system that automatically measures and corrects draft-length tables for speculative decoding in large language model serving. It profiles decoding performance across batch sizes and draft lengths, generates an optimized table in minutes, and a controller plugin continuously updates the table based on live traffic. Experiments on H100 GPUs show that TableTune consistently achieves at least 94% of the best fixed draft-length throughput, outperforms other speculative control methods, and can recover up to 96% of performance lost by stale tables, delivering significant speedups in real deployments.

By Abhishek Malvankar, Paul Castro, Nick Mitchell, Lionel Villard