Choosing Quantized Code LLMs by File Size and Energy per Solved Task
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Tetra introduces a new Leech‑lattice based codebook that reduces the memory footprint of quantized LLMs to about 2.15 bits per weight, enabling efficient 2‑bit quantization without a massive lookup table. The method employs a 64‑state Golay trellis and a shared 16 KiB table, decoding each 24‑weight block with only six table loads and two small lookups. When applied to Qwen3 models (4B, 8B, 14B), Tetra achieves 2.70–2.73 bits per parameter, scoring 63–75 on MMLU and generating 57–114 tokens per second, while maintaining close performance to 4‑bit AWQ and outperforming llama.cpp’s IQ2_XXS on 4B.
The paper introduces DuoRate, a method for allocating precision in Mixture-of-Experts (MoE) models that separates storage and decoding rates. It stores a uniform tier of weights once and then adapts read precision per workload using a Lagrangian rate‑distortion optimization, reducing the store‑read gap. Experiments on Qwen3-30B-A3B and GSM8K show that DuoRate achieves higher accuracy at lower bit‑per‑weight (bpw) rates compared to fixed‑format baselines.
TableTune is a system that automatically measures and corrects draft-length tables for speculative decoding in large language model serving. It profiles decoding performance across batch sizes and draft lengths, generates an optimized table in minutes, and a controller plugin continuously updates the table based on live traffic. Experiments on H100 GPUs show that TableTune consistently achieves at least 94% of the best fixed draft-length throughput, outperforms other speculative control methods, and can recover up to 96% of performance lost by stale tables, delivering significant speedups in real deployments.
arXiv:2608. 08910v1 Announce Type: cross Abstract: PTQTP decomposes LLM weight matrices into two ternary (trit) planes with two free per-group scales.
arXiv:2608. 20210v1 Announce Type: cross Abstract: Small language models are usually built like large ones and then squeezed onto a CPU afterwards.
Coding agents re-send large file reads and tool outputs to a frontier LLM every turn, and this context dominates their token bill. General-purpose prompt compressors are trained on prose and suit code...