← Back to all news
Hugging Face Trending Papers

Tetra: Serving Leech-Lattice Quantized LLMs at 2.7 Bits per Parameter

Tetra introduces a new Leech‑lattice based codebook that reduces the memory footprint of quantized LLMs to about 2.15 bits per weight, enabling efficient 2‑bit quantization without a massive lookup table. The method employs a 64‑state Golay trellis and a shared 16 KiB table, decoding each 24‑weight block with only six table loads and two small lookups. When applied to Qwen3 models (4B, 8B, 14B), Tetra achieves 2.70–2.73 bits per parameter, scoring 63–75 on MMLU and generating 57–114 tokens per second, while maintaining close performance to 4‑bit AWQ and outperforming llama.cpp’s IQ2_XXS on 4B.

Related stories

arXiv Machine Learning
Sep 3

Unfolding the Leech Lattice: Fused Multi-Shell Decoding and VRAM Layouts for 2-Bit LLM Weights

The paper introduces a multi‑shell decoder for Leech‑lattice vector quantization, achieving the best reported 2‑bit quality under its evaluation protocol. It presents a GPU‑friendly layout that fuses dequantization with matrix‑vector multiplication, demonstrating significant speed and memory advantages over traditional one‑hot masks and other 4‑bit methods. Experiments show the new kernel outperforms baseline approaches across multiple model sizes, with measurable gains in throughput and reduced byte traffic.

By Pier-Jean Malandrino (Scub)
llmsefficiencybenchmarks
More like this →
arXiv AI
Sep 3

Post-Training Ternarization of Qwen3-4B Capability, Effective Bit Budget, Storage Compression, and Deployment

The paper reports a post‑training ternarization of the 4‑billion‑parameter Qwen model, achieving an effective 1.641‑bit representation for 81.62 % of its weights while keeping activations at 16‑bit precision. Accuracy drops from 64.5 % to 54.7 % across ten capability tests, with uneven degradation (e.g., BoolQ 84.6 % of teacher performance, ARC‑Challenge 43.8 %). After packing the ternary planes, the model size shrinks from 8.29 GiB to 3.96 GiB with negligible change in perplexity, though inference speed is not improved.

By Anirudh Malik, M Sparsh Mehra, Poojith Devan
llmsefficiency
More like this →
arXiv AI
Sep 11

Scaling Post-Training Ternarisation to Qwen3-8B Capability Retention, Reproduction, Lossless Packing, and Packed Execution

The paper reports a large‑scale post‑training ternarisation of the Qwen3 language model, extending a conversion pipeline from the 4B to the 8B variant. Using KOTMS rotation, E2M‑ATQ adaptive ternarisation, and GPTQ‑style error compensation, the authors achieve a 1.361× perplexity ratio across three corpora and retain 78.5% of the FP16 accuracy on zero‑shot tasks, with the 8B model outperforming the 4B by 8.9 percentage points. The study also demonstrates lossless lattice‑aware packing, producing an 8.24 GiB checkpoint that preserves perplexity, and shows that direct packed execution can reach 15.52 tokens/s in 7.35 GiB, though packed GEMV remains slower than FP16 cuBLAS.

By Anirudh Malik, M Sparsh Mehra, Poojith Devan
llms
More like this →
arXiv AI
Aug 20

Cacheable by Design? Training Mixture-of-Experts Routers for Locality Against the Edge Memory-Bandwidth Wall: A Pre-Registered Negative Result with a Systems Measurement Study

The paper investigates whether training Mixture-of-Experts (MoE) routers can improve memory‑bandwidth locality on consumer GPUs. Using a new zero‑surgery telemetry tool, the authors measure that a large Qwen3‑235B model is bottlenecked by disk‑based expert access, and that an LRU cache can serve a majority of requests. They pre‑register experiments training 137 M‑parameter MoE models with locality‑aware losses, finding that while cache misses can drop up to 60 % (99 % static‑pin hit rate), every configuration fails to meet a strict 1 % perplexity threshold, indicating a tight coupling between cache efficiency and model quality.

By Shriniwas Ramesh Suram
llms
More like this →