← Back to all news
Hugging Face Trending Papers

Tetra: Serving Leech-Lattice Quantized LLMs at 2.7 Bits per Parameter

Read the original on Hugging Face Trending Papers →

Tetra introduces a new Leech‑lattice based codebook that reduces the memory footprint of quantized LLMs to about 2.15 bits per weight, enabling efficient 2‑bit quantization without a massive lookup table. The method employs a 64‑state Golay trellis and a shared 16 KiB table, decoding each 24‑weight block with only six table loads and two small lookups. When applied to Qwen3 models (4B, 8B, 14B), Tetra achieves 2.70–2.73 bits per parameter, scoring 63–75 on MMLU and generating 57–114 tokens per second, while maintaining close performance to 4‑bit AWQ and outperforming llama.cpp’s IQ2_XXS on 4B.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at Hugging Face Trending Papers.

Related stories

arXiv Machine Learning
Sep 3

Unfolding the Leech Lattice: Fused Multi-Shell Decoding and VRAM Layouts for 2-Bit LLM Weights

The paper introduces a multi‑shell decoder for Leech‑lattice vector quantization, achieving the best reported 2‑bit quality under its evaluation protocol. It presents a GPU‑friendly layout that fuses dequantization with matrix‑vector multiplication, demonstrating significant speed and memory advantages over traditional one‑hot masks and other 4‑bit methods. Experiments show the new kernel outperforms baseline approaches across multiple model sizes, with measurable gains in throughput and reduced byte traffic.

By Pier-Jean Malandrino (Scub)
llmsefficiencybenchmarks
More like this →
arXiv AI
Sep 3

Post-Training Ternarization of Qwen3-4B Capability, Effective Bit Budget, Storage Compression, and Deployment

The paper reports a post‑training ternarization of the 4‑billion‑parameter Qwen model, achieving an effective 1.641‑bit representation for 81.62 % of its weights while keeping activations at 16‑bit precision. Accuracy drops from 64.5 % to 54.7 % across ten capability tests, with uneven degradation (e.g., BoolQ 84.6 % of teacher performance, ARC‑Challenge 43.8 %). After packing the ternary planes, the model size shrinks from 8.29 GiB to 3.96 GiB with negligible change in perplexity, though inference speed is not improved.

By Anirudh Malik, M Sparsh Mehra, Poojith Devan
llmsefficiency
More like this →
arXiv AI
Sep 11

Scaling Post-Training Ternarisation to Qwen3-8B Capability Retention, Reproduction, Lossless Packing, and Packed Execution

The paper reports a large‑scale post‑training ternarisation of the Qwen3 language model, extending a conversion pipeline from the 4B to the 8B variant. Using KOTMS rotation, E2M‑ATQ adaptive ternarisation, and GPTQ‑style error compensation, the authors achieve a 1.361× perplexity ratio across three corpora and retain 78.5% of the FP16 accuracy on zero‑shot tasks, with the 8B model outperforming the 4B by 8.9 percentage points. The study also demonstrates lossless lattice‑aware packing, producing an 8.24 GiB checkpoint that preserves perplexity, and shows that direct packed execution can reach 15.52 tokens/s in 7.35 GiB, though packed GEMV remains slower than FP16 cuBLAS.

By Anirudh Malik, M Sparsh Mehra, Poojith Devan
llms
More like this →