← Back to all news
arXiv Machine Learning By Pier-Jean Malandrino (Scub)

Unfolding the Leech Lattice: Fused Multi-Shell Decoding and VRAM Layouts for 2-Bit LLM Weights

Read the original on arXiv Machine Learning →

The paper introduces a multi‑shell decoder for Leech‑lattice vector quantization, achieving the best reported 2‑bit quality under its evaluation protocol. It presents a GPU‑friendly layout that fuses dequantization with matrix‑vector multiplication, demonstrating significant speed and memory advantages over traditional one‑hot masks and other 4‑bit methods. Experiments show the new kernel outperforms baseline approaches across multiple model sizes, with measurable gains in throughput and reduced byte traffic.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

Hugging Face Trending Papers
Sep 28

Tetra: Serving Leech-Lattice Quantized LLMs at 2.7 Bits per Parameter

Tetra introduces a new Leech‑lattice based codebook that reduces the memory footprint of quantized LLMs to about 2.15 bits per weight, enabling efficient 2‑bit quantization without a massive lookup table. The method employs a 64‑state Golay trellis and a shared 16 KiB table, decoding each 24‑weight block with only six table loads and two small lookups. When applied to Qwen3 models (4B, 8B, 14B), Tetra achieves 2.70–2.73 bits per parameter, scoring 63–75 on MMLU and generating 57–114 tokens per second, while maintaining close performance to 4‑bit AWQ and outperforming llama.cpp’s IQ2_XXS on 4B.

llmsragefficiencybenchmarks
More like this →
arXiv Machine Learning
Oct 19

Memory Bandwidth, Not FLOPs: Profiling and Accelerating Query-Based Segmentation Decoders on Edge GPUs

The paper shows that query‑based segmentation decoders on edge GPUs are limited by memory bandwidth rather than FLOPs. Profiling reveals that non‑tensor‑core operations consume a large fraction of decoder time and bytes, saturating the DRAM bus. CompactInst mitigates this by using parameter‑free spatial pooling and a lightweight refinement head, achieving significant speedups on Jetson Orin Nano while maintaining accuracy.

By Oliver Busemann (Osnabr\"uck University, CLAAS E-Systems GmbH), Torben T\"oniges (CLAAS E-Systems GmbH), Martin Atzmueller (Semantic Information Systems Group, Osnabr\"uck University, German Research Center for Artificial Intelligence)
llmscomputer-vision
More like this →