arXiv Machine Learning
Extreme Compression of Per-Layer Embeddings via Cross-Layer Latent Vector Quantization
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv AI
3d ago
E8-QAT: Efficient 2-Bit Quantization-Aware Training with Implicit Lattice Codebooks
arXiv:2610.27699v1 Announce Type: new Abstract: Low-bit quantization reduces the memory footprint of large language models, but preserving accuracy with efficient training remains challenging. Scalar...
arXiv Machine Learning
Jul 7
Variable Bit-width Quantization: Learning Per-Group Precision for "Bigger-but-Smaller" Language Models
arXiv:2607. 02893v1 Announce Type: new Abstract: Low-bit quantization shrinks language models but treats precision as a single global hyper-parameter: every weight uses the same bit-width.
arXiv AI
Oct 15
E8 Lattice Key Quantization for Low-Bit KV-Cache Compression
arXiv:2610.15384v1 Announce Type: cross Abstract: Compressing the key-value (KV) cache is essential for serving long-context large language models (LLMs) under memory constraints. While current scala...
arXiv AI
Oct 5
MRVQ: One Resident Index for Dimension- and Rate-Elastic Vector Search
MRVQ (Matryoshka Residual Vector Quantization) is a single post‑hoc residual quantizer designed for frozen embeddings that can be truncated to support any combination of embedding dimension and code rate. By dropping residual stages or embedding coordinates, MRVQ provides a single resident artifact that serves all evaluated (dimension, rate) pairs, achieving the lowest memory usage among tested designs. Although it uses 17.8–22.0× less memory than three separately trained QINCo2 indices and 1.89–2.02× less than a lean shared‑model steelman, it trades off a small drop in nDCG@10 on FiQA compared to per‑rate QINCo2, yet still outperforms PQ, OPQ, and AdANNS‑OPQ at matched code size.
arXiv AI
Oct 15
SAGE: Pushing 2-Bit Vector-Quantized LLMs toward Real-World Usability
arXiv:2610.14745v1 Announce Type: new Abstract: Extreme low-bit weight-only quantization is an effective approach to reducing the deployment costs of large language models. Below 3 bits, scalar quant...
arXiv Machine Learning
Sep 3
Unfolding the Leech Lattice: Fused Multi-Shell Decoding and VRAM Layouts for 2-Bit LLM Weights
The paper introduces a multi‑shell decoder for Leech‑lattice vector quantization, achieving the best reported 2‑bit quality under its evaluation protocol. It presents a GPU‑friendly layout that fuses dequantization with matrix‑vector multiplication, demonstrating significant speed and memory advantages over traditional one‑hot masks and other 4‑bit methods. Experiments show the new kernel outperforms baseline approaches across multiple model sizes, with measurable gains in throughput and reduced byte traffic.