← Back to all news
arXiv Computation and Language By Stephen Chung, Wenyu Du

Low-Bit N-Gram Embeddings for Language Models

Read the original on arXiv Computation and Language →

The paper investigates reducing memory usage of n‑gram embedding tables in open‑weight language models by storing them in low precision. Experiments on nanochat, Llama, and GPT‑2 families show that four‑bit tables cut table memory by about 75% while retaining over 99% of the validation‑loss improvement. The study also demonstrates that re‑allocating the saved memory to larger tables can surpass the performance of full‑precision tables, and that similar savings are achievable in large models like DeepSeek‑V4.1‑Flash and Qwen3.8‑Flash‑Next with minimal impact on perplexity and downstream tasks.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Related stories

arXiv Machine Learning
Oct 16

N-Gram Memories at Two Bits: Quantizing Lookup Tables in Large Language Models

The paper introduces Conditional Program (CP), a compressed representation for large language model memory tables that reconstructs only the rows requested by an input. CP encodes each row with a short two‑bit code and a local scale, using a shared reconstruction program based on a Gaussian trellis. Experiments on three n‑gram memory families show CP reduces output KL divergence by 41–63% compared to 2‑bit round‑to‑nearest, and for DeepSeek it cuts table storage from 203 GB to 51 GB, enabling larger workloads on GPU.

By Donghoon Han, Minseop Jung, Dong Woo Kwak, SeungJae Lee, Junhoo Lee
llmsrag
More like this →
arXiv Machine Learning
Oct 15

The COUNT: Counting N-grams Unlocks Scaling on Host Memory and Training-Free Adaptation

The paper introduces COUNT, a method that builds n‑gram embeddings without adding learned parameters by storing precomputed n‑gram counts on host memory. By weighting the language model’s own unembeddings with corpus frequencies, COUNT can be added as an extra input embedding during pretraining, improving validation loss equivalent to a 57% increase in compute for a 1B‑parameter model. Additionally, COUNT enables efficient GPU memory usage and allows domain‑specific n‑gram tables to be swapped in at inference time for training‑free adaptation.

By Ian Magnusson, Noah A. Smith, Pang Wei Koh
llmsrag
More like this →