The paper introduces Conditional Program (CP), a compressed representation for large language model memory tables that reconstructs only the rows requested by an input. CP encodes each row with a short two‑bit code and a local scale, using a shared reconstruction program based on a Gaussian trellis. Experiments on three n‑gram memory families show CP reduces output KL divergence by 41–63% compared to 2‑bit round‑to‑nearest, and for DeepSeek it cuts table storage from 203 GB to 51 GB, enabling larger workloads on GPU.
By Donghoon Han, Minseop Jung, Dong Woo Kwak, SeungJae Lee, Junhoo Lee
Conditional memory expands language-model capacity through selective lookup, but the full embedding table must remain available for every possible input. We introduce Conditional Program (CP), a compr...
arXiv:2608.23843v1 Announce Type: new
Abstract: Long-context inference in large language models (LLMs) is increasingly limited by the memory required for the key-value (KV) cache. KV cache compressio...
By Zizhong Wang, Jieying Wang, Zhao Zhang, Jiajia Li
The paper investigates reducing memory usage of n‑gram embedding tables in open‑weight language models by storing them in low precision. Experiments on nanochat, Llama, and GPT‑2 families show that four‑bit tables cut table memory by about 75% while retaining over 99% of the validation‑loss improvement. The study also demonstrates that re‑allocating the saved memory to larger tables can surpass the performance of full‑precision tables, and that similar savings are achievable in large models like DeepSeek‑V4.1‑Flash and Qwen3.8‑Flash‑Next with minimal impact on perplexity and downstream tasks.
By Stephen Chung, Wenyu Du
FastE is a training‑free, plug‑and‑play method that compresses token prefixes in large language model (LLM) embedding inference. It uses a shared fixed threshold on batch‑mean readout‑prefix alignment to decide when to compress and ranks prefix states by readout attention scores to keep the most important ones. Experiments on Qwen3‑Embedding models show that FastE can cut decoder‑backbone FLOPs by over 40% while preserving more than 99% of the original ranking quality across multiple benchmarks and tasks.
By Jinsong Shu, Jinyong Wen, Baokun Wang, Zhongle Xie, Lidan Shou, Weiqiang Wang, Gang Chen
arXiv:2605. 20689v2 Announce Type: replace-cross Abstract: High-dimensional language-model embeddings increase storage and search costs, while supervised compressors can overfit when relevance labels are scarce.
By Dongfang Zhao