The paper introduces Conditional Program (CP), a compressed representation for large language model memory tables that reconstructs only the rows requested by an input. CP encodes each row with a short two‑bit code and a local scale, using a shared reconstruction program based on a Gaussian trellis. Experiments on three n‑gram memory families show CP reduces output KL divergence by 41–63% compared to 2‑bit round‑to‑nearest, and for DeepSeek it cuts table storage from 203 GB to 51 GB, enabling larger workloads on GPU.
By Donghoon Han, Minseop Jung, Dong Woo Kwak, SeungJae Lee, Junhoo Lee
Conditional memory expands language-model capacity through selective lookup, but the full embedding table must remain available for every possible input. We introduce Conditional Program (CP), a compr...
arXiv:2608.23843v1 Announce Type: new
Abstract: Long-context inference in large language models (LLMs) is increasingly limited by the memory required for the key-value (KV) cache. KV cache compressio...
By Zizhong Wang, Jieying Wang, Zhao Zhang, Jiajia Li
The paper investigates reducing memory usage of n‑gram embedding tables in open‑weight language models by storing them in low precision. Experiments on nanochat, Llama, and GPT‑2 families show that four‑bit tables cut table memory by about 75% while retaining over 99% of the validation‑loss improvement. The study also demonstrates that re‑allocating the saved memory to larger tables can surpass the performance of full‑precision tables, and that similar savings are achievable in large models like DeepSeek‑V4.1‑Flash and Qwen3.8‑Flash‑Next with minimal impact on perplexity and downstream tasks.
By Stephen Chung, Wenyu Du
FastE is a training‑free, plug‑and‑play method that compresses token prefixes in large language model (LLM) embedding inference. It uses a shared fixed threshold on batch‑mean readout‑prefix alignment to decide when to compress and ranks prefix states by readout attention scores to keep the most important ones. Experiments on Qwen3‑Embedding models show that FastE can cut decoder‑backbone FLOPs by over 40% while preserving more than 99% of the original ranking quality across multiple benchmarks and tasks.
By Jinsong Shu, Jinyong Wen, Baokun Wang, Zhongle Xie, Lidan Shou, Weiqiang Wang, Gang Chen
arXiv:2605. 20689v2 Announce Type: replace-cross Abstract: High-dimensional language-model embeddings increase storage and search costs, while supervised compressors can overfit when relevance labels are scarce.
By Dongfang Zhao
arXiv:2510. 20535v2 Announce Type: replace-cross Abstract: Recent techniques such as retrieval-augmented generation or chain-of-thought reasoning have led to longer contexts and increased inference costs.
By Hippolyte Pilchen, Edouard Grave, Patrick P\'erez
arXiv:2610.31309v1 Announce Type: new
Abstract: Scaling embedding capacity alongside Transformer computation is an emerging trend in language models. This trend makes high compression rates increasin...
By Jiacheng Liu, Yaxin Luo, Xiaohan Zhao, Peng Sun, Xinyi Shang, Jiacheng Cui, Yonina C. Eldar, Zhiqiang Shen
The paper introduces zip2zip++, a method for inference‑time token compression that reduces token counts using a dynamic Lempel‑Ziv‑Welch (LZW) codec while preserving language model quality. It identifies four problems with prior LZW‑based compression—embedding conflict, improper RoPE application, harmful digit merges, and segmentation ambiguity—and proposes solutions: separate input/output embeddings, size‑aware RoPE, and a selective‑merge LZW codec. Experiments on four models show that these changes halve the quality gap in perplexity and reasoning accuracy compared to the uncompressed baseline, achieving similar compression ratios.
By Saibo Geng, Andrea Gentilini, Xinxian Ma, Nathan Ranchin, Yunzhen Yao, Robert West
The paper introduces COSA-GS, a new compression method for 3D Gaussian Splatting that avoids spatial aggregation by using anchor-wise causal factorization. It builds a compact learnable anchor latent from geometry context and fuses it with the geometry context to create an anchor context for attribute coding, employing only linear transformations and activations. The method is trained with rate–distortion optimization, adaptive Gaussian pruning, and quantization-aware training to ensure bit‑exact entropy decoding across platforms, achieving state‑of‑the‑art compression performance with fast, consistent cross‑platform decoding.
By Pengpeng Yu, Yueru Chen, Fei Song, Tai Qin, Qi Zhang, Jing Wang, Yulan Guo
arXiv:2607. 24176v1 Announce Type: cross Abstract: Compressed short-text generators can fail in two different places: the codec may discard information before generation starts, or the latent generator may produce weak codes.
By Alexey Gavrilov, Alan-Barsag Gazzaev, Sergey Muravyov
arXiv:2606. 09659v1 Announce Type: cross Abstract: Long-context language model inference is bottlenecked by memory, as the KV cache grows with context length.
By Ang Li, Sean McLeish, Haozhe Chen, Nimit Kalra, Zaiqian Chen, Artem Gazizov, Venkata Anoop Suhas Kumar Morisetty, Bhavya Kailkhura, Harshitha Menon, Zhuang Liu, Brian R. Bartoldson, Tom Goldstein, Sanae Lotfi, Micah Goldblum, Pavel Izmailov