arXiv:2609. 16391v1 Announce Type: cross Abstract: Weight-only post-training quantization is the cheapest way to shrink a retrieval embedder, and the received advice for applying it -- protect the embedding table, allocate bits by module sensitivity, prefer a ranking-aware objective over weight reconstruction -- was carried into LLM quantization largely intact.
By Hyojung Han
arXiv:2607. 07144v1 Announce Type: new Abstract: The key-value (KV) cache dominates the memory cost of long-context autoregressive inference, and a growing body of work compresses it through quantization, eviction, or offloading.
By Vladimir Gusev
BellowsKV introduces a training‑free approach that maps the hidden states of multiple adjacent layers in a large language model to a single shared latent representation per token, allowing each layer to retrieve its own keys and values via a layer‑specific projection. The method retains the latent at full dimension and offers two compression strategies: BellowsKV‑Quant, which stores the latent at low precision, and BellowsKV‑Evict, which makes a shared eviction decision per block. Experiments on Llama‑3.1‑8B, Mistral‑7B, and Qwen2.5‑7B show that BellowsKV‑Quant preserves about 99% of full‑cache accuracy at 3.6× compression and 97‑98% at 13‑14×, outperforming existing cross‑layer methods that drop to at most 84% accuracy at 16× compression.
By Jiamu Zhang, Hanjie Chen
The paper evaluates FlyHash, a locality‑sensitive hashing scheme inspired by the fruit fly’s olfactory circuit, for use in large language model (LLM) agent memory retrieval. It demonstrates that FlyHash maintains a quality advantage over SimHash when bits are matched, and that this advantage persists with real agent‑memory relevance labels. However, FlyHash’s larger, slower codes and the dominance of dense‑vector indexing (HNSW) limit its practical benefits in terms of memory and latency savings.
By Mohan Bheemadolu, Chrisogonas O. Odhiambo
arXiv:2510. 04127v2 Announce Type: replace-cross Abstract: Approximate nearest neighbour (ANN) search underpins large-scale retrieval, increasingly within the retrieval-augmented generation pipelines that ground large language models, yet the methods that address it have multiplied across communities until they are seldom read as a single field.
By Sean Moran
arXiv:2607. 02893v1 Announce Type: new Abstract: Low-bit quantization shrinks language models but treats precision as a single global hyper-parameter: every weight uses the same bit-width.
By Hamish Ogilvy
MRVQ (Matryoshka Residual Vector Quantization) is a single post‑hoc residual quantizer designed for frozen embeddings that can be truncated to support any combination of embedding dimension and code rate. By dropping residual stages or embedding coordinates, MRVQ provides a single resident artifact that serves all evaluated (dimension, rate) pairs, achieving the lowest memory usage among tested designs. Although it uses 17.8–22.0× less memory than three separately trained QINCo2 indices and 1.89–2.02× less than a lean shared‑model steelman, it trades off a small drop in nDCG@10 on FiQA compared to per‑rate QINCo2, yet still outperforms PQ, OPQ, and AdANNS‑OPQ at matched code size.
By Sean Culatana, Shang-En Huang, Kang Li
arXiv:2610.27699v1 Announce Type: new
Abstract: Low-bit quantization reduces the memory footprint of large language models, but preserving accuracy with efficient training remains challenging. Scalar...
By Xinyu Lei, Hong Huang, Jiebin Zhang, Hongyu Li, Jinlong Song, Decheng Wu, Guanghua Yu, Jianchen Zhu, Sujian Li
arXiv:2608.20873v2 Announce Type: new
Abstract: A deployed language model is a file that many things depend on - a benchmark report, a certification, a fleet of devices - and every way of teaching it...
By Zifeng Liu, Yaxin Lu, Xuanhan Wu, Zhiyong Du, Yiming Mao, Zhenhe Wang, Wenqi Shi, Zhengkun Jing, Linwei Liu
arXiv:2607. 08032v1 Announce Type: new Abstract: Large language models, and the agents built on them, spend an ever-growing share of their compute and memory on remembering: caching attention keys and values, carrying long prompts, maintaining recurrent state, and storing what happened in previous turns and sessions.
By Ashwin Gerard Colaco, Nada Lahjouji
arXiv:2610.33958v1 Announce Type: cross
Abstract: Coding agents are given embedding search and code-graph tools on the premise that a repository's structure is something the model lacks and the tool...
By Saurav Bellam (The University of Texas at Austin)
arXiv:2607. 01065v1 Announce Type: new Abstract: The deployment of Large Language Models (LLMs) with extended context windows is increasingly constrained by the linear growth of Key-Value (KV) cache memory.
By Soosung Kim, Minjae Park, Eui-Young Chung, Jaeyong Chung