The paper introduces LOHA, a context layout that compresses older tool observations into soft tokens while keeping the agent’s own turns and the last K observations in plain text, and ACD, a training method that distills full‑text predictions into this latent representation while anchoring behavior on plain text. This approach reduces context per call by up to 57% without significant loss in resolve rates, and improves instance throughput in single‑GPU serving. Experiments on SWE‑bench Verified show that K=3 yields a 43–57% compression with only modest performance impact, while larger windows favor task performance over compression.
By Zhensheng Zou (Peking University), Guoqing Wang (Peking University), Dan Hao (Peking University)
arXiv:2609.22114v1 Announce Type: new
Abstract: Context compression is widely proposed as a way to cut the token bill of LLM coding agents, and public benchmarks report that aggressive compression pr...
By Luzhuo Chen, Jiayu Shi
arXiv:2606. 10932v1 Announce Type: cross Abstract: We present Density Field State Space Models (DF-SSM), a framework for compressing SSMs to a 1-bit scaffold with int8 low-rank correction.
By Chirag Shinde
The paper introduces the Tri‑Metric Router, a deterministic, training‑free policy that chooses among Raw, Neural, and Lexical pipelines for retrieval‑augmented generation on commodity GPUs. It uses three CPU‑side signals—spatial complexity, syntactic density, and type‑token ratio—to balance VRAM headroom and latency, calibrated on LongBench qasper. The method eliminates out‑of‑memory failures and improves alignment and F1 scores compared to always‑on lexical compression without extra VRAM or training costs.
By Saipraveen Vabbilisetty, Ajay Kumar Boddepalli, Deep Narayan Mishra, Shashank Kapadia, Haoan Wang, Anupriya Sharma
arXiv:2609.21281v1 Announce Type: cross
Abstract: Embedding-based retrieval on user-generated content at the trillion-document scale exposes a sharp conflict between two production demands: deep, exp...
By Hao Fu, Jichao Sun, Baiting Zhu, Qiaoling Liu, Yan Shi, Cheng Lu, Liu Liu, Yubo Wang, Xin Yao, Xiangyu Niu, Xu Dong, Wenhan Lyu, Chiyao Shen, Yinjie Huang, Minglei Chen, Shuai Ding, Li Fan, Xiao Kong
arXiv:2606. 06302v2 Announce Type: replace Abstract: Multi-turn LLM serving accumulates dialogue history whose Key-Value (KV) cache grows with every turn and every user, quickly exceeding the model weights themselves and making memory -- not compute -- the binding constraint on throughput.
By Hyungmin Kim, Minsoo Kim, Hongseok Kim, Jungwook Choi
arXiv:2608. 03796v1 Announce Type: cross Abstract: Small language models are often the only option for deployment under tight latency, cost, and on-premises constraints, but they are rarely trained from scratch: a compressed model is usually recovered through knowledge distillation (KD).
By Bakbergen Ryskulov, Iker Garc\'ia-Ferrero, David Montero, David Jansen, Ali Hashemi, Jezabel R. Garcia, Antonio Tiene, Rom\'an Or\'us
arXiv:2610.34582v1 Announce Type: new
Abstract: We describe HobbyLM, a family of small sparse Mixture-of-Experts (MoE) models trained from scratch with a few hundred to a few thousand dollars of rent...
By Harish Santhanalakshmi Ganesan, Hariharan M. S., Prabhurajhan R
arXiv:2606. 31145v2 Announce Type: replace Abstract: Large language models increasingly operate over long contexts, where the KV cache grows linearly with sequence length and dominates memory during decoding.
By Amirhossein Abaskohi, Giuseppe Carenini, Peter West, Yuhang He
arXiv:2606. 26666v1 Announce Type: new Abstract: Autoregressive large language model (LLM) serving is increasingly limited by key-value (KV) cache movement rather than dense matrix multiplication.
By Muhammad Ahmed
Minima-KV introduces a retention‑preserving hierarchy for mixed‑format paged attention that keeps recent and protected anchor pages in FP8 while older pages are compressed into packed TQ3, allowing every live‑request page to remain addressable. The approach uses format‑specific kernels and a globally normalized online‑softmax merge to compute partial attention states, enabling direct heterogeneous decoding without a dense shadow cache. Experiments on Qwen3.6‑27B on a 96‑GB NVIDIA RTX PRO 6000 Blackwell GPU show 3.50× compression over BF16 and 1.75× over FP8, with minimal impact on performance across long‑context benchmarks.
By Sergii Kozyrev (Minima AI, Inc), Davyd Maiboroda (Minima AI, Inc)
arXiv:2609.39096v1 Announce Type: new
Abstract: Autoregressive video diffusion supports streaming generation and interactive control, but its KV cache grows with the generated history. Existing compr...
By Zeqi Xiao, Qingle Liu, Kaiwen Zhang, Yifan Zhou, Zihan Ding, Xingang Pan