arXiv Computation and Language

LooM: Loop-Invariant Memory for Causal Encoder-Decoder Language Models

LooM (Loop‑Invariant Memory) is a technique for causal encoder‑decoder language models that loops only the decoder layers after the first, using a global memory written once per token and publishing key‑value entries only on the first visit. This design prevents the cache from growing with the loop count, allows independent looping of layers to provide any budget from 3 to 21 layer executions per token, and enables a single model trained on randomly sampled budgets to cover all budgets without a router. Experiments show that with enough loops, LooM outperforms a single‑pass baseline on language modeling, zero‑shot accuracy, and long‑context perplexity while using the same cache size.

arXiv Machine Learning
6d ago

Looping Deeper Without Growing the KV Cache: Recurrent Depth for Causal Encoder-Decoders

The paper introduces a looped causal encoder-decoder (CED) architecture that increases model depth without enlarging the key-value (KV) cache across layers and rounds. By placing recurrence in the encoder and allowing sparse encoder layers to generate global KV, the design achieves language‑modeling gains while maintaining long‑context scaling. Experiments show that the Looped CED matches the memory footprint of an unlooped CED and recovers most of a looped Transformer’s performance with significantly less KV, enabling higher throughput on large‑token prompts.

By Jiaqi Leng, Minshen Zhang, Zhaofeng Sun, Yu-Yang Qian, Peng Zhao, Hao Zhang
arXiv AI
Oct 5

The Surprising Effectiveness of Shared Memory in Looped Transformers

The paper introduces a method for looped transformers that share a key‑value cache across recursions, reducing memory usage without sacrificing quality. Experiments show that the Looped Prediction Transformer (LPT) and its hybrid variant achieve lower perplexity on FineWeb‑Edu while using 76‑79% less context memory compared to standard transformers. Analysis reveals that shared memory develops distinct representations and serves as a gradient highway, enabling later recursions to focus on shared information.

By Giovanni Monea, Keshav Ramji, Yousef El-Kurdi, Luis A. Lastras, Yoav Artzi, Nathan Godey, Ram\'on Fernandez Astudillo
Hugging Face Trending Papers
Jun 8

End-to-End Context Compression at Scale

Long-context language model inference is bottlenecked by memory, as the KV cache grows with context length. Recent techniques to compress the KV cache fall short: they either degrade model quality substantially or require considerable time and compute to compress a single long prompt.

arXiv Machine Learning
Sep 17

GroupKV: Hierarchical KV Cache Management for Long-Context Diffusion LLM Inference

GroupKV is a lightweight hierarchical KV cache management system designed for long‑context diffusion large language model (dLLM) inference. It partitions the context into contiguous groups and uses coarse‑to‑fine sparse selection, cross‑layer consistency for predictive prefetching, and a staleness correction mechanism to keep the cache coherent amid dynamic KV updates. The approach also incorporates streaming prefill to lower peak memory usage, achieving up to 48× longer serviceable context, 3.73× faster inference in offload‑based settings, and competitive task accuracy.

By Jinhao Wang, Zhexin Hu, Kangjie Zhou, Xin Zhou, Fangfang Liu
arXiv Machine Learning
Sep 24

Shared Global KV with Layer-Specific Local History

The paper investigates how to combine shared global key‑value (KV) caches with layer‑specific local history in decoder‑only Transformer language models. By separating historical content from the input source used to form it, the authors show that adding local history can reduce held‑out test perplexity by about 1.4% compared to a current‑token local branch, while also demonstrating benefits in capacity, entry‑count, and training‑compute controls. Experiments on a 126M‑parameter model with 2K context reveal that local history remains valuable even when adjacent layers share local inputs, and that a sufficient suffix schedule can reduce upper‑layer construction work without losing cache completeness.

By Xinglang Xian
arXiv AI
Jun 9

End-to-End Context Compression at Scale

arXiv:2606. 09659v1 Announce Type: cross Abstract: Long-context language model inference is bottlenecked by memory, as the KV cache grows with context length.

By Ang Li, Sean McLeish, Haozhe Chen, Nimit Kalra, Zaiqian Chen, Artem Gazizov, Venkata Anoop Suhas Kumar Morisetty, Bhavya Kailkhura, Harshitha Menon, Zhuang Liu, Brian R. Bartoldson, Tom Goldstein, Sanae Lotfi, Micah Goldblum, Pavel Izmailov
arXiv AI
Jul 8

DepthWeave-KV: Token-Adaptive Cross-Layer Residual Factorization for Long-Context KV Cache Compression

arXiv:2607. 06523v1 Announce Type: new Abstract: Long-context language model inference is increasingly limited by the memory bandwidth and capacity required to store key-value caches, yet existing compression methods often apply uniform budgets across layers or tokens and degrade retrieval when lexical cues and semantic states require different preservation.

By Anna Cordoba, Adam Puente Tercero, Nerea Angulo Hijo, Mar Linares Tercero, Julia Barrientos, Ainhoa Miranda, Jesus Olivera