LooM (Loop‑Invariant Memory) is a technique for causal encoder‑decoder language models that loops only the decoder layers after the first, using a global memory written once per token and publishing key‑value entries only on the first visit. This design prevents the cache from growing with the loop count, allows independent looping of layers to provide any budget from 3 to 21 layer executions per token, and enables a single model trained on randomly sampled budgets to cover all budgets without a router. Experiments show that with enough loops, LooM outperforms a single‑pass baseline on language modeling, zero‑shot accuracy, and long‑context perplexity while using the same cache size.
By Lele Yang, Muxi Diao, Zhonghao Yan, Jiankang Chen, Bin Wen, Tingting Gao, Wentao Zhang, Kongming Liang, Zhanyu Ma, Weiran Xu
Looped Transformers increase effective depth by re-executing shared layers, but three limitations hinder their adoption: memory grows with the loop count, compute can only be adjusted in whole-block s...
Looped transformers are a parameter-efficient alternative to standard transformers for language modeling. By reapplying the same layers across multiple loop steps, recent looped language models match...
The paper introduces BeNice!, a two‑stage approach that enables KV‑shareability in looped neural architectures. First, stochastic depth during pretraining drives representations toward a common limit across recurrences; second, a post‑training broadcast of the final recurrence’s KV‑cache to earlier recurrences trains the model to produce compatible residuals while preserving quality. BeNice! achieves KV‑shareability without sacrificing validation perplexity and can be combined with other memory‑efficient techniques, yielding significant gains in cache hit rates, throughput, and latency in low‑memory, long‑context inference.
By Hayden Prairie, Jacob M. Ryan, Taylor Berg-Kirkpatrick, Daniel Y. Fu
GroupKV is a lightweight hierarchical KV cache management system designed for long‑context diffusion large language model (dLLM) inference. It partitions the context into contiguous groups and uses coarse‑to‑fine sparse selection, cross‑layer consistency for predictive prefetching, and a staleness correction mechanism to keep the cache coherent amid dynamic KV updates. The approach also incorporates streaming prefill to lower peak memory usage, achieving up to 48× longer serviceable context, 3.73× faster inference in offload‑based settings, and competitive task accuracy.
By Jinhao Wang, Zhexin Hu, Kangjie Zhou, Xin Zhou, Fangfang Liu
arXiv:2607. 15456v1 Announce Type: new Abstract: Looped, weight-tied Transformers reduce parameters by reusing a block, but decoding still stores a separate K/V cache for every recurrence step.
By James O' Neill, Fergal Reid