LooM: Loop-Invariant Memory for Causal Encoder-Decoder Language Models
Read the original on Hugging Face Trending Papers →The Flow has not summarised this story yet — read it at Hugging Face Trending Papers.
The Flow has not summarised this story yet — read it at Hugging Face Trending Papers.
LooM (Loop‑Invariant Memory) is a technique for causal encoder‑decoder language models that loops only the decoder layers after the first, using a global memory written once per token and publishing key‑value entries only on the first visit. This design prevents the cache from growing with the loop count, allows independent looping of layers to provide any budget from 3 to 21 layer executions per token, and enables a single model trained on randomly sampled budgets to cover all budgets without a router. Experiments show that with enough loops, LooM outperforms a single‑pass baseline on language modeling, zero‑shot accuracy, and long‑context perplexity while using the same cache size.
The paper introduces a looped causal encoder-decoder (CED) architecture that increases model depth without enlarging the key-value (KV) cache across layers and rounds. By placing recurrence in the encoder and allowing sparse encoder layers to generate global KV, the design achieves language‑modeling gains while maintaining long‑context scaling. Experiments show that the Looped CED matches the memory footprint of an unlooped CED and recovers most of a looped Transformer’s performance with significantly less KV, enabling higher throughput on large‑token prompts.
arXiv:2606. 06467v1 Announce Type: cross Abstract: Long-context inference in modern LLMs is increasingly constrained by decoding efficiency, especially in reasoning-heavy settings where models generate long intermediate chains of thought.
Long-context language model inference is bottlenecked by memory, as the KV cache grows with context length. Recent techniques to compress the KV cache fall short: they either degrade model quality substantially or require considerable time and compute to compress a single long prompt.
Decoder-only Transformer language models cache keys and values (KV) to reuse past computation during generation. Sharing KV across layers saves storage but reduces the diversity of representations ava...
arXiv:2606. 09514v1 Announce Type: new Abstract: Large language models (LLMs) incur high inference cost due to their depth and parameter scale.