The Flow has not summarised this story yet — read it at arXiv Computation and Language.
The paper introduces Declarative Attention (DA), a protocol that lets language models explicitly declare which parts of their context to focus on during generation. By partitioning decoding into full-context, region-specific, and recent-output-only modes, the inference engine can skip large portions of the KV cache, dramatically reducing attended tokens. Experiments on 15 long-context tasks with off-the-shelf models show significant savings (52.0% and 31.1% reductions) with only modest accuracy drops that diminish as model size increases.
The paper compares two low‑budget methods for converting a 30B Mixture‑of‑Experts autoregressive language model into a diffusion language model. One method updates a subset of the model’s weights in‑place, while the other freezes the context tower and conditions on a frozen causal copy via cross‑attention. With only 1B training tokens, the frozen‑tower approach achieves a HumanEval pass@10 score of 71.60 versus 6.19 for the in‑place method, and retains 95% of the parent’s GSM8K and 99% of its MMLU‑Pro performance.
The paper introduces LRE (Learned Relevance Eviction), a lightweight, CPU‑only, language‑model‑free scorer that learns which parts of an agent’s interaction history are task‑critical and preserves them verbatim. In experiments, LRE matches or surpasses baseline eviction policies on accuracy‑cost trade‑offs, recovers 93% of full‑history accuracy, reduces worst‑case prompt size by 52%, and outperforms dense and token‑pruning encoders in conversational memory while being 295–1569× smaller. The method also achieves superior budgeted answer quality on LoCoMo reading and can be trained annotation‑free, recovering 95% of supervised scorer performance.
arXiv:2610.14754v1 Announce Type: new Abstract: Hybrid language models interleave Gated DeltaNet (GDN) layers, whose recurrent memory has a fixed size, with full-attention layers, whose key-value (KV...
Delta N-gram introduces a forward‑only write path into a language model’s native hashed N‑gram table, preserving the pretrained table while adding evidence through a sparse patch Δ. The method uses a 4.1 M‑parameter writer that emits gated targets per event and applies a block‑averaged delta rule to keep each row a convex combination of its old value and the new targets. Experiments on a 0.6 B prototype and Qwen3.8‑Flash‑Next show that the delta writer maintains more document‑specific memory and reduces drift compared to online gradient writes, achieving stable performance over thousands of writes. whyItMatters":"The approach allows continual updates to a language model’s memory without altering its pretrained parameters, enabling efficient, stable, and document‑specific knowledge retention during inference."
arXiv:2606. 06712v1 Announce Type: cross Abstract: We study the transformation of autoregressive models (ARLMs) into diffusion language models (DLMs).