The Flow has not summarised this story yet — read it at Hugging Face Trending Papers.
arXiv:2610.33025v1 Announce Type: cross Abstract: Letting a language model take gradient steps on the text it is reading, known as dynamic evaluation, lowers its loss on the rest of the document. Its...
The paper introduces Declarative Attention (DA), a protocol that lets language models explicitly declare which parts of their context to focus on during generation. By partitioning decoding into full-context, region-specific, and recent-output-only modes, the inference engine can skip large portions of the KV cache, dramatically reducing attended tokens. Experiments on 15 long-context tasks with off-the-shelf models show significant savings (52.0% and 31.1% reductions) with only modest accuracy drops that diminish as model size increases.
arXiv:2607. 01002v1 Announce Type: cross Abstract: In long-context use, large language models frequently synthesize answers from the meaning of a relevant context span rather than literally copy-pasting them.
arXiv:2609.37879v1 Announce Type: cross Abstract: How many tokens from its context does a language model actually use, and what determines that number? We study this question through self-attention....
The paper investigates how the dimension of attention heads influences long‑context retrieval in language models. Experiments show that while pretraining loss and short‑context tasks are largely unaffected, wider heads significantly improve retrieval performance at fixed model sizes. Ablation studies indicate that this benefit persists even when attention weights are frozen, suggesting that wider heads enhance adaptation of other model components such as embeddings and MLPs.
arXiv:2608. 03297v1 Announce Type: new Abstract: A standard claim in the literature on retrieval-augmented and memory-augmented language models is that shorter context is better when the relevant information is preserved.