arXiv AI
When Attention Closes: How LLMs Lose the Thread in Multi-Turn Interaction
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Jul 24
Attention Degradation, Function Token Anchoring, and the Limits of Attention-Based Intervention in Large Language Models
arXiv:2607. 20524v1 Announce Type: new Abstract: Mean cross-positional attention degradation is widely reported in transformer interpretability, yet whether it causally limits contextual retrieval remains untested.
arXiv Machine Learning
3d ago
LS-AR: Future-Predictive Latent Steering in Autoregressive LLMs
The paper introduces LS-AR, a dual‑channel architecture that separates continuous goal steering from discrete token decoding using FiLM conditioning. It employs a Static Goal Encoder for long‑term macro‑objective retention and a Dynamic State Tracker for recurrent latent updates. Experiments show LS‑AR achieves perfect target recall on long‑horizon retrieval tasks, improves throughput by ~35%, reduces VRAM usage by 52.8%, and maintains higher completion rates in Blocksworld planning under perturbations.
arXiv AI
Jul 7
Attending to Multimodal Generation One Token at a Time
arXiv:2607. 03738v1 Announce Type: cross Abstract: Multimodal large language models (MLLMs) generate responses autoregressively, integrating visual and linguistic information in an evolving context.
arXiv AI
Sep 3
Language Models Can Control Their Own Attention
The paper introduces Declarative Attention (DA), a protocol that lets language models explicitly declare which parts of their context to focus on during generation. By partitioning decoding into full-context, region-specific, and recent-output-only modes, the inference engine can skip large portions of the KV cache, dramatically reducing attended tokens. Experiments on 15 long-context tasks with off-the-shelf models show significant savings (52.0% and 31.1% reductions) with only modest accuracy drops that diminish as model size increases.
arXiv AI
Aug 11
ATMA: Long-Context Language Modeling via Polar Attention and Gated-Delta Compression Memory
arXiv:2606. 25156v3 Announce Type: replace-cross Abstract: Native length extrapolation remain a weakly solvable problem in language modeling due to trade-off balancing between exact retrieval fidelity, long-document likelihood, and inference efficiency.
arXiv Machine Learning
Jun 2
Resonant Context Anchoring: Decoupling Attention Routing and Signal Gain at Inference Time
arXiv:2606. 01923v1 Announce Type: cross Abstract: Large Language Models (LLMs) frequently exhibit "contextual disregard" when faced with input evidence that conflicts with their internal parametric memory, leading to persistent factual hallucinations.