arXiv Machine Learning
ARM: Attention with Routed-Memory for Learnable Sparse Control
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Machine Learning
Aug 14
MARCH: Scaling Recurrent Memory with Content-Routed State Anchors
arXiv:2608. 12435v1 Announce Type: new Abstract: Transformers owe much of their strong long-context retrieval capability to a token-level memory that grows with context length.
arXiv AI
Jul 7
Kwai Summary Attention Technical Report
arXiv:2604. 24432v2 Announce Type: replace-cross Abstract: Long-context ability, has become one of the most important iteration direction of next-generation Large Language Models, particularly in semantic understanding/reasoning, code agentic intelligence and recommendation system.
arXiv AI
Jul 29
Raven: High-Recall Sequence Modeling with Sparse Memory Routing
arXiv:2607. 25357v1 Announce Type: cross Abstract: Long-context recall in linear-time sequence models highlights a tradeoff in how they write to memory.
arXiv AI
Jul 29
GLIDE: Guided Layerwise Hybrid Attention for Efficient LLM Inference
arXiv:2607. 24788v1 Announce Type: new Abstract: As Large Language Models scale to increasingly long contexts, the memory I/O and computational overhead of the Key-Value (KV) cache during decoding emerges as the primary throughput bottleneck.
arXiv Machine Learning
Sep 24
Attention Routing Stabilizes Early: Working-Set Inference for Recurrent Language Models
The paper investigates how attention dynamics evolve across recurrent depth in language models, finding that attention support stabilizes early while hidden states and outputs take longer. It proposes WISE, a training‑free method that uses full attention in early steps and then reuses the discovered sparse working set for later steps, preserving performance on multi‑hop QA tasks. Experiments show that WISE maintains quality up to 2K context, offers measurable speedups, and highlights the importance of recurrent discovery of attention support.
arXiv Machine Learning
Jul 7
Learning When to Attend: Conditional Memory Access for Long-Context LLMs
arXiv:2603. 17484v2 Announce Type: replace-cross Abstract: Language models struggle to generalize beyond pretraining context lengths, limiting long-horizon reasoning and retrieval.