The paper introduces a family of adapters that enhance language model reasoning by adding selective state-space control at token and context levels. The token-level MaLoRA makes the adapter’s scaling factor dynamic and recurrent, improving over static low‑rank adaptation. The context-level MaRA tracks cross‑segment reasoning state and retrieves relevant segments, outperforming an eight‑billion‑parameter dense retriever and boosting reasoning accuracy by an average of +6.4 F1 over LoRA.
By Atahan Dokme, Larry Heck
arXiv:2608. 12435v1 Announce Type: new Abstract: Transformers owe much of their strong long-context retrieval capability to a token-level memory that grows with context length.
By Ming Zhang, Kaisen Yang, Shu Yu, Ermo Hua, Ning Ding, Xia Hu, Bowen Zhou, Chaochao Lu, Youbang Sun
The study investigates how the position of evidence documents relative to a prompt affects long‑context question answering across different sequence mixers—Transformer, state‑space, and hybrid Mamba models. Results show that Transformers exhibit a beginning‑position advantage that state‑space models lack, while both families tend to favor evidence near the end. Additional experiments reveal that prompt design, corpus choice, and model capability influence performance, and that position bias remains decodable across architectures even when response curves differ.
By Aman Behera, Namit Solanki, Mehul Anand
The paper introduces Declarative Attention (DA), a protocol that lets language models explicitly declare which parts of their context to focus on during generation. By partitioning decoding into full-context, region-specific, and recent-output-only modes, the inference engine can skip large portions of the KV cache, dramatically reducing attended tokens. Experiments on 15 long-context tasks with off-the-shelf models show significant savings (52.0% and 31.1% reductions) with only modest accuracy drops that diminish as model size increases.
By Namgyu Ho, Huzama Ahmad, Woosung Koh, Se-Young Yun, Tal Schuster, Cicero Nogueira dos Santos
arXiv:2606.16661v2 Announce Type: replace-cross
Abstract: Fixed-length chunking in Retrieval-Augmented Generation (RAG) often leads to boundary fragmentation, where critical evidence is split across...
By Nathana\"el Langlois
arXiv:2610.08463v1 Announce Type: cross
Abstract: Long-context inference and Retrieval-Augmented Generation (RAG) handle evidence selection at vastly different scales, from a single long prompt to an...
By Edan Kinderman, Elad Hoffer, Yochai Blau, Brian Chmiel, Ron Banner, Daniel Soudry, Boris Ginsburg