The paper introduces Internalizer, a portable Context-to-Parameter Mapping hypernetwork that generates document‑specific LoRA adapters for the 284B‑parameter DeepSeek v4 Flash model. Unlike previous work limited to models up to 14B parameters, Internalizer’s architecture places most parameters in a model‑agnostic trunk, allowing cheap training on small models before porting to the large one. On unseen 4096‑token documents, the generated adapters achieve 84.9% top‑1 and 97.8% top‑5 teacher‑forced accuracy, far surpassing the base model’s 63.4% and 83.5% respectively, using only a three‑word instruction in the context window.
By Peter Devine, Nick Ryan, Benjamin Sirb, Alex Chiocchi
Select‑TTT introduces a query‑conditioned rule for In‑Place Test‑Time Training (TTT) that scales the write strength of each context chunk based on its relevance to the user query. By using the model’s frozen output head to compute relevance scores and applying a temperature‑scaled softmax as learning‑rate multipliers, the method preserves the efficient, chunk‑wise computation of In‑Place TTT while preventing irrelevant content from diluting fast‑weight memory. Experiments on two benchmarks and two backbones show consistent improvements in long‑context question answering.
By Yue Wu, Yang Zhang, Weinan Gan, Chongming Gao, Xinsen Guo, Xiaoyuan Li, Fuli Feng
The paper introduces Declarative Attention (DA), a protocol that lets language models explicitly declare which parts of their context to focus on during generation. By partitioning decoding into full-context, region-specific, and recent-output-only modes, the inference engine can skip large portions of the KV cache, dramatically reducing attended tokens. Experiments on 15 long-context tasks with off-the-shelf models show significant savings (52.0% and 31.1% reductions) with only modest accuracy drops that diminish as model size increases.
By Namgyu Ho, Huzama Ahmad, Woosung Koh, Se-Young Yun, Tal Schuster, Cicero Nogueira dos Santos
arXiv:2602. 06358v3 Announce Type: replace-cross Abstract: We propose SHINE (Scalable Hyper In-context NEtwork), a scalable hypernetwork that can map diverse meaningful contexts into high-quality LoRA adapters for large language models (LLMs).
By Yewei Liu, Xiyuan Wang, Yansheng Mao, Yoav Gelbery, Haggai Maron, Muhan Zhang
arXiv:2608.17050v3 Announce Type: replace-cross
Abstract: Methods for improving knowledge use in large language models typically fall into two regimes. Non-parametric retrieval offers flexible access...
By Mingyuan Li, Guangsheng Yu, Xu Wang, Shaoxiong Ji
arXiv:2610.33006v1 Announce Type: new
Abstract: Transformer-based large language models (LLMs) impose substantial computation and memory costs as context sizes increase, motivating methods that compr...
By Marc Serra Ortega, Andrey Barsky, Carlos Boned Riera, Tom\'as Ockier, Dimosthenis Karatzas
Split-KV introduces a context‑parallelism technique for training diffusion drafters, keeping key/value shards local and limiting per‑layer attention communication to query‑sized tensors. This approach exploits the asymmetry between short draft queries and long contexts, enabling efficient load balancing via a cost‑aware longest‑processing‑time‑first strategy. Experiments show that at a CP size of 64, Split‑KV achieves 85–91 % weak‑scaling efficiency and up to 3.67× speedup over the best baseline, with less than 1 % load imbalance.
By Geng Zhang, Fangzhao Zhang, Cong Xie, Tiantian Fan, Yang You
arXiv:2603. 01097v3 Announce Type: replace Abstract: Continuous knowledge updating for pre-trained large language models (LLMs) is increasingly necessary yet remains challenging.
By Seungju Back, Dongwoo Lee, Naun Kang, Taehee Lee, S. K. Hong, Youngjune Gwon, Sungjin Ahn
arXiv:2610.19210v1 Announce Type: new
Abstract: Transformers are efficient to pre-train at scale because attention permits parallel computation across sequence positions, but long-context inference r...
By Sarah Pan, Ali Behrouz
The paper investigates how Engram-style hashed memory can be transferred between different language model backbones. By freezing a memory table trained on a source model and attaching it to a target model with only a lightweight reader, the authors find that both the memory content and correct addressing are important, but the reader must be aligned to the target to make the memory useful. In question‑answering experiments, a dual‑layer, four‑branch reader nearly matches same‑model performance, and when the reader interface is directly compatible, the frozen memory alone provides substantial benefit, with optional reader adaptation offering further gains.
By Mingyuan Li, Guangsheng Yu, Xu Wang, Shaoxiong Ji
arXiv:2609.07093v2 Announce Type: replace
Abstract: Retrieval-augmented generation (RAG) enables large language models (LLMs) to answer questions by accessing external knowledge and has been widely a...
By Yifan Wang, Xinkui Lin, Yongxiu Xu, Shen Gao, Ruochen Yang, Kun Huang, Yubin Wang, Jie Wu, Wei Liu, Jian Luan, Hongbo Xu, Shuo Shang
arXiv:2510. 20535v2 Announce Type: replace-cross Abstract: Recent techniques such as retrieval-augmented generation or chain-of-thought reasoning have led to longer contexts and increased inference costs.
By Hippolyte Pilchen, Edouard Grave, Patrick P\'erez