← Back to all news
arXiv AI By Xinhao Zhong, Mengxuan Wu, Chonghua Liao, Linxi Xie, Ruicheng Feng, Yanpei Guo, Hao Tan, Jiaxin Hong, Jason Li, Panpan Zhang, Ziheng Qin, Qinglin Lu, Yuzhang Shang, Tongliang Liu, Damian Borth, Haggai Maron, Yongxiang Hua, Zhixiang Wei, Jian Hu, Yunfei Wu, Xing Sun, Xin Li, Haoyu Cao, Zhangyang Wang, Kai Wang

Encode Once, Answer Many: Internalizing Context via Parametric Flow

Read the original on arXiv AI →

Encode Once, Answer Many: Internalizing Context via Parametric Flow proposes CLIK, a context-conditioned parameter generator that transforms a given context into LoRA parameters in a single forward pass, allowing a frozen base model to answer queries without re-reading the context. By internalizing knowledge into model weights, CLIK eliminates per-task optimization and test-time gradient updates, amortizing the cost of reading a document across multiple queries. Experiments show CLIK improves over a no-context base model and outperforms other hypernetwork generators, though it still lags behind models that read the context directly, and it generalizes to multi-turn dialogue without specialized training.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv AI
Oct 9

Internalizer: Portable Context-to-Parameter Mapping for Very Large Language Models

The paper introduces Internalizer, a portable Context-to-Parameter Mapping hypernetwork that generates document‑specific LoRA adapters for the 284B‑parameter DeepSeek v4 Flash model. Unlike previous work limited to models up to 14B parameters, Internalizer’s architecture places most parameters in a model‑agnostic trunk, allowing cheap training on small models before porting to the large one. On unseen 4096‑token documents, the generated adapters achieve 84.9% top‑1 and 97.8% top‑5 teacher‑forced accuracy, far surpassing the base model’s 63.4% and 83.5% respectively, using only a three‑word instruction in the context window.

By Peter Devine, Nick Ryan, Benjamin Sirb, Alex Chiocchi
llmsfine-tuningbenchmarks
More like this →
arXiv AI
Oct 28

Select-TTT: Query-Conditioned Selective Fast Weight Updates for In-Place Test-Time Training

Select‑TTT introduces a query‑conditioned rule for In‑Place Test‑Time Training (TTT) that scales the write strength of each context chunk based on its relevance to the user query. By using the model’s frozen output head to compute relevance scores and applying a temperature‑scaled softmax as learning‑rate multipliers, the method preserves the efficient, chunk‑wise computation of In‑Place TTT while preventing irrelevant content from diluting fast‑weight memory. Experiments on two benchmarks and two backbones show consistent improvements in long‑context question answering.

By Yue Wu, Yang Zhang, Weinan Gan, Chongming Gao, Xinsen Guo, Xiaoyuan Li, Fuli Feng
llmsagentsnlpbenchmarks
More like this →
arXiv AI
Sep 3

Language Models Can Control Their Own Attention

The paper introduces Declarative Attention (DA), a protocol that lets language models explicitly declare which parts of their context to focus on during generation. By partitioning decoding into full-context, region-specific, and recent-output-only modes, the inference engine can skip large portions of the KV cache, dramatically reducing attended tokens. Experiments on 15 long-context tasks with off-the-shelf models show significant savings (52.0% and 31.1% reductions) with only modest accuracy drops that diminish as model size increases.

By Namgyu Ho, Huzama Ahmad, Woosung Koh, Se-Young Yun, Tal Schuster, Cicero Nogueira dos Santos
llmsefficiency
More like this →