arXiv:2503. 06211v3 Announce Type: replace-cross Abstract: Text-pretrained language models (LMs) encode rich world knowledge, but adapting them to process and generate perceptual modalities such as audio and images while effectively leveraging that knowledge remains challenging.
By Santiago Cuervo, Adel Moumen, Yanis Labrak, Sameer Khurana, Antoine Laurent, Mickael Rouvier, Phil Woodland, Ricard Marxer
arXiv:2608. 15022v1 Announce Type: new Abstract: Language models hold latent quantities in a form they can report on, and more of a quantity is present in that form when the task requires reusing it flexibly.
By Parsa Mazaheri
The paper proposes Bypass Observation, a non‑intrusive layer‑wise readout architecture that attaches read‑only observation heads to selected Transformer layers without feeding their outputs back into the backbone. Three variants are explored: a shared language‑model head across layers, layer‑specific heads, and a layer‑ or step‑adaptive head. The authors provide a closed‑form overhead estimate (≈ V/(12d)) and discuss ways to reduce cost, while distinguishing bypass chain‑of‑thought from conventional chain‑of‑thought and outlining potential applications to looped and recurrent‑depth Transformers.
By Haibin Tong, Jiang Yu
arXiv:2603.18908v5 Announce Type: replace
Abstract: Independently trained language models often learn compatible late-stage representations, despite differences in training objectives, architectures,...
By Matt Gorbett, Suman Jana
The paper introduces Declarative Attention (DA), a protocol that lets language models explicitly declare which parts of their context to focus on during generation. By partitioning decoding into full-context, region-specific, and recent-output-only modes, the inference engine can skip large portions of the KV cache, dramatically reducing attended tokens. Experiments on 15 long-context tasks with off-the-shelf models show significant savings (52.0% and 31.1% reductions) with only modest accuracy drops that diminish as model size increases.
By Namgyu Ho, Huzama Ahmad, Woosung Koh, Se-Young Yun, Tal Schuster, Cicero Nogueira dos Santos
The paper presents a mean‑field analysis of attention in language models, defining an average attention kernel that propagates representations layer by layer. When conditioned on a whole corpus, the kernel predicts the average evolution of representation geometry; when conditioned on a single context, it predicts the expected geometry for that context. The difference between actual attention and the mean‑field prediction—called the mean‑field deviation—captures context‑specific computation, revealing how models diverge from average behavior during training and in few‑shot tasks.
By Micah Adler, John W. Byers, Mark Crovella