arXiv Machine Learning September 16, 2026 By Micah Adler, John W. Byers, Mark Crovella

Attention Mean Fields Predict Average Representation Dynamics and Reveal Context-Specific Computation

Read the original on arXiv Machine Learning →

The paper presents a mean‑field analysis of attention in language models, defining an average attention kernel that propagates representations layer by layer. When conditioned on a whole corpus, the kernel predicts the average evolution of representation geometry; when conditioned on a single context, it predicts the expected geometry for that context. The difference between actual attention and the mean‑field prediction—called the mean‑field deviation—captures context‑specific computation, revealing how models diverge from average behavior during training and in few‑shot tasks.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv Machine Learning

Sep 1

Context Staircase: Signature-Aligned Dynamics of Token Embeddings under Small Initialization

arXiv:2608.30315v1 Announce Type: new Abstract: Token embeddings are the basic representational units that connect discrete tokens with continuous computation in language models. Although modern lang...

By Junjie Yao, Liangkai Hang, Zhi-Qin John Xu

llms rag safety

arXiv AI

Sep 3

Language Models Can Control Their Own Attention

The paper introduces Declarative Attention (DA), a protocol that lets language models explicitly declare which parts of their context to focus on during generation. By partitioning decoding into full-context, region-specific, and recent-output-only modes, the inference engine can skip large portions of the KV cache, dramatically reducing attended tokens. Experiments on 15 long-context tasks with off-the-shelf models show significant savings (52.0% and 31.1% reductions) with only modest accuracy drops that diminish as model size increases.

By Namgyu Ho, Huzama Ahmad, Woosung Koh, Se-Young Yun, Tal Schuster, Cicero Nogueira dos Santos

llms efficiency

arXiv AI

Jun 9

Contribution Weights: A Geometrical Analysis of Self-Attention Transformers

arXiv:2606. 07604v1 Announce Type: cross Abstract: Analyzing attention weights has become a standard approach for interpreting the information flow of Large Language Models (LLMs).

By Harry Jake Cunningham, Nicola Muca Cirone

llms safety

Hugging Face Trending Papers

Aug 4

Attention is Case-Sensitive

In human visual perception, uppercase lettering serves as a natural salience cue that captures attention within lowercase text. In this paper, we present a systematic empirical characterization study revealing that Large Language Models (LLMs) exhibit an analogous property: letter casing modulates internal attention allocation.

llms nlp fine-tuning multimodal

arXiv AI

Aug 12

Attention-Path Fragility as an Uncertainty Signal in Large Language Models

arXiv:2608. 11138v1 Announce Type: cross Abstract: We propose that a model's uncertainty about a token is reflected not only in the breadth of its output distribution but also in whether a confident prediction is \emph{fragile} under perturbation of its attention pathways.

By Minsoo Kim, Sungyoung Ji, Kisung Moon, Ilyong Yoon

llms benchmarks

arXiv Machine Learning

Aug 5

Attention is Case-Sensitive

arXiv:2608. 03711v1 Announce Type: cross Abstract: In human visual perception, uppercase lettering serves as a natural salience cue that captures attention within lowercase text.

By Maximilian Dillitzer, Tin Stribor Sohn, Jason J. Corso, Michael Auerbach

llms nlp fine-tuning multimodal

One email a morning, machine-written

Related stories

Context Staircase: Signature-Aligned Dynamics of Token Embeddings under Small Initialization

Language Models Can Control Their Own Attention

Contribution Weights: A Geometrical Analysis of Self-Attention Transformers

Attention is Case-Sensitive

Attention-Path Fragility as an Uncertainty Signal in Large Language Models

Attention is Case-Sensitive