arXiv AI October 31, 2026 By Giacomo Greco

Emergence of the transformer architecture as the score of entropic laws of random measures

The paper demonstrates that the transformer architecture naturally arises as the score of last‑token laws in a statistical model of language where sentences are treated as empirical measures of tokens. By applying the maximum‑entropy principle to a prior that assumes no inherent sentence structure, the authors derive a statistic capturing token interactions via Luce’s choice axiom, leading to the softmax choice rule and attention mechanisms. The resulting framework explains key transformer components—query‑key and value matrices as Lagrange multipliers, causal masking, multi‑head attention, and residual blocks—as consequences of entropic optimization.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

llms

arXiv AI

Aug 7

Beyond Sequence Order: Syntax-Informed Positional Embeddings for Transformers

arXiv:2608. 06111v1 Announce Type: cross Abstract: Positional embeddings (PE) in Transformers encode token distance and order but are largely agnostic to \textit{syntactic structure}.

By Haris Riaz, Hyungji Kim, Mihai Surdeanu

llms rag benchmarks

arXiv Machine Learning

Sep 16

Attention Mean Fields Predict Average Representation Dynamics and Reveal Context-Specific Computation

The paper presents a mean‑field analysis of attention in language models, defining an average attention kernel that propagates representations layer by layer. When conditioned on a whole corpus, the kernel predicts the average evolution of representation geometry; when conditioned on a single context, it predicts the expected geometry for that context. The difference between actual attention and the mean‑field prediction—called the mean‑field deviation—captures context‑specific computation, revealing how models diverge from average behavior during training and in few‑shot tasks.

By Micah Adler, John W. Byers, Mark Crovella

llms rag

arXiv Statistics ML

Sep 28

On the Expressive Power of Transformers for Contextual Relations

The paper investigates the theoretical expressive power of Transformers in modeling contextual relations. By framing a text as a distribution of representations and attention as a probabilistic relation, it connects attention normalization to optimal transport: softmax yields conditional relations, while Sinkhorn yields joint relations with fixed marginals. The authors prove universal approximation results, showing that Transformers with Sinkhorn normalization can represent any joint probability relation, whereas standard softmax Transformers can represent any conditional probability relation.

By Demi\'an Fraiman

llms

arXiv Machine Learning

Jul 22

Elicitation without Backpropagation: Steering Model Behavior by Optimizing the Latent Posterior

arXiv:2607. 18804v1 Announce Type: new Abstract: In the \emph{latent posterior model} of transformer behavior, the next-token distribution arises from a posterior over latent predictive models conditioned on the context, mixed to generate continuations.

By Garrett Baker, Vinayak Pathak, Daniel Murfet, Susan Wei

llms

arXiv Computation and Language

Oct 5

Emergent Structure in the Marginal Attention Space of Language Models

The paper investigates how post‑softmax attention weights behave across different language models by marginalizing over query positions to create a joint token‑head space. Across more than 60 diverse LLMs, token‑wise marginal attention reveals a text‑intrinsic signal that is conserved across models, while head‑wise marginal attention yields a model‑private signature that is consistent across documents. This insight enables a practical method for estimating per‑head budgets for key‑value cache eviction, achieving competitive performance on standard benchmarks without per‑document recomputation or training.

By Valentino Maiorca, Walter Nelson, Francesco Locatello

llms benchmarks

arXiv AI

Jul 24

Attention Degradation, Function Token Anchoring, and the Limits of Attention-Based Intervention in Large Language Models

arXiv:2607. 20524v1 Announce Type: new Abstract: Mean cross-positional attention degradation is widely reported in transformer interpretability, yet whether it causally limits contextual retrieval remains untested.

By Sagar Dangal, Manoj Shakya

llms safety