arXiv Statistics ML September 28, 2026 By Demi\'an Fraiman

On the Expressive Power of Transformers for Contextual Relations

Read the original on arXiv Statistics ML →

The paper investigates the theoretical expressive power of Transformers in modeling contextual relations. By framing a text as a distribution of representations and attention as a probabilistic relation, it connects attention normalization to optimal transport: softmax yields conditional relations, while Sinkhorn yields joint relations with fixed marginals. The authors prove universal approximation results, showing that Transformers with Sinkhorn normalization can represent any joint probability relation, whereas standard softmax Transformers can represent any conditional probability relation.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Statistics ML.

llms

arXiv Machine Learning

Jul 2

Ghost in the Kernel: In-Context Learning with Efficient Transformers via Domain Generalization

arXiv:2607. 00479v1 Announce Type: new Abstract: Transformer-based large models have demonstrated remarkable generalization abilities across different tasks by leveraging a context-aware attention module for in-context learning.

By Peilin Liu, Ding-Xuan Zhou

llms

arXiv AI

Jun 30

Transformer Architectures as Complete Bayes Processes: A Formal Proof in the Measure-Theoretic Kernel Framework

arXiv:2606. 30440v1 Announce Type: cross Abstract: We present a complete formal proof that transformer architectures, when their internal update mechanisms satisfy a Bayes joint-distribution condition, implement exact Bayesian posterior inference.

By Haobo Yang

llms

arXiv Machine Learning

Jun 30

Generalization Analysis of Transformers in Distribution Regression

arXiv:2606. 29256v1 Announce Type: cross Abstract: In recent years, models based on the Transformer architecture have seen widespread applications and have become one of the core tools in the field of deep learning.

By Peilin Liu, Ding-Xuan Zhou

llms nlp fine-tuning

arXiv Machine Learning

Aug 11

Training-Free Universal Approximation by Prompting Random Transformers

arXiv:2608. 09558v1 Announce Type: new Abstract: How expressive is prompting a transformer?

By Alexander Hsu, Rongjie Lai

llms

arXiv Machine Learning

Jun 17

Softmax as Linear Attention in the Large-Prompt Regime: a Measure-based Perspective

arXiv:2512. 11784v2 Announce Type: replace Abstract: Softmax attention is a central component of transformer architectures, yet its nonlinear structure poses significant challenges for theoretical analysis.

By Etienne Boursier, Claire Boyer

llms

arXiv Machine Learning

Sep 10

Conditioned Initialization for Attention

arXiv:2609.07086v1 Announce Type: new Abstract: Transformers are a dominant architecture in modern machine learning, powering applications across vision, language, and beyond. At the core of their su...

By Hemanth Saratchandran, Simon Lucey

llms safety

One email a morning, machine-written

Related stories

Ghost in the Kernel: In-Context Learning with Efficient Transformers via Domain Generalization

Transformer Architectures as Complete Bayes Processes: A Formal Proof in the Measure-Theoretic Kernel Framework

Generalization Analysis of Transformers in Distribution Regression

Training-Free Universal Approximation by Prompting Random Transformers

Softmax as Linear Attention in the Large-Prompt Regime: a Measure-based Perspective

Conditioned Initialization for Attention