arXiv AI June 30, 2026 By Haobo Yang

Transformer Architectures as Complete Bayes Processes: A Formal Proof in the Measure-Theoretic Kernel Framework

arXiv:2606. 30440v1 Announce Type: cross Abstract: We present a complete formal proof that transformer architectures, when their internal update mechanisms satisfy a Bayes joint-distribution condition, implement exact Bayesian posterior inference.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

llms

arXiv Statistics ML

6d ago

On the Expressive Power of Transformers for Contextual Relations

The paper investigates the theoretical expressive power of Transformers in modeling contextual relations. By framing a text as a distribution of representations and attention as a probabilistic relation, it connects attention normalization to optimal transport: softmax yields conditional relations, while Sinkhorn yields joint relations with fixed marginals. The authors prove universal approximation results, showing that Transformers with Sinkhorn normalization can represent any joint probability relation, whereas standard softmax Transformers can represent any conditional probability relation.

By Demi\'an Fraiman

llms

arXiv Machine Learning

Sep 1

Tracing distinguishability through transformer processing with stochastic LayerNorm

arXiv:2608.30720v1 Announce Type: new Abstract: Representational similarity is foundational to analyses of deep networks, yet distances between point-valued representations are not intrinsically tied...

By Kieran Murphy

llms fine-tuning efficiency safety

arXiv Machine Learning

Jul 21

What does a Bayes-filtered transformer believe? A predictive Monte Carlo approach

arXiv:2607. 17060v1 Announce Type: new Abstract: A Bayes-filtered transformer (BFT) is a transformer trained on sequences that are generated in two steps: first a latent task is drawn from a prior, then observations are drawn conditional on that task.

By Afiq Abdillah Effiezal Aswadi, Haotong Ma, Susan Wei

llms diffusion safety

arXiv Machine Learning

Jul 7

Induction Heads Interpolate N-Grams

arXiv:2607. 02800v1 Announce Type: new Abstract: Induction heads are attention circuits believed to underlie in-context learning in transformers, yet a precise characterization of the estimators they implement remains elusive.

By Francesco D'Angelo, Oguz Kaan Yuksel, Swathi Shree Narashiman, Nicolas Flammarion

llms safety

arXiv Machine Learning

Jun 11

Kalman Linear Attention: Parallel Bayesian Filtering For Efficient Language Modelling and State Tracking

arXiv:2602. 10743v2 Announce Type: replace Abstract: State-space language models such as Mamba and gated linear attention (GLA) offer linear-complexity, parallelisable alternatives to transformers, but their linear state updates limit expressivity and robust state tracking.

By Vaisakh Shaj, Cameron Barker, Aidan Scannell, Andras Szecsenyi, Elliot J. Crowley, Amos Storkey

llms robotics benchmarks

arXiv Machine Learning

Aug 10

The Expressive Power of Low Precision Softmax Transformers with (Summarized) Chain-of-Thought

arXiv:2605. 18079v2 Announce Type: replace Abstract: Existing expressivity results for transformers typically rely on hardmax attention, high precision, and other architectural modifications that disconnect them from the models used in practice.

By Moritz Br\"osamle, Stephan Eckstein

llms safety

One email a morning, machine-written

Related stories

On the Expressive Power of Transformers for Contextual Relations

Tracing distinguishability through transformer processing with stochastic LayerNorm

What does a Bayes-filtered transformer believe? A predictive Monte Carlo approach

Induction Heads Interpolate N-Grams

Kalman Linear Attention: Parallel Bayesian Filtering For Efficient Language Modelling and State Tracking

The Expressive Power of Low Precision Softmax Transformers with (Summarized) Chain-of-Thought