arXiv Machine Learning June 11, 2026 By Itay Lavie, Kirsten Fischer, Andrey Lekov, Frederic Van Maele, Zohar Ringel, Moritz Helias

Phase Transitions in Attention: A Bayesian Theory of Copy Head Emergence

Read the original on arXiv Machine Learning →

arXiv:2606. 12058v1 Announce Type: cross Abstract: Attention is the key mechanism underlying in-context learning in transformers, and attention patterns have been observed empirically to emerge abruptly during training.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

llms

arXiv Machine Learning

Jul 7

Incremental Learning of Sparse Attention Patterns in Transformers

arXiv:2602. 19143v2 Announce Type: replace Abstract: This paper studies simple transformers trained on a high-order Markov chain, where the model must incorporate information from multiple past positions, each with different statistical importance.

By O\u{g}uz Kaan Y\"uksel, Rodrigo Alvarez Lucendo, Nicolas Flammarion

llms

arXiv Machine Learning

Jun 5

Specialization of softmax attention heads: insights from the high-dimensional single-location model

arXiv:2603. 03993v2 Announce Type: replace Abstract: Multi-head attention enables transformer models to represent multiple attention patterns simultaneously.

By M. Sagitova, O. Duranthon, L. Zdeborov\'a

llms

arXiv Machine Learning

Jun 17

Softmax as Linear Attention in the Large-Prompt Regime: a Measure-based Perspective

arXiv:2512. 11784v2 Announce Type: replace Abstract: Softmax attention is a central component of transformer architectures, yet its nonlinear structure poses significant challenges for theoretical analysis.

By Etienne Boursier, Claire Boyer

llms

arXiv Machine Learning

Jun 25

Emergent Capabilities Arise Randomly from Learning Sparse Attention Patterns

arXiv:2606. 25010v1 Announce Type: new Abstract: Neural scaling laws for transformer language models predict smooth improvements in pretraining loss with increasing parameters, but downstream capabilities such as in-context learning are known to emerge abruptly past a certain model scale.

By Vatsal Baherwani, Zixi Chen, Shikai Qiu, Andrew Gordon Wilson, Pavel Izmailov

llms efficiency

arXiv Machine Learning

Sep 10

Conditioned Initialization for Attention

arXiv:2609.07086v1 Announce Type: new Abstract: Transformers are a dominant architecture in modern machine learning, powering applications across vision, language, and beyond. At the core of their su...

By Hemanth Saratchandran, Simon Lucey

llms safety

arXiv Machine Learning

Aug 11

Training-Free Universal Approximation by Prompting Random Transformers

arXiv:2608. 09558v1 Announce Type: new Abstract: How expressive is prompting a transformer?

By Alexander Hsu, Rongjie Lai

llms

One email a morning, machine-written

Related stories

Incremental Learning of Sparse Attention Patterns in Transformers

Specialization of softmax attention heads: insights from the high-dimensional single-location model

Softmax as Linear Attention in the Large-Prompt Regime: a Measure-based Perspective

Emergent Capabilities Arise Randomly from Learning Sparse Attention Patterns

Conditioned Initialization for Attention

Training-Free Universal Approximation by Prompting Random Transformers