← Back to all news
arXiv Machine Learning By Jasper van Doornmalen, Alexander Kozachinskiy, Corinna Mathwieser, Tomasz Steifer, Felipe Urrutia, Jos\'{e} Verschae, Przemys{\l}aw Andrzej Wa{\l}\c{e}ga

Transformer Heads Looking for Order

Read the original on arXiv Machine Learning →

The paper demonstrates that a single-head, single-layer transformer cannot determine whether a bit sequence is ordered, whereas a two-head, single-layer transformer can. This distinction is shown under a model where transformers include an output MLP. The study provides a concrete example of how increasing the number of heads can enhance a transformer’s computational capability.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv AI
Sep 10

Parity, Sensitivity, and Transformers

arXiv:2602.05896v3 Announce Type: replace-cross Abstract: Understanding what neural architectures can and cannot compute is a central challenge in the theory of AI. One of the fundamental problems in...

By Alexander Kozachinskiy, Tomasz Steifer, Przemys{\l}aw Wa{\l}\c{e}ga
llms
More like this →
arXiv Machine Learning
Aug 11

How Many Different Outputs Can a Transformer Generate?

arXiv:2605. 22223v2 Announce Type: replace Abstract: We study how we can leverage only a handful of characteristics of a transformer's architecture to closely predict the number of different sequences it can output, both qualitatively and quantitatively.

By Maxime Meyer, Mario Michelessa, Caroline Chaux, Vincent Y. F. Tan
llms
More like this →