arXiv Machine Learning August 10, 2026 By Moritz Br\"osamle, Stephan Eckstein

The Expressive Power of Low Precision Softmax Transformers with (Summarized) Chain-of-Thought

Read the original on arXiv Machine Learning →

arXiv:2605. 18079v2 Announce Type: replace Abstract: Existing expressivity results for transformers typically rely on hardmax attention, high precision, and other architectural modifications that disconnect them from the models used in practice.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv Machine Learning

Aug 11

Barriers to Universal Reasoning With Transformers (And How to Overcome Them)

arXiv:2604. 25800v2 Announce Type: replace Abstract: Chain-of-Thought (CoT) has been shown to empirically improve Transformers' performance, and theoretically increase their expressivity to Turing completeness.

By Oliver Kraus, Yash Sarrof, Yuekun Yao, Alexander Koller, Michael Hahn

llms

arXiv Machine Learning

Aug 11

Training-Free Universal Approximation by Prompting Random Transformers

arXiv:2608. 09558v1 Announce Type: new Abstract: How expressive is prompting a transformer?

By Alexander Hsu, Rongjie Lai

llms

arXiv Machine Learning

Sep 2

Performance-Efficiency Tradeoffs in Transformers: An Approximation Theory Perspective

The paper examines how to allocate attention heads and head dimensions across Transformer layers to balance expressivity and efficiency. It provides a mathematical analysis of early layers’ role in information extraction and characterizes the trade‑off between head count and dimension under a fixed parameter budget. The authors prove a saturation effect of softmax activations, showing that increasing head dimensions yields diminishing returns, especially for long sequences, and propose strategies for efficient parameter allocation across layers.

By Ruoxi Yu, Haotian Jiang, Jingpu Cheng, Penghao Yu, Qianxiao Li, Zhong Li

llms

arXiv AI

Jun 18

Explaining Attention with Program Synthesis

arXiv:2606. 19317v1 Announce Type: cross Abstract: A longstanding goal of research on interpretable deep learning is to replace opaque neural computations with human-meaningful symbolic descriptions.

By Amiri Hayes, Belinda Li, Jacob Andreas

llms nlp benchmarks

arXiv Machine Learning

Jul 21

Can Transformers Really Do It All? On the Compatibility of Inductive Biases Across Tasks

arXiv:2607. 17624v1 Announce Type: new Abstract: Transformers are remarkably versatile and their design is largely consistent across a variety of applications.

By Damien Teney, Liangze Jiang, Hemanth Saratchandran, Simon Lucey

llms

arXiv AI

Jun 30

Emergence of Minimal Circuits for Indirect Object Identification in Attention-Only Transformers

arXiv:2510. 25013v2 Announce Type: replace-cross Abstract: Mechanistic interpretability aims to reverse-engineer large language models (LLMs) into human-understandable computational circuits.

By Rabin Adhikari

llms rag benchmarks safety

One email a morning, machine-written

Related stories

Barriers to Universal Reasoning With Transformers (And How to Overcome Them)

Training-Free Universal Approximation by Prompting Random Transformers

Performance-Efficiency Tradeoffs in Transformers: An Approximation Theory Perspective

Explaining Attention with Program Synthesis

Can Transformers Really Do It All? On the Compatibility of Inductive Biases Across Tasks

Emergence of Minimal Circuits for Indirect Object Identification in Attention-Only Transformers